STEP: a scripting language for embodied agents

Zhisheng Huang, Anton Eliens, and Cees Visser

(Scripting Technology for Embodied Persona)

http://wasp.cs.vu.nl/step

Embodied Agent and 3D Web Agent• Embodied agents are autonomous

agents which have bodies by which the agents can perceive their world directly through sensors and act on the world directly through effectors.

• 3D Web agents are embodied agents whose experienced worlds are the Web, and whose 3D avatars can interact with each other or with users via Web browsers

MotivationWhy Embodied Agents

• Sound, graphics, and knowledge could convey ideas and feelings faster than technical documents.[Bickmore(1998)]

• Multimodal human-computer interface, in particular, human-Web interface

MotivationWhy Embodied Agents

• User interface is better if implemented with respect to what people would expect from the same kind of creature in the real world, regarding personality and emotion[Reeves(1996)]

• The use of an anthropomorphic agent may affect the way in which a user interacts with an application or interface[Koda(1996)]

MotivationWhy Scripting Languages

• Embodied agent oriented. More convenience and efficiency for authoring.

• Reusability of scripts for different agents in different application contexts.

MotivationWhy Scripting Language

• Scripting languages focus on communicative acts, like gesture and posture, facial expression, and speech.

• The specification of external-oriented communicative acts are separated from the specification of the internal states of embodied agents to reduce the complexity

Principles• Convenience: natural-language-like terms

turn(Agent, left_arm, front, slow)

turn(Agent, left_arm, rotation(1,0,0,1.57), 3)

• Compositional semantics: combining operators.

par([turn(Agent, left_arm, front, slow),

turn(Agent, right_arm, front, slow)]).

• Re-definability: for high-level specification of actions

run_then_kick(Agent)=seq([run(Agent),kick(Agent)]).

• Parametrization: for the adaptation of actions

Introduce variables in actions.

• Interaction: with a (virtual) environment

STEP and Dynamic Logic• Distinction between an action and a state

• Semantically a state represents the properties at a particular moment, whereas an action consists of a set of state pairs, which represent a relation between two states.

Dynamic Logic

• [a] if the state is true, then after doing action a, the state is true.

• Pre-condition: and post-condition: • two sub languages: a language for action

specification, and a meta language for state specification

STEP and its Meta Language

• STEP is a scripting language for action specification. In order to interact with virtual worlds/states, STEP needs a meta language, like DLP, Prolog, Java, etc.

Main Features of STEP

• X3D/VRML H-anim avatars based, the standard Web3D Technology.

• Dynamic logic based, for solid semantic foundation.

• Declarative semantics, for knowledge representation and reasoning, and for agent technology

Implementation

• STEP has been implemented by using Distributed Logic Programming Language (DLP)

• STEP (version 1.0) and its authoring tools are available from the STEP website:

http://wasp.cs.vu.nl/step

Why DLP

• Object-oriented parallel Prolog

• Web3D support

• TCP/IP support for network communication

• Java-based. All DLP programs are compiled into Java classes

• Efficiency for programming

• Reasonably good performance

FACT : STEP Source Code (1.0)

FACT: Web_oriented• Web deliverable animation. No need to

install any software or tool to run and test STEP scripts. No DLLs are needed to run STEP facial animation engine.

• Animation and presentation can be created on-the-fly, well suitable for web agents which need interaction with users in real time with reasonably good performance.

Functionalities

• Gesture and Posture (version 1.0)• Inverse Kinematics I(version 1.0)• XSTEP (version 1.0)• Text to Speech (version 1.1)• Facial Animation and Emotion

Expression (version 2.0)

Reference System of STEPDirection Reference for Humanoid

Combination of Directions

Body Referencebased on H-anim specification

Time Reference

•Duration: very_fast, fast, slow, very_slow, very_very_fast

•Time: time(3, second)

Action Operators

• turn(Agent, BodyPart, Direction, Duration).

• move(Agent, BodyPart, Direction, Duration).

• seq([Action1, …, Actionm])

• par([Action1, …,Actionm])

• repeat(Action, T).

Interaction Operators

scripting language versus meta language

• execution: do(State) where State is a formula in the meta language:

do(setSFVec3f(Agent,position,X,Y,Z))

• conditional: if_then_else(Condition, Action1, Action2)

• until: until(Action, UntilCondition)

Actions in Script LibrariesScript(action_name(Agent, OtherVariables), Action):-

Action = ComplexAction.

Prolog-like syntax

Example: Walk

Walk in STEP script(walk_pose(Agent), Action):- Action = seq([par([ turn(Agent,r_shoulder,back_down2,fast), turn(Agent,r_hip,front_down2,fast), turn(Agent,l_shoulder,front_down2,fast), turn(Agent,l_hip,back_down2,fast)]), par([turn(Agent,l_shoulder,back_down2,fast), turn(Agent,l_hip,front_down2,fast), turn(Agent,r_shoulder,front_down2,fast), turn(Agent,r_hip,back_down2,fast)])]).

Script(walk(Agent, N), Action):-Action = repeat(walk_pose(Agent),N).

Variants of walk

script(walk_forward_step0(Agent,StepLength),Action):- Action = par([walk_pose(Agent), move(Agent,increment(0.0,0.0,StepLength),fast)]).

Execution of scripting actionsIn DLP

Format: do_script(Script)

Example: do_script(walk(humanoid, 10))

STEP Authoring Tools

Run

Run in STEPscript(start_run(Agent), Action) :-

Action = seq([par([turn(Agent, r_elbow, front, fast), turn(Agent, l_elbow, front, fast)]),

par([turn(Agent, l_hip, front_down2, fast),turn(Agent, r_hip, front_down2, fast),turn(Agent, l_knee, back_down, fast),turn(Agent, r_knee, back_down, fast)])

]),!.

script(run(Agent), Action) :- Action = seq([par([turn(Agent,r_shoulder,back_down2,fast),

turn(Agent,r_hip,front_down,fast),turn(Agent,l_shoulder,front_down2,fast),turn(Agent,l_hip,back_down2,fast)]),par([turn(Agent,l_shoulder,back_down2,fast),turn(Agent,l_hip,front_down,fast),turn(Agent,r_shoulder,front_down2,fast),turn(Agent,r_hip,back_down2,fast)])]),!.

Example: Taichi

STEP script for Taichiscript(taichi(Agent,beginning_posture),Action):- Action =seq([ turn(Agent,l_hip,side1_down,fast), turn(Agent,r_hip,side1_down,fast), par([turn(Agent,l_shoulder,front,slow), turn(Agent,r_shoulder,front,slow)])]).

script(taichi(Agent,push_down_posture),Action) :- Action =seq([ par([turn(Agent,l_shoulder,front_down,slow), turn(Agent,r_shoulder,front_down,slow), turn(Agent,l_elbow,front_right2,slow), …...

Variables and ParametrizationAgent names and their corresponding parameters

are specified as variables in script libraries, by which the same scripting actions can be re-used for different embodied agents under different situations by different authors. It improves the reusability of scripting actions for the purpose of productivity.

Example: Interaction with virtual worlds

STEP script: look-at-ballscript(turnToDirection(Object,SrcVector,DestVector),Action):- Action = seq([ do(vectorCrossProduct(SrcVector,DestVector,vector(X,Y,Z),R)), do(setRotation(Object,X,Y,Z,R))]).

script(look_at_position(Agent,X1,_Y1,Z1),Action):-

Action = seq([do(getPosition(Agent,X,_Y,Z)),

do(Xdif is X1-X),

do(Zdif is Z1-Z),

turnToDirection(Agent,vector(0.0,0.0,1.0),

vector(Xdif,0.0,Zdif))]).

Interaction Operators for Computation

High-level interaction operators can be used to access the computational capabilities of the meta language in order to find the computation solutions, which involve inverse kinematics.

Example: Touchan inverse kinematics problem

STEP script: Touch

script(getABvalue(Agent,position(X0,Y0,Z0),Hand,A,B),Action):- Action = seq([getDvalue(Agent,position(X0,Y0,Z0),Hand, D), get_upperarm_length(Agent,L1), get_forearm_length(Agent,L2), do(D1 is L1 + L2), if_then_else(sign(D1-D)>sign(0.001-D), seq([do(cosine_law(L1,L2,D,A)), do(cosine_law(L1, D, L2, B))]), seq([do(A is 1.57*(1+sign(D-0.001))), do(B is 0.0)]))]).

script(getVvalue(Agent,position(X0,Y0,Z0),Hand,V),Action):- Action = seq([get_shoulder_center(Agent,Hand, position(X2,Y2,Z2)), do(direction_vector(position(X2,Y2,Z2),position(X0,Y0,Z0),V))]).

script(touch(Agent, position(X0,Y0,Z0),l),Action):- Action = seq([getABvalue(Agent,position(X0,Y0,Z0),l,A,B), do(R1 is 3.14-A), getVvalue(Agent,position(X0,Y0,Z0),l,V), get_arm_vector(Agent,l,V0),

do(vector_cross_product(V0,V,vector(X3,Y3,Z3),C)), do(R2 is C-B),

par([turn(Agent,l_shoulder,rotation(X3,Y3,Z3,R2),fast), turn(Agent,l_elbow,rotation(X3,Y3,Z3,R1),fast), turn(Agent,l_wrist,rotation(X3,Y3,Z3,-0.5),fast)])]).

script(touch_absolutePosition(Agent,position(X1,Y1,Z1),Hand),Action):-

Action = seq([do(getPosition(Agent,X,Y,Z)), do(getRotation(Agent, X2,Y2,Z2,R)),

do(X3 is X1-X), do(Y3 is Y1-Y),

do(Z3 is Z1-Z), do(R1 is -R),

do(position_rotation(position(X3,Y3,Z3), rotation(X2,Y2,Z2,R1),position(X4,Y4,Z4))),

touch(Agent,position(X4,Y4,Z4), Hand)]).

STEP Architecture

STEP Engine

XSTEP: A Markup Language for Embodied Agent

• XSTEP is XML-based STEP.

• For XSTEP we have developed an XSLT style sheet, using Saxon package, that transforms an XSTEP specification into STEP, thus DLP.

An Example in XSTEP<action name="walk(Agent)"><seq><par> <turn actor="Agent" part="r_shoulder"> <dir value="back_down2"/><speed value="fast"/></turn> <turn actor="Agent" part="r_hip"><dir value="front_down2"/> <speed value="fast"/></turn> <turn actor="Agent" part="l_shoulder"><speed value="fast"/> <dir value="front_down2"/></turn> <turn actor="Agent" part="l_hip"><dir value="back_down2"/> <speed value="fast"/></turn></par> <par>……</par></seq></action>

XSTEP Editor

Installation of STEP on DLP

1. Download step1.0 Lite (which includes STEP engine, templates, authoring tools, and the avatar ‘yt’) and unzip it to the DLP directory dlp. Make sure that do not overwrite the existing files in your system when you use your own unzip tool.

2. STEP templates :

the application template ‘steptest.pl’

the script library template ‘stepscriptlib.pl’

the step setting template ‘stepsetting.pl’

Default Script Library Namestep_scriptlib

Problem: Due to the limit of the Java virtual machine, the size of each script library cannot be larger than 64KB after the compiling.

Solution: Split a large library into multiple ones.

Hierarchical Library Claims

:-object step_scriptlib : [ bcilib ].

script(ScriptAction, Action):-

step_myscriptlib <- script(ScriptAction,Action),

!.

:-end_object step_scriptlib.

Your own library object name

Humanoid Objects

• humanoid object: humanoid is defined based on a prototype

• non humanoid object: humanoid is not defined based on a prototype.

stepsetting.pl:-object stepsetting.

humanoid_objects(humanoid1, false):-!.

% humanoid1 is not a humanoid object

humanoid_objects(humanoid2, true):-!.

% humanoid2 is a humanoid object

:-end_object stepsetting.

STEP application template:-object my_step_application : [bcilib, step_kernel].

var url = './yt/yt.wrl'.

main:-

text_area(BrowserStream),

set_output(BrowserStream),

loadURL(url),

sleep(6000),

script_threads, /* initialize parallel processing*/

do_script(walk(humanoid)),

do_script(run(humanoid)).

:-end_object my_step_application.

Exercises•Define at least five postures/gestures in STEP from the following set: kick, beat, sit, kneeing, squat, crouch, crawl, climb, lie, sleep, fall, stretch, greeting, hello, bye-bye, ...