1
Toward a Unified Scripting LanguageToward a Unified Scripting Language
Toward a Unified Scripting Language : Toward a Unified Scripting Language : Lessons Learned from Developing CML Lessons Learned from Developing CML
and AMLand AML
Soft computing LaboratoryYonsei University
October 1, 2004
2
Toward a Unified Scripting LanguageToward a Unified Scripting Language
1. Introduction1. Introduction
• Two approaches to specifying scripting languages for character animation– Character Markup Language(CML)
Top-down approach by defining high-level attributes for character personality, emotion and behavior that are integrated to form the specification of synchronized animation script
– Avatar Markup Language(AML)Bottom-up approach in which the language provides a generic
mechanism for the selection and synchronized merging of animations
• Need for powerful yet generic scripting languages to bridge the gap between behavior generation and animation tools
3
Toward a Unified Scripting LanguageToward a Unified Scripting Language
• Visual Behavior Definition– A third factor that governs character behavior
The actions an agent needs to perform in a session to achieve given tasks
Personality and current mental stateThe role the agent is given
– The behaviors are defined as XML tags
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
CMLProcessor
High-level behavior tags
(map appropriate action point parameters)
Animation Script
4
Toward a Unified Scripting LanguageToward a Unified Scripting Language
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
• Classification of Motion– Derived from Blumberg and Russell’s architecture (three-
layer structure : geometry, motor, behavior system)
– The initial set of CML base motions is classified by the goal of the motion as follows:
Movement defines motion that require the rotation or movement of a character from one position to another. Positions are defined by exact coordinates, an object position, or a character position. (move-to, turn-to)
Pointing defines a pointing gesture toward a coordinate, object, or character. (point-to)
Grasping defines motions that require the character to hold, throw, or come in contact with an object or another character (grasp, throw, touch)
Gaze defines the movements related to the head and eyes. (Gaze are gaze, track, blink, look-to, and look-at. )
Gesture includes motions that represent known gestures like hand movement to convey an acknowledgment, a wave, etc. (gesture-at)
5
Toward a Unified Scripting LanguageToward a Unified Scripting Language
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
• CML Specification– Animated character behavior is expressed through the
interpretation of XML Schema structures
– The language containsLow-level tags (specific character gesture representations
defining movements, intensities, and explicit expression)High-level tags (commonly used combinations of these low-
level tags
– Synchronization between the audio and visual modalities is achieved through the use of SMIL ( <seq>, <par>)
6
Toward a Unified Scripting LanguageToward a Unified Scripting Language
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
• CML Representation Language– Head Gesture Taxonomy
Symbolic gestures, Iconic gestures, Deictic gesture
– Hand Gestures TaxonomyPosture, Motion, Orientation, ‘Gestlets’, Fingers
– Body Gestures TaxonomyNatural, Relax, Tense, Iconic, Incline
– Emotions (based on the OCC theory of emotion)Class, Valence, Subject, Target, Intensity, Time-stamp,
Origin
7
Toward a Unified Scripting LanguageToward a Unified Scripting Language
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
• CML Scripting Language– Face Animation Scripting
Head Movement tilt : movement in a slant with often subtle or superficial neck movement
turn : require more profound movement of the neck<turn-to>
<order {0 to n/before/after} /><priority 0 to n/><begin {ss:mmm:/before/after/object} /><end {ss:mmm:/before/after/object} />
<speed {0.n to n.n (unit) /defualt/slow/fast} />
<target {x,y,z/object/character} /><direction (rightside, leftside, frontside, backside) /><degree (n%) />
<repeat {0 to n/dur} /><interrupt {yes/no} />
<transAnimat {head groups} /><transPos {x,y,z/object/character} /><transSpeed default/slow/ intermediate/fast} />
</turn-to>
8
Toward a Unified Scripting LanguageToward a Unified Scripting Language
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
Head Gesture Deictic
Symbolic
Face Movement and GestureDefine the elements and behaviors for specific parts of the face including the Brow, Gaze, and Mouse
<disagree><tilt-to-right>…</tilt-to-right><tilt-to-left>…</tilt-to-left>
</disagree>
9
Toward a Unified Scripting LanguageToward a Unified Scripting Language
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
– Body Animation ScriptingA set of defined body elements which are low level tags
based on MPEG-4 BAPsA set of high-level tags representing body parts which are
grouped from a set of respective low-level tags
Base elements are..Movement (moving, bending, turning)
Gesture defines body postures that include motions representing common Iconic, Symbolic, or Deictic body gestures
Posture (expression) defines a set of high-level tags representing general body gesture
» Natural ,Relax ,Tense, Incline
10
Toward a Unified Scripting LanguageToward a Unified Scripting Language
CML Generation Animation Generation
2. Scripting with the Character Markup Language2. Scripting with the Character Markup Language
• Generating Script
State and Context
Input
Emotion Signal
Behavior/Action
Speech Text
PDTD
EDTD
BDB
FDB
CML Processor
Action Composition
UtteranceComposition
Synchronisation
CM
L D
ecod
er
CMLScript
FAP
Audio/TTS
FAP
BAP
Audio/TTS
Face
Face&Body
11
Toward a Unified Scripting LanguageToward a Unified Scripting Language
3. Avatar Markup Language3. Avatar Markup Language
• Objective : to design and develop a full end-to-end MPEG-4 multimedia framework to support, amongst other features, 3D avatar-based multi-user chat rooms and autonomous synthetic characters
• Three component– A database of basic facial and body animation units(can
extended by third party)– A rendering system capable of merging multiple face and
body animation units and text to speech input in realtime– A high-level scripting language designed to allow animators
to specify which animations to use together with timing, priority, and decay information
12
Toward a Unified Scripting LanguageToward a Unified Scripting Language
3. Avatar Markup Language3. Avatar Markup Language
• AML Specification
– Flexiblity : as many Expression Tracks as required, each containing as many Expressions as required
<AFML><Settings> …… </Settings>…<ExpressionsTrack name=“Track name”>
<Expression><StartTime>mm:ss:mmm</StartTime><ExName>”name”</ExName><Envelope> …… </Envelope>
</Expression></ExpressionTrack>…
</AFML>
<AFML syntax>
13
Toward a Unified Scripting LanguageToward a Unified Scripting Language
3. Avatar Markup Language3. Avatar Markup Language
<BodyAnimationTrack name=“char, name of track”>…
<UserAnimation type=“bap|trk|wrl” filename=“char”><StartTime> mm:ss:mmm|autosync|autoafter</StartTime><Speed>normal|slow|fast</Speed><Intensity>float, 0 to fn </Intensity><Priority>integer, 0 to n</Priority>
</UserAnimation><Emotion>…</Emotion><StandardGesture>… </StandardGesture>
</BodyAnimationTrack>
<ABML’s Body AnimationTrack Syntax>
<BodyAnimationTrack name=“char, name of track”><FacingAction bodypart=“body|headonly”>
…<XCoor>float, target’s X coordinate in meters</XCoor><YCoor>float, target’s Y coordinate in meters</YCoor><ZCoor>float, target’s Z coordinate in meters</ZCoor>
</FacingAction><PointingAction handconfig=“onefinger|open”>…</ PointingAction><WalkingAction mode=“default|run”>…</WalkingAction><ResettingAction>…</ResettingAction>
</BodyAnimationTrack>
<ABML’s Body AnimationTrack Behavior>
14
Toward a Unified Scripting LanguageToward a Unified Scripting Language
4. CML and AML Applied4. CML and AML Applied
• CML– Sample CML – Happy Move and Point Script
<cml> <character name=“James” personality=“extravert” role=“psa” gender=“m” base-animation-file=“butler.liv”> <happy intensity=“0.3” decay=“0.5” target=“goal” priority=“1”> <move-to order=“0” priority=“0” speed=“default” object=“product1” /> <sync type=“par” order=“1” priority=“0”> <point-to object=“product1” /> <utterance> </sync> </happy> </character></cml>
15
Toward a Unified Scripting LanguageToward a Unified Scripting Language
4. CML and AML Applied4. CML and AML Applied• AML
– Sample AML – Walk and Point<AML face_id=“x” body_id=“y” root_path=“c:\” name=“point to phone”> <FA start_time=“00:00:000”> <TTS output_fap=speech.fap output_wav=speech.wav> <Text>”Let me show you another phone over here”</Text> </TTS> <AFML> <Settings> <Fps>25</Fps> <Duration>00:06:000</Duration> <FADBPath>.\Expressions\</FADBPath> <SpeechPath>.\Speech\</SpeechPath> </Settings> <ExpressionsTrack name=“smile”> <Expresion> <StartTime> 00:00:800 </StartTime> <ExName> smile.ex </ExName> <Envelope> <Point>… </Point> <Point>…</Point> <Point>…</Point> </Envelope> </Expression> </ExpressionTrack> <SpeechTrack> …</SpeechTrack> </AFML> </FA> <BA> <ABML>….</ABML> </BA></AML>
16
Toward a Unified Scripting LanguageToward a Unified Scripting Language
5. Discussion and Lessons Learned5. Discussion and Lessons Learned
• A Comparison
AML MPML MURML TVML APML CMLSTEPXTEP
VHML HumanML PAR RRL
objective animationpresatatio
n
verbal andnon-verbal
utterances
presatation
discourse animation animation animationhuman
representation
parameterised
actiondiscourse
XML V V V V V V V V V
Character Definition V V V V V V VAnimation V V V V V V V V V VDialogue Acts V V V VWorld V V V V VActions/Behavior V V V V V V V V V VVoice Controls V V
Inhibiting animation VMerging V V V V V V V V V ?Synchronisation V V V V V V V V V VAdditional Parameters V V V V V V V V VParameterised actions V V V V V V V V VFeedback to application V V V
Extensibility V V V V V V V V VMacro Elements V V V V V V V V V VMicro Elements V V V V V
Emotions V V V V V V V V VPersonality V V V V V V V
Human- like V V V V V V V V V V VNon- Human V V V V V
Face V V V V V V V VBody V V V V V V V V VSpeech V V V V V V V V V V
V
Character Parts/Modules
Specification Element
Format
Animation Control
Specification Granularity
Believability Attributes
Representation LanguagesBoth Scripting and Representation LanguagesScripting Languages
approach
17
Toward a Unified Scripting LanguageToward a Unified Scripting Language
5. Discussion and Lessons Learned5. Discussion and Lessons Learned
• Towards a Unified Language– Objectives of a Unified Language
Define a framework to decouple embodied agent animation tools and the underlying affect and planning engines
Establish a formal specification for unified /consistent interpretation
Provide for modular developmentCreate a markup language based on XML (providing
semantic and scripting annotations)– Language Requirements
High LevelUsabilityExtensibilityParameterizedSynchronization SupportConsistencyDomain
18
Toward a Unified Scripting LanguageToward a Unified Scripting Language
6. Conclusion6. Conclusion
• There is a trade-off to be made between higher levels of control (high granularity) and the complexity of the resulting language
• Striving toward a unified scripting and representation language may be the catalyst for much needed agreement