Designing and Evaluating Life-like Agents as Social Actors
Helmut PrendingerDept. of Information and Communication Eng.
Graduate School of Information Science and Technology
University of Tokyo
[email protected]://www.miv.t.u-tokyo.ac.jp/~helmut/helmut.html
Short Bioeducation, experience Master’s in Logic (1994)
U. of Salzburg, Austria, Dept. of Logic and Philosophy of Science Dynamic modal logic (completeness, decidability) Non-degree studies in Psychology, Linguistics, Literature
Ph.D. in Artificial Intelligence (1998) U. of Salzburg, Dept. of Logic and Philosophy of Science and
Dept. of Computer Science; U. of California, Irvine Incomplete reasoning (deduction, hypothetical reasoning, EBL)
Post doctoral research U. of Tokyo, Ishizuka Lab JSPS Fellowship (4/1998-3/2000): Knowledge compilation,
hypothetical reasoning “Mirai Kaitaku” project (since 4/2000): Life-like characters,
affective communication with animated agents, markup languages for animated agents, emotion recognition
Social Computingmain objective and task
Social Computing aims to support the tendency of humans to interact with
computers as social actors.
Develop technology that reinforces human bias towards social interaction by appropriate feedback
in order to improve the communication betweenhumans and computational devices.
Social Computingrealization
Most naturally, social computing can be realized
by using life-like characters.
Life-like Characters at Worksample applications
Sales, DFKI
Tutoring, USC Knowledge Sharing, ATR
Presentation, U. of Tokyo
Entertainment, MIT
Life-like Charactersdesiderata
Life-like characters should be emphatic and engaging as tutors trustworthy as sales persona entertaining and consistent as actors stimulating as match-makers convincing as presenters (in short) … social actors [… and competent ]
Life-like characters should enable effective and natural communication with humans
Backgroundcomputers as social actors
Humans are biased to treat computers like real people
Psychological studies show that people tend to treat computers as social actors (like other humans)
Tendency to be nicer in “face-to-face” interactions, ...
Animated agents may support this tendency if they are designed as social actors
Ref.: B. Reeves and C. Nass, 1998. The Media Equation. Cambridge University Press, Cambridge.
Animated Agents as Social Actorsrequirements for life-likeness
Synthetic bodies Emotional facial
display Communicative
gestures Posture Affective voice
Embodiment
Features of Life-like Characters
Artificial Emotional Mind
Affect-based response Personality Response adjusted to
social context social role awareness
Adaptive behavior social intelligence
Outlinedesigning and evaluating life-like characters
The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting
Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game
Book project - character scripting languages and applications
Appraisal Modulethe cognitive structure of emotions
Evaluates external events according to their emotional significance for the agent
Outputs emotions joy, distress happy for, sorry for angry at resent, gloat … 22 in total
Ref.: A. Ortony, G. Clore, A. Collins, 1988. The Cognitive Structure of Emotions. Cambridge UniversityPress, Cambridge.
Social Filter Moduleemotion expression modulating factors
Ekman and Friesen’s facial “Display Rules” (’69)
Expression and intensity of emotions is governed by social and cultural norms
Brown and Levinson (’87) on linguistic style
Linguistic style is determined by social variables: power, distance, imposition of speech acts
Agent Modelcharacter profile, affect processing
Character Profile static and dynamic features
Static features personality traits, standards
Dynamic features goals, beliefs, attitudes
Attitudes (liking/disliking) are an important source of emotions toward other agents
an agent’s attitude decides whether it has a positive or negative emotion (toward another agent)
“happy for”– resent; “sorry for”– gloat an agent’s attitude changes as a result of communication
dependent on “affective interaction history”
Signed Summary Recordcomputing attitude from affective interaction history
joy (2)
distress (1)
distress (3)
angry at (2)
hope (2)
good mood(1)
gloat (1)
happy for (2)
winning emotionalstates
time
positive emotions
negativeemotions
joy (2)
hope (2)
good mood(1)
happy for (2)
distress (1)
distress (3)
angry at (2)
gloat (1)
+ Liking if positiveDisliking if negative
Attitudesummaryvalue
=
Ref.: A. Ortony, 1991. Value and emotion. In: W. Kessen, A. Ortony, and F. Craik (eds.), Memories,Thoughts, and emotions: Essays in the honor of George Mandler. Hillsdale, NJ: Erlbaum, 337-353.
inte
ract
ion
his
tory
<emotion,intensity> pairs
If a high-intensity emotion of opposite sign occurs – e.g., a liked interlocutor makes the agent very angry
Agent ignores “inconsistent” new information Agent updates summary value by giving greater weight to
“inconsistent” information (“primacy of recency”, Anderson ’65)
Updating Attitudeweighted update rule
disliking liking h-weight angry r-weight
3 = (3 0.25) (5 0.75)
Consequence for future interaction with interlocutor Momentary disliking: new value is active for current situation Essential disliking: new value replaces summary record
(Sitn) = (Sitn1) h + w
(Sitn) r
w: intensity of
(winning) emotion
, {+,}h/r: historical/recency
weight
Life-like Agentsmaking them act and speak
Realization of embodiment 2D animation sequences Synthetic affective speech
Technology Microsoft Agent package (installed client-side) JavaScript based interface in Internet Explorer
Microsoft Agent package Controls to trigger character actions Text-to-Speech (TTS) Engine Voice recognition
Multi-modal Presentation Markup Language (MPML) Easy-to-use XML-style authoring tool Interface with SCREAM system
Life-like Characters in Interactionsome demos
ComicsScenario
CasinoScenario
Life-like characters that change their
attitude during interaction
Animated comics actors engaging in developing social
relationships
BusinessScenario
Animated agents that storify tacit
corporate knowledge
Casino Scenariolife-like characters with changing attitude
Animated advisor (“Genie”) Emotion, personality Changes attitude dependent
on interaction history with user
Dealer (“James”), player (“Al”) Pre-scripted behavior
Implemented with MPML and SCREAM
Genie‘s Character Profile% Personality specificationpersonality_type(genie,agreeableness,3).personality_type(genie,extraversion,2).% Social variables specificationsocial_power(genie,user,0,_).social_distance(genie,user,1,_).% Goalswants(genie,user_wins_game,1,_).wants(genie,user_follows_advice,4,_).% Attitudeattitude(genie,user,likes,1,init).
User in the role of player of Black Jack game
Emotional Arcadvisor’s dominant emotions depending on attitude
sorry for (4)distress (4) gloat (5) sorry for (5) good mood (5)
ignores advice
pos. attitude
user looses
ignores advice
pos. attitude
user looses
ignores advice
neg. attitude
user looses
follows advice
pos. attitude
user looses
ignores advice
pos. attitude
user wins
Internal intensity values
Round 1 Round 2 Round 3 Round 4 Round 5
advisor has agreeable personality
advisor has agreeable personality, is socially slightly distant to user
sorry for (5)distress (1) gloat (2) sorry for (5) good mood (5)
Intensity values of expressed emotions
Agent Scriptingsimple MPML script
<!--Example MPML script --><mpml>… <scene id=“introduction” agents=“james,al,spaceboy”> <seq> <speak agent=“james”>Do you guys want to play Black Jack?</speak> <speak agent=“al”>Sure.</speak> <speak agent=“spaceboy”>I will join too.</speak> <par> <speak agent=“al”>Ready? You got enough coupons?</speak> <act agent=“spaceboy” act=“applause”/> </par> </seq> </scene>…</mpml>
Mind-Body Interfaceinterface SCREAM MPML
<!--MPML script showing interface with SCREAM --><mpml>… <consult target=”[…].jamesApplet.askResponseComAct(‘james,’al’,’5’)”> <test value=“response25”> <act agent=“james” act=“pleased”/> <speak agent=“james”>I am so happy to hear that.</speak> </test> <test value=“response26”> <act agent=“james” act=“decline”/> <speak agent=“james”>We can talk about that another time.</speak> </test> … </consult>… </mpml>
Alternative Viewsmart characters vs. smart environments
“Sense-think-act” cycle Classical AI approach Internet softbots search for
information on the web, robots explore their environment
All the intelligence is agent-side
“Annotated” environments Shift from agent intelligence to
environment intelligence Semantic web, ubiquitous
computing, affordance theory Agents and environments can
be developed independently
“perceives”game state
infers “I am happy”
“acts”expresseshappiness
behaviorrepository
“tells”available behaviors
environment instructs agent “be happy now”
Outline revisiteddesigning and evaluating life-like characters
The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting
Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game
Book project - character scripting languages and applications
Affective Computingwhy should a computer recognize user emotions?
Human-human communication Based on efficient grounding mechanisms
including the ability to recognize interlocutors’ emotions (frustration, confusion,…)
Humans may react appropriately upon detection of an interlocutor’s emotion (clarification upon confusion)
Human-computer communication Computers typically lack ability to
recognize user emotions Ignoring users’ emotions causes users’
frustration Recognizing and responding to users’
(often) negative emotions may improve users’ interaction experience
Ref.: R. Picard, 1997. Affective Computing. The MIT Press.
Emotion Recognitionhow can computers recognize users’ emotions?
Stereotypes A typical visitor of a casino wants… (to win)
Communicative modalities Facial display (face recognition) Prosody (speech analysis) Linguistic style (NLU) Gestures (gesture recognition) Posture (posture recognition)
Physiological data Biosignals
Physiological Data AssessmentProComp+ unit
EMG: Electromyography EEG: Electroencephalography EKG: Electrocardiography BVP: Blood Volume Pressure GSR: Galvanic Skin Response Respiration Temperature
GSRBVP
sensors
Inferring Emotions from Biosignals Lang’s 2-dimensional emotion model
Lang’s two dimensions Valence - positive or negative
dimension of feeling Arousal - degree of intensity
of emotional response Biometric measures
Skin conductivity increases with arousal (Picard ’97)
Heart rate increases with negatively valenced emotions
Note introverts reach a higher level
of emotional arousal than extroverts
enraged
Valence
Arousal
excited
joyful
sad
relaxeddepressed
Ref.: Lang, P. 1995. The emotion probe: Studies of motivation and attention. American Psychologist 50(5):372–385.
some named emotions in thearousal-valence space
Experimental Studyeffects of a character-based interface
Aim of study Show that a character with affective expression may improve
users’ experience (= reduce frustration) of a simple quiz game Method
Biosignals to measure skin conductance and blood volume pressure (`objective’ assessment of user experience)
Questionnaire (users’ subjective assessment) Instruction
Addition/subtraction task (short-term memory load) Solve a series of 30 quizzes correctly and as fast as possible Frustration is deliberately caused by delay (in 6 out 30 quizzes)
Subjects 20 university students (all male Japanese, approx. 24 years old) JPY 1000.- for participation, JPY 5000.- for best score
Junichiro Mori -Experimenter
Analyser
Instructionmathematical quiz game
Add 5 numbers and subtract the i-th number (i < 5)
1 + 3 + 8 + 5 + 4 = [21]21] E.g.: subtract the 2nd number Result: 18
Select the correct answer by clicking the radio button next to the number
Then the character tells whether answer is correct
It is correct.(polite language)
timer
sometimes delayhere (6 – 14 sec.)
Two Versions of the Gameaffective vs. non-affective (independent variables)
Affective Version Non-Affective Version
Description
Character expresses happiness (sorriness) for correct (wrong) answer Character shows empathy (when delay occurs) Character expresses affect both verbal and nonverbal
Character does not show affective response Character ignores occurrence of delay
Hypotheses
Character may reduce user stress (SC) and decrease negative valence (heart rate)
Character has no significant effect on user emotion (SC, heart rate)
Character Responsesexamples of affective/non-affective feedback
I am sorry. It is wrong.(hyper-polite language)
Hanging shoulder gesture toexpress sorriness non-verbally
I am sorry for the delay.(polite language)
Character apologizes for thedelay
Non-affective feedback“Wrong.” No non-verbalemotion expression.
Non-affective feedbackCharacter ignores the occurrenceof delay.
Analyzing Physiological User Data
BVP
GSR
delaystarts
delayends
DELAYsegment
RESPONSEsegment
userresponse
agentresponse
BiographSoftware(ThoughtTechnologies)
BVP could not be takenreliably
Preliminary Findings9 subjects in each version (data of 2 subjects discarded)
Hypothesis (main): affective agent behavior reduces user frustration
Hypothesis (design): delay induces frustration in subjects All 18 subjects showed significant rise of SC in DELAY segment Corresponds to finding in behavioral psychology (if an individual is prohibited
from attaining a goal, the individual experiences primary frustration)
Preliminary evaluation suggests that an animated character expressing emotions and empathy may undo some of the user’s frustration.
DELAYsegment
mean values sf SC (BVP could not be taken reliably)
RESPONSEsegment
Non-affective version: mean = 0.05Affective version: mean = 0.2
t-test (assuming unequal variance)t(16)=2.57; p = .01
Agents Adapting to User Emotionassumes real-time recognition of user emotions
emotional state
user’straits
ti
bodily expressions
user’saction
ti+1
agent’s actions
sensors
emotional state
user’straits
bodily expressions
sensors
user modeluser model
DynamicDecisionNetwork(simplified)
learning learning
U
evidencenodes
evidencenode
QUESTION:Given user’s state at ti, which agent action will maximize agent’sexpected utility at ti+1, in terms of, e.g., user’slearning and emotion?
Dynamics of User Emotionsuser personality
bodily expressions
extraversion
skin conductivity
eyebrows position
agent’s action
pos valence
vision basedrecognizer
EMG
sensors
GSR
ti+1
ti
reproach
shame
joy
user’s emotional state at ti
user’s emotionalstate at ti+1
neg valence
highdown(frowning)
heart rate
BVP
high
provide help
do nothing
agreeableness
user goals
succeed bymyself
have fun
arousal
reproach shame joy
Ref.: Conati, C. 2002. Probabilistic assessment of user’s emotions in educational games. Applied Artificial Intelligence 16(7-8):555–575.
Outline revisiteddesigning and evaluating life-like characters
The mind of life-like agents Emotion, social role awareness, attitude change Demo - Casino scenario Implementation and character behavior scripting
Evaluating life-like characters Using biosignals to detect user emotions Experimental study with character-based quiz game
Book project - character scripting languages and applications
Book Projectcharacter scripting languages and applications
Wide dissemination of life-like character technology requires
standardized ways to represent the behavior of agents
Book will offer state-of-the-art on XML-based markup languages and tools
Scripting languages for face animation, body animation and gestures, emotion expression, synthetic speech, interaction with environment,…
Characters are already used in a wide variety of applications
Book contains some of the most successful character-based applications
Synopsis chapters on character design
H. Prendinger, M. Ishizuka (Eds.)Life-like Characters. Tools, Affective
Functions and ApplicationsSpringer Hardcover
(in preparation)
useful asStandard/Reference Book
State-of-the-Art in Life-like AgentsCourse Book
for HCI, HAI, multimedia, life-like agentapplications, scripting languages,…
Conclusion Social Computing
Human-computer interaction as social interaction Designing life-like characters as social actors
Believability-enhancing agent features Emotion, personality, social role awareness, attitude
change, familarity change Casino demo Future avenues – “smart” environments (character &
annotated environments) Evaluating life-like characters as social actors
Experimental study using user’s biosignals Life-like characters’ affective response may undo some
of the user’s negative feeling Future avenues – real-time adaptivity of agent
behavior to user’s emotion, decision-theoretic approach to agent behavior