- 1. Affective Computing Quantifying and Augmenting Peoples
ExperiencesHyungil Ahn, Shani B Daily, Rana el Kaliouby, Micah
EckhardtMIT Media Laboratory
2. Outline of Workshop
- The challenge of measuring customer experiences
- Affective sensors and technologies + Demo
3. Measuring Beverage Taste Preference (with Pepsi)
- Evaluation method used today:
-
- Cleanse palate (crackers, water)
-
- Consume at least half of the beverage (food)
-
- Fill out questionnaire about liking/buying
-
- Other method: Focus Groups
- Problem: These do not predict marketplace buying
- Challenge: Construct an evaluation method and a computational
model to accurately predict consumer preferences and marketplace
buying behavior.
4. Self-Report
- Most attempts to date that try to predict marketplace decisions
are based on studies where people are asked what they would do
- Self-report captures cognitive elements of liking or wanting
(what you think you should say you like or want) but may or may not
capture the actual feelings of liking or wanting.
- Self-reported liking can be rated instantly (after any sip),
but obtaining an accurate value for wanting or motivation may
require a longer experience.
5. Why Technology to Tag Experiences?
- Measuringin situExperiences: MIT Media Lab has over 100
industry sponsors -> unique opportunity to develop and test with
real-world applications
Advertising Customer delight Customer preferences Learning
Medical training - empathy Cognitive load measurement 6. Why
Technology to Tag Experiences?
-
- Advertising, marketing, usability, Learning, Training, Customer
relationship management
-
- Advance social-emotional intelligence in machines to improve
peoples experience with technology
-
- Enhance peoples ability to connect with others (autism spectrum
disorders)
7. Physiology Sensing 8. Traditional SC sensing off the fingers,
wired to a box Current version:wirelessly communicating SC off the
wrist,Media Lab Galvactivator LED on the hand reflects SC (1999)
Skin Conductance (SC) Sensors 9. Mindreader Platform 10. Automated
Facial Analysis Affective + Cognitive States Concentrating
Disagreeing Interested Thinking Unsure Absorbed Concentrating
Vigilant Disapproving Discouraging Disinclined Asking Curious
Impressed Interested Brooding Choosing Thinking Thoughtful Baffled
Confused Undecided Unsure Agreeing Assertive Committed Persuaded
Sure Happy Happy Delighted Enjoying Content 11. A Smile for Every
Emotion!! 12. Mindreader Platform Feature point tracking
(Nevenvision) Head pose estimation Facial feature extraction Head
& facial action unit recognition Head & facial display
recognition Mental state inference Hmm Let me think about this 13.
Demo 14. Generalization Level 1 Train and test on Mind Reading DVD
15. Generalization Level 2 Train>Mind Reading DVD; Test>posed
corpus
- A challenge with machine learning + with interventions for
autism
- 96 videos from CVPR 2004 tested on our system and on 18
people
- Accuracy: 80% for best 11% of videos
Agreeing Disagreeing Confused Concentrating Thinking Interested
16. Generalization Level 2 Train>Mind Reading DVD; Test>posed
corpus
- 96 videos from CVPR 2004 tested on our system and on 18
people
- Accuracy: 80% for best 11% of videos
Accuracy of panel of 18 people Average = 54.5% Accuracy of
computerAverage = 63.5% (better than 17 of 18 panelists) 17.
Mindreader Platform Components MindReaderAPI Wrappers SDK (for
developers) Application(for non-developers) Sample apps Tracker
OpenCV nPlot External Libs/APIs MindreaderPlatform Downloadables
18. Features of the platform
- Facial information accessible at multiple levels (action units,
expressions and head gesture, affective and/or cognitive
states)
- Interface for training of new and existing mental states
- Sensor support (skin conductance, motion, temperature)
- LINUX (ubunto) version coming soon
- Porting to camera phone, xo
19. To use the platform
- Accessible to sponsors for download (the stand-alone
application or SDK for developers)
- Email: Rana el Kaliouby,[email_address] . mit . edu
20. ConsumerDecision -making 21. Consumer Decision-Making
BehavioralMeasure (Number of Choices, Amount Consumed, etc.)
CognitiveMeasure (Self-Reports,Focus Groups, etc.) AffectiveMeasure
(Facial Valence, Skin Conductance,etc.) Decision -Making
ComputationalModel 22.
- Experienced utility: affective or hedonic experience, which can
occur in a moment-based measure or memory-based measure.
- Decision utility is inferred from observed behavioral
choices.
- Predicted utility is a belief about future experienced
utility.
- Moment utility is a measure of current affective or hedonic
experience.
- Total utility is derived from statistically aggregating a
series of moment utilities.
- Remembered utility is a single memory-based measure of
affective or hedonic experience, which is based on retrospective
assessments of episodes or periods of life.
23. Wanting vs. Liking
- Rather, multiple valuation systems such as cognitive and
affective
- processing systematically influence human decision-making.
Also, the neural substrates of liking (pleasure) are separate from
those of wanting (motivation) in the human brain, so there is
evidence from neuroscience that supports treating these concepts
differently when modeling how people make decisions. Our model will
separate these.
24. Experienced Utility
- The memory-based approach accepts a persons retrospective
evaluations of past episodes and situations as valid data.
Theremembered utility of an episode of experienceis defined by
retrospective global assessment of it the self-reported
recollection ofliking, e.g., how much did you like the drink?
- (ii) The moment-based approach derives the experienced utility
of an episode, e.g., a
- series of sips, from real-time measures of the pleasure and
pain that the participant
- experienced during that episode. Moment utility refers to the
valence (good or bad) and
- to the intensity (mild to extreme) of current affective or
hedonic experience, e.g. the
- experience of the current sip. Kahnemanstotal utilityof an
episode is derived
- exclusively and statistically from the record of moment
utilities during that episode. We
- measure moment utility by measuring emotional expressions
elicited by the episode in
- People sometimes also attempt to forecast the affective or
hedonic experience the
- experienced utility that is associated with various life
circumstances. These are called
- predicted utilityor affective forecasting.
25. Experiment Setup 26. Experiment Setup Machine SelectionSip
on Resulted BeverageAnswer Questions (each trial) 27. 28.
AffectiveMeasure (Facial Valence, Skin Conductance,etc.)
AnticipationDisappointment - Satisfaction Liking / Disliking 25
consumers, 30 trials, 30 min. videos! 29. BehavioralMeasure (Number
of Choices, Amount Consumed, etc.) Choosing a Vending Machine on
the Computer Screen 30. BehavioralMeasure (Number of Choices,
Amount Consumed, etc.) Beverage Outcome and Sipping 31.
CognitiveMeasure (Self-Reports,Focus Groups, etc.)? Asking
Self-Reported Beverage Liking (asked every trial) 32.
CognitiveMeasure (Self-Reports,Focus Groups, etc.)? Asking Machine
Liking (asked every 5 trials) 33. Asking Expectation Comparison and
Purchase Intent (asked every 5 trials) 34. Participants Men
WomenTotal participants 17 22 Video analysis 15 19 Facially
expressive 7 10 35. Analysis overview
- Cognitive - Self-Report on Questionnaire:
-
- valence (positive or negative) from the facial expression
-
-
- Affective outcome valence (satisfaction/disappointment,
associated with affective wanting value)
-
-
- Affective evaluation valence (liking/disliking, associated with
affective liking value)
-
- arousal (calm or excited) from the skin conductance data
-
-
- Affective anticipatory arousal (associated with risk and
uncertainty, possibly valenced feelings of hope or dread)
-
-
- Affective outcome arousal (associated with affective wanting
value)
-
-
- Affective evaluation arousal (associated with affective liking
value)
36. Cognitive Analysis
- Three different ways of looking questionnaire data:
-
- Ultimate after the 30 trials: 20:18
- People were fairly evenly split in preferring Vanilla or
SummerMix, with only a slight preference for Vanilla
37. Behavioral Analysis
-
- an average of 15 sips of each beverage
-
- slightly more Vanilla (6.7 oz) than SummerMix (6.1 oz)
-
- Vanilla favorers reported lower purchase intent than those who
preferred SummerMix
-
- suggests that the SummerMix favorers, while a slightly smaller
group, were even more likely to buy the soda.
-
- a subtle bias (54.6%) toward the SummerMix machine
38. Cognitive + Behavioral Anslysis
- SummerMix should have had nearly as good of a chance to succeed
in the marketplace as Vanilla had.
- Going to market looks like a reasonable decision.
- But this turns out to be only part of the story.
39. Physiology Sensing 40. Traditional SC sensing off the
fingers, wired to a box Current version:wirelessly communicating SC
off the wrist,Media Lab Galvactivator LED on the hand reflects SC
(1999) Skin Conductance (SC) Sensors 41. Patriots Touchdown Doritos
Mouse Patriots 1st Down End Zone Overthrown Leading up to
PatriotsTouchdown Super Bowl XLII 42. Sponsor Week Event:Randi vs.
Raphael with live audience feedback Magician correctly identifies
search term
-
-
http://web.media.mit.edu/~sbdaily/AaI/AaIGroupVisualizationSkin.html#
43. 44. FacialAnalysis 45. Satisfaction DisappointmentLiking
Disliking 46. (Obtained Outcome What She Wanted) Satisfaction
DisappointmentLiking Disliking 47. Satisfaction
DisappointmentLiking Disliking 48. (Obtained Outcome = What She
Wanted) Satisfaction DisappointmentLiking Disliking 49.
Satisfaction DisappointmentLiking Disliking 50. (Sipped Soda = What
He Disliked) Satisfaction DisappointmentLiking Disliking 51.
Satisfaction DisappointmentLiking Disliking 52. (Sipped Soda = What
He Liked) Satisfaction Disappointment Liking Disliking 53. Facial
Valence Analysis 54. Asymmetry
- Vanilla favorers showed absolutely no positive expressions
while tasting SummerMix, while nearly half of the SummerMix
favorers showed something positive while tasting Vanilla.
- The complete lack of any positive expressions in the Vanilla
group may be a red flag.
- If positive FVs were mapped into purchasing behavior, then one
might expect slightly less than half the SummerMix favorers to buy
both products, while no Vanilla favorers would buy SummerMix.
55. 56. Automated Facial Analysis Tracker Accuracy Average =
77.4%; highest = 96.87%; lowest = 23.98% 57. Automated Facial
Analysis Sip Detection 58. Automated Facial Analysis Sip Detection
700 Sips, 82% of the sips detected 59. Automated Facial Analysis
Real-time Analysis 60. Automated Facial Analysis Speedup over
manual coding
- 3 minutes on average for each minute of video.
- At least 2 or 3 coders are needed to establish validity of the
coding, resulting in 6 to 9 minutes of coding per minute of
video
- A very labor-intensive and time-consuming approach
- Each participant has about 30 minutes of video, so we can
expect two coders to take over 180 minutes (3 hours) to code the 30
minutes of data.
61. Automated Facial Analysis Speedup over manual coding
- With our automated sip detection algorithm, the algorithm can
fast forward to sip events and can be made to look at outcome
segments, the only two places where we coded for FVs in the
analysis above.
- If positioned to the right spots in the video, the coder only
has to look at about 20 seconds of video per trial. Over 30 trials
per participant, this is 600 seconds, or 10 minutes of video to
code, instead of 30.
- At 3 minutes of coding time per minute of video, plus
occasional breaks, this results in ~30-40 minutes for each coder to
code each participants experience, or ~60-80 minutes if we use two
coders.
- Combining our sip detection algorithm with human coding would
cut the total manual coding time required (man hours) for sip
events down by at least a factor of three.
62. Conclusion Next steps 63. Affect asIndex 64. Multi-person
aggregate
- X People are watching same video, or doing same task
(+physiology) can we aggregate this data in real-time?
- Advertising and TV/Cable programming
65. Patriots Touchdown Doritos Mouse Patriots 1st Down End Zone
Overthrown Leading up to PatriotsTouchdown Super Bowl XLII 66.
Sponsor Week Event:Randi vs. Raphael with live audience feedback
Magician correctly identifies search term
-
-
http://web.media.mit.edu/~sbdaily/AaI/AaIGroupVisualizationSkin.html#
67. Shani to add more slides here? 68. ResearchRoadmap
forQuantifyingExperiences 69. Two-person interaction
- For service interactions (e.g. Bank of America)
- For autism (e.g., monitoring real-time social interactions,
parent-child interactions Baby Siblings project (UK), Playlamp
70. Mobile Phone Application
- Port API to Googles Android platform (Java)
- Develop some applications to work on mobiles
- HTC Android phones coming out Dec 08
71. Mobile Phone Application
- Port API to Googles Android platform (Java)
- Develop some applications to work on mobiles
- HTC Android phones coming out Dec 08
72. Sociable Robots 73. Acknowledgements
http://affect.media.mit.edu [email_address] MIT Media Lab Affective
Computing GroupComputer Science, American University in Cairo Abdel
Rahman Nasser, Youssef Abdallah. Mina Mikhail, Tarek
HefniSponsors:National Science Foundation Nancy Lurie Marks Family
Foundation MIT Media Lab TTT Consortium Seagate, Google, Robeez