Post on 20-Dec-2015
transcript
Modalities vs Media
Modalities are ways of encoding informatione.g. graphics
Media are instantiations of modalitiese.g. a particular image
How Do Multimodal Systems Differ?
Domain/application Available media Modeling of context/environment Modeling of user Focus of research
Example Multimodal Systems
Not speech-centricMIT paintbrush, soundbrush
• http://www.youtube.com/watch?v=04v_v1gnyO8
• http://www.youtube.com/watch?v=iZbe3t8YSf4
• http://www.youtube.com/watch?v=18RY8Jgid20
Wearables• http://www.gatech.edu/innovations/wearable/
MSOIP Keywords
Multimodal mobile dialog Integration of speech and pen input User modeling for presentations
Johnston et al. 2001
About MATCH
What input modalities? What output modalities? What application(s)? What aspects of context?
COMIC Keywords
Ambient intelligence HHI/HCI research
Collaborative problem solving User modeling Avatar
Alexandersson et al. 2004
About COMIC
What input modalities? What output modalities? What applications? What aspects of context?
SmartKom Keywords
Multimodal dialog across applications devices and situations
Avatar Situation aware
Alexandersson et al., Reithinger et al. 2003
SmartKom Video
http://www.smartkom.org/start_en.htmlI showed the SK-Mobile one, but theother one is also interesting.
About SmartKom
What input modalities? What output modalities? What applications? What aspects of context?
Parts of a Multimodal System
Interpreter
DialogManager
Generator
KnowledgeBase
SpeechIn
SpeechOut
GestureIn
PresentOut
TextIn
TextOut
HCI and Multimodal Systems
Input integration/fusion Representations Effective help Quality presentations Managing context Understanding the user
Input Integration/Fusion
Key elements:TimeMultiple uses of some modalitiesError rates
Typical approach is to map straight to semantics if possible
Representation
Increasing use of XML-based languages (SMIL, EMMA)But these don’t solve the semantic
problems Keep ‘backbone’ knowledge separate
from ‘peripheral’ information (Alexandersson et al.)
Quality Presentations
Talking headsAdvantagesDisadvantages
Informative presentations are key User modeling/adaptive presentations
are a bonus These systems go beyond scripts
Understanding the User
What kinds of information can we gather about users in general?
About one user in particular? How can we use this information?
Commercial Multimodal Systems Most are for research
Military• Training and battlefield
Education• Tutoring systems
Commercial ones include: Wii: http://www.youtube.com/watch?
v=n4nZVAEeitU Microsoft surface:
http://www.youtube.com/watch?v=rP5y7yp06n0
TradeOffs
You get:More intuitive technologyMore information, more easilyLess (dumb stuff) for you to do
You trade:PrivacyControl
Towards the Future
DesignMultimodal systems in virtual worlds,
or crossing over from virtual to real worlds
Ambient multimodal interaction Implementation
Mashups – user controlledPervasive multimedia
Towards the Future
http://www.youtube.com/watch?v=FMJwURqpFWs
http://www.programmableweb.com/mashups
SciFi?
Lathe of Heaven by Ursula LeGuin Summa Technologiae by Stanislaw Lem Fast Times at Fairmont High by Vernor
Vinge The Human Machine Merger, talk by
Raymond Kurzweil (at http://www.kurzweilai.net/meme/frame.html?main=memelist.html?m=6%23581)