60
Intelligent User Interfaces Dr. Bowen Hui University of Bri8sh Columbia Okanagan [email protected] 1
Intelligent User Interfaces
Dr. Bowen Hui University of Bri8sh Columbia Okanagan
1
Intelligent User Interfaces (IUI)
• User interfaces – a soDware interface for human users – Very concrete but open-‐ended
2
Intelligent User Interfaces (IUI)
• User interfaces – a soDware interface for human users – Very concrete but open-‐ended
• Intelligent – does something the user wants, even when not asked – Completely subjec8ve
3
Why IUI?
• Increasing complexity in soDware • Less 8me to do stuff • Lower cost of development • Increase soDware acceptance • Applica8on for AI techniques
4
Evolu8on of Email
5
Pine Email
6
Email Threads
7
Email Priori8es
8
Email Tracking
9
Objec8ves of IUI
• Increase produc8vity • Decrease expenditures • Lower level of exper8se required to use soDware
• Overall: to improve efficiency, effec8veness, and naturalness of interac8on
• How? – Use AI representa8on, reasoning, models
10
Ex: Mail Filters
• What is the UI aspect? • What is the intelligent aspect? • What are the poten8al user benefits?
11
Ex: Speech Recogni8on SoDware
• What is the UI aspect? • What is the intelligent aspect? • What are the poten8al user benefits?
12
Ex: Desktop Assistant SoDware
• What is the UI aspect? • What is the intelligent aspect? • What are the poten8al user benefits?
13
What Could Go Wrong?
• Risks:
14
What Could Go Wrong?
• Risks: – Don’t do what the user wants
• Some8mes this is okay
– Interrupt the user at a bad 8me – Frustrate user – Lose use trust/loyalty
• How should the system decide?
15
Design Opportuni8es
• Needs-‐driven: – Consider interface aspects – Consider domains of applica8ons – What can be beier?
• Faster, cheaper, more “naturally”
• Technique-‐driven: – How can I use XX technique fruilully?
16
User Interface Components?
• What are some specific UI components that you/others interact with? – Desktop/Web apps – Ubiquitous and pervasive apps – Mobile apps – Very large displays
17
Applica8on Domains?
• What do you use soDware for? – School – Work
• What do you want to use soDware for? – Games – Entertainment – Socializa8on
18
IUI Architecture (1999)
19
Medium and Modality
• Medium – Material object used for presen8ng, saving, or handling informa8on
– E.g., paper, CD, microphone, mouse
• Modality – Human senses used to process informa8on
• Vision, audi8on, olfac8on, touch, taste – Also called mode
20
Why So Complex?
• How would you respond to someone saying to you, “I want to go to the bank”?
21
Why So Complex?
• How would you respond to someone saying to you, “I want to go to the bank”? – Disambigua8on: Which bank? – Task: What for? – Planning: By when? – Formality: How do you respond back? – Relevant Context: …?
22
AI Meets HCI
23
Movement Towards Automa8on
• Roughly, input processing requires system to “understand” input
• Likewise, output display requires system to “generate” output – Historically, canned output used – Genera8on on the fly – Requires common representa8on of knowledge – Towards model-‐based solu8ons
24
The Famous Eliza Agent
• Psychotherapist who repeats your thoughts – “Yes, tell me more about ______ ” – “Do you think it is ______ to ______ ” – “I understand. I am listening.”
• Try it out at: hip://nlp-‐addic8on.com/eliza/
25
Sample Conversa8on with Eliza
26
Ex: Sentence Genera8on
• Solu8on 1: Hardcode a bunch of sentences • Solu8on 2: – Define S = N V NP – Specify list of nouns, verbs, model for NP, etc.
• Benefits – More structure (beier understanding) – Easier to change – Easier to extend
27
Range of Informa8on Presenta8on
• Formal representa8on of info to be conveyed (1999) – Pure linguis8c form – … (spectrum) – Pure visualiza8on
• Missing dimensions beyond the visual modality
28
Design of Basic Intelligent Agent
29
Environment?
• Everything “outside” the agent soDware • Examples: – Opera8on system – Other soDware – The Internet – Input and output devices – User
30
Percepts?
• Anything that can be monitored (observable) • Examples: – Keystrokes and mouse movements – Gestures – Facial expressions – Conversa8onal syntax – Dialogue structure
31
Ac8ons?
• Anything that the agent soDware can do to the environment
• Examples: – Pop up dialog box – Ask ques8on or follow-‐up ques8on – Show hint balloon – Auto-‐comple8on – Change layout
32
Ex: Simple Reflex Agent
Look up table implementa8on 33
Ex: Reflex Agent with Internal State
Gives memory capability 34
Ex: Goal-‐Based Agent
Gives goals modeling + planning capability 35
Ex: U8lity-‐Based Agent
Gives decision making capability 36
General IUI Architecture
37
General IUI Architecture
38
Need help, Frustrated
Typing, Menu selec8on, Schedule mee8ng
General IUI Architecture
Moio: Agent is happy if the user is happy 39
Pr(X)
Offer help, Do nothing (listen), Complete task for user
40
b
41
b
a1
42
b
. . . b1,1 b1,m
o1 om
a1
43
b
. . . b1,1 b1,m
o1 om
bk,1 bk,m . . .
o1 om
a1 ak . . .
. . .
.
.
.
.
.
.
.
.
.
.
.
.
Complex! Layers of uncertain8es! In-‐depth calcula8ons! Future es8ma8ons!
Mixed-‐Ini8a8ve (MI) Interac8on • Historically, user-‐ini8ated commands only • Intelligent aspect is to have mixed-‐ini8a8on • Dialogue analogy:
• Imagine system-‐ini8ated commands only! 44
Ex: Command Line Interface
45
MI Goals • Aims to support efficient, naturally interleaving contribu8ons of work
• Both par8es converge to same solu8on for a problem
• Objec8ves: – Solve a problem together – Achieve the same goal – Come to a common understanding
• Improve quality of interac8on flow when working with computers
46
Difficul8es
• How to establish common ground? • What is the user’s current focus of aien8on? • What is the user’s level of understanding of the problem?
• What are the user’s beliefs? • What are the user’s abili8es and inten8ons to contribute to solu8on?
47
Mutual Beliefs between A and B
48
A and B believe p A believes that B believes p A believes that B believes that A believes p . . .
Ex: Intelligent Tutoring System
49
I don’t get it ?
Ex: Intelligent Tutoring System
• Student’s current level of understanding? – Student’s focus, abili8es, inten8ons – Student’s past successes, mistakes, learning paierns
50
I don’t get it ?
Ex: Intelligent Tutoring System
• Student’s current level of understanding? • How best to help student? – Watch, show similar example, give hint, give complete solu8on
51
I don’t get it ?
Is It Any Good?
• Recall: IUIs do what users want • General evalua8on methods – Simula8on across variety of scenarios – Empirical studies with variety of users
• Criteria based on metrics – E.g., 8me to select target – E.g., accuracy percentage in target selec8on – E.g., level of frustra8on perceived
• Comparisons of methods (benchmarking)
52
Major Challenges
• Modeling what users want • Elici8ng what users want • Not knowing the “true” world state • Planning and reasoning ahead • Con8nually learning model parameters
53
Major Challenges
• Modeling what users want – Quan8fying using a u8lity func8on – Shape of func8on? Addi8vity? – Based on what interac8on principles? – Learning model parameters offline vs. online
54
Major Challenges
• Modeling what users want • Elici8ng what users want – What to ask, when, how oDen – What is the value of informa8on? – What if they change? How do you know?
55
Major Challenges
• Modeling what users want • Elici8ng what users want • Not knowing the “true” world state – Par8al observability vs. full observability – Uncertainy about the state – Noise in the model – Complex mathema8cal models required
56
Major Challenges
• Modeling what users want • Elici8ng what users want • Not knowing the “true” world state • Planning and reasoning ahead – Expected costs and benefits of each ac8on – Myopic decision making vs. N-‐step look-‐ahead vs. long term decision making
– Complex mathema8cal models required
57
Class Topics
• Adaptable vs. adap8ve interfaces • Natural language interfaces • Social media analysis • User types • Ac8vity recogni8on
58
Logis8cs
• Next class: – Systems that Adapt to their Users (Bowen) – Socially-‐Adaptable Interfaces: Crowdsourcing Customiza8on (?)
• Presenta8on rota8on • Paper review form • Study material: logic, probability
59
Addi8onal Image References • hip://www.deepakarora.com/how-‐to-‐hide-‐your-‐email-‐address-‐
from-‐spam-‐robots/ • hip://media-‐dis-‐n-‐dat.blogspot.ca/2010/07/speech-‐recogni8on-‐
soDware-‐con8nues.html • hip://www.8me.com/8me/specials/packages/ar8cle/
0,28804,1991915_1991909_1991755,00.html • hip://www.cs.berkeley.edu/~russell/aima1e/chapter02.pdf • hip://themediaonline.co.za/2012/01/fun-‐client-‐conversa8ons-‐or-‐
nobody-‐likes-‐to-‐be-‐first/ • hips://www.msimaging.com/services/ediscovery/advanced-‐
review-‐tools/ • hip://email-‐organiza8on.com/email_produc8vity_solu8ons/
email_produc8vity_best_prac8ces.html
60
![Creating [User] Interfaces](https://static.documents.pub/doc/80x56/56813cf9550346895da69cd5/creating-user-interfaces-568da88f9f659.jpg)
