Post on 10-Oct-2020
transcript
Page 1
1 CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University.
Lecture 5: Input Basics
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 2
GUI Input Modalities
• How can a user provide input to the GUI? – keyboard – mouse / joystick – touch pad / screen – pen / stylus – speech – other…
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 3
Keyboard input
• Short history of the keyboard – typewriter patented by Christopher Latham
Sholes in 1868 – Sholes patent sold to Remington & Sons in 1873
• Remington adopted the QWERTY layout
– companies held contents to see whose typewriters & typists could enter the fastest
– why did the QWERTY keyboard win out? • people disagree... seemingly part marketing, part luck
“Typing errors,” by Stan Liebowitz and Stephen E. Margolis. Available at http://reason.com/9606/Fe.QWERTY.shtml
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 4
Keyboard input
• Short history of the keyboard (cont.) – Dvorak Simplified Keyboard patented by
August Dvorak in 1936 • “key” feature: common keys on home row
Page 2
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 5
Keyboard input
• So how fast can people type anyway?
• Of course, this is touch typing normal text...
“Typing Speed: How Fast is Average,” by Teresia R. Ostrach. Available at http://www.fivestarstaff.com/publication_typing.htm
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 6
Mouse input
• Short history of the mouse – Douglas Engelbart invented the mouse in 1963
in a larger project to “augment human intellect” • and thereby invented “point and click”
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 7
Direction manipulation
• Why is the mouse so effective? – at least part of the answer lies in the facilitation of
“direction manipulation”
• Direction manipulation involves... – visual representation of the manipulated objects – physical actions instead of text entry – immediately visible impact of the operation
• Actually, “direct manipulation” is a misnomer – obviously it’s not direct, but indirect
• your hand might move 3”; the cursor moves 10” (dependent on your display size)
– but the key is that it feels direct — because of the real-world metaphor
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 8
Direction manipulation
• Example: “Hand” dragging
Page 3
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 9
Direction manipulation
• Example: Desktop folder & file manipulation
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 10
Related input
• Trackball – nice for games, small footprint – why aren’t they mainstream? metaphor?
• Touchpad / trackpad – small & thin — perfect for laptops – integrates notions of mouse & pen – metaphor is maintained
• Trackpoint (IBM) – pencil-eraser nub – some love ’em, some hate ’em
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 11
Pen input
• Pen-based interfaces in mobile computing
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 12
Pen input
• Pen input has lots of advantages... – users can hold a familiar input device – users can write in a familiar input language – users can carry with them
• ... and some major disadvantages:
Eat up Martha Kearney Newton
Page 4
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 13
Pen input recognition
• Handwriting – very general, well-developed human skill
• thus, make use of what users can already do!
– but hard to recognize (for people & machines)
• Gestures – gesture alphabets
• Palm Pilot graffiti
– editing gestures – easier to recognize
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 14
• Off-line recognition – examine static output of handwriting,
i.e., the end result of the writing
• On-line recognition – examine dynamic movement of handwriting,
i.e., the strokes, pen up/downs involved
• Which is more “informed”? more useful?
Off-line vs. on-line recognition
like OCR, but much harder!
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 15
Recognition techniques
• Neural networks – neurally-inspired computational models – input: bitmap, or “vectorized” strokes – output: probably characters – best for off-line recognition
• Hidden Markov models (HMMs) – powerful probabilistic models – input: vectorized strokes – output: full recognition of chars, words, etc. – best for on-line recognition
And hybrid
methods!
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 16
On-line feature extraction
• On-line strokes -> feature vectors – basic features: pen up/down, direction, velocity – useful features: curvature, reversal, … – other features: ascender/descender? …
Page 5
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 17
On-line recognition
• Hidden Markov models (HMMs) – probabilistic models for dynamic behavior
• Set of N states with – a(i,j) = probability of state transition ij – b(o,i) = probability of seeing o in state I
• can be discrete or continuous prob. distributions
s1 b(o,1)
s2 b(o,2)
s3 b(o,3)
a(1,1) a(2,2) a(3,3)
a(1,2) a(2,3)
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 18
Hidden Markov models
• Let’s say we have – M = HMM representing predicted behavior – O = observation vector sequence O
• Three problems – evaluation: find Pr(O|M) – decoding: find the state sequence Q that
maximizes Pr(O|M,Q) – training: adjust parameters of M to
increase Pr(O|M)
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 19
Hidden Markov models
• HMM decoding (Viterbi algorithm) – find best state sequence through HMM,
maximizing the probability of the sequence – assuming O = < x x y x y > , try to decode…
s1 x .2 y .8
s2 x .8 y .2
s5 x .2 y .8
s7 x .5 y .5
s3 x .2 y .8
s6 x .4 y .6
s8 x .5 y .5
s4 x .8 y .2
s9 x .5 y .5
s10 x .5 y .5
.3
.4
.3 1
1
1
1
.3
1
.7
1
1
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 20
Hidden Markov models
• HMM training (Baum-Welch / EM algorithm) – re-adjust a(i,j), b(o,i) to increase Pr(O|M) – iterative procedure – allows for fine-tuning of HMM parameters for
particular observation sets • what’s in the set? everything? a single person?
a specific group of people? (e.g., R/L-handed for writing, male/female for speech)
– susceptible to local minima! • and this is often a problem!
Page 6
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 21
Hidden Markov models
• Composing HMMs – we can add “sub-” HMMs into larger HMMs,
creating a model hierarchy at different levels
• For instance, we can create three levels – strokes – letters – words
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 22
On-line recognition
• Stroke HMMs with states – up-down loop
• s1: up, + curvature, hi velocity • s2: down, + curvature, hi velocity
– up-down cusp • s1: up, + curvature, , hi velocity • s2: 0 velocity • s3: down, + curvature, hi velocity
– up-down rhamphoid • s1: up, – curvature, hi velocity • s2: 0 velocity • s3: down, + curvature, hi velocity
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 23
On-line recognition
• Letter HMMs based on stroke HMMs
ramphoid cusp
cusp cusp
loop down-up loop
(‘o’ ?)
(‘w’ ?)
(‘g’ ?)
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 24
On-line recognition
• Word HMMs based on letter HMMs
– basic idea is straightforward – but it’s deceptively tricky — why??
H E L L O
P L A T Y P U S
Page 7
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 25
On-line recognition
• Putting it all together – compacting states – taking word frequencies into account – where do frequencies come from?
Note: Probabilities aren’t real…
H
E L …
.07
I
…
A
…
.43
.31
.04
.27
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 26
Handwriting issues
• Vocabulary size • Individual variability
– writer dependent / ind.? adaptive?
• Signal segmentation – isolated words, continuous
• Speed-accuracy tradeoff • Printed vs. handwritten?
– often some combination of the two!!
• Dotting i’s, crossing t’s, … for on-line recog. • Mixing language with graphics & gestures
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 27
Speech input
• Another up & coming input modality • A few benefits of speech...
– like handwriting, very natural – hands-free
• you can avoid carpal tunnel • you can drive while writing email (ok, benefit??)
• ... and a few drawbacks: – some people hate talking – some people hate other people talking – and again, the recognition problem
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 28
Speech recognition
• Limited speech recognition – only allow small sets of words/phrases
• e.g., “one” – “nine”
• Full speech recognition – again, general, well-developed skill – full standard vocabulary (1000-10000+ words) – specialized vocabularies (research, medical, …) – “editing” vocabularies (back, delete, …)
• Same basic ideas for recognition – convert to recognizable signal (transforms) – recognize using hybrid methods along with
hierarchy of phonemes, words, etc.
Page 8
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 29
Speech issues
• Speech has many of the same issues as pen and handwriting input – vocabulary size – individual variability
• speaker dependent, adaptive, independent
– signal segmentation • isolated words, continuous
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 30
Research Question of the Day
• How quickly can users enter input given the various input modalities?
• There’s lots of research out there about such questions
• Let’s focus on one: using a laptop trackpad • What’s faster for button pressing?
– pressing a button with the thumb? – lifting off and tapping the trackpad? – some other method?
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 31
MacKenzie, I. S., & Oniszczak, A. (1998). A comparison of three selection techniques for touchpads. In CHI 98 Conference Proceedings (pp. 336-343). New York: ACM Press.
Research Question of the Day
• MacKenzie & Oniszczak (1998) compared: – “button”: pressing a
button with the thumb – “lift & tap”: lifting off
and tapping the trackpad
– “tactile”: sensing pressure on trackpad
• shown to right...
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 32
Research Question of the Day
• With a simple questionnaire, people showed some interesting opinions:
Page 9
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 33
Research Question of the Day
• Here are the quantitative results:
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 34
Research Question of the Day
• And when measuring “throughput,” which combines speed & accuracy:
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 35
Research Question of the Day
• How does this throughput compare with other input modalities? – trackpad 1.0 – 1.5 bps (what we just saw) – trackball 2.0 - 3.5 bps – mouse 3.0 – 4.5 bps
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 36
Multimodality
• Today’s desktops mostly use mouse & keys • We’ve already started expanding the horizon
– speech input is more common – pen tablets work & feel better
• Off the desktop, it’s even more important – virtual worlds
• allow for pointing, grabbing, etc. • “direct manipulation” becomes even more direct
– in-vehicle devices • visual & manual channels are busy • can interface exploit other channels? • what are the cognitive implications?
Page 10
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 37
Other input modalities
• Eye-movement input
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 38
Other input modalities
• Sign-language input
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 39
Redundancy
• Redundancy is a great thing • Use many input devices together • Redundancy is a great thing • Allow the user to choose between them • Redundancy is a great thing • And not choose once, but continuously • Redundancy is a great thing • With freedom comes speed & usability • Redundancy is a great thing
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 40
Redundancy
• Command examples of redundancy – mouse & keyboard
• e.g., menu commands & keyboard shortcuts
– commands in different places • e.g., menu commands & toolbar items
– mouse & eye ??
Page 11
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 41
Input in Swing
• Swing monitors many types of input • Most common input
– mouse: click, motion, drag – keyboard: keypresses, “focus” – (file: read, write)
• Other inputs – speech: through speech recognition API – pen: through 3rd-party APIs and toolkits
• e.g., SATIN developed at UC Berkeley
CS 338: Graphical User Interfaces. Dario Salvucci, Drexel University. 42
Input in Swing
• So how do I write the code? – Swing prefers not to give you input directly
• e.g., mouse position, keystroke
– this is a good thing! • you can still read such input; it’s just not preferable
– instead (as you know) it’s based on events that are processed in the context of a GUI component
• e.g., mouse click as an event on a button • e.g., keystroke as an event in a text component • e.g., keystroke as an shortcut to a menu command