© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
expressive interfacesandruid kerne
interface ecology labdepartment of computer sciencetexas a&m university
temporal composition
Media Workbenchediting, annotation
locative mediacollection server
contextsemantics
media presentation
GP
S
mobile media
capture
site-specificmobile
locative
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
expanding the roles of digital systems in human life
support a broad range of activities
emphasize human expression
ecosystems approach: combine methods (art, science)
interface design, knowledge representation, conceptual art
information visualization, pattern recognition, information retrieval
new hybrid methods of evaluationqualitative data (ethnographic)
quantitative data
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
research areas
combinFormation
represent collections through mixed-initiative composition of image and text surrogates
embodied interfaces
choreographic buttons
location-aware gaming
locative media collectiontemporal
composition
Media Workbenchediting, annotation
locative mediacollection server
contextsemantics
media presentation
GP
S
mobile media
capture
site-specificmobile
locative
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
combinFormation
represent collections through mixed-initiative composition of image and text surrogates
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
information discovery
not just finding information
having ideas in the context of the stimulus of new information
creativity
collecting new sets of existing materials
seeing them in new ways
search 1 search 2 search n
User:forming goals ,finding, restructuring,understanding
composition ofhybrid
surrogates
...
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
library science
a replacement for an original item, which gives some description of the
item, and how it can be obtained
surrogate
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
digital surrogate(aka hypersign)
represents aninformation resource
special type of hyperlink
formed from metadata
tightly linked to the original
the surrogate is the object!?
amazon catalogue entry
google gists
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
prior representations of collections
textual list
spatial hypertext
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
evidence for image + text
working memory
phonological loopprocesses auditory & text information
visuo-spatial sketchpad
dual coding strategy for representation of surrogates
marchionini et al 1999-2003
“combined” surrogates for video -> better comprehension, reduced processing time
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
composition
“putting together”
assemble collection elements to form a connected whole
visual design principles represent relationships
spatial organization
compositing
fading
expressive
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
combinFormationmixed-initiative composition of
surrogates
initiatives
user
agent
processes
information finding / choosing / collecting
visual composition
semantic model
Generative Information Collecting
GenerativeTemporal Visual
Compositioncomposition space
Direct Manipulation Information Collecting
Directing the Agent
Direct Manipulation Composition
Seeding
Agent InitiativesUser Initiatives
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
direct manipulation
information collecting
visual composition
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
direct manipulationinformation collecting
drag and drop
Firefox Extension
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
metadata system
labels
values
types
details on demand
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
direct manipulationcomposition
rearrange, remove, edit, color, resize
navigate
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
generative agents
model
information collecting
temporal visual composition
semantic model
Generative Information Collecting
GenerativeTemporal Visual
Compositioncomposition space
Direct Manipulation Information Collecting
Directing the Agent
Direct Manipulation Composition
Seeding
Agent InitiativesUser Initiatives
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
interface design to direct the agent
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
example
paper thin displays
wearable computing
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
experiment 1direct manipulation only
source format only
create a 3-level collection of psychology resources in 3 formats
formats of surrogates
text
images+text
formats of collection
linear list
spatial composition
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
results:subjective measures
liked best [X2(2) = 34.88, p < .001]
easiest to use [X2 (2) = 16.12, p < .001]
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80
like the most easiest to use most helpful Surrogate Format
Prop
ortio
n of
resp
onse
s
Spatial Labeled Images Spatial Text
Linear Text
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
experiment 1 addressed
format for presenting information collectionto users
but: for information discovery questions, the answer is also a collection!
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
4 divergent thinking questions
input surrogate collection format
linear text
composition of image+text
output surrogate collection format
linear text
spatial image+text
experiment 2direct manipulation only
source format + answer format
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
experiment 2 results
initial participants ratings
23
18
0
5
0
5
10
15
20
25
Which answer format did youlike better?
Which answer format wasbest for expressing your
ideas?
# of
Res
pons
es
Image+TextComposition(cF)
Linear text
Answer format
3.02 2.772.602.33 2.952.98
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
expressive communicative informative
text lists
image+textcomposition
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
experiment 3 - mixed-initiativesfield study - initial deployment
ENDS 101/200 The Design Process
Hybrid and Invention Assignments
182 students
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
prior work Hybrid
scor
es
0.00
0.50
1.00
1.50
2.00
2.50
3.00
3.50
4.00
prior work Inventionsc
ores
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
embodied interfacechoreographic buttons
interactive art installation
body-based interface
promoting social interaction
collaborative art creation
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
a tale of two grids
two 3x3 grids
physical
visualized
direct spatial mapping: grid cells
choreography gridphysical point of interaction
movements captured via cameras
imagery gridvisualization of image collection (collage)
cross fading transition between images
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
interaction
recognizedmovements
jumpingcrouching
system actionsselection
locking image (persistence)fast forwarding
(finding specific images)
mappingsposition = selection
jump = lockcrouch = fast forward
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
affordances
yellow border = selection
state indicator iconsappear during selection
each has two visible states (i.e. open and closed lock)spatially positioned to afford activating movements
images in collection represent literal and metaphorical up and down movements
Icon for locked or unlocked
Icon for fast forwarding
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
evaluation
art exhibition /party / user study
food and musicpost-questionnaire
video-taped
movement-based vs. mouse-basedin same room, projected on two different walls
observing engagement and social interaction
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
movement-based versionmore socially engaging
“It's funny to see people jumping and concentrated in looking at the screen. It seems more natural to talk and interrupt people in the
movement-based than in the mouse-based.”
evaluation
Which was more entertaining to watch others use?
0
2
4
6
8
10
12
14
16
18
20
movement-based mouse-based equal
Responses
Num
ber o
f Par
ticip
ants
Which involved more social interactions?
0
2
4
6
8
10
12
14
16
movement-based mouse-based
Response
Num
ber o
f Par
ticip
ants
© 2006 by Andruid Kerne, Eunyee Koh, Ross Graeber, Andrew Webb, & Zach Toups. Use only as per license on last page.
embodied interfacelocative media collection
collect images, soundsw GPS identifiers
additional metadata
retrieval by location
visualizations including temporal composition