Using color in complex visualizations

Greg TraftonNaval Research Laboratory

Using color in complex visualizations

Greg TraftonNaval Research Laboratory

Complex Visualizations,Spatial Transformations,

and Uncertainty

Greg TraftonNaval Research Laboratory

Collaborators (reverse alphabetical)

• Susan Trickett (GMU, NRL postdoc)• Chris Schunn (LRDC, University of Pitt.)• Cara Stitzlein (GMU)• Lelyn Saner (University of Pittsburgh)• Raj Ratwani (GMU) • Paula Raymond (Center for Applied Research)• Farilee Mintz (ITT/NRL)• David Jones (APL, U. of Wash)

• Funded by the Office of Naval Research

Undisplayed Uncertainty

• Notice these visualizations show no uncertainty

Displaying uncertainty in this workshop

• Captain Knight’s COP visualization with color-coded dots showing nationality of each ship (lots of data, little uncertainty)

• Data fusion examples (Duquet)– Uncertainty was not displayed at the visualization

level, but rather calculated “under the hood” by various classifiers (e.g., 80% merchant, 20% unknown)

• Col (Ret’d) Johansen’s comment: display that uncertainty so people can make the decisions!

Problem

• How do experts reason with uncertainty in these highly spatial domains that display little or no explicit uncertainty?

• Is there a relationship between uncertainty and spatial reasoning? What is that relationship? What’s the process that experts go through to deal with uncertainty?

Framework

• Since uncertainty is critical in complex visual/spatial domains (Schunn, Kirschenbaum, & Trafton, under review-a, under review-b), people must understand how displayed uncertainty effects their task.

• If uncertainty is not explicitly displayed, the person must add their own understanding of uncertainty to the visualization itself

• Even if it is displayed, they may need to modify the displayed/represented uncertainty in some way if the person does not agree with it.

Hypothesis

• Hypothesis: Experts must mentally add their own understanding of uncertainty to the visualizations by spatial transformations– A front may be displayed in one location but the

forecaster is unsure where it will be. She may play with several possibilities (geographical or extent) and mentally move the front around until she’s happy with it.

– This moving / extending / deleting of part of an onscreen object is a spatial transformation (Trafton et al., 2000; 2002; under review).

• Specifically, experts use spatial transformations to create certainty out of uncertain data visualizations (qual vs. quant)

Theoretical Framework:Spatial Transformations

• A spatial transformation is a mental operation the user performs on a visualization or an image (happens inside the user’s head)

• The visualization or image can be either internal (mental representation) or external (on-screen visualization) (for an excellent review of cognition on different diagrammatic representations, see Hegarty, 2004)

• (Can be seen as a measure of spatial problem solving / work)

Spatial Transformations

• Mental rotation – Shepard & Metzler, 1985

• Time series extrapolation

• Object movement

• Object animation – Bogacz & Trafton, 02; Hegarty, 94

• 2D <-> 3D transformation– St. John et al., 2000; Trafton, 2002

• Comparisons between images– Trafton et al., 2002

• Other mental manipulations

0

20

40

60

0 1 2 3 4

# study repetitions

Spatial Transformations and Uncertainty

• Difficulty of task will increase when uncertain and will show itself in increased number of spatial transformations (spatial reasoning)

Specific Hypothesis for this project:

• When working with these complex visualizations, as people become uncertain, they will perform more ST than when they are certain.

Sidebar

• I’m interested in how experts deal with these issues.

• All experts performing their own data analyses in their own domain with their own tools (Dunbar’s in vivo approach)– Requires protocol analysis, detailed examination of

what people do and say (tons of rich data, not for the FOH)

– Interviews after the analysis (retrospective protocols can be suspect!)• In general, believe what people do, not what they say they

do and not what they remember

• I’ll be examining two different domains (meteorology and fMRI) to show generalizability, etc.

Study 1 (in vivo)Method (fMRI and METOC)

• 14 experts examined while they were “thinking with their data” (took about an hour for each expert)– fMRI scientists were analyzing their data for the first time– Navy forecasters were predicting what the weather

would be as part of their job

• Talk-aloud protocols given• Coded spatial transformations of in vivo fMRI and

in vivo METOC data [Trafton lab]• Coded (un)certainty of each in vivo minute

[Schunn lab]– Certain– Mixed– Uncertain

Predictions

• The more uncertain, the more spatial transformations

Example of ST coding

QuickTime™ and aDV/DVCPRO - NTSC decompressor

are needed to see this picture.

Example of ST coding(Transcription)

Utterance Comments/Code

you also have a 12 max 14 Extract info from vis

winds are not supporting that Uncertainty

the next chart has it moving down further to the south

Remembering previous visualization; comparison

there is a low coming off the coast that is probably getting around

Mentally creating and moving the low around

so I would move it further to the south

Mentally moving the low south

and that just supports what I said about ours, OK

Resolution of uncertainty

METOC and fMRI Results (Spatial Transformations)

Other times spatial transformations are used?

• Are spatial transformations used when thinking about space?

• YES!

Spatial Gestures

QuickTime™ and aDV/DVCPRO - NTSC decompressor

are needed to see this picture.

More Spatial Gestures

QuickTime™ and aDV/DVCPRO - NTSC decompressor

are needed to see this picture.

More Spatial Gestures

QuickTime™ and aDV/DVCPRO - NTSC decompressor

are needed to see this picture.

Summary

• Uncertainty in these complex domains has a very strong spatial component

• In these kinds of complex domains, when people are uncertain, they use more spatial transformations than when they are certain

• Specifically, in order to understand and deal with the uncertainty in these complex, visual/spatial domains that do not show uncertainty, experts must use spatial transformations to create certainty out of uncertain data visualizations

• People’s gestures also tell us what they’re thinking

So what?

• General info about collecting data from people– People’s memories aren’t that great – believe what they do,

not what they remember doing• They may tell you how they are supposed to do a task (or how

they think you want to hear it), not how they actually do it• They may not be able to describe how they solve problems in their

domain of expertise (like driving a car)

– Multiple converging experimental methods (protocol analysis, eye-tracking, gesture, etc.) are critical to triangulate

• Specific info about these studies– Many military visualizations do not show uncertainty– Many COP systems emphasize data display, accuracy of data,

etc. There should also be a strong emphasis on spatial cognition/spatial transformations

– How you display uncertainty is a big question – minimize spatial transformations

Fini