Post on 13-Jan-2015
description
transcript
MUTUALLY LINKED STUDIESBalancing Threats to Internal and
Ecological Validity in InfoVis Evaluation
NIKLAS ELMQVISTSchool of Electrical & Computer Engineering
Purdue University
2
The Problem of Validity
• InfoVis evaluation often comes in two flavors:– Rigorous + unrealistic• Scientific rigor• Toy dataset• => Lacking in ecological validity
– Realistic + ad hoc• Real dataset• Imperfect experimental design• => Lacking in internal validity
3
Example: The ColorLens[McDonnel & Elmqvist 2009]
4
Example: Evaluating the ColorLens
Realistic photographs:• Ecologically valid (+)• Learning effects (-)• Cannot control images (-)• Does not “feel” canonical! (-)
Perlin Noise:• Full control over dataset (+)• Eliminate learning (+)• Canonical and abstract (+)• Not realistic! (-)
?
5
Mutually Linked Studies
• Why choose one type of study? Do both! • Idea: Mutually linked studies– 1 x toy study – canonical and exhaustive– 1 x realistic study – prove it works– Complement each other– Recycle much of the mechanics for both
• Experimental design• Tasks• Analysis• Participants (!)
Reduces time and paper space requirements for conducting two studies (they are not separate)!
6
Questions?
• Acknowledgments: Color Lens co-authors(coming soon to a journal near you)Pierre DragicevicHM Jean-Daniel Fekete
7
Validity Primer
• Several different types of validity…• I will discuss the following here:– Internal validity• Degree to which the outcome is a function of the
controlled parameters of experiment
– Ecological validity• Degree to which the results of the experiment can be
applied to realistic situations