Date post: | 13-Jan-2015 |
Category: |
Documents |
Upload: | beliv-workshop |
View: | 196 times |
Download: | 2 times |
Pragmatic Issues in the Evaluation
of Interactive Visualization Systems
April 10, 2010
Youn-ah Kang, Carsten Görg, John Stasko
School of Interactive ComputingGeorgia Institute of Technology
A comparative lab study with an investigative analysis scenario
VAST 2009
Paper Desktop
Entity Jigsaw
Pragmatic Challenges
1. Student as Subject– Graduate students vs. domain experts– Difficulty in accessing domain experts – To what extent are the results valid?
2. Quantitative vs. Qualitative– Are the two really incompatible?– Using a mixed method to further
validate and enrich findings
Pragmatic Challenges
3. Statistical Power vs. Analysis Depth– Hard to have both large participant
population and in-depth analysis of each session
– Need to focus on either side
4. Scalability– Balancing between feasibility and
ecological validity– Chose the document size to make the study
feasible in a given time, but in reality…
Pragmatic Challenges
5. Familiarity with the Tool– Good usability is critical in a utility evaluation – Unfamiliarity with the tool blurs potential
benefits– Big learning curve but limited time
6. Comprehensiveness of the Task– Only targeting scenario for this study– Challenges in developing evaluation
strategies and measures for less focused scenarios
Answers
• ???