HCI Methods for Pathway Visualization Tools

Purvi Saraiya, Chris North, Karen Duca*

Virginia Tech

Dept. of Computer Science,Center for Human-Computer Interaction

*Virginia Bioinformatics Institute

What is HCI?

• All aspects of human interaction with a computer system

• A discipline concerned with design, implementation, and evaluation of computer systems for human use

Two Perspectives For Pathway Tools

• Developers:– What tool will be most helpful to

users?• Users:

– Many options, how to select the most appropriate tool?

Software Development Cycle

Reqs Analysis

Evaluate Design

Develop

Requirement Analysis Using Informal Interviews

Research Questions: • What kinds of tasks do

the users perform?• How do the tools fit with

overall research goals of the users?

Outcomes:• Usage Scenarios• Requirements

Method: 1-1 Interviews

Usage Scenarios

• Scenarios: stories about users and their work activities

• User-oriented: focus on needs and concerns of users

• User-perception: how do users perceive their problems?

User Requirements for Pathway Visualization Tools

Categories Requirements

Tasks

Pathway Construction

1. Create & Update

Collect and link pathways from multiple resources

2. Context Provide information about pathways

3. Uncertainty Show alternate hypotheses and information reliability

4. Collaboration

Enable group work

Questionnaires and SurveysResearch Question: Which requirements are more important?

Pathway Questions Strongly Agree

Agree

Neutral

Disagree

Category: Pathway AssemblyR1: Create & Update

1

In my work, the entire pathway(s) is generally not available from a single source. 6 4

R1: Collaborate

2

I collaborate with others and need a tool to let them enter changes from remote sites 1 1 4 4

Focus Groups• Research Questions:

– What is user feedback for the selected systems?

• Method:– Group discussion with

users • No of users:

– 5 - 10

Systems Evaluated

GenMapp, PathwayAssist, Cytoscape, Patika, GScope

Results• Example User Responses:

– Users were excited about the NLP features provided by PathwayAssist but were skeptical about its reliability

– Need more biological context for the pathways

End-Users:– Which system should I use for my work?

Longitudinal Studies • Research Question:

– How are the tools used in actual real world scenarios?

– What interactions and features were actually used by the users for their tasks?

• Method:– Users: Log Keeping– Evaluators: Discuss user logs

Example Log

Heatmap + Pathway Visualization in PathwayAssist

Date Visualization

Insight Value

9/01 Heatmap A list of genes that are suppressed by smoking but up-regulated by flu.

4

9/12 Pathway Visualization

The up-regulation of Mx by flu is suppressed by smoking even though smoking itself did not have an effect on basal Mx activity.

3

Example log for data analysis with PathwayAssist

Empirical Studies

• Research Question: – Are the tools preferred by users actually better?

• Method:– Task-Based Method– Insight-Based Method

Task-Based Studies

• Typical question:– Which visualization is better for defined

tasks?

Vs.

Vis. 1 Vis. 2

Variables• Independent Variables:

• Visualization tool» GenMapp, PathwayAssist, etc.

• Task type» Find, count, pattern, compare

• Data size (# of items)» 100, 1000, 1000000

• Dependent Variables:• Task completion time• Errors• Subjective satisfaction (survey)

Experiment Protocol

Participants may be videotaped

Evaluator takes notes

Data Analysis

• t-test:• Compares 1 dep var on 2 treatments of 1 ind var

• ANOVA:• Compares 1 dep var on n treatments of m ind vars

Task1: Find the node with max value?

Task2 Task3

Vis 1 123245

… …

Vis 2 … … …

Ind Var 1: Vis. Tool

Ind Var 2: Task Type

Dep var: user performance times (3 users per cells)

Insight-Based Method

Motivation:• Task-based methods uses pre-selected

tasks• Often not representative of the real world

visualization tool usageIssues:• How to eliminate benchmark tasks?• What is Insight? How to codify and

quantifiably measure insights across participants?

Insight Characteristics• Insight = an individual data observation

•Observation•Time to discover•Domain value (importance)•Hypotheses generated?•Directed vs. unexpected•Correctness •Category (overview, details)

• Can be recognized by thinking-aloud

Experiment Design

• Independent Variables:– Visualization Tools– Datasets– Participant Background

• Dependent Variables:– Insights– Time at which insights were reported– User feedback

Tool 1 Tool 2Tool 3 Tool 4 Tool 5

Count of insights

Domain value of insights

Average timeto first insight(in mins)

18

21

14

25

20

0

5

10

15

20

25

48 51

34

66

40

0

15

30

45

60

75

4.5

7

14

8

16

0

5

10

15

20

Data Bias

The tool works better for categorical data

36

20

9

26

42 2

14

0

10

20

30

Count of Insights Domain value of Insight Avg Time to 1st Insight(mins)

Time-series Categorical Multi-Categorical

ConclusionsDevelopers:• Informal 1-1 interviews & questionnaires • Focus group meetings• Longitudinal StudiesUsers:• Controlled studies to evaluate tools• User studies may not always be feasible• Insight characteristics can be used as

checklists for selecting a tool

Thank you