Neuro–Information Science
Jacek Gwizdka & Michael Cole iSchool @ Rutgers University, NJ, USA
http://jsg.tel
June 5, 2012
Towards
Information Science
Another IS Information Science is about : ◦ understanding information seeking behavior (why/how/where/…) ◦ helping people find information they need
2
Information Systems vs. iScience
A lot of common concerns and constructs: ◦ information is digital accessed via information systems ◦ technology – task – individual ◦ IT usefulness, user interface design, usability … ◦ trust … ◦ decision making … ◦ affective and cognitive factors ◦ information search (e.g., stopping behavior) …
3
Information Systems vs. iScience
Also new opportunities: ◦ neural-correlates of constructs specific to Information Science ◦ Information Relevance : most commonly refers to topical relevance
or aboutness, that is: to what extent the content of a search result matches the topic of the query or a person’s information need (e.g., Saracevic, 2007) relevance judgment decision making information stopping
4
Opportunities for Neuroscience to Inform IS Seven opportunities for cognitive neuroscience to inform IS research: 1. localize the neural correlates of IS constructs to better understand
their nature and dimensionality; 2. complement existing sources of IS data with neuroscientific data; 3. capture hidden (automatic) processes that are difficult to measure
with existing measurement methods; 4. identify antecedents of IS constructs by exploring the specifics of
how IT stimuli (e.g., the design of graphical user interfaces) are processed by the brain;
5. test the outcomes of IS constructs by showing how brain activation predicts behavior (e.g., decisions);
6. infer causality among IS constructs by examining the timing of brain activations due to a common stimulus;
7. challenge existing IS assumptions and enhance IS theories that do not correspond to the brain’s functionality
(Dimoka et al. 2010)
5
Neuroscience and Information Science?
Eye-tracking +++ Galvanic skin response (GSR) ++ Heart –rate variability (HRV) + EEG fNIRS fMRI
Recent and Current Projects
1. eye-tracking: modeling reading + cognitive effort 2. fMRI + eye-tracking: information relevance
7
Part I: Eye-tracking
General research goal: infer and predict mental states and context of a person engaged in interactive information searching
Influence system design adaptive systems
Macro user task characteris�cs, cogni�ve effort, domain knowledge
Meso reading pa�erns
Micro eye-‐gaze posi�ons + �ming
8
Eye-movement Presentation
9
Eye-tracking Data
10
State2 State1
State3
àà Patterns
Eye-movement Patterns
New methodology to analyze eye-movement patterns ◦ Model reading and Measure cognitive effort ◦ Correlate with higher-level constructs
user task characteristics, user knowledge, etc.
11
Reading Model Origins
12
Based on E-Z Reader model Rayner , Pollatsek, Reichle
◦ Serial reading
◦ Words can be identified in parafovial region
◦ Early lexical access (word familiarity) + Complete lexical processing (word identification)
2o (70px) foveal region parafoveal region
MORE…
Two-State Reading Model
13
◦ Filter fixations < 150ms (min time required for lexical processing) ◦ Model states characterized by: probability of transitions; number of lexical fixations; duration length of eye-movement trajectory, amount of text covered
Scan Read
1-q
p
1-p
q
MORE…
isolated fixations fixation
sequences
Example Reading Sequence
14
Fixation sequence: (F F F) F (F F F) F F F F (F F F F F F) F Reading model states: R S R S S S S R S
Reading state – R | Scanning state – S
Cognitive Effort Measures of Reading
Reading Speed
Fixation Regression
Perceptual Span
Fixation Duration (“lexical processing excess”)
15
foveal region
a b c d
Perceptual span = Mean(a,b,c,d)
regression
excess
User Study 1: Cognitive Effort and Tasks
16
OBI: advanced obituary INT: interview prepara�on CPE: copy edi�ng BIC: background informa�on
N = 32
MORE…
Journalists’ Information Search
Do the cognitive effort measures correlate with: task difficulty (by design), observable search effort, user’s subjective perception of task difficulty
Can we detect differences between task characteristics from eye-movement patterns?
Eye-data and Cognitive Effort Measures
Cognitive effort measures derived from eye-tracking
reading speed mean fixation duration perceptual span total fixation regressions
Task difficulty by design Copy Editing (CPE) Advance Obituary (OBI)
Search effort task time pages visited queries entered
Subjective Task Difficulty
CPE INT BIC OBI
As expected: Copy Editing CPE easiest Advance Obituary OBI most difficult Sig: Kruskal-Wallis χ2 =46.1, p<.0001
17
Eye-data and Task Characteristics
18
Measure Related Task Characteris�cs
Frequency of reading state transi�ons
SR bias to read Advanced obituary and Interview prepara�on tasks: search for document; task goal not specific
RS bias to scan Copy Edi�ng task: search for segment and task goal specific
Scan Read
1-q
p
1-p
q
MORE…
Copy Edi�ng Interview prepara�on
Summary: Eye-tracking Methodology
Domain independent ◦ Document content is not involved
Culturally* and individually independent Real-time modeling of user and tasks is possible Adaptive systems feasible Eye-tracking is coming to us!
Tobii
19
Part II: Current fMRI+eye-tracking Study
Information Relevance : refers to topical relevance or aboutness, that is: to what extent the content of a document (webpage) matches the topic of the query or a person’s information need (e.g., Saracevic, 2007) ◦ Relevance multi-dimensional: topical, meaningful, useful, trust, affective…
Neural correlates of topical relevance judgments Hypothesis ◦ Brain regions that are activated when relevant information is found are
different from regions activated when no relevant info is found and when person does “low-level” visual word search (orthographic matching) but no hypothesis in a sense where the brain activity is located
Exploratory research
(also: a similar experiment with eye-tracking, EEG, GSR) 20
fMRI + eye-tracking lab Lab Equipment: ◦ fMRI: 3T Siemens TRIO ◦ eye-tracker: Eyelink-1000 non-ferromagnetic optimized design; up to 2000 Hz sampling rate
21
fMRI + eye-tracking
22
fMRI + eye-tracking
Eye-tracking imposes additional constraints on projection (geometry)
projected screen
mirror
eye-tracker
23
Current Experimental Design Two blocks (types of tasks, balanced) ◦ WS – word search: find target word in a short news story – press yes/no ◦ IS – information search: find information that answers given question –
press yes/no. Three types of trials: relevant (R), topical (T), irrelevant (I) ◦ TR cycle: 2s
xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek dkdkdkkd kdkddk dkdkdk dkdkdkd kkdkd d d dd d djdj djdjdj rjrjr rjjweje ejejej ejej kek ekeke wej e ejej eje j
xmx ssms nsns snsns jsdjsd ke ekek dkdkdkk kdkddk dkdkdkdkdkdkd kkdkd d rjr jweje ejeje ekeke wej e ejej fjfjf fjfjfjfjf fjfjrjr rreje j
xmx ssms nsns snsns jsdjsd ke ekek dkdkdkkd
kdkddk dkdkdk dkdkdkd kkdkd d rjr jweje ejejej ejej kekekek ekeke wee ejej fjfjf fjfjfjfjf fjfjrjr rreje j
+ target: info
target: info
target: info
xmx ssms nsns snsns jsdjsd djdjd djdj dkke ekek kdkddk dkdkdk dkdkdkd kkdkd d d dd d djdj djdjdj rjrjr rjr jweje ejejej ejej kekekek ekeke wej e eej eje j
+ + target: word
21 x
21 x
WS task instruc-
tions
IS task instruc-
tions
+
+ + + + + +
30s 4s 6s 4s 20s max 4s
30s 4s 8s 20s max 4s 20s max 4s 20s max
24
Planned Analysis
Two blocks (types of tasks, balanced) ◦ WS – word search: find target word in a short news story ◦ IS – information search: find information that answers given question –
Three types of trials: relevant (R), topical (T), irrelevant (I)
The main contrasts of interests are: ◦ IS-R - WS ◦ IS-R - IS-T ◦ IS-R - IS-I
25
A Very, Very Preliminary Analysis
For one participant, aggregated for all trials in each of two blocks (tasks)
Word search (WS)
Information Search – Relevant (IS-R)
26
Stay Tuned for Results…
Neuro – Information Science
28
Funding: Google, HP, IMLS (now funded by IMLS CAREER) Collaborators: Drs. Nicholas Belkin, Art Chaovalitwongse (U Wash), Xiangmin Zhang,
Ralf Bierig (Post Doc); PhD students: Michael Cole (co-author), Chang Liu, Jingjing Liu, Irene Lopatovska + many Master and undergraduate students …
Acknowledgements:
Fragen?
29
More info & contact http://jsg.tel