IMPRINT models of training: Digit Data Entry and RADAR

MURI Annual Meeting

September 7, 2007

Carolyn Buck-GenglerDepartment of Psychology and

Center for Research on Training

University of Colorado at Boulder

September 7, 2007

Overview: Modeling effort

• Digit Data Entry (DDE; with Bill Raymond):– Completed model– Did goodness of fit evaluations (also Bengt Fornberg)– Reported at BRIMS (Buck-Gengler, Raymond, Healy,

& Bourne, March, 2007)– Bengt and Bill: Model comparisons

• RADAR: First start in February– Beginning structure of model created; old IMPRINT– Resumed when new IMPRINT Pro available in

April/May

September 7, 2007

Summary of DDE modelwhen different, last year’s model in blue italics

• Modeled both experiments of Healy, Kole, Buck-Gengler, & Bourne (2004) in one model (only Experiment 2)

• Model based on cognitive model of DDE– Cognitive and physical components assigned to different keystrokes

– Chunking

• Difference between hands (R faster than L) and hand switching

• Learning (RT improvement) due to repetition of numbers and due to general practice (only unique numbers; improvements made to how learning modeled)– Differences in cognitive and physical learning, assigned to different

keystrokes

• Cognitive Fatigue (not in previous model)

• Accuracy over the course of the experiment– Overall, and by output length

September 7, 2007

Picture of RADAR taskone frame

September 7, 2007

Shift type

• 4 different shift types, created by crossing mapping type and load– Mapping: Consistent (CM) vs. Varied (VM)

• CM: target(s), distractors from DIFFERENT character sets (letters, digits)

• VM: target(s), distractors from SAME set

– Load: light (1-1) or heavy (4-4)• 1-1: one possible target; one blip has a character• 4-4: four possible targets; all 4 blips have

characters

September 7, 2007

Other details

• Particular target(s) different every shift• 2 sessions (training, test)

– Each session 8 blocks of 20 shifts of one type• 15 shifts have a target, 5 have only distractors• IF target, in only one of the frames, rest only distractors

– Block order each session:CM1-1 CM4-4 VM1-1 VM4-4 VM4-4 VM1-1 CM4-4 CM1-1

• Half the subjects also have secondary tone-counting task (tone/no tone crossed between sessions)

September 7, 2007

Picture of RADAR taskCM 4-4 shift example – one frame

target

September 7, 2007

Subject response

• GO-NO GO (at frame level): Press space bar if see target character in the frame, otherwise ignore

• Scored on:– Response time to correctly hitting space bar

(when target) from time target frame starts– Accuracy in correctly identifying target– Correct rejection of non-target frames (by

non-response)

September 7, 2007

Scoring

Non-target shiftEvery frame non-target

Target shift

No response Correct reject Miss

Response

- any frame False alarm NA

- before target frame NA False alarm

- target frame NA Hit

- after target frame NA Miss

September 7, 2007

Results to be modeled• RT:

– CM faster than VM, 1 target faster than 4

– CM 1-1 and VM 1-1 about same RT; CM 4-4 a bit slower, VM 4-4 much slower

• Hit rate:– CM more accurate than VM, 1 target more accurate than 4

– VM 4-4 is only shift type to show accuracy degradation

– Small improvement over time

• False alarm rate:– sharp learning curve between first and second blocks

– some penalty for VM and more for VM 4-4

– lots of learning between 1st and 2nd block of VM 4-4 (blocks 4 and 5) and 1st and 2nd blocks of CM 1-1 (blocks 1 and 8)

• Tone counting hurts both accuracy and RT– If done during training, hurts performance at test regardless of whether

tone counting at test

September 7, 2007

Current modeling approach• Similar approach to how things done in digit data

entry model– Simplest subset of data modeled first, then after that

works, add new ones• CM/VM x 1-1/4-4; no tone counting• Tone counting• Learning• Add test session to training session

– RT and accuracy• Statistical Ss will vary around the mean S RT and accuracy

for the experiment• RTs vary for each trial around the base time for the S• Response results from random selection of trial as response

or no response based on hit and false alarm rates

September 7, 2007

Model

• Two networks– Computer:

• presents frames within shifts; 15 of 20 shifts per block contain target; target randomly assigned to one of frames 2-6

• “presents” tones to tone-counting “subjects” (will be half of subjects in each session)

– Subject goal network: wait until a frame is presented, then respond (or not) to that frame

• Third network planned– Probable second subject goal network: count tones

that are different from the reference tone

September 7, 2007

Main network

Do One Shift subnetwork

Subject goal network

September 7, 2007

Do one shift subnetworkSubject network responds here

Parallel paths forpresenting targets

and presenting tones

Parallel pathsrejoin here

September 7, 2007

CM vs. VM

• CM: Targets from different character set than distractors very easy and fast discrimination for CM

• VM: Targets from same character set as distractors slower decision

• MODEL: Two separate paths; decision time longer on VM path than CM path; time accrued for each blip looked at

September 7, 2007

1 target/blip filled invs. 4

• Eye movement: Occurs for each blip looked at• Decision:

– 1: once there, just have to make the decision– 4: assumption: sequential search/decision process

because blips start outside fovea• Target frames: Target occurs equally in any of the 4

locations, and subjects could focus first equally on any of the 4 locations. So 1/4 of the time, should need to look at just 1, 1/4 of time look at 2, etc. time to make decision accrues for each looked at

• Non-target frames: Subject should have to look at all 4; when they don’t false alarm, they don’t respond and time will be maximum frame time

September 7, 2007

Subject goal network Respond to frame

Decide on whether to respond

Respond (or not)

Response time (positive response) =N*(eye movement + decision time) + time to press space bar

N = number of blips looked at to make decision

CM

VM

September 7, 2007

Tone counting

• Making the go-no go target detection decision (primary task) incurs cognitive workload

• Keeping track of tones (secondary task) adds cognitive workload

• Assumption: when workload added, if workload threshold reached, something suffers; accuracy, RT, or both (on primary task)

September 7, 2007

Learning

• Learning is our term for the observation of improved RT or accuracy over the sessions

• Assumption: Learning only occurs on correct trials

• Data:– No consistent pattern of improvement for RTs – Small improvement in hit rates (by shift type) (silent

group)– More complex pattern in false alarm rates

• Tone counting, to the extent it decreases (detection) accuracy, will decrease improvement due to learning

September 7, 2007

Current state of model

• Only silent group, only first (training) session

• RT, Hit rate, False Alarm rate– Incorporates both hit rate and false alarm rate

into decision making

• Trial-level variability of RTs

• NOT doing tone counting or learning yet• Not yet incorporating subject-level

variability

September 7, 2007

RTs for correct shifts

0

200

400

600

800

1000

1200

1400

1600

CM 1-1 CM 4-4 VM 1-1 VM 4-4

Shift type

RT (ms)

DataModel 24 Ss

September 7, 2007

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

CM 1-1 CM 4-4 VM 1-1 VM 4-4

Shift type

Hit rate (proportion of targets correctly

identified)

DataModel 24 Ss

Hit rate

September 7, 2007

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

CM 1-1 CM 4-4 VM 1-1 VM 4-4

Shift type

False Alarm Rate

DataModel 24 Ss

False Alarm Rate

September 7, 2007

Planned

• Learning/training– Program it similarly to method used in DDE

model– Explore possibility of using new ARL training

plug-in

• Incorporate IMPRINT workload– Tone counting impact on performance

• Add second (test) session– Tone vs. non-tone manipulation– Effects of training over longer period

September 7, 2007

Contributions of IMPRINT modeling of DDE and RADAR• Translatable to MATLAB – allows using MATLAB for model comparison and parameter optimization

• Allows exploration of feasibility of modeling basic low-level cognitive tasks in IMPRINT

• Leads to insights concerning underlying cognitive processes

• Leads to predictions concerning training effects and suggestions for further experiments