Acquisition and Acquisition and Retention of Team UAV Retention of Team UAV
SkillsSkills
Nancy J. CookeNancy J. Cooke1,21,2, Olena Connor, Olena Connor1,31,3, & Harry Pedersen , & Harry Pedersen 1, 3
11Cognitive Engineering Research InstituteCognitive Engineering Research Institute22Arizona State UniversityArizona State University
33New Mexico State UniversityNew Mexico State University
May 25, 2005May 25, 2005
SponsorsAir Force Office of Scientific ResearchAir Force Office of Scientific ResearchAir Force Research LaboratoryAir Force Research LaboratoryOffice of Naval ResearchOffice of Naval Research
AFRLAFRLDee AndrewsDee Andrews
CERTT Developer: US PositioningCERTT Developer: US PositioningSteven M. ShopeSteven M. Shope
Post Doctoral AssistantPost Doctoral AssistantPreston A. KiekelPreston A. Kiekel
ASU FacultyASU FacultyNia AmazeenNia AmazeenTom TaylorTom TaylorKevin DooleyKevin DooleySteve CormanSteve Corman
Research TeamResearch TeamNMSU Students
Olena ConnorJanie DeJoode
Jamie C. GormanRebecca KeithHarry Pedersen
ASU StudentsLeah Rowe
Jennifer WinnerChristy Caballero
Pat FitzgeraldJasmine DuranKaren KieslingTamica Smith
NMSU FacultyPeter Foltz
May 25, 2005May 25, 2005
OverviewOverview
UAV operations and team coordinationUAV operations and team coordinationStudies on team skill acquisition and Studies on team skill acquisition and retentionretentionOngoing effortsOngoing effortsConclusions & implicationsConclusions & implications
May 25, 2005May 25, 2005
UAV Operations UAV Operations and and
Team CoordinationTeam Coordination
May 25, 2005May 25, 2005
UAV Operations as a Team UAV Operations as a Team TaskTask
Team MembersTeam MembersGround control stationGround control station
Air Vehicle Operator/PilotAir Vehicle Operator/PilotSensor/Payload OperatorSensor/Payload OperatorDEMPCDEMPC
MaintenanceMaintenanceAir Operations CenterAir Operations CenterIntelligenceIntelligence
Characteristics
Heterogeneous
Teams of Teams
Fast Tempo
Distributed
Team Cognition in PracticeTeam Cognition in Practice
May 25, 2005May 25, 2005
Focused on process (vs. Focused on process (vs. product/knowledge)product/knowledge)It is more than the sum of the cognition of It is more than the sum of the cognition of individual team membersindividual team membersIt emerges (Gestalt It emerges (Gestalt -- like) from the like) from the interplay of the individual cognition of interplay of the individual cognition of each team member or cognitive entity each team member or cognitive entity Interaction, communication, coordination, Interaction, communication, coordination, pushpush--andand--pull of info. is teampull of info. is team--level level cognitive processingcognitive processingTeam cognition (or team mind) is Team cognition (or team mind) is observable in the teamobservable in the team’’s coordination s coordination behaviorbehaviorPoor coordination contributes to systemPoor coordination contributes to system--wide failure and can be improved through wide failure and can be improved through training and designtraining and design
Emergent Team CognitionEmergent Team Cognition
+ +
May 25, 2005May 25, 2005
Team CognitionTeam Cognition……
Is Is not in the headsnot in the heads of the individual team of the individual team membersmembersIt is in the It is in the interactions among team interactions among team membersmembers
Team coordination IS team cognition and a critical team-level skill
May 25, 2005May 25, 2005
Coordination is Critical Coordination is Critical to UAV Team to UAV Team EffectivenessEffectiveness
RQ-1 Predator According to the accident investigation board report, the Predator experienced a fuel problem during its descent. Uponentering instrument meteorological conditions and experiencing aircraft icing, the Predator lost engine power. The two Predator pilots, who control the aircraft from a ground station, executed critical action procedures but were unable to land the aircraft safely. It crashed in a wooded area. According to the report, the pilots' attention became too focused on flying the Predator in icing and weather conditions they had rarely encountered. The report also cites lack of communication between the two pilots during the flight emergency as a cause of the accident.
May 25, 2005May 25, 2005
Studies on Studies on Team Skill Acquisition Team Skill Acquisition
and Retentionand Retention
May 25, 2005May 25, 2005
CERTT Lab MoveCERTT Lab Move
December 2004
5810 S. Sossaman, Mesa AZ
May 25, 2005May 25, 2005
CERTT LabCERTT LabCCognitiveognitive EEngineeringngineering RResearchesearch onon TTeameam TTasksasks
In our UAV STE three operators must coordinate in order to maneuver their UAV to take pictures of ground targets
Unmanned Aerial Vehicle Synthetic Task
Environment
May 25, 2005May 25, 2005
The Synthetic TaskThe Synthetic TaskAir Vehicle Operatorcontrols UAV airspeed, heading, and altitude and monitors air vehicle systems
Payload Operatorcontrols camera settings, takes photos, and monitors camera systems
DEMPC navigator, mission planner, plans route from target to target under constraints
Based on a cognitive task analysis done on Predator operations at at Indian Springs, NV
May 25, 2005May 25, 2005
Examples of CERTT UAVExamples of CERTT UAV--STE STE CoordinationCoordination
Good Coordination Poor Coordination
May 25, 2005May 25, 2005
Six CERTT UAV ExperimentsSix CERTT UAV ExperimentsThree team members (AVO, PLO, DEMPC) maneuver UAV to take reconnaissance photos
Independent Variables: knowledge sharing, workload, dispersion
Primary Measures: performance, process, cognition (teamwork knowledge, taskwork knowledge, SA)
Experiment 1: 11 teams, 10 missionsExperiment 2: 18 teams, 5 missions, shared vs. unsharedExperiment 3: 20 teams, 7 missions, 5-7 high workload, distributed vs. co-locatedExperiment 4: 20 all-male teams, 5 missions, 5th high workload, distributed vs. co-locatedExperiment 5: Benchmarking, 5 “expert” teams, 5 missions, 5th high workloadExperiment 6: Team member familiarity & retention interval, 40 teams, 8 missions
May 25, 2005May 25, 2005
Team Skill AcquisitionTeam Skill AcquisitionExperiment 1Experiment 1
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10
Mission
Team
Per
form
ance
Tm 1Tm 2Tm 3Tm 4Tm 5Tm 6Tm 7Tm 8Tm 9 Tm 10Tm 11
Individuals are trained to criterion prior to M1
Asymptotic team performance after 4 40-min missions (robust finding)
Knowledge changes tend to occur in early learning (M1) and stabilize
Process improves and communication becomes more standard over time
May 25, 2005May 25, 2005
Team RetentionTeam RetentionExperiment 1Experiment 1
5510.8610.868.718.719.639.63Long (8Long (8--11 11 weeks)weeks)
446.576.574.714.715.865.86Short (4Short (4--7 7 weeks)weeks)
NNMaxMaxMinMinMeanMean(# of (# of
weeks)weeks)
Retention Retention Interval Interval GroupGroup
The question of retention of C2 skill is of great practical importance, but little is known.
May 25, 2005May 25, 2005
Expert Teams & Skill AcquisitionExpert Teams & Skill AcquisitionExperiment 5Experiment 5
150
200
250
300
350
400
450
500
1 2 3 4
UAV mission
perfo
rman
ce ra
te BenchmarkExp1Exp2Exp3Exp4
Teams take four 40-minute missions (post individual taskwork training) to reach asymptotic performance. Experienced teams reach the same level in 1-2 missions.
May 25, 2005May 25, 2005
Conclusion from StudiesConclusion from Studies
Coordination involves the timely and adaptive passing of information among cognitive entities in a systemTeams acquire team skill over time and development of coordination seems critical to this processPerformance (coordination skill) declines over periods of nonuseCoordination skill may transfer across tasks
But…Coordination not directly measured (just communication and process)Team familiarity confounded with coordination skill at another task
May 25, 2005May 25, 2005
Ongoing EffortsOngoing Efforts
May 25, 2005May 25, 2005
Project ObjectivesProject Objectives
Collect empirical data on acquisition and Collect empirical data on acquisition and retention of team coordinationretention of team coordination
Little/no existing team retention dataLittle/no existing team retention dataWill also manipulate team member familiarityWill also manipulate team member familiarity
Will model optimal coordination at local target Will model optimal coordination at local target waypoints and derive a metric by which to waypoints and derive a metric by which to quantify coordinationquantify coordinationWill model development of coordination over Will model development of coordination over time and under various conditionstime and under various conditionsWill use data and model to inform intervention Will use data and model to inform intervention to improve retention of coordinationto improve retention of coordination
May 25, 2005May 25, 2005
Empirical Study of Acquisition and Empirical Study of Acquisition and Retention of Team CoordinationRetention of Team Coordination
120 unfamiliar individuals assigned to 120 unfamiliar individuals assigned to 40 UAV teams and a role40 UAV teams and a roleIntervals and familiarity randomly Intervals and familiarity randomly assignedassignedTeam members will be unfamiliar with Team members will be unfamiliar with one anotherone another
Familiar Condition
Session 1 Session 2
Retention
Interval
Unfamiliar Condition
Session 1 Session 2
Retention
Interval
AVO PLO DEM
10 Teams10 Teams10 Teams10 Teams
10 Teams10 Teams10 Teams10 Teams
3-6 weeks 10-13 weeks
Fam
iliar
Unf
amili
arFa
mili
arity
Retention Interval
Current N = 4 Teams Current N = 2 Teams
Current N = 5 Teams Current N = 5 Teams
AVO PLO DEM
AVO PLO DEM AVO PLO DEM
Team Performance
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8
Mission
Mea
n Pe
rfor
man
ce S
core
Long Interval-FamiliarLong Interval-UnfamiliarShort Interval-UnfamiliarShort Interval-Familiar
Mean Performance Scores Based On Current Data Collected (N = 16 Teams)
First mission after delay
May 25, 2005May 25, 2005
Local optimal modeling Coordination metric
Local Coordination MetricExperimental observations are fed into an optimal coordination model in order to measure distance from optimal
What is optimal?F (ordering, timing, mode) =
AVODEM
PLO
1
23
3
e-λ(T)i
nff
The Coordination Logger was developed by CERI in order to record the interaction dynamics of specific UAV task elements.
May 25, 2005May 25, 2005
Dynamical Systems Model of Team Coordination Over Time
Model with controls for…
a) Retention interval
b)Familiarity
Observe team coordination (x) over time
log ∆<
x2 >
log ∆time
a
b
May 25, 2005May 25, 2005
Conclusions and Conclusions and ImplicationsImplications
May 25, 2005May 25, 2005
ConclusionsConclusions
Coordination involves the timely and adaptive passing of information among cognitive entities in a systemIn the context of our synthetic UAV operations task,
coordination is key to effective team performanceTeams acquire team skill over time and development of coordination seems critical to this processPerformance (coordination skill) declines over periods of nonuseCoordination skill may transfer across tasksOngoing studies are measuring coordination directly and manipulating team member familiarity
May 25, 2005May 25, 2005
ImplicationsImplications
Design of UAV technology needs to consider coordination requirements (don’t forget the communication function)
Staffing of UAV ground control needs to consider the need to coordinate and relevant factors such team member familiarity
Training for UAV ground control also needs to consider appropriate training regime for preserving coordination over planned retention interval
Coordination metrics, if embedded and automated in the operational or training environment, can facilitate monitoring, assessing, and improving team performance
May 25, 2005May 25, 2005
Questions or Comments?Questions or Comments?email:
website:
www.certt.com
Cognitive Engineering Research Institute
Mesa, AZ
www.cerici.org
