SingleOperatorControlofMul2pleUAS:A
SupervisoryDelega2onApproach
PresentedtoUASEXCOMScienceandResearchPanel(SARP)WorkshoponSingleOperatorControlofMul2pleUAS
JayShivelyNASA-AmesResearchCenter
UAS INTEGRATION IN THE NAS
JayShivelyDAASub-ProjectManager
1
Levelsofautoma2onofDecisionandAc2onSelec2on(Sheridan&
Verplanck,1978)
1Thecomputeroffersnoassistance,humanmusttakealldecisionsandac2ons2Thecomputeroffersacompletesetofdecision/ac2onalterna2ves,or3Narrowstheselec2ondowntoafew,or4Suggestsonealterna2ve,and5Executesthatsugges2onifthehumanapproves,or6Allowsthehumanarestrictedveto2mebeforeautoma2cexecu2on7Executesautoma2cally,thennecessarilyinformsthehuman,and8Informsthehumanonlyifasked,or9Informsthehumanonlyifit,thecomputer,decidesto10Thecomputerdecideseverything,actsautonomously,ignorestheHuman
SupervisoryControlSheridan(2002)definedsupervisorycontrolasanarrangementinwhich“oneormorehumanoperatorsareintermiaentlyprogrammingandcon2nuallyreceivinginforma2onfromacomputerthatitselfclosesanautonomouscontrolloop,”buthealsoaccentuatedthehumansystemrela2onshipunderlyingthedefini2on:“Supervisorycontrolderivesfromthecloseanalogybetweenasupervisor’sinterac2onwithsubordinatepeopleinahumanorganiza2onandaperson’sinterac2onwithintelligentautomatedsubsystems”Supervisorycontrolisageneraltermforcontrolofmanyindividualcontrollersorcontrolloops,suchaswithindistributedcontrolsystem.Itreferstoahighlevelofoverallmonitoringofindividualprocesscontrollers,whichisnotnecessaryfortheopera2onofeachcontroller,butgivestheoperatoranoverallplantprocessview,andallowsintegra2onofopera2onbetweencontrollers.
Delega2onControl:Playbook®• Delega2on:onewayhumansmanagesupervisory
controlwithheterogeneous,intelligentassets
• Playbook®:onesmeansofdelega2on
• Plays:analogoustofootball– Quickcommands–complex
ac2ons
• APlayprovidesaframework– Referencesanacceptablerange
ofplan/behavioralterna2ves– Requiressharedknowledgeof
domainGoals,TasksandAc2ons– Supervisorcanfurtherconstrain/
s2pulate• Poten2allyfacilitatesintui2vecoopera2vecontrolof
UnmannedSystems
• Drill-downandmodifyasrequiredbycontext
ApagefromAlonzoStagg’s1927Playbook
TANGO
Predator Provides Overwatch and Hellfire strike capability
Shadow designates target
Firescout Does Quick Med Drop
Example:TroopsinContactTango
Example:ProsecuteTarget
Tools:Armlaser➔Lasetarget➔SendcoordinatestoweaponizedUAV➔ToggleUAVs➔Armmissile➔Fire
Scripts:Select‘Lase’script➔ToggleUAVs➔Armweapons➔Fire
Plays:Select‘ProsecuteTarget’play➔Fire
0%
10%
20%
30%
40%
50%
60%
70%
80%
Plays Scripts Tools
% C
orre
ct
Control Mode
Secondary Task Performance (% Correct Hits)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Plays Scripts Tools
Ove
rall
Rat
ing
Control Mode
NASA-TLX Ratings
0
10
20
30
40
50
60
Plays Scripts Tools
Rea
ctio
n Ti
me
Control Mode
Primary Task Performance (RT)
Target Acquisition
Target Prosecution
ShorterReac2onTimeforPlays
PlayshadlowerworkloadHigherAccuracyforPlays
LevelsofAutomaConSimulaCon
Manned-UnmannedTeaming:MUM
Level IV Control: Control of Payload and Vehicle Excluding Take-off and Landing
Extend to simultaneous control of multiple heterogeneous UAS
Manned-UnmannedTeaming:MUM
Goals:
• ApplyPlaybook®methodologyandDelConlessonslearnedtohelicoptercockpit;Testinsimula2on
• IncreasecapabilityandefficiencyofUAScontrolbyhelicopterpilots
• Supervisorycontrolofmul2ple,heterogeneousUAS
• Developinfrastructureandlayfounda2onforlaterefforts
Propor2onofTargetsMarkedbyControlMode(OutofTotalPossible)
0
0.2
0.4
0.6
0.8
1
NoUAV Manual Playbook
Prop
orCo
n
ControlMode
p < .05
0
5
10
15
20
25
Manual Playbook
Time(s)
ControlMode
UASRoutePlanningTimebyControlMode
Results
0
2
4
6
8
10
Temporal Frustra2on Performance Overall
AverageRa
Cng
WorkloadDimension
NoUAV Manual Playbook
NASA–TLXRa2ngs
p < .05 p < .05 p < .05
p < .05
HigherAccuracyPlaybook
LowerRoutePlanningTimeforPlaybook
LowerworkloadforPlaybookonseveraldimensions
FlightDemonstra2on2009Ft.OrdCA,23APR2009
Goal:• Demonstratesini2alproofofconceptof
Delega2onControl(Playbook)inflight–supervisorycontrolofmul2pleair/groundassetsinMOUTScenario
Method:• Live/VirtualDemo–ControllingRMAX,CMU
MAXRoverand2virtualUASwithDelega2onControl
• VoiceRGNControl(USAF)
Features:• Delega2oncontrolhuman-machineinterface
supportscontrolandmonitoring4payloads• Automa2onTransparency• LiveUGV-UAVcoordina2onforslungload
drop• Reducedoperatorworkload/highsitua2on
awareness
Live RMAX Virtual Shadow
Virtual Sky Warrior
Live CMU
MAX Rover
• Troopsincontact
• RouteRecon
• AreaRecon
• Convoysupport
• QuickMeds
TopPlays
FlightDemonstra2on2011Ft.Hunter-Ligge9CA,19May2011Purpose:• Buildonprevioussimula2onsandflighttest
examiningsingleoperatorcontrolofmul2pleheterogeneousground/airunmannedsystemsthroughdelega2oncontrolemployment– Operatorperformancedatacollec2on/workload
assessments– Heterogeneousflightassets:BoeingScanEagle
andYamahaRMAX;twovirtualUAS– Tes2nginopera2onallyrelevantmissionscenarios– Mul2-sensorcross-cueinsupportofboth
targe2ngandconvoysupport• ArmyAFDD/BoeingCRADAKeyObjec2ve:• DevelopandtestDelConTopPriorityPlays;
routerecon,convoysupport,troopsincontact
Demonstratedinnumeroussimula2onsandflighttests(evenNOPEsimula2ons)
• AFRL–Basesecurity,UASgroundsta2on
• RCO–Dispatch,cockpit
• HumanAutoma2onTeaming(HAT)
SupervisoryControlSummary
13
CivilUASPlays
• MonitorBorder– Flydesignatedborder– Alertany“signsoflife”
• UAS1–flywaypointatob• UAS2–flyWPbtoC• UAS3–follow-upwithanyalerts
• Evaluatepowerlines• Transitairspace
CivilPlays
• SearchandRescue– Flydesignatedareasofsearchzone–lawnmowerpaaern,alertshapes,colors,etc.
– Survivaldrop–assoonasWPisdesignated• Meds• Radio• Food/water• Shelter
HATAgent
HAT Agent
Operator
Interface
DisplayAudioVisual
Context - Time Pressure- User Info- more
Queries/Requests - A v. B- Why?- What If?
Automation
Plays- Goals- Risks to
achieving goals- Mitigations
AlertsContextResponses to Queries- Alternatives- Transparency info - Predicted Outcomes - Reasoning - Confidence level
Transparency Info
RequestsPolling for Risks
Etiquette Rules/Contextual Sensitivity
Authority Info
Scratch Pad
16
• Pilotdirectedinterface
– Nointentinferencing– Directedbypilotac2ons– Nosetrolesandresponsibli2es– Playbook
• Bi-direc2onalCommunica2on
– Why?– Howconfident?– Whatif?– Addinforma2on
• Transparency
– Calibratedtrust– Granularity– Timepressure
HATAaributes
17
ProblemswithAutoma2on
• Briale– Automa2onotenoperateswellforarangeofsitua2onsbutrequireshuman
interven2ontohandleboundarycondi2ons(Woods&Cook,2006)
• Opaque– Automa2oninterfacesotendonotfacilitateunderstandingortrackingofthe
system(Lyons,2013)
• MiscalibratedTrust– Disuseandmisuseofautoma2onhaveleadtoreal-worldmishapsand
tragedies(Lee&See,2004;Lyons&Stokes,2012)
• Out–of-the-LoopLossofSitua2onAwareness– Trade-off:automa2onhelpsmanualperformanceandworkloadbut
recoveringfromautoma2onfailureisotenworse(Endsley,2016;Onnasch,Wickens,Li,Manzey,2014)
18
HATSolu2onstoProblemswithAutoma2on
• Briale– NegoCateddecisionsputsalayerofhumanflexibilityintosystembehavior
• Opaque– Requiresthatsystemsbedesignedtobetransparent,presentraConaleand
confidence– Communica2onshouldbeintermstheoperatorcaneasilyunderstand
(sharedlanguage)
• MiscalibratedTrust– Automa2ondisplayofraConalehelpshumanoperatorknowwhentotrustit
• Out–of-the-LoopLossofSitua2onAwareness– Userdirectedinterface;adaptable,notadap2veautoma2on– Greaterinterac2on(e.g.,negoCaCon)withautoma2onreduceslikelihoodof
beingoutoftheloop
19
WorkingAgreements
• Pre-determinedauthoritysharingagreementswithautoma2on– Ifthewatercoolingleveldropsbelowacertainvalue,openvalvestoemergencycooling
• Autonomy– Notmuchintoday’s“approved”UAS– WordsMaaer
• ICAO
• Businesscaseforsingleoperatorsupervisorycontrolofmul2pleUAS– Playbookdelega2onisonesuccessfulmethod
• HAT– Coopera2veagentwithknowledgeofworkdomain– Sharedworldknowledge– Canbeextendedtonetworksupervision
Summary
21