AugCog System Architecture

Semi-ProcessedSensor Data

Gauge Data

Tasks

Cognitive StateAssessment

Task SystemSensor Data

Sensor Data Processing

Polly Tremoulet, PhD. Lockheed MartinAdvanced Technology Laboratories

Performance Augmentation through Cognitive Enhancement (PACE)

HCI International / AugCog InternationalJuly 25, 2005

Overview• Background

– Augmented Cognition– Sensors and Cognitive States– Mitigation Strategies

• System– Design Goals– Component descriptions– Task selection – Modality selection

• Ongoing and future work

Sensors

Background: Augmented Cognition

• Goal: Maximize operator cognitive performance in dynamic, complex operational environments

• Approach: Biophysical sensor technology assesses operator cognitive state– Detects, predicts, avoids overload to reduce

operator error and maximize effectiveness

• Benefit: Improve operator performance– Increase situation understanding– Reduce errors – Improve accuracy

DomainSimulation

Cognitive State Assessor

User

Current Suite of Sensors • EEG

– Placement:• Monopolar placement of sensors along middle of head• Bipolar placement of sensors on both sides of the head

– Sensors: Electrodes– Preprocessing: None

• EKG – Placement: Traditional placement on left and right shoulders and abdomen– Sensors: Electrodes– Preprocessing: None

• GSR – Placement: Non-traditional placement on toes rather than fingers– Sensors: Electrodes– Preprocessing: None

• Down-selection criteria:– Correlation with performance– Interoperability with other sensors – Physical discomfort for users/subjects– Portability and robustness in operational environments

Sensor Data Processing Path

Gauges MitigationsCSASensorsPre-Processing

EEG

fNIR

GSR

EKG

Pupil

Sequencing

Pacing

Reinforce-ment

ModalitySwitch

Spatial WM

Verbal WM

Workload

Arousal

NeuralNetwork

Mitigation Strategies• Pacing

– Delegation– Defer– Decomposition

• Intelligent Sequencing– Ordering based on modality and priority

• Modality switching– Changing presentation modality based upon capacity

• Multi-modality reinforcement

Initial Gauge and Mitigation OptionsGauge Trigger Mitigation logicWorkload Above

thresholdPacing = change timing of Secondary tasks Decomposition = break down Primary and/or Secondary tasks

Arousal Below Range

Above threshold

Request attention / alertOffload/delegate workOffload/delegate workDecomposition

Spatial WM

High compared to Verbal WM

SequencingVerbal Modality ShiftChunking

Verbal WM High compared to Spatial WM

SequencingSpatial Modality ShiftChunking

PACE High-Level Software Architecture

ExternalApplication

User Environment Director (ED)

Active Task Manager (ATM)

Task Information Manager (TIM)

Cognitive State Assessor (CSA)

System Interface Director (SID)

Adaptive Workload Director (AWD)

Delegation Manager (DM)

DelegatedTasks

ProposedTasks

CognitiveState

SensorData

TaskInteractions

PresentedTasks

ProposedTasks

NewTasks

UserActions

UserPerformance

ConfigurationFiles

TaskAttributes

Overarching Architectural Concepts

• Domain Neutrality– In order to provide the most generally useful and reusable system, as many

components as possible are written without reference to domain.– Configuration files allow tasks, priorities, and application information to be

specified per-domain– Certain components include domain-specific extensions to manage domain-

specific logic• Component Separability

– CommsProvider interface allows easy exchange of underlying communications layer

– All components operate independently, subscribing for and publishing particular types of messages through CommsProvider

– Allows reconfiguration of system to separate machines and eases integration with other applications

Configuration Files• Purpose: Allow per-domain and

run-time configuration of tasks

• Used primarily by TIM but also usedby Environment Director and potentially others

• XML-based formats for each configuration file:– Augmentation – configuration and selection of mitigation strategies– Modalities – specification of modalities in which tasks may be presented– Presentation – specification of modalities supported by external

applications– Priorities – assignment of priorities and urgency of different types of tasks

App

User ED ATM TIM

CSA SID AWD DM

Conf

Task Information Manager• Purpose: Manage the creation,

evaluation, and decomposition of individual tasks

• Creates new tasks in response to external stimulus

• Implementation for TTWCS experiments creates tasks based on a scenario script

• Monitor performance of the user to provide feedback and potentially influence mitigations

• Perform task decomposition and combination (not currently being used as a mitigation)

App

User ED ATM TIM

CSA SID AWD DM

Conf

Adaptive Workload Director• Purpose: Manage the set of tasks

awaiting user attention

• Maintains priority-based queue ofpending tasks

• Maintains dependency graph indicating tasks which are dependent upon the completion of other tasks before they may be presented to the user

• Proposes tasks to present to the System Interface Director

• Tasks are proposed upon completion of a task, rejection of a proposed task, and on a periodic update (10 sec.)

• Tasks to propose are selected based on their priority and how long they’ve been waiting in the queue

• Tasks which are rejected can be replaced on the queue, sent to the TIM for decomposition into smaller tasks, or sent to the Delegation Manager for the task to be handled elsewhere

App

User ED ATM TIM

CSA SID AWD DM

Conf

Delegation Manager

• Purpose: Reassign tasks to a peer, either another human user or an intelligent agent

• The functionality of the DM is notbeing used for TTWCS, as only a single operator is responsible for handling all tasks

App

User ED ATM TIM

CSA SID AWD DM

Conf

System Interface Director

• Receives periodic updates of cognitive state from Cognitive State Assessor

• Receives task proposals from the Adaptive Workload Director

• Using cognitive state and currently active mitigation strategies decides whether to accept the proposed task or to reject the task, sending it back to the Adaptive Workload Director

• Accepted tasks are passed on to the Environment Director

App

User ED ATM TIM

CSA SID AWD DM

Conf

•Purpose: Perform mitigations based on the current cognitive state of the user

• Purpose: Evaluate the currentcognitive state of the user

• Currently implemented as Proxy to Labview implementation

• Labview performs data exchange with sensor systems via established protocol and executes neural network function

• Gauge values are sent out of CSA to the System Interface Director

• Also includes capability to provide current performance as inputs to neural network, but this is not currently used in TTWCS domain

App

User ED ATM TIM

CSA SID AWD DM

ConfCognitive State Assessor

Environment Director

• Purpose: Manage the presentation oftasks through the external application

• Monitors the modalities currentlybeing used on all external applications by tasks which currently have user attention

• Receives proposed tasks from the System Interface Director

• Examines tasks and attempts to select a presentation modality based on the task’s preferred modality as well as the application’s modality capabilities

• If no available modality can be found to successfully present the task, it will be rejected and sent back to the Adaptive Workload Director

App

User ED ATM TIM

CSA SID AWD DM

Conf

Active Task Manager

• Purpose: Manage the progression of actions associated with individual tasks

• Receives newly presented tasks and user actions associated with tasks from Environment Director

• Determines the appropriate next step in the task whenever a user takes an action, sending out system actions to the Environment Director

• For TTWCS, interacts with Expert Model to generate a score of the user’s performance on completed tasks

App

User ED ATM TIM

CSA SID AWD DM

Conf

jTTWCS Application• Purpose: Provide to the operator an

interface through which experimental tasks may be performed

• Recognize and forward user-initiated actions– Alert Responses– Retargetting Solutions– Coverage Indications

• React to system-initiated actions– Begin New Scenario– Add Emergent Target– Display Alert Question

• Provide Expert Model to score user responses for each type of task

App

User ED ATM TIM

CSA SID AWD DM

Conf

Launch Area

PreplannedHealth and Statuspoints

Primary (Default)d- Target

Guidance Waypoint

Loiter Pattern

Alternate (Flex)f-Target

BranchPoint

Time-critical (emergent) e-Target

The Tactical Tomahawk cruise missile represents the next generation of cruise missiles with:

– On-board mission planning– Inflight retargeting– Battle damage assessment

This weapon will now be able to service high-priority, time-critical targets, more quickly and effectively.

3. Emergent (e-target) Missions

1. Default (d-target) Missions

2. Flex (f-target) Missions

Tactical Tomahawk Application Domain

TTWCS Problem Space: increasing cognitive demands

• Launch Area Coordinator (LAC) acting as strike controller will need to:– Review Exception Reports– Re-allocate missions to shooters on ships– Review Waiver Reports– De-conflict and re-allocate missiles & air tracks – Review shooter casualty reports– Re-allocate and order backup– Monitor missiles– Re-target and Re-strike

• Apply learned heuristics:• Who’s in range? Who’s been on station longer? Who will be off-

station earliest? What is my resource availability?

Task Selection in PACE

• Tasks are inserted by application or TIM’s task generator

• Tasks are decomposed into forest of subtasks, as needed– E.g. two button clicks two trees

• (Sub)tasks assigned priorities and inserted into a queue– Prioritization function of insertion time, urgency, etc

• Proposed tasks are examined by SID and compared to CSA’s most recent assessment of cognitive workload

• Appropriate modality for next task in queue is selected

App

User ED TIM

CSA SID AWD DM

Conf

ATM

Modality Selection in PACE• Each task is defined with a preferred modality

– E.g. alerts prefer text-window panels, but may be delivered via speech

• Application interface specifies all possible modalities for each task and quality rankings for each modality

• SID examines available modalities and proposed task.– Task rejected if no slots available in any possible modality,

o/w• SID accepts task and designates it for modality of

greatest utility– Utility = combination of task preference and application’s

modality quality and user’s cognitive capacity for task

Additional LM ATL Components Developed

• Log Analyzer– Data combined from multiple ACES XML log files into one, easy to read

spreadsheet– ACES logs quickly distributed to Sub-Contractors

• Scenario Generator– Enables realistic, rapid creation of scenarios by all groups– Playback enables review of scenarios at different speeds

• ACES (AugCog Experimental System) Launcher and Distribution Tool– Every component of the ACES system can be started up or shutdown by

pressing a button– Simple install script

• All required libraries are included• Runs “out of the box” with no compilation or compatibility issues

Future Directions• Mitigation Strategy research

– Appropriate application of delegation– Multi-modal reinforcement strategies

• Using task context to control application of mitigations

• Transitioning PACE to the field: – HCI evaluation: work in progress– Training operators to use complex applications – Improving command and control operator

performance in operational environments

Why this is NOT just Advanced HCI

Cognitive ModelMeasured Verbal Task Performance is Optimal

Task System inhibits Mitigation

CSA

Hysteresis and Smoothing

Neural Network

Verbal Gauge

Spatial Gauge

Sensors

Verbal Only Task

1. Anticipates when gauge will reach threshold

2. Threshold is set to avoid becoming seriously overloaded.

Task System turns on Mitigation

PACE Architecture

Cognitive Workload Assessor

ExternalActuators/Sensors

Tasks delegated to other operators or software agents

NewTasks

Maintains a virtual work environment that optimizes communication between operator and machine

Maintains definition and state of all operator tasks both current and historic

Measures the operator’s ability to handle the current and projected workload

Optimizes presentation of current tasks within the operator’s virtual work environment

Maintains a plan that optimizes the operator’s ability to handle the current workload

Human Work Space Task Space External

TaskInformationManager

SystemInterfaceDirector

EnvironmentDirector Operator

Actions

Domain & Application IndependentDomain & Application Dependent

AdaptiveWorkloadDirector

• Manages Tasks, Alerts and Contexts

• Monitors User Performance

• Listens to Cognitive Workload Level

• Directs Cognitive Augmentations– Sequencing– Pacing– Modality Shifts– Chunking– Delegation

LM ATL AugCog Environment:Augmented Cognition Experimental System – ACES

• Experiment environment– Controlled– Repeatable– Scorable– Portable

• Provides realistic and discrete events• Isolates memory-intensive tasks• Separable spatial and verbal activities• Modular: able to gradually increase

complexity

CLIP System Configuration

Semi-ProcessedSensor Data CSA System

Gauge DataTask System

Tasks

SensorData

Sensor DataProcessing System

Sensor Data Processing Systems

• Sensor Data Processing Systems connect directly to a set of sensors• Minimal processing is performed on that data, producing a periodic report

on all pertinent sensor values• Sensor data is passed through the network to the CSA System

Sensors

Sensor DataProcessingSystem

Semi-ProcessedSensor Data

CSASystem

CSA System

• The CSA System receives sensor data from the various Sensor Systems.• Using an ANN, these sensor values are processed into a set of Gauge

values.• Current gauge values are periodically sent to the Task System to affect its

mitigation strategy.

CSASystem

Semi-ProcessedSensor Data

TaskSystem

GaugeDataSensor Data

ProcessingSystem

Task System

• Task System receives Gauge Data from CSA System.• Based on current Cognitive State, additional tasks are proposed to the user or

rescheduled if Cognitive State indicates a potential overload• Tasks which are deferred due to Cognitive State are retained and re-proposed

at a later time when the user’s state is more conducive to completing that task.

TaskSystem

Tasks

CSASystem

Gauge Data

User

Neural Network Vital Statistics• Inputs: 234 excluding fNIR, 252 including fNIR

– For each feed, 3 inputs: now, 0.5 sec ago, 1.0 sec ago– GSR: 1 x 3– IBI: 1 x 3– fNIR: 6 x 3– EEG: 74 x 3

• Combination of direct measurements and calculated values such as vigilance

– Pupillometry: 2 x 3 • Outputs: 2

– Spatial Working Memory– Verbal Working Memory

• Hidden/Internal Nodes: 200 (single hidden layer)• Type: Feed-forward• Training Method: Standard Back-propagation

Building the Neural Network• Data Collection

– Collected data during several scenario runs for 3 subjects– Subjects performed same types of tasks to be used during CVE

• Training– Untrained network created in Stuttgart Neural Network Simulator

(SNNS) – SNNS provided with data files from scenarios– 1000 training epochs executed

• Standard back-propagation, no momentum factor, learning rate = 0.2

– Resulting network converted to C-function to be embedded within Labview sensor pre-processing system

• Other experimentation– Other networks and training methods were attempted and this was the

best combination found

Using the Neural Network• CSA System

– Reads sensor values – Passes them to them to the Neural Network every 0.5 seconds

• Neural Network– Processes sensor data and returns gauge value estimations

• PACE System Interface Director– Examines current cognitive state– Perform hysteresis and smoothing on cognitive state values

• If user has been in high verbal memory workload for at least 5 seconds, postpone low-priority verbal tasks

• If user has been in high spatial memory workload for at least 5 seconds, postpone low-priority spatial tasks

Task Description and StimuliRetarget task

– Reassign missiles to service higher priority emergent targets instead of their default target destinations.

• Goal is to service as many emergent targets as possible, while maintaining coverage on as many high and medium default targets as possible.

– Tactical TargetingAlert task

– Respond to questions from a commanding officer about an ongoing strike– Commander and Team Online Interruptions

Location task– Upon Inquiry, determine what targets can/cannot be covered based on

missile coverage zones. – Situation Awareness

Benefits of AugCog in TTWCS domain• Augmented Cognition system in TTWCS environment will increase

operator performance– Number of missiles simultaneously monitored– Number of alerts handled successfully– Overall number of emergent targets handled correctly – Enable operators to employ new capabilities effectively:

• Redirection and flex missions• Multiple engagements• Overlapping strike packages

• Augmented Cognition system in TTWCS environment will reduce manning