Supporting European

Aviation

Interactive Performance DashboardsMaking sense of Big Human Performance Data in large-scale Real-Time SimulationsEUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’ Atlantic City, USA, 16/17 October 2019

Bernd LorenzHuman Performance & Validation Expert

Overview

How we do Real-Time Simulations (RTS)

The Challenge: Large-scale RTS

Examples of Interactive Performance Dashboards

The way forward

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019 2

E-OCVM: Performance-driven Operational Concept Development

3

R&D

Full-scale RTS

PeopleSystem

Procedures

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019

Staff adaptation

Validation Activities in RTS Life-cycle

Kick-off MeetingState Concept and Assumptions

Validation StrategyIdentify stakeholders, issues, aims, objectives, claims, indicators, maturity criteria, cost-benefit mechanisms and expected outputs of the validation process

Validation Plan

2-stage Acceptance Testing

Simulation Conduct

Final ReportProvide a documented body of evidence in conclusion on the claimsInteractive Validation Data Repository

Core team Review

Core team Review

Core team Review

Final Review

Data Analysis&

Initial Reporting

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019 4

Visualisation in Data Analysis and Claim Evaluation

Quantitative DataSubjective ratings

Platform recordings

Qualitative DataQuestionnaire comments

DebriefingsInterviews

SharedSense-Making

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019 5

The challenge: Large-scale RTS

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019

165ATCOs

25Controller Working Positions

11Scenarios

88Exercises

1900Questionnaires

1335Debriefing log entries

616Interview

log entriesTMA re-organisation supporting Istanbul New Airport (2018)

6

Interactive Sector Performance Dashboard

(1) Sector SEN

7

Interactive Sector Performance Dashboard

(1) Sector SWF

8

Interactive Sector Performance Dashboard

(1) Sector SEL

9

Drilling into details … what exactly was the issue?

10

Drilling into further details … what exactly was the issue?

11

Dashboard: Targeted browsing through ATCO comments made in Post-Exercise-

Questionnaires

12

Dashboard: LOS analysis

13

13

Dashboard: LOS analysis

14

14

15

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019

Dashboard: Inspection of a more

complicated statistical evaluation of workload

16

17

Our experience: Interactive Performance Dashboards …

Provide an actionable insight into the collected data

Provide data visualisations at the right level of detail

Help to better grasp connections between different data-sets

Help to better communicate the data

Help to better prompt expert reasoning about operational issues

Support deeper human performance analysis

Make data integrity checks more efficient

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019 18

The way forward

Data-base Management

Live-connect to the SIM platform to support online monitoring dashboards

Extending the Metric Toolbox

ATCO monitoring/information acquisition/intervention Eye-tracking Mouse-position/click tracking HMI/support tool usage

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019 19

(2) Extending the metric tool box

20

Eye/mouse - tracking HMI input

Sector load

Fixed vs. Free Route

R/T activity

ISA –WL rating

Hilburn, 2004

EUROCONTROL-FAA Technical Interchange Meeting ‘Modelling and Simulation’: Atlantic City, USA, 16/17 October 2019 21

Thank you for your attention