@) mTe GROUP FOR AERONAUTICAL RESEARCH AND TECHNOLOGY IN EUROPE

FRANCE. GERMANY. ITALY. M E NETHERLANDS . SPAIN. SWEDEN . UNITED KINGDOM

ORIGINAL: ENGLISH July 4 ~ , 2003

GARTEUR Open

Final Report for GARTEUR Flight Mechanics Action Group FM AG13

GARTEUR Handbook of Mental Workload Measurement

by

GARTEUR Action Group FM AG 13

GARTEUR aims at stimulating and co-ordinating co-operation between Research Establishments. Industry and Academia in the areas of Aerodynamics. Flight Mechanics. Helicopters. Structure & Material and Propulsion Technology.

mTg GROUP FOR AERONAUTICAL RESEARCH AND TECHNOLOGY IN EUROPE

FRANCE. GERMANY. ITALY. THE NETHERLANDS . SPAIN. SWEDEN . UNITED KINGDOM

ORIGINAL: ENGLISH July 4th, 2003

GARTEUR Open

GARTEUR TP 145

Final Report for GARTEUR Flight Mechanics Action Group FM AG13

GARTEUR Handbook of Mental Workload Measurement

GARTEUR Action Group FM AG 13

This report has been published under auspices of the Flight Mechanics Group o f Responsables of the Group for Aeronautical Research and

Technology in ELRope (GARTEUR)

Group of Resp. : FM GoR Report Resp. : Martin Castor, FOI Project Manager : Martin Castor, FOI Monitoring Resp. : Bertil Brannstriim, FMV

Action Group : FMAG13 Version : 2.0 Completed : [030704] 0 GARTEUR [2003]

GlIRTEUR GARTEUR FM AG 13 FINAL REPORT - GARTEUR TP 145

List of Authors

Martin Castor Eamonn Hanson Erland Svensson Staffan Niihlinder Patrick Le Blaye Iain MacLeod

Nicky Wright Jens Alfredson Lotta Agren Peter Berggren Val&e Juppet Brian Hilburn Kjell Ohlsson

FOI NLR FOI FOT ONERA AeII Atkins Aviation and Defence Systems QINETIQ F01 SAAB FOI Dassault Aviation NLR LiU

ClrnTEWR GARTEUR FM AG13 FINAL REPORT - GARTEUR TP 145

Summary

The mental workload problems experienced by current and future pilots of fixed and rotary wing aircraft constitute a major limiting factor on information processing capabilities and mission performance. Studies of mental workload issues are therefore highly important. However, current workload measurement techniques still need refinement, and as the methods used are not standardised it is problematic to compare results from different studies.

The present report summarises the work executed in the GARTELJR Action Group on Mental Workload Measurement (FM AG13). The objectives of the present Action Group were to make an inventory of mental workload measurement methods and techniques, and present a method to choose between different methods and advise on their use in various operational settings. In order to be able to recommend suitable measures for different studies, the Action Group has developed Measures Assessment Matrices (MAMs) that assist in the selection of appropriate measures fiom the workload "toolbox". The report also describes a number of example studies performed by the Action Group members where issues such as experimental protocol and how results are put in context are discussed. Finally, the report describes a number of available approaches to summarise data when several different types of measures have been used in an experiment.

C ~ I R ~ ~ U R GARTEUR FM AG 13 FINAL REPORT - GARTEUR TP 145

Distribution List

GARTEUR Executive Committee (XC) B. Oskam (NL) P. Garcia Samitier (SP) - XC Chairman T. J. Birch (UK) D. Nouailhas (FR) W. Riha (DE) L. Falk (SE) A. Amendola (IT)

GARTELTR XC Secretary F. Merida Martin (SP)

GARTEUR Flight Mechanics Group of Responsables (FM GoR) C. Banouil (FR) P. Hecker (DE) W.P. de Boer (NL) B. Brknstrom (SE) - Chairman FM GoR M. Hagstrom (SE) J. Keirl (UK) F. Muiioz Sanz (SP) L. Verde (IT) A. Kriiger (DE)

NLR INT A DSTL ONERA DLR FMV CIRA

INTA

ONERA DLR NLR FMV FOI DSTL rNTA CIRA Airbus

GARTEUR Flight Mechanics Industry Points of Contact (FM IPoC) R. Carabelli (IT) Alenia J. Enhagen (SE) SAAB L. Goerig (FR) Dassault Aviation

GARTEUR FM AG13 Members Lotta Agren (SE) Jens Alfiedson (SE) Martin Castor (SE) - Chairman FM AG13 Xavier Chalandon (FR) Carole Deighton (UK) Eamonn Hanson (NL,) Brian Hilburn (NL) Valkrie Juppet (FR) Patrick Le Blaye (FR) Iain MacLeod (UK)

Staffan Nalinder (SE) Nicolas Maille (FR) Kjell Ohlsson (SE) Fredrik Romare (SE) Erland Svensson (SE) Nicky Wright (UK)

SAAB FOI FOI Dassault Aviation AeI NLR NLR Dassault Aviation ONERA AeI/ Atkins Aviation and Defence Systems FOI ONERA LiU SAAB FOI Q W T I Q

ClRTEUrn GARTEUR FM AG13 FINAL REPORT - GARTEUR TP 145

Contents List of Authors

Summary v

Distribution List vi

List of Abbreviations xii

1 Introduction 14

1.1 Background 14

1.2 The workload concept 14

1.3 Approaches to mental workload 15

1.4 A comprehensive approach to workload 16

1.5 Creating the Methods Assessment Matrices 18

1.6 Choosing the best instrument for the job 21

2 Description of workload assessment tools 24

2.1 Criteria 24

2.2 Summary table of additional requirements 28

2.3 Descriptions of the measures 29

2.3.1 Bedford Scale 3 0

2.3.2 Modified Cooper-Harper Scale 3 2

2.3.3 NASA TLX 3 5

2.3.4 FOI Pilot Performance Scale 40

2.3.5 Rating Scale Mental Effort 42

2.3.6 DRAWS - DRA (Defence Research Agency) Workload Scales 44

2.3.7 Instantaneous Self-Assessment (ISA) 46

2.3.8 SWAT 48

2.3.9 Task Analysis 52

2.3.1 0 Usability methods 56

2.3.1 1 Heart rate (HR) and Heart Rate Variability (HRV) 5 8

2.3.12 EPOG and scan patterns 62

2.3.13 Blink rate 65

2.3.14 Eye movements via electro-oculogram (EOG) 67

2.3.15 Blood pressure and ear pulse 70

2.3.16 Brain activity: electroencephalogram (EEG) & event-related potentials (ERP)72

2.3.1 7 Respiration 76

2.3.1 8 Secondary Embedded Task 78

2.3.19 MOEsIMOPs 8 1

2.3.20 "Second pilot" or instructor assessment of performance 83

2.3.2 1 Subjective assessment of performance 85

G@lmv@am GARTEUR FM AG13 FINAL REPORT - GARTEUR TP 145

3 Example studies

3.1 NLR contribution

3.1.1 Mission

3.1.2 Experimental protocol

3.1.3 Methods and analysis

3.1.4 Results in context

3.2 SAAB contribution

3.2.1 Mission

3.2.2 Experimental protocol

3.2.3 Methods and analysis

3.2.4 Results in context

3.3 AeI contribution

3.3.1 Mission

3.3.2 Experimental protocol

3.3.3 Methods and analysis

3.3.4 Results in context

3.4 FOI contribution

3.4.1 Mission

3.4.2 Experimental protocol

3.4.3 Methods and analysis

3.4.4 Results in context

3.5 ONERA contribution

3.5.1 Mission

3.5.2 Experimental protocol

3.5.3 Methods and analysis

3.5.4 Results in context

4 Mission examples

4.1 Introduction

4.2 Military fixed wing mission

4.3 Civil fixed wing mission

4.4 Military rotor wing mission

4.5 Differences between military and civilian missions

5 Aggregation of results

5.1 Standardization of psychophysiological data

5.1.1 Other uses

5.1.2 Theoretical illustration

5.1.3 Conclusion

5.2 Generalised Formal Concept Analysis

$@RT@UR GARTEUR FM AG13 FINAL REPORT - GARTEUR TP 145

5.3 Triangulation

5.4 Statistical techniques for data reduction and modelling

5.4.1 Factor analysis (FA)

5.4.2 Multidimensional scaling (MDS)

5.4.3 Illustrations of the techniques

6 Modelling of operator performance

6.1 Conceptual modelling

6.2 Computer based modelling

6.3 Data-based modelling

6.4 Applied examples of data-based modelling

6.5 Conclusions on modelling

7 Concluding remarks

8 References

APPENDIX 1: The modified Cooper-Harper scale

APPENDIX 2: The Bedford scale

APPENDIX 3: Rating Scale Mental Effort

APPENDIX 4: Online Use of ISA Ratings

APPENDIX 5: NASA TLX

APPENDIX 6: DRA Workload Scales (DRAWS)

Q@lmvgUm GARTEUR FM AG13 FINAL REPORT . GARTEUR TP 145

List of Figures

Figure 1 . Four main approaches to workload ............................................................................. 15

Figure 2 . A simplified model of the human operator environment illustrating the importance of

................................................. a multi-dimensional approach to workload measurements 16

Figure 3 . Examples of Methods Assessment Matrices (MAM) .................................................. 18

Figure 4 . Four-leafed clover representing the important sources of Human Factors expertise .. 88

....................................................................................................... Figure 5 . Gaze scan patterns 91

Figure 6 . Spheres of activity at NLR .......................................................................................... 93

Figure 7 . Galois Lattice of the Bourget'2001 experiment ........................................................ 106

......................................................................... Figure 8 . A suggested standardization method 116

............................................................................ Figure 9 . Heart Rate responses of two pilots 117

Figure 10 . Standarized Heart Rate of the same two pilots ....................................................... 117

Figure 1 1 . The latent construct temperature and (some of) its manifest measures ................... 123

Figure 12 . Plot of eigenvalues extracted from successive residual correlation matrices .......... 127

........................................................................................................ Figure 13 . Plot of loadings 128

Figure 14 . A two-dimensional MDS ........................................................................................ 128

Figure 15 . A MDS solution separating markers for cognitive and perceptual processes ......... 129

Figure 16 . The structural model of example 4 .......................................................................... 130

. .......................................................................... Figure 1 7 The structural model of example 5 131

Figure 18 . A structural LISREL model from Svensson et al, 1993 ......................................... 137

Figure 19 . A structural LISREL model from Angelborg-Thanderz. 1997 ............................... 138

Figure 20 . A causal model from Svensson et al, 1997 .............................................................. 140

Figure 21 . The final structural model presented in Svensson et al, 1999 ................................. 141

Figure 22 . A submodel from from Svensson et al, 1997 .......................................................... 142

Figure 23 . A model based on the relationships between mental workload, eye fixation rate, heart

rate. situational awareness, and pilot performance ........................................................... 143

Figure 24 . A model based on the relationships between mental workload, heart rate, situational

..................................................................................... awareness, and pilot performance 144

QlRTGUrn GARTEUR FM AG13 FINAL REPORT . GARTEUR TP 145

List of Tables

Table 1 . The Workload Decomposition.matrix .......................................................................... 19

Table 2 . The Measures, Methods and Instruments-matrix .......................................................... 22

Table 3 . The Additional Requirements.matrix ........................................................................... 28

Table 4 . Experiment environments at NLR ................................................................................ 89

Table 5 . Examples of parameters that can be measured, necessary equipment and the underlying

constructs or topics of interest for the three ........................................................................ 90

Table 6 . Differences between military and civilian missions ................................................... 112