Detecting Fatigue:Lessons Learned

James C. Miller, Ph.D., CPEMiller Ergonomics

ICASM 2014 Ernsting Panel

In the Beginning…1939. Williams, MacMillan and Jenkins of U. Maryland, sponsored by the U.S. Civil Aeronautics Administration and the U.S. National Research Council Committee on Aviation Psychology:

Performed the first inflight physiological measurements of “tension” in student pilots.

1947. In Fatigue and Impairment in Man, Bartley and Chute analyzed the nature of fatigue, discriminating at least three distinct classes of fatigue: –Physiological,–Performance decrement, and–Subjective.

Williams, A. C., Jr., MacMillan, J. W., & Jenkins, J. G. (1946). Preliminary experimental investigation of "tension" as a determinant of performance in flight training (Report 54, Publication Bulletin L 503 25). Washington, DC: Civil Aeronautics Administration, Division of Research.

Three-Part ApproachMackie RR, Miller JC (1978). Effects of hours of service, regularity of

schedules, and cargo loading on truck and bus driver fatigue. Human Factors Research, Inc.

Seminal highway study that:(1) Demonstrated that EEG data could be acquired reliably during open-highway driving;(2) Demonstrated the sensitivity and reliability of the standard deviation of lane position (SDLP) measure; and (3) Popularized the three-part data acquisition methodology in driver fatigue research that included performance, physiological and quantified subjective data.

Three-Part Approach

3. Quantified subjective ratings:

very valuable as screening tool, but not discussed in this presentation.

Miller JC (2012). An historical view of operator fatigue. Chapter 2 in G Matthews, PA Desmond, CE Neubauer, PA Hancock (ed.), Handbook of Operator Fatigue, Ashgate Publishing Limited, Surrey, England.

1. Physiology: EEG, EOG/Eyeblink, ECG/HRV (vagal tone), sleep and activity logging.2. Task Performance: SDLP, steering and control variability. Vigilance, attention, perception.

Discussed Here

Physiology

• EEG

• EOG/Eyeblink

• ECG/HRV (vagal tone)

• Sleep and activity logging

Task Performance

• SDLP, steering and control variability

• Vigilance and attention

Test sensitivity and specificity

EEG

EEGJuly 1962. Major Milt DeLucchi of AFOSR observed a demonstration of EEG helmet on a Los Angeles freeway. Helmet created by Dr. W. Ross Adey, Ray Kado and Rod Zweizig of the Space Biology Laboratory at UCLA, and pre-tested on the freeway in Dr. Miller’s 1953 Ford.Made possible by the invention of the transistor.

Kado, R. T., Adey, W. R., & Zweizig, J. R. (1964). Electrode system for recording EEG from physically active subjects. In Annual Conference on Engineering in Medicine and Biology. Cleveland, OH.

EEG1974-1976. Dr. Robert R. Mackie and Miller demonstrated EEG acquisition on the open highway:• “Low cortical arousal on

the 1st day and high cortical activation on the 6th day … on the regular schedule.”

• “Greater cortical activation was required to maintain performance.”

Mackie & Miller (1978), ibid.

EEG1980s. Oxford Medilog, ElectroCap, etc.

The Medilog pre-amp incorporated some characteristics of the UCLA and HFR pre-amps: 100 gain, 80 dB CMMR.

EEG• In 1983, five Air Force test pilots practiced a

battery of 4 tasks for about 6 hours, until the learning curves for response error stabilized.

• Testing began at about 1:30 p.m. the following day with a 5- to 8-hour work period, a brief dinner break, then another 5- to 7-hour work period which ended when the subject was too exhausted to continue.

• During the session each subject performed several types of tracking and cognitive tasks, including about 1400 visuomotor memory task trials.

Gevins, A. S., Bressler, S. L., Cutillo, B. A., Illes, J., Miller, J. C., Stern, J. A., & Jex, H. R. (1990). Effects of prolonged mental work on functional brain topography. Electroencephalography and Clinical Neurophysiology, 76(4), 339–350.

Gevins

Stern

Jex

EEG“[S]ince dramatic changes in brain activity patterns occurred before performance deteriorated significantly, measures of brain activity may be more sensitive indicators of the deleterious effects of sustained mental work than measures of overt behavior.

“This evidence could explain, in part, the increased vulnerability to life threatening accidents that exists during the early stages of prolonged mental activity, when performance has not yet become significantly degraded.”

Gevins et al., 1990, ibid.

EEG1990s. U.S. Department of Transportation Driver Fatigue and Alertness

Study (DFAS):• 80 drivers; 5 round trips, St Louis-Kansas City or Montreal-Toronto.

Regular and irregular schedules.• EEG during driving (4,600 hours) and during major sleep periods

(3,200 hours).• No detections of sleep stages during driving.

Wylie CD, Shultz T, Miller JC, Mitler MM, Mackie RR (1996). Commercial Motor Vehicle Driver Fatigue and Alertness Study: Project Report. Federal Highway Administration, Department of Transportation, Washington DC,.

Miller JC (1996). “Descriptive quantitative analysis of 4,000 hours of day and night EEG recorded from truck drivers on the open-highway.” 36th Annual Meeting of the Society of Psychophysiological Research.

Miller JC (1997). Quantitative analysis of truck driver EEG during highway operations. Proc. 35th Annual Rocky Mountain Bioengineering Symposium, Biomed. Sci. Instrum. 34:93-98.

EEG TodayExample: Advanced Brain Monitoring, Inc. (ABM). Drs. Chris Berka and Daniel J. Levendowski.• B-Alert Wireless Sensor Headset:

easy to administer platform for acquiring high quality electroencephalographic (EEG) and electrocardiographic (ECG) signals.

• Alertness and Memory Profiler: neurocognitive battery of vigilance, attention, learning, and memory tests to simultaneously acquire and synchronize data on brain function and performance.

EEG Today• EEG may serve as a standard for the assessment of cortical

activation or non-activation during actual and simulated flight.• Detection of sleep stages* on the highway during transportation

operations is rare.• To date, EEG is not practical for use in day-to-day operations as a

fatigue monitor.

*Rechtschaffen, A., & Kales, A. (1968). A Manual of Standardized Terminology, Techniques and Scoring System for Sleep Stages of Human Subjects (No. 204). Washington DC: National Institutes of Health.

EOG/Eyeblink

EOG/Eyeblink1980s. Stern, J. A., Walrath, L. C., & Goldstein, R. (1984). The endogenous eyeblink. Psychophysiology, 21(1), 22–33.• “Allocation of attentional resources, transition points in

information processing flow, and possibly processing mode, are indexed by blink parameters.”

• “Endogenous blinks are coordinated with oculomotor activity in such a way as to minimize interference with information intake.”

EOG/Eyeblink1980s. Terry Morris:

“a number of components of the EOG signal predicted performance changes due to fatigue. … [Blink Amplitude] was the best single predictor of [flying] performance. However, no single variable accounted for more than 41% of the variability in the regression. This suggested that a combined metric based on the changes in all four EOG measures may be a good alternative for future investigations.”

Morris, T. L. (1985). Electrooculographic indices of changes in simulated flying performance. Behavior Research Methods, Instruments & Computers, 17, 176–182.

Morris, T. L., & Miller, J. C. (1996). Electrooculographic and performance indices of fatigue during simulated flight. Biological Psychology, 42(3), 343–360.

EOG/Eyeblink1990s-2000s. PMI’s FIT: “[R]esting pupil diameter was smaller, constriction amplitude was greater, and saccade velocity was slower at 03:00 than at 21:00.”

Miller, J. C., Eddy, D. R., & Fischer, J. (2004). The Sensitivity and Specificity of Oculometrics Under Fatigue Stress Compared to Performance and Subjective Measures (No. 2004-0056, ADA425455). Brooks City-Base, TX: Air Force Research Laboratory.

Eyelid Closure (PERCLOS)

PERCLOS: the proportion of total time that the eyelids are closed 80% or more.

From Akrout & Mahdi, 2013.

Dingus, T. A., Hardee, H., & Wierwille, W. W. (1987). Development of models for on-board detection of driver impairment. Accident Analysis & Prevention, 19(4), 271-283.

PERCLOS TodaySeven systems that have been tested and may be ready for commercial use soon:DD850 Driver Fatigue Monitor (DFM; Attention Technology, Inc.) uses video to track eye position and eyelid closure. Driver Monitoring System (DMS; Siemens VDO). Uses video. Calcualtes AVECLOS, the percentage of time the eyes are fully closed during a one-minute period.Drowsy Driver Detection System (DDDS; Johns Hopkins University Applied Physics Laboratory) uses Doppler radar to monitor the speed, frequency, and duration of eyelid closure, and respiration, and pulse rates.faceLAB™ system (Seeing Machines, Canberra, Australia) uses video to track the head and face, eye gaze, and eyelid, and calculates PERCLOS.

PERCLOS TodaySeven systems that have been tested and may be ready for commercial use soon (continued):InSight™ (SensoMotoric Instruments GmbH) computer vision measures head position and orientation, gaze direction, eyelid opening, and pupil position and diameter.Prototype (Bergasa & Nuevo; U. Alcala, Madrid) computer vision based upon PERCLOS, eye closure duration, blink frequency, nodding frequency, face position, and fixed gaze. Prototype (Rensselaer Polytechnic Institute) computer vision system tracks PERCLOS and average eye closure speed.

Barr, L., Howarth, H., Popkin, S., & Carroll, R. J. (2005). A review and evaluation of emerging driver fatigue detection measures and technologies. In Proceedings of the 2005 International Conference on Fatigue Management in Transportation Operations (Vol. TP 14620E). Ottawa: Transport Canada.

ECG/HRV (Vagal Tone)

ECG/HRV (Vagal Tone)1970s. Dr. Steve Porges, U. Maryland:

“(a) following the onset of the warning signal, heart rate variability increased; (b) in anticipation of the termination of the preparatory interval (PI), heart rate variability decreased; and (c) following the onset of the respond signal, both heart rate and heart rate variability increased.”

1978. Mackie & Miller (ibid.)“HRV decreased with fatigue, indicating greater driver arousal during driving to combat fatigue effects.”

Porges, S. W. (1972). Heart rate variability and deceleration as indexes of reaction time. Journal of Experimental Psychology, 92(1), 103–110.

ECG/HRV (Vagal Tone)1980s. Dr. Glenn Wilson, USAF:

“Four emergency situations occurred during inflight and simulated air-to-ground training missions. … Fifty percent increases in heart rate were found to occur only during actual flight but not during simulated flight emergencies. Heart rate variability decreased in all cases but to a greater extent during the [two] inflight emergencies.”

Wilson, G. F., Skelly, J., & Purvis, B. (1989). Reactions to emergency situations in actual and simulated flight. In Human Behaviour in High Stress Situations in Aerospace Operations (Vol. AGARD-CP-458, AD-A212884, pp. 9–1 to 9–13). The Hague, The Netherlands: AGARD Aerospace Medical Panel.

ECG/HRV Today• Studies of functional neuroanatomy support a model of pre-frontal

cortex effects on heart rate.• Julian Thayer: high vagal tone was associated with better go/no-go

task performance and better situation awareness (SA).• Vagal tone has been only marginally useful in the classification of

levels of mental workload difficulty.• The LF/RSA (< 0.15 Hz) and HF (> 0.15 Hz) components of HRV are

marginally useful in the detection of high mental demand upon the pilot.

Sleep and Activity Logging

Sleep and Activity Logging1992. The Cole-Kripke alorithm for the Actigraph brand of wrist activity monitor (WAM) agreed 80% with polysomnogrpahy (Rechtschaffen & Kales, 1968, ibid.).

Today (example). “the Readiband can collect sleep data and convert it into an effectiveness score – which is viewable by the user at any time with the push of a button” (Fatigue Science, Vancouver, BC, Canada). This device is in use by several professional sports teams that travel across time zones to compete. Uses the SAFTE/FAST fatigue simulation.

Cole, R. J., Kripke, D. F., Gruen, W., Mullaney, D. J., & Gillin, J. C. (1992). Automatic sleep/wake identification from wrist activity. Sleep, 15(5), 461–469.

SDLP, Steering and Aircraft Control

SDLP, Steering and Aircraft Control1940s. The Cambridge Cockpit Studies showed that Muscio’s idea of defining fatigue as a measurable, time-dependent change from before to after a period of performance was applicable to simulated flying performance (Bartlett, 1942; Davis, 1948; Drew, 1940).In the 1950s and 1960s, the late Duane McRuer, the late Henry Jex and others at Systems Technology Inc. (STI) helped solve the problem of pilots not responding quickly enough to prevent a swept-wing aircraft from swapping ends in flight.From this work came models of manual control and the Critical Tracking Task (CTT), a device used to measure an individual’s maximum capacity in joystick control at any given time.

SDLP, Steering and Aircraft Control1970s-1990s. I helped study the effects of fatigue on manual control through the use the Critical Tracking Task in various studies in which human fatigue was a manipulated variable: the US Navy’s Surface Effect Ship Habitability study, the Gevins et al. fatigue study, DoT studies of over-the-road trucking, etc.

1974-1976. Mackie and Miller (1978) demonstrated the usefulness of the standard deviation of lane position (SDLP) and steering measures as indicators of driver fatigue:• “Lower levels of fine steering…”• “Significantly more coarse steering…”• “Significantly increased lane tracking variability…”

Mackie & Miller (1978), ibid.

SDLP, Steering and Aircraft Control1980s. Terry Morris demonstrated the usefulness of variability measurement in the study of fatigue effects upon performance in both straight & level and turning flight in the GAT-1 simulator:• Windows: airspeed ±5 miles per hour, heading ±2.5 degrees, altitude ±40 feet, vertical velocity ±150 feet per minute.

• Error = SD + ([X - A] / W)2, where SD = standard deviation, X = measured value, A = assigned value and W = window value.• “error increased [significantly] until the next-to-the-last segment (epoch 7), then decreased slightly” • “a particularly good measure” “face validity is high” ”showed changes in performance which corresponded to predictions based on [others’] investigations”

Morris, T. L. (1985), ibid.

Morris, T. L., & Miller, J. C. (1996), ibid.

SDLP, Steering and Aircraft Control2000s. Chris McClernon demonstrated again the usefulness of variability measurement in the study of fatigue effects upon flight performance in a Cessna 172 simulator:• Calculated the variance (sd2) around assigned altitude, heading, pitch angle and roll angle.• Compared to constant (mean) error and RMS error, “in an aviation context a variable-error measure of performance is sensitive to research manipulations, precise at quantifying performance, and robust within a dynamic, three-dimensional space.“

McClernon, C. K., & Miller, J. C. (2011). Variance as a Measure of performance in an aviation context. The International Journal of Aviation Psychology, 21(4), 397–412.

Control Variability Today• Investigators may measure aircraft control variability in simulators.

It is an excellent fatigue metric.• Control variability is now monitored in some high-end automobiles

for the purposes of impairment detection and lane departures.• Theoretically, one may also acquire aircraft control variability data

from aircraft computers. However, commercial airline pilots spend little time hand-flying.

• Control variability might be a useful, real-time fatigue detection tool in cargo, business and/or general aviation operations.

Vigilance and Attention

Vigilance and Attention1930s. Arthur G. Bills, U. Chicago, demonstrated the brief mental block, or lapse (the “Bills Block”).1982. Wilkinson & Houghton developed the unprepared simple reaction time test (USRT), a simple, portable 10-min test of “arousal” and “continuous, concentrated attention,” using a cassette tape deck.1985. Dinges & Powell introduced a solid-state version of the USRT called the Psychomotor Vigilance Task (PVT), now used widely and successfully in sleep loss and aviation fatigue studies.

The USRT/PVT has a signal probability of 1.0, much unlike many classic vigilance tests that have signal probabilities of less than 0.05.

Miller JC (2012). An historical view of operator fatigue. Chapter 2 in G Matthews, PA Desmond, CE Neubauer, PA Hancock (ed.), Handbook of Operator Fatigue, Ashgate Publishing Limited, Surrey, England.

Vigilance and AttentionVigilance: Remaining alert for a rare but important signal embedded in a background of frequent but unimportant similar events. Includes a visual or auditory search component.1940s. Dr. Norman Mackworth, UK MRC, demonstrated the 20-minute vigilance decrement in signal detection accuracy with the Clock Test.For decades, it was thought that the vigilance decrement was caused in large part by low mental arousal.

Miller JC (2012). An historical view of operator fatigue. Chapter 2 in G Matthews, PA Desmond, CE Neubauer, PA Hancock (ed.), Handbook of Operator Fatigue, Ashgate Publishing Limited, Surrey, England.

Vigilance and Attention1970s. Dr. Robert R. Mackie, HFR, Inc.:• Extended vigilance research to include transportation operations.• Chaired the 1977 NATO conference on vigilance research.• Discussed a number of theories about causes of the vigilance decrement.• Questioned the parallelism of vigilance decrements for simple and complex tasks.

Mackie, R. R. (ed.) (1977). Vigilance: Theory, Operational Performance, and Physiological Correlates. Springer.

Miller, J. C., & Mackie, R. R. (1980). Vigilance Research and Nuclear Security: Critical Review and Potential Applications to Security Guard Performance. National Bureau of Standards contract NBS-GCR-80-201 for the Defense Nuclear Agency. Goleta CA: Human Factors Research Inc.

Vigilance and Attention1980s and 1990s. Drs. Roy Davies, Joel Warm and Raja Parasuraman:

1982. The vigilance decrement depends upon task demand factors such as memory load and event rate, variables that influence motivation such as performance feedback, and adverse environmental conditions. A critical factor is whether the discrimination required is successive or simultaneous.1998. Subjective reports show that the workload of vigilance is high and sensitive to factors that increase processing demands. Vigilance requires hard mental work and is stressful.

Davies, D. R., & Parasuraman, R. (1982). The Psychology of Vigilance. Academic Press.

Warm, J. S., Parasuraman, R., & Matthews, G. (2008). Vigilance Requires Hard Mental Work and Is Stressful. Human Factors, 50(3), 433–441.

Vigilance and Attention TodayTo date, there is no specific instrumentation that could be embedded into cockpit operations to detect too many lapses in attention or the possibility of a vigilance decrement.

Such a device would have to be implemented as a secondary task, which might compromise flight safety.

Parallel and Serial Measurement

Parallel and Serial MeasurementTest sensitivity (true positive rate):• The sensitivity of a screening test increases with the number of

samples, due to decreasing sampling variance of the estimate.• Test sensitivity is enhanced as the useful data set is increased and is

diminished as the useful data set is reduced when effect size is held constant.

• A desirable test provides sensitivity within a short test period, but is not so short that it is insensitive.

• Another way to increase test sensitivity is to measure several performance, physiological and/or subjective functions in parallel, and then look for any one or two of them to fall outside their respective normal limits.

Parallel and Serial MeasurementTest specificity (true negative rate):• One way to increase test specificity is to use serial, test-retest

strategies. • For example, Wade Allen and Henry Jex at Systems Technology, Inc.,

described a "1-of-n" strategy.• In the 1-of-n strategy, the test is usually passed by an un-impaired

person in their first trial. Failure of the first trial requires a rest break and then a re-test.

• Test failure requires failing every one of several sequential trials.• This strategy emphasizes the correct identification of un-impaired

workers.

Parallel and Serial Measurement

1990s. I implemented the serial-plus-parallel measurement approach with excellent success in two fitness-for-duty (FFD) detection devices:• Factor 1000, a desktop tacking task using STI’s Critical Tracking Task

technology• ReadyShift, an in-cab driving simulation based upon the standard

deviation of lane position (SDLP)• Eighteen references

Parallel and Serial MeasurementASTiD™, an excellent example of parallel measurement:The Advisory System for Tired Drivers (Fatigue Management International, UK). Designed by Professor Jim Horne and Dr. Louise Reyner of the Loughborough Sleep Research Centre.Unobtrusive, predictive system for driver sleepiness.Uses parallel approaches to predict and detect driver fatigue:

A fatigue model and prior sleep quality data provide hour by hour estimates of the likelihood of the driver falling asleep.A direct measurement of vehicle lateral acceleration allows a calculation of an analog of lateral lane position variability (weaving).

What Won’t Work

What Won’t Work

Considering several decades of lessons learned about fatigue detection in operations through:

– Physiology: EEG, EOG/Eyeblink, ECG/HRV (vagal tone), sleep and activity logging; and

– Task Performance: SDLP, steering and control variability. Vigilance, attention, perception.

In real-time, day-to-day cockpit operations, these approaches are not practical at present:• EEG measurement • ECG measurement• Aircraft control measurements• Measures of attention

RecommendationFatigue detection in aviation operations could be implemented by a system that incorporates both parallel and serial measurements with inputs that include:

Sleep and time zone logging,A two- or three-process sleep and fatigue model, andA PERCLOS-like device.

Detecting Fatigue:Lessons Learned

James C. Miller, Ph.D., CPEMiller Ergonomics

jcmiller@millergonomics.com