Real-Time Analysis of EEG Indexes of Alertness,Cognition, and Memory Acquired With a Wireless

EEG Headset

Chris BerkaDaniel J. LevendowskiMilenko M. Cvetinovic

Miroslav M. PetrovicGene Davis

Michelle N. LumicaoVladimir T. Zivkovic

Advanced Brain Monitoring, Inc., Carlsbad, CA

Miodrag V. PopovicFaculty of Electrical Engineering, University of Belgrade, Belgrade, Serbia & Montenegro

Richard OlmsteadVeterans Affairs Greater Los Angeles Healthcare System

The integration of brain monitoring into the man–machine interface holds great prom-ise for real-time assessment of operator status and intelligent allocation of tasks be-tween machines and humans. This article presents an integrated hardware and soft-ware solution for acquisition and real-time analysis of the electroencephalogram(EEG) to monitor indexes of alertness, cognition, and memory. Three experimentalparadigms were evaluated in a total of 45 participants to identify EEG indexes associ-ated with changes in cognitive workload: the Warship Commander Task (WCT), asimulated navy command and control environment that allowed workload levels tobe systematically manipulated; a cognitive task with three levels of difficulty and con-sistent sensory inputs and motor outputs; and a multisession image learning and rec-

INTERNATIONAL JOURNAL OF HUMAN–COMPUTER INTERACTION, 17(2), 151–170Copyright © 2004, Lawrence Erlbaum Associates, Inc.

We thank Tim Zavora and Roy Dalati for their assistance in data collection and analysis, and PhilipWestbrook for his editorial comments. We also thank Mark St. John, Dave Kobus, Jeff Morrison, andtheir colleagues at Pacific Science and Engineering for organizing and hosting the pre-TIE and TIE dataacquisition sessions.

This research was supported by grants and contracts from DARPAand the National Institute of Neu-rological Disease and Stroke, the National Institute of Mental Health, and the National Heart Lung andBlood Institute divisions of the National Institutes of Health.

Requests for reprints should be sent to Chris Berka, Advanced Brain Monitoring, Inc., 2850 Pio PicoDrive, Suite A, Carlsbad, CA 92008. E-mail: chris@b-alert.com

Do Not

Cop

y

ognition memory test. Across tasks and participants, specific changes in the EEG wereidentified that were reliably associated with levels of cognitive workload. The EEG in-dexes were also shown to change as a function of training on the WCT and the learn-ing and memory task. Future applications of the system to augment cognition in mili-tary and industrial environments are discussed.

1. INTRODUCTION

The technical complexity and 24-hr schedule of contemporary industrial and mili-tary operations increasingly demand rapid skill acquisition and the ability to sus-tain high levels of performance for extended periods of time (Moore–Ede, 1993). Ef-ficient and reliable interfaces between human operators and machines are requiredto facilitate the integration and adoption of sophisticated technologies. With in-creased automation, the operator’s role is often simply to monitor, and maintainingvigilance becomes more difficult, with performance decrements increasing withtime-on-task (Parasuraman, Molloy, & Singh, 1993; Singh, Molloy, & Parasuraman,1993). The integration of physiological monitoring into the man–machine interfaceholds great promise both for real-time assessment of operator status and for pro-viding a means to allocate tasks between machines and humans based on the oper-ator status. Once meaningful real-time monitoring is achieved, intelligent feedbackor “closed-loop” systems can facilitate active intervention by the operator orthrough a third party, increasing safety, efficiency, and productivity (Parasuraman,Bahri, Deaton, Morrison, & Barnes, 1992; Parasuraman et al., 1993; Parasuraman,Mouloua, & Molloy, 1996; Singh et al., 1993).

Research conducted over the past 40 years has established electroencephalogra-phy (EEG) and event-related potentials (ERPs) as the primary tools available to sci-entists investigating neural indexes of cognition (Fabiani, Gratton, & Coles, 2000).Characteristic changes in the EEG and ERPs that reflect levels of alertness anddrowsiness, selective attention, workload, memory, and executive function havebeen identified (Akerstedt & Folkard, 1997; Fabiani et al., 2000; Gevins et al., 1996;Gevins et al., 1998; Gevins, Smith, McEvoy, & Yu, 1997; Hillyard, Hink, Schwent, &Picton, 1973; Polich, 2002; Polich & Herbst, 2000). EEG measures have been appliedin areas as diverse as human factors, lie detection, pharmacological research, andassessment of cognitive dysfunction caused by neurological disorders (Fabiani etal., 2000). EEG indexes of real-time cognitive state changes offer the capability ofproviding real-time input to adaptively automated systems (Parasuraman,Mouloua, & Hilburn, 1999). This article presents an integrated hardware and soft-ware solution for acquisition and real-time analysis of EEG to monitor indexes ofalertness, cognition, and memory. The system will translate the data into interpret-able feedback for state modification by the operator, an observer, or a machine.

EEG has traditionally been confined to laboratory settings due to the technicalobstacles of recording high-quality data and the computational demands ofreal-time analysis. A reasonable expectation for success with ambulatory EEG ap-plications in operational environments is that the setup and acquisition of highquality recordings can be obtained by the user without technical assistance. A re-

152 Berka et al.

Do Not

Cop

y

cently developed wireless EEG sensor headset achieves this goal by combiningbattery-powered hardware with a sensor placement system to provide a light-weight, easy-to-apply method to acquire and analyze six channels of high-qualityEEG (Figure 1). The EEG sensor headset requires no scalp preparation and pro-vides a comfortable and secure sensor-scalp interface for 12 to 24 hr of continu-ous use. The headset was designed with fixed sensor locations for three sizes(e.g., small, medium, and large). Sensor placement was determined using a data-base of over 225 participants so that each sensor is no more than one centimeterfrom the international 10 to 20 system coordinates. The workload studies de-scribed in this article required only three EEG channels. The remaining channelsare utilized to monitor other cognitive states including attention, learning, andmemory.

Amplification, digitization, and radio frequency (RF) transmission of the signalsare accomplished with miniaturized electronics in a portable unit worn on thehead. The combination of amplification and digitization of the EEG close to the sen-sors and wireless transmission of the data facilitates the acquisition of high-qualitysignals even in high electromagnetic interference environments.

Quantification of the EEG in real time, referred to as the B-Alert® system, isachieved using signal analysis techniques to identify and decontaminate fast andslow eye blinks, and identify and reject data points contaminated withelectromyography (EMG), amplifier saturation, or excursions due to movement ar-tifacts. Each 1-sec EEG epoch is then classified into one of four states of alertness:“high vigilance,” “low vigilance,” “relaxed wakefulness,” and “sleepy.” These fourstates were empirically derived using EEG acquired from individuals participatingin sleep deprivation studies. The high and low vigilance states were modeled byvarying the level of task engagement. Relaxed wakefulness is the state inducedwhen participants are instructed to relax with eyes closed and is generally charac-terized by predominance of EEG in the alpha frequency band (8–12 Hz.). Data forthe sleepy class were obtained using EEG samples acquired just subsequent tosleep onset.

The classification model utilizes discriminant function analysis derived from alarge normative database and is fitted to each individual’s unique EEG patterns

Real-Time EEG Indexes of Cognition 153

FIGURE 1 Wireless EEG sensor headset: (a) Front view, (b) Rear view, and (c) Aug-mented-Cognition Technical Integration Experiment participant.

Do Not

Cop

y

with data acquired from three baseline conditions. The B-Alert® model was devel-oped and evaluated for its capability to provide early warnings of the onset ofdrowsiness. The B-Alert® system was validated in sleep deprivation studies withperformance in a driving simulator (Levendowski, Berka, Olmstead, & Jarvik,1999), accuracy and reaction time during a psychomotor vigilance task, behavioralevidence as measured by cessation of finger tapping, visually scored observationsof facial signs of drowsiness (eye closures, head nods), and responses to a subjec-tive sleepiness questionnaire (Levendowski et al., 2001; Levendowski, Olmstead,Konstantinovic, Berka, & Westbrook, 2000). The B-Alert® model was independ-ently validated with visual inspection of the EEG signals and observations of facialsigns of drowsiness conducted by two board-certified sleep specialists(Levendowski et al., 2000; Mitler et al., 2002). The model was demonstrated to be ef-fective in characterizing excessive daytime drowsiness in patients with sleep apnea(Westbrook et al., 2002). Analysis of the B-Alert® indexes during 44 hr of sleep de-privation revealed that changes in the indexes could predict performance deficits(Mitler et al., 2002) and confirmed the previously reported observation (Balkin,2001; Doran, Van Dongen, & Dinges, 2001) that individuals differ in their vulnera-bility to sleep deprivation. Sleep deprivation studies (Levendowski et al., 2000;Mitler et al., 2002) revealed that highly engaging or difficult tasks induce higheroverall levels of vigilance as measured by B-Alert®, suggesting that the B-Alert® in-dexes may have utility in monitoring cognitive workload.

Other investigators have reported EEG measures of workload that reflected dif-ferences in task-related cognitive resource allocation, task mastery, and task over-load (Byrne & Parasuraman, 1996; Kramer, 1991; Pope, Bogart, & Bartolome, 1995;Prinzel, Freeman, Scerbo, Mikulka, & Pope, 2000; Sterman, 1995). The EEG vari-ables employed in these models to monitor workload included alpha suppression,increased beta, increased frontal midline theta, and ratios such as beta–alpha plustheta and alpha plus theta–beta. The B-Alert® model incorporates informationfrom the alpha, beta, and theta bands in assigning a class to each 1-sec of EEG. Itwas hypothesized that increasing the workload would result in increasing levels ofhigh vigilance as measured by the percentage of B-Alert® high vigilance.

The studies presented in this article were conducted to evaluate the efficacyof the sensor headset and B-Alert® system in monitoring mental workload dur-ing complex cognitive tasks for the Defense Advance Research Projects Agency(DARPA) Augmented Cognition (Aug-Cog) program. Figure 1c presents a par-ticipant during a typical Aug-Cog data acquisition session with the EEG sensorheadset, eye tracking and pupillometry (Marshall, St. John, Knust, & Binning,2003), and functional near infrared imaging headgear (Onaral, 2003). Three cog-nitive test paradigms were employed to evaluate workload measures. The War-ship Commander Task (WCT) simulated a multitasking Navy command andcontrol environment and allowed workload levels to be systematically manipu-lated (St. John, Kobus, & Morrison, 2002). A cognitive task with three levels ofdifficulty provided consistent sensory inputs and motor outputs while varyingworkload levels (Mathiak et al., 2002). Image Learning and Recognition Mem-ory Tests (Levendowski et al., 2002) were utilized to evaluate the B-Alert® in-dexes during learning and recognition memory.

154 Berka et al.

Do Not

Cop

y

2. METHODS

2.1. Acquisition Hardware System

The sensor headset acquires six channels of EEG or electrooculography (EOG) us-ing either a unipolar or bipolar montage. Data are sampled at 256 samples per sec-ond with a bandpass from 0.5 Hz and 65 Hz (at 3 dB attenuation) obtained digitallywith Sigma-Delta A/D converters. The RF link is frequency-modulated to transmitat a rate of 57 kBaud in the 915 MHz ISM band. When utilized in the bidirectionalmode, the firmware allows the host computer to initiate impedance monitoring ofthe sensors, select the transmission channel (so two or more headsets can be used inthe same room), and monitor battery power of the headset. Data are acquiredacross the RF link on a host computer via an RS232 interface. Data acquisition soft-ware stores the EEG data and synchronizes event codes generated by the cognitivetasks in the EEG record for ERP analysis. For all of the results presented later, thestandard hardware montage includes bipolar recordings from Fz to POz and Cz toPOz (required for the B-Alert® system), unipolar recordings from Fz, Cz, and POzreferenced to linked mastoids (available for ERP analysis), and a bipolar configura-tion for horizontal and vertical EOG (to confirm the accuracy of the eye blink iden-tification and decontamination algorithm).

2.2. Artifact Identification and Decontamination, and Signal Processing

Artifacts automatically detected and decontaminated in the time-domain EEG sig-nal include 3, 5, or 7 data point spikes with amplitudes greater than 40 mV (causedby tapping or bumping of the sensors), amplifier saturation, and excursions thatoccur during the onset or recovery of saturations. For each of these artifacts, datapoints with 0 µ V are inserted, starting at the last zero crossing prior to and endingat the first zero crossing after the artifact. A 60 Hz notch filter is applied to all EEGdata. Three sets of filtered EEG data are then derived using a 0.5 Hz 256th orderhigh-pass FIR filter, a 4 Hz 640th order FIR high-pass filter, and a 7 Hz IIR low-passfilter. To obtain faster computations, both high-pass filters are realized by subtract-ing the output of the corresponding low-pass filter from the original signal.

Identification of eye blinks in the EEG without the use of a reference EOG chan-nel is achieved by filtering the fast component of the FzPOz channel with a 7 Hz IIRlow-pass filter, applying cross-correlation analysis to the filtered signal using thepositive half of a 40 µ V 1.33 Hz sine wave as the target shape, and applying thresh-olds to the outputs from the cross-correlation analysis. Minima and maxima analy-sis in each direction from the point of maximum correlation is used to identify thedata points corresponding to the range between the start and end of each eye blink.Once eye blink ranges have been determined, the 0.5 Hz high-pass filtered EEG sig-nal from each channel is decontaminated by replacing the data points in the eyeblink region with the corresponding data after application of the 4 Hz filter.

Decontaminated EEG is then segmented into overlapping 256 data-point win-dows called overlays. An epoch consists of three consecutive overlays. Fast-Fourier

Real-Time EEG Indexes of Cognition 155

Do Not

Cop

y

transform is applied to each overlay of the decontaminated EEG signal multipliedby the Kaiser window (α = 6.0) to compute the power spectral densities (PSD). ThePSD values are adjusted to take into account zero values inserted for artifact-con-taminated data points. The PSD between 70 and 128 Hz is used to detect EMG arti-fact. Overlays with excessive EMG artifact (“EMG”) or with fewer than 128 datapoints (“missing data”) are rejected. The remaining overlays are then averaged toderive PSD for each epoch with a 50% overlapping window. Epochs with two ormore overlays with EMG or missing data are classified as invalid. For each channel,PSD values are derived for each 1-Hz bin (“bin”) from 3 Hz to 40 Hz and the totalPSD from 3 Hz to 40 Hz (“band”).

2.3. Classification Model

The four-class model was developed with data from three 5-min baseline condi-tions (i.e., baseline conditions = finger-tapping with eyes open [EO] and eyesclosed [EC] and three-choice psychomotor vigilance task [PVT]) and sleepy epochsselected from sleep-deprivation data using a database of 150 healthy participants.For each epoch from these four conditions, five variables were computed for each1-Hz bin between 3 Hz and 40 Hz (5 variables × 38 bins) for each channel: thelogged PSD, the relative power compared to the total power between 3 Hz and 40Hz, and the z scores for EO, EC, and PVT. The z scores were computed using themeans and standard deviations from each of the three baseline conditions. Theidentification of fast blinks was also used as a predictive variable. A total of 381variables were available for each epoch based on a two-channel classificationmodel (i.e., 190 variables each for FzPOz and CzPOz, plus fast-blink).

The variables from each artifact-free epoch for the four conditions were submit-ted to stepwise analysis to select those variables most predictive in a four-classmodel (high vigilance, low vigilance, relaxed wakefulness, and sleepy). A total of19 variables were selected. The most predictive variables were as follows: (a) the zscore of the 10 Hz bin from CzPOz relative to the PVT task (partial r2 = 0.46), (b) thepresence of a fast blink (partial r2 = 0.09), and (c) the z score of 11 Hz from CzPOzrelative to eyes closed (partial r2 = 0.06). The r2 for each of the remaining 16 vari-ables was less than 0.03.

The approach developed for the B-Alert® system was to utilize population datato establish the underlying model and then refine the discriminant function by ad-justing for individual differences in the EEG using data from the three baseline con-ditions. Although baseline data could be readily acquired for development of theclassification models for new individuals, sleep data could not be obtained a priori.Rather, the mean values of all variables from the three baseline conditions (190 vari-ables × two channels × three conditions) for all participants in the database weresubmitted to stepwise linear regressions to derive equations to predict thediscriminant function coefficients for the “sleepy” classification for the each of the19 predictive variables. Matrices were then derived using the aforementioned anal-yses to fit the four-class model to the individual and compute the probability of cor-

156 Berka et al.

Do Not

Cop

y

rect classification into each of the four output classes on a second-by-second basisoffline or in real time.

2.4. Warship Commander Task

The WCT requires the user to monitor groups (“waves”) of incoming aircraft(“tracks”), to identify tracks as friendly or hostile, and to warn and then destroyhostile tracks. Each wave is 75 sec in duration with workload level manipulated byincreasing numbers of tracks per wave (6, 12, 18, or 24 tracks). The learning curvefor WCT is steep due to the complexity of task demand. Performance measurescomputed for each wave, including reaction times to identify, warn, and destroytracks and total game scores (as a percentage of possible points), correlate directlywith workload levels (number of tracks) and can be used as behavioral measures ofparticipant workload (St. John et al., 2002).

Of a total of 15 “novices” evaluated in an initial study, one was eliminated fromthe study for fatigue and another for failing to reach proficiency on WCT, leaving atotal of 13 participants in the analysis. Novices were given 35 min of training toachieve sufficient proficiency to complete the basic protocol that included threewaves of 6, 12, and 18 tracks. As part of the DARPA Augmented Cognition Techni-cal Integration Experiment (TIE), 10 participants were studied with additional lev-els of WCT training, ranging from 1 hr to over 300 hr. This group of 10 participantscompleted multiple sessions with 3 waves of 6, 12, 18, and 24 tracks (presented inthe order 6, 18, 12, 24).

2.5. Three-Level Cognitive Task

One of the limitations of the WCT in validating a cognitive workload measure wasthe fact that the number of stimuli and the amount of required motor activity (mouseand keypress) varied directly as a function of the “workload” manipulations. Toevaluate the B-Alert® EEG indexes without the sensory and motor confounds associ-ated with workload levels in the WCT, a three-level cognitive task, developed as partof the DARPAAugmented Cognition project (Mathiak et al., 2002), was evaluated in16 healthy participants. In this task, both the stimuli and motor demands are keptconstant during three levels of increasing task difficulty (easy, moderate, hard) byemploying the same number of stimuli and targets for each level of difficulty. Foreach levelofdifficulty,250 trialsof single integersbetween1and8arepresentedwitha 1.6-sec Interstimulus Interval (ISI). The order of the digit presentation is identicalfor each of the levels of difficulty to maintain consistent visual inputs. For level 1(easy), participants are instructed to press the space bar with the index finger of bothhands only when they see the number 5. In level 2 (moderate), participants respondonly after any three consecutive even numbers and for level 3 (hard), they respondonly to a number the same as the number 2 trials earlier (2-back task). Performance

Real-Time EEG Indexes of Cognition 157

Do Not

Cop

y

measures, including reaction times and percentage of correct responses, were com-puted to verify the levels of task difficulty.

2.6. Image Learning and Recognition Memory Task

The recognition memory task requires participants to memorize 20 images, andthen recognize those images when randomly interspersed among 80 new images ofthe same category (e.g. animals, food, travel, sports). The categories were selectedand evaluated for equivalence in test performance across categories. During thetraining period, each of the 20 images is presented sequentially twice for 1.25 secwith a 1.5-sec ISI. During the testing period, each of the 100 images is presented for100 msec. with a 2.1-sec ISI, while the participant identifies the training and newimages. Each participant completed two recognition memory sessions with twodifferent image categories.

2.7. Participants

A total of 45 fully-rested, healthy individuals (ages 18–50) participated in the WCTnovice (n = 13), three-level cognitive task (n = 16), Image Learning and RecognitionMemory Task (n = 19), and DARPA Augmented Cognition TIE (n = 10) studies.Thirteen participants took part in both the WCT novice and three-level cognitivetask All studies were conducted at the Advanced Brain Monitoring laboratory inCarlsbad, CA, between the hours of 9 a.m. and 2 p.m. with the exception of theDARPA Augmented Cognition TIE, conducted at Pacific Science and Engineeringin San Diego, CA.

3. RESULTS

3.1. WCT Novices

B-Alert® percentage high vigilance (HV) and WCT performance measures wereaveraged across the three waves for each of the workload levels (i.e., 6, 12, 18tracks) by and across participants. A repeated-measures analysis of variance(ANOVA) applied across the three WCT workload levels revealed an increasingpercentage of HV classifications as a result of increasing workload with a signifi-cant main effect for workload (F = 7.369, p < 0.005; Figure 2). Significance levelsreflect an adjustment to the degrees of freedom using the Greenhouse–Geisserprocedure to correct for violations of the sphericity assumption in repeated-mea-sures designs, when appropriate.

Post hoc comparisons between the easy, moderate, and hard levels of the WCTrevealed significant differences for easy versus moderate (F = 5.095, p < 0.05) andeasy versus hard (F = 10.423, p < 0.01), with the moderate versus hard (F = 4.074, p <0.066) approaching significance. As expected, repeated measures ANOVAs across

158 Berka et al.

Do Not

Cop

y

the three levels showed significant increases in the reaction time to identify tracks(WCT-Rtiff; F = 88.913, p < 0.001) and decreases in the game scores (WCT-%score; F= 117.78, p < 0.001) across the three levels (Figure 3). The Pearson product correla-tions between HV and at least one of the WCT performance measures were r ≥ 0.85in the majority of participants.

3.2. WCT TIE

Repeated measures ANOVAs across the four WCT workload levels revealed an in-creasing percentage of high vigilance classifications (Figure 4; F = 3.573, p < 0.001),significant increases in the reaction time to identify tracks (WCT-Rtiff; F = 27.485, p< 0.001), and decreases in the game scores (WCT-%score; F = 14.585, p < 0.001)across the four levels of increasing workload with a significant main effect for

Real-Time EEG Indexes of Cognition 159

FIGURE 2 Mean ± SE percentage of EEG epochs classified as high vigilance (% HV) dur-ing WCT in 13 novices for easy, moderate, and hard levels (6, 12, and 18 tracks, respectively).

FIGURE 3 Mean ± SE percentage game scores (%Score) and reaction time (RTiff) duringWCT in 13 novices for easy, moderate, and hard levels (6, 12, and 18 tracks, respectively).

Do Not

Cop

y

workload. Figure 5 presents the wave-by-wave relation between the mean B-Alert®

indexes and mean WCT-RTiff across participants, illustrating that the B-Alert® in-dexes maintained the temporal characteristics of the WCT protocol. The correlationbetween B-Alert® indexes and the WCT-RTiff across waves and participants wassignificant but weak (r = .432, p < 0.01) as a result of within- and between-subjectvariability.

3.3. Effects of increasing levels of training in WCT

An inspection of data from individual participants suggested that the percentageof HV classifications during WCT was influenced by the level of training. Figure 6illustrates this observation with data from four individual participants with in-

160 Berka et al.

FIGURE 4 Mean ± SE percentage of epochs classified as high vigilance (% HV) dur-ing WCT in 10 Technical Integration Experiment participants for 6, 12, 18, and 24tracks.

FIGURE 5 Mean reaction time (RTiff) and percentage of epochs classified as highvigilance (% HV) across 12 waves in 10 Technical Integration Experiment participantsduring WCT.

Do Not

Cop

y

creasing levels of WCT training: 35-min training, 5-hr training, 10-hr training, andover 40 hr of training. The workload effect is demonstrated by an increasing per-centage of high vigilance at each of the levels and the training effect is indicated byan overall decrease in the percentage of HV. These findings suggest that differencesin training among the TIE participants (ranging from 1 hr to 300 hr) might explainsome of the between-subject variability. To investigate this hypothesis, the 10 TIEparticipants were stratified into three groups based on amount of WCT training(i.e., 1–3 hr, n = 3; 6–8 hr, n = 4; and 40–300 hr, n = 3), group means were calculatedfor the percentage of HV, and an ANOVA was applied across workload levels bygroup (see Figure 7). Despite the small sample size, the results revealed a signifi-

Real-Time EEG Indexes of Cognition 161

FIGURE 6 Percentage epochs classified as high vigilance (% HV) in 6, 12, 18, and 24tracks for four participants with 0.6, 5, 10, and 200 hr of WCT training, respectively.

FIGURE 7 Mean ± SE percentage of epochs classified as high vigilance (% HV) in 6,12, 18, and 24 tracks for three groups stratified based on WCT training: group 1 with 1to 3 hr (n = 3), group 2 with 6 to 8 hr (n = 4), and group 3 with 40 to 300 hr (n = 3) of WCTtraining. Probabilities for the comparisons between workload levels are designated as*p < .05 and **p < .01.

Do Not

Cop

y

cant interaction between level of expertise and workload level (number of tracks; F= 3.118, p < 0.01).

Figure 7 illustrates that the B-Alert® percentage HV increases across WCT work-load levels for all three groups, however, the overall levels of HV across the fourworkload levels decreased dramatically across groups as a result of training. Thissuggests that as participants gain expertise, the level of vigilance is modulated tomeet task demands. For the participants with over 40 hr of training, there were nostatistically significant differences between the 12- to 18-track and 24-track condi-tions (Figure 7). Due to the small sample size (n = 3), it is premature to draw anyconclusions from these data.

3.4. Three-Level Cognitive Task

The mean percentage of B-Alert® HV classifications for each of the three difficultylevels is illustrated in Figure 8. Repeated measures ANOVAs across the three levelsof difficulty revealed a significant increase in the percentage of high vigilance clas-sifications as a result of increasing task difficulty (F = 21.962, p < 0.001). Compari-sons between the easy, moderate, and hard levels revealed significant differencesfor easy versus moderate (F = 6.419, p < 0.05), easy versus hard (F = 24.608, p <0.001), and moderate versus hard (F = 27.321, p < 0.001). Repeated measuresANOVAs across the three difficulty levels revealed significant increases in reactiontime (F = 17.267, p < 0.001) and decreases in the percentage of correct responses (F =38.217, p < 0.001), confirming the actual increase in the levels of task difficulty (Fig-ure 9). The correlations between HV and at least one of the performance measureswere r ≥ 0.85 in the majority of participants.

162 Berka et al.

FIGURE 8 Mean ± SE percentage of epochs classified as high vigilance (% HV) forthree levels of cognitive task.

Do Not

Cop

y

3.5. Image Learning and Recognition Memory Task

Mean performance (percentage of correct responses) was computed for each of thetwo sessions. Mean percentages of B-Alert® HV classifications were computed forthe image memorization period and the recognition memory period for each of thetwo sessions of the memory test. Comparison of the performance results betweenthe two sessions revealed significant improvements in performance between ses-sion 1 and session 2 (Figure 10), suggesting a practice effect (t = –2.983, p < .01). Sim-ilar practice-related results were obtained for the B-Alert® indexes with percentageHV decreasing from session 1 to session 2 (t = 4.765, p < .001). In addition, signifi-

Real-Time EEG Indexes of Cognition 163

FIGURE9 Mean± SEforpercentageofcorrect responses (%correct)andreactiontime(RT) during easy, moderate, and hard difficulty levels of a three-level cognitive task.

FIGURE 10 Mean ± SE for percentage of correct responses (% correct) during imagelearningandrecognitionmemorytest fortwosessions(categories:animalsandtravel).

Do Not

Cop

y

cant within-session differences in the B-Alert® HV indexes were obtained withhigher levels of HV during the image memorization period when compared to therecognition memory period (Figure 11). This effect was enhanced for session 2 (t =3.291, p < .005) in comparison to session 1 (t = 1.989, p = .06).

4. DISCUSSION

The wireless sensor headset provided a reliable method for EEG acquisition andanalysis even within the constraints of the challenging TIE environment (see Figure1c). Although previous investigators have reported high quality EEG acquisition inoperational settings including airplane cockpits, long haul truck cabins, and trainoperator quarters (Caldwell, 1995; Kecklund & Akerstedt, 1993; Miller, 1995;Mitler, Miller, Lipsitz, Walsh, & Wylie, 1997; Sterman & Mann, 1995), the wirelesssensor headset represents a significant advance in technology for operational mon-itoring. The limited channel montage has now been demonstrated effective for thedetection of EEG indexes of alertness–drowsiness (Levendowski et al., 2001;Levendowski et al., 2000), attention- and memory-related ERPs (Levendowski etal., 2000; Mitler et al., 2002), and the quantification of cognitive workload duringthe WCT and the three-level cognitive task. The limited channel approach has alsobeen utilized by other investigators to achieve highly sensitive and reliable correla-tions between EEG and performance, including the ability to predict performanceon a second-by-second basis (Makeig & Inlow, 1993; Makeig & Jung, 1995, 1996;Sterman & Mann, 1995; Torsvall & Akerstedt, 1987).

The B-Alert® system was designed to detect and predict vigilance decrements,and was validated in sleep deprivation studies using a variety of objective and sub-jective measures to confirm its sensitivity and reliability (Levendowski et al., 1999;Levendowski et al., 2001; Levendowski et al., 2000). In studies of more than 400healthy participants and sleep disorder patients, a number of tonic influences onthe B-Alert® indexes were observed including homeostatic and circadian effects,

164 Berka et al.

FIGURE 11 Mean ± SE percentage of epochs classified as high vigilance (% HV) for theimage memorization period and the recognition memory period for the two sessions.

Do Not

Cop

y

individual differences in vulnerability to sleep deprivation, and disease states suchas sleep and neurological disorders (Westbrook et al., 2002). Quantifiable changesin the B-Alert® indexes were predictive of fatigue-related decrements in accuracyand reaction time that occurred hours before the actual onset of sleep. B-Alert® in-dexes also identified lapses in alertness on a second-by-second basis that were pre-dictive of missed responses or errors on vigilance tests (Levendowski et al., 2001;Levendowski et al., 2000; Mitler et al., 2002).

These studies also revealed a number of phasic influences on the B-Alert® in-dexes including the type of task the in which the participant was engaged, task dif-ficulty, and novelty (Levendowski et al., 2001; Levendowski et al., 2000; Mitler etal., 2002). Caffeine and nicotine were shown to affect the B-Alert® indexes (Berka etal., 2000). Although the B-Alert® model was designed for maximum sensitivity tothe vigilance and performance decrements resulting from the transition from wak-ing to drowsiness and sleep onset, these data suggested that the B-Alert® indexesmay have utility for monitoring workload.

4.1. B-Alert® Indexes as Measures of Cognitive Workload

Cognitive workload has been conceptualized as the allocation of mental re-sources or effort required to maintain adequate performance on one or moretasks (Kahneman, 1973; Kramer, 1991; Wickens, 1984, 1992; Wickens & Holland,1984). Several investigators have reported EEG measures of workload and taskdifficulty, and in a review of studies of EEG and workload, Sterman and Mann(1995; Sterman, 1995) concluded that EEG had the potential to be a valid and ob-jective measure of mental workload. Studies of air traffic controllers (Brookings,Wilson, & Swain, 1996), airline pilots (Sterman & Mann, 1995; Sterman, Mann, &Kaiser, 1992;), drivers (Brookhuis & de Waard, 1993), and participants performingcognitive tasks (Smith, Gevins, Brown, Karnik, & Du, 2001), have related EEGshifts to changes in task complexity and task difficulty. The EEG variables com-monly employed by these investigators (e.g. changes in alpha, beta, and thetabands; Byrne & Parasuraman, 1996; Kramer, 1991; Pope et al., 1995; Prinzel et al.,2000; Sterman & Mann, 1995) are incorporated into the B-Alert® classificationmodel.

The WCT developed for the DARPA project had several advantages for validat-ing workload indexes: The task was highly engaging with a series of multitaskingsubtests, the workload levels could be systematically varied, and the task resem-bled a navy command and control scenario (St. John et al., 2002). One problem withthe WCT was that the number of stimuli and the motor demands increased as afunction of increasing workload levels, leaving open the possibility that perceptualprocesses or motor activity were being quantified and not cognitive workload perse. Although changes in sensory processing and motor output are important ele-ments of increasing engagement in a task, it has been proposed that the neural re-sources involved in cognitive processes are relatively independent of those in-volved in the motor output (Donchin, Karis, Bashore, Coles, & Gratton, 1986;Gopher, 1992; Wickens, 1991).

Real-Time EEG Indexes of Cognition 165

Do Not

Cop

y

The three-level cognitive task allowed a more precise definition of the validity ofthe B-Alert® indexes in the measurement of workload by keeping constant thenumber and type of stimuli as well as the motor demands while manipulating thecognitive requirements. The combined results from these two tasks confirm thatthe B-Alert® indexes are related to cognitive effort associated with task difficultyand not to the number of sensory inputs or the amount of motor output requiredfor the different levels.

The image memory and recognition task offered a third paradigm for evaluatingthe B-Alert® indexes. The percentage of B-Alert® HV was significantly higher dur-ing the image memorization period when compared to the recognition memory pe-riod (Figure 11), presumably as a result of increased effort during the encoding pe-riod in comparison to the less demanding recognition period.

4.2. Training Effects

Although the WCT experiments were not designed specifically to evaluate changesin EEG parameters as a function of training, the level of WCT expertise clearly hada substantial impact on the B-Alert® indexes, with overall high vigilance percent-ages decreasing dramatically as a result of training. Thus, in the context of theWCT, the B-Alert® indexes reflect changes in task difficulty as well as the total ef-fort expended by the operator. These results, in conjunction with performance dataand subjective reports, confirm that the cognitive, and by inference, neural, re-sources required for WCT decrease as a function of practice and are consistent withcurrent theories in cognitive skill acquisition (Fisk & Schneider, 1984; Schneider &Fisk, 1982; Schneider & Shiffrin, 1977). Similarly, brain imaging studies show a re-duction in the distribution and the magnitude of cortical activation as skill acquisi-tion progresses (Haier et al., 1992; Raichle et al., 1994). Further investigation is re-quired to determine whether the B-Alert® indexes can be used as predictivevariables to estimate, for example, the number of tracks that can be successfullyhandled by an operator at a specified time. Interestingly, the WCT “workload ef-fect” appeared to diminish after 40 to 300 hr of training. It would be of value forpractical applications to determine whether additional training would ultimatelyeliminate this distinction.

The image learning and recognition memory tests provide additional supportfor the B-Alert® indexes reflecting practice effects that were correlated with im-proved performance (Figures 10 and 11). It is tempting to speculate that these dataimply that the B-Alert® indexes reflect practice-related changes in cognitive re-source allocation and that, as participants gain expertise, the level of “vigilance” asmeasured by B-Alert® is modulated to meet task demands. To support this conclu-sion, additional studies are required utilizing repeated measures testing on indi-viduals at regular intervals during the skill acquisition process.

If the B-Alert® indexes related to skill acquisition can be replicated in a con-trolled experiment, they may have utility for monitoring progress during a varietyof computer-based training exercises. An intelligent brain-computer interfacecould be designed to assure that trainees move efficiently through various levels of

166 Berka et al.

Do Not

Cop

y

expertise using the B-Alert® indexes to maintain an optimal pace of informationpresentation.

5. CONCLUSION

The wireless sensor headset and the B-Alert® EEG analysis software provide a ro-bust and reliable method for monitoring alertness and cognitive workload in oper-ational environments. The B-Alert indexes are sensitive to long-term and tran-sient fluctuations in the neural activity involved in alertness and cognitiveworkload. In future applications, the global B-Alert indexes could be applied incombination with other analytic techniques such as ERPs (Fabiani et al., 2000; Kok,2001; Levendowski et al., 2000) or event-related desynchronization (Klimesch,Russegger, Doppelmayr, & Pachinger, 1998) to capture a more detailed image of in-formation processing in the brain.

Additional research is required to develop an intelligent interface that utilizesthe outputs from the B-Alert® to drive an adaptive automation system. The datasuggest that in addition to predicting performance decrements resulting from ex-cessive workload, the B-Alert® EEG indexes could have utility in optimizing thespeed and efficiency of computer-based training programs.

REFERENCES

Akerstedt, T., & Folkard, S. (1997). The three-process model of alertness and its extension toperformance, sleep latency, and sleep length. Chronobiolology International, 14(2), 115–123.

Balkin, T. J. (2001, August). Sleep deprivation research at WRAIR. Paper presented at theDARPA Workshop, Las Vegas, NV.

Berka, C., Levendowski, D. J., Konstantinovic, Z. R., Olmstead, R. E., Davis, G., & Lumicao,M. N. (2000, July). Detection of drowsiness with realtime analysis of the electroencephalogram(EEG). Paper presented at the National Institutes of Health SBIR Conference, Bethesda,MD.

Brookhuis, K. A., & de Waard, D. (1993). The use of psychophysiology to assess driver status.Ergonomics, 36, 1099–1110.

Brookings, J. B., Wilson, G. F., & Swain, C. R. (1996). Psychophysiological responses tochanges in workload during simulated air traffic control. Biological Psychology, 42,361–377.

Byrne, E. A., & Parasuraman, R. (1996). Psychophysiology and adaptive automation.Biological Psychology, 42, 249–268.

Caldwell, J. (1995). Assessing the impact of stressors on performance: Observations on levelsof analyses. Biological Psychology, 40, 197–208.

Donchin, E., Karis, D., Bashore, T. R., Coles, M. G., & Gratton, G. (1986). Cognitivepsychophysiology and human information processing. In M. G. Coles, E. Donchin, & S.Porges (Eds.), Psychophysiology: Systems, processes and applications (pp. 244–266). NewYork: Guilford.

Doran, S. M., Van Dongen, H. P., & Dinges, D. F. (2001). Sustained attention performanceduring sleep deprivation: Evidence of state instability. Archives Italiennes de Biologie,139(3), 253–267.

Real-Time EEG Indexes of Cognition 167

Do Not

Cop

y

Fabiani, M., Gratton, G., & Coles, M. G. (2000). Event-related brain potentials. In J. T.Caciooppo, L. G. Tassinary, & G. G. Berntson (Eds.), Handbook of psychophysiology (pp.53–84). Cambridge, England: Cambridge University Press.

Fisk, A. D., & Schneider, W. (1984). Memory as a function of attention, level of processing,and automatization. Journal of Experimental Psychology, Learning, Memory, and Cognition,10, 181–197.

Gevins, A., Smith, M. E., Le, J., Leong, H., Bennett, J., Martin, N., et al. (1996). High resolutionevoked potential imaging of the cortical dynamics of human working memory. Electroen-cephalography and Clinical Neurophysiology, 98, 327–348.

Gevins, A., Smith, M. E., Leong, H., McEvoy, L., Whitfield, S., Du, R., et al. (1998). Monitoringworking memory load during computer-based tasks with EEG pattern recognition meth-ods. Human Factors, 40, 79–91.

Gevins, A., Smith, M. E., McEvoy, L., & Yu, D. (1997). High-resolution EEG mapping of corti-cal activation related to working memory: Effects of task difficulty, type of processing,and practice. Cerebral Cortex, 7, 374–385.

Gopher, D. (1992). The skill of attention control: Acquisition and execution of attention strat-egies. In D. E. Meyer & S. Kornblum (Eds.), Attention and performance XIV (pp. 299–322).Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Haier, R. J., Siegel, B. V., Jr., MacLachlan, A., Soderling, E., Lottenberg, S., & Buchsbaum, M.S. (1992). Regional glucose metabolic changes after learning a complex visuospatial/mo-tor task: A positron emission tomographic study. Brain Research, 570, 134–143.

Hillyard, S. A., Hink, R. F., Schwent, V. L., & Picton, T. W. (1973). Electrical signs of selectiveattention in the human brain. Science, 182, 177–180.

Kahneman, D. (1973). Attention and effort. Englewood Cliffs, NJ: Prentice Hall.Kecklund, G., & Akerstedt, T. (1993). Sleepiness in long distance truck driving: An ambula-

tory EEG study of night driving. Ergonomics, 36, 1007–1017.Klimesch, W., Russegger, H., Doppelmayr, M., & Pachinger, T. (1998). A method for the cal-

culation of induced band power: Implications for the significance of brain oscillations.Electroencephalography and Clinical Neurophysiology, 108, 123–130.

Kok, A. (2001). On the utility of P3 amplitude as a measure of processing capacity.Psychophysiology, 38, 557–577.

Kramer, A. F. (1991). Physiological metrics of mental workload: A review of recent progress.In D. L. Damos (Ed.), Multiple task performance (pp. 279–328). Washington, DC: Taylor &Francis.

Levendowski, D. J., Berka, C., Olmstead, R. E., & Jarvik, M. (1999, October). Correlations be-tween EEG indices of alertness measures of performance and self-reported states while operating adriving simulator. Paper presented at the 29th annual meeting of the Society for Neurosci-ence, Miami, FL.

Levendowski, D. J., Berka, C., Olmstead, R. E., Konstantinovic, Z. R., Davis, G., Lumicao, M.N., et al. (2001). Electroencephalographic indices predict future vulnerability to fatigueinduced by sleep deprivation. Sleep, 24(Abstract Suppl.), A243–A244.

Levendowski, D. J., Olmstead, R. E., Konstantinovic, Z. R., Berka, C., & Westbrook, P. R.(2000). Detection of electroencephalographic indices of drowsiness in realtime usinga multi-level discriminant function analysis. Sleep, 23(Abstract Suppl. 2),A243–A244.

Levendowski, D. J., Westbrook, P., Berka, C., Popovic, M. V., Pineda, J. A., Zavora, T. M., et al.(2002). Event-related potentials during a test of working memory differentiate sleepapnea patients from healthy subjects. Sleep, 25(Abstract Suppl.), A460–A461.

Makeig, S. (1993). Lapses in alertness: Coherence of fluctuations in performance and EEGspectrum. Electroencephalography and Clinical Neurophysiology, 86, 23–35.

168 Berka et al.

Do Not

Cop

y

Makeig, S., & Jung, T. P. (1995). Changes in alertness are a principal component of variance inthe EEG spectrum. Neuroreport, 7, 213–216.

Makeig, S., & Jung, T. P. (1996). Tonic, phasic, and transient EEG correlates of auditoryawareness in drowsiness. Brain Research, Cognitive Brain Research, 4, 15–25.

Mathiak, K., Hertrich, I., Rothe, S., Kincses, W. E., Lutzenberger, W., & Ackerman, H. (2002).Involuntary crossmodal enhancement of preattentive auditory processing is selective in space.Manuscript submitted for publication.

Miller, J. C. (1995). Batch processing of 10,000 h of truck driver EEG data. Biological Psy-chology, 40, 209–222.

Mitler, M. M., Miller, J. C., Lipsitz, J. J., Walsh, J. K., & Wylie, C. D. (1997). The sleep oflong-haul truck drivers. New England Journal of Medicine, 337, 755–761.

Mitler, M. M., Westbrook, P., Levendowski, D. J., Ensign, W. Y., Olmstead, R. E., Berka, C., et al.(2002). Validation of automated EEG quantification of alertness: Methods for early identificationof individuals most susceptible to sleep deprivation. Sleep, 25(Abstract Suppl.), A147–A148.

Moore–Ede, M. C. (1993). The twenty-four-hour society: Understanding human limits in a worldthat never stops. Reading, MA: Addison–Wesley.

Parasuraman, R., Bahri, T., Deaton, J. E., Morrison, J. G., & Barnes, M. (1992). Theory and designof adaptive automation in adaptive systems (Progress Rep. No. NAWCADIWAR–92033–60).Warminster, PA: Naval Air Warfare Center, Aircraft Division.

Parasuraman, R., Molloy, R., & Singh, I. L. (1993). Performance consequences of automationinduced “complacency”. International Journal of Aviation Psychology, 3, 1–23.

Parasuraman, R., Mouloua, M., & Hilburn, B. (1999). Adaptive aiding and adaptive task allo-cation enhance human–machine interaction. In M. W. Scerbo & M. Mouloua (Eds.), Auto-mation technology and human performance: Current research and trends (pp. 119–123).Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Parasuraman, R., Mouloua, M., & Molloy, R. (1996). Effects of adaptive task allocation onmonitoring of automated systems. Human Factors, 38, 665–679.

Polich, J. (2002). Neuropsychology of P3a and P3b: A theoretical overview. Advanced inElectrophysiology in Clinical Practice and Research.

Polich, J., & Herbst, K. L. (2000). P300 as a clinical assay: Rationale, evaluation, and findings.International Journal of Psychophysiology, 38, 3–19.

Pope, A. T., Bogart, E. H., & Bartolome, D. S. (1995). Biocybernetic system evaluates indicesof operator engagement in automated task. Biological Psychology, 40, 187–195.

Prinzel, L. J., Freeman, F. G., Scerbo, M. W., Mikulka, P. J., & Pope, A. T. (2000). A closed-loopsystem for examining psychophysiological measures for adaptive task allocation.International Journal of Aviation Psychology, 10, 393–410.

Raichle, M. E., Fiez, J. A., Videen, T. O., MacLeod, A. M., Pardo, J. V., Fox, P. T., et al. (1994).Practice-related changes in human brain functional anatomy during nonmotor learning.Cerebral Cortex, 4, 8–26.

Schneider, W., & Fisk, A. D. (1982). Concurrent automatic and controlled visual search: Canprocessing occur without cost? Journal of Experimental Psychology: Learning, Memory, andCognition, 8, 261–278.

Schneider, W., & Shiffrin, R. M. (1977). Controlled and automatic human information pro-cessing I: Detection, search, and attention. Psychological Reviews, 84, 1–66.

Singh, I. L., Molloy, R., & Parasuraman, R. (1993). Automation-induced “complacency”: De-velopment of the complacency-potential rating scale. International Journal of Aviation Psy-chology, 3, 111–121.

Smith, M. E., Gevins, A., Brown, H., Karnik, A., & Du, R. (2001). Monitoring task loadingwith multivariate EEG measures during complex forms of human–computer interaction.Human Factors, 43, 366–380.

Real-Time EEG Indexes of Cognition 169

Do Not

Cop

y

St. John, M., Kobus, D. A., & Morrison, J. G. (2002). Amulti-tasking environment for manipu-lating and measuring neural correlates of cognitive workload. In Proceedings of the 2002IEEE 7th Conference on Human Factors and Power Plants (pp. 7.14–7.14). New York: IEEE.

Sterman, M. B., & Mann, C. A. (1995). Concepts and applications of EEG analysis in aviationperformance evaluation. Biological Psychology, 40, 115–130.

Sterman, M. B., Mann, C. A., & Kaiser, D. A. (1992). Quantitative EEG patterns of differentialin-flight workload (Abstract from Soar ’92). Sepulveda, VA: Sepulveda VAMedical Center.

Torsvall, L., & Akerstedt, T. (1987). Sleepiness on the job: Continuously measured EEGchanges in train drivers. Electroencephalography and Clinical Neurophysiology, 66, 502–511.

Westbrook, P., Berka, C., Levendowski, D., Lumicao, M. N., Davis, G., Olmstead, R. E., et al.(2002). Biobehavioral quantification of alertness and memory in patients with sleepapnea. Sleep, 25(Abstract Suppl.), A49–A50.

Wickens, C. D. (1984). Engineering psychology and human performance. Columbus, OH: Merrill.Wickens, C. D. (1991). Processing resources and attention. In D. L. Damos (Ed.), Multiple-task

performance (pp. 1–34). London: Taylor & Francis.Wickens, C. D. (1992). Engineering psychology and human performance (2nd ed.). New York:

HarperCollins.Wickens, C. D., & Holland, J. G. (1984). Engineering psychology and human performance (3rd

ed.). Englewood Cliffs, NJ: Prentice Hall.

170 Berka et al.

Do Not

Cop

y