EFFECTS OF LATENCY ON FLIGHT INFORMATION DISPLAYS Nicholas M. Lorch, and Thomas Schnell, Operator Performance Lab, Iowa City, Iowa

Marty Steffensmeier, Rockwell Collins, Cedar Rapids, Iowa

Abstract New display architectures will be required to

efficiently handle video and graphics from a wide variety of sources with minimal latency in order to minimize operator error and reduce the likelihood that pilot-induced oscillations (PIO) will occur. These new display systems must be able to handle up to sixteen video and graphics sources, including forward-looking infrared (FLIR) and weather information.

We designed a pilot-in-the-loop flight simulator study to determine the effects of latency for a head-down Primary Flight Display (PFD) with a large field of view Synthetic Vision surround, a central inset containing sensor imagery (Forward Looking Infrared), and a non latent 2D guidance symbology layer. The investigation was designed in two parts, a breadth study that served as an overall screening for the effectiveness of a relatively large number of variables, and a depth study that explored deeper into effects caused by variables that were deemed significant and important from the breadth study.

The results of our study indicate the need to synchronize the latencies of a large field of view attitude indicator and a central inset FLIR overlay. Unsynchronized overlays cause an effect that we refer to as “swimming”, which is annoying to the pilot, leads to higher workload, and causes distractions that may lead to some amount of flight technical deterioration as demonstrated by our results. Within the limits of the latencies that were administered in this study, we found only a very small effect on actual flight technical performance. Positional accuracy and heading alignment was not strongly affected and generally remained well within limits specified by most practical test standards (PTS). Pilots tended to compensate by operating the flight controls with more gain and with larger amplitudes. This change of the controlling strategy indicates a reduction in aircraft controllability and may not be desirable for prolonged operations.

Introduction System latencies usually have a negative effect

on the controllability of a system. Aircraft control is certainly no exception, and the consequences of poor controllability are especially high in airborne control applications. A detailed review of the literature that was used to design the study shown herein is provided by Schnell (2004). System latencies in aircraft can result from a number of factors, including:

• Aircraft Dynamics: Large and heavy aircraft have an inherent latency in reacting to control inputs. Pilots of heavy aircraft adapt by anticipating that a given input will, in time, have the desired effect. Pilots of large and sluggish aircraft will instinctively avoid any form of unusual aircraft attitudes or maneuvers near the edge of the operational envelope. Pilots of such aircraft are better able to operate in an open-loop feed-forward mode than pilots who are used to flying small and highly maneuverable aircraft. Pilots of such small aircraft rely on timely feedback information in the control of the inputs. We hypothesize that display latencies have a bigger effect on small and nimble aircraft control than on large and sluggish aircraft control.

• Display Latencies: Flight by reference to primary flight instrumentation depends, among many factors, on the latency of the display system. Any form of latency in the chain from the sensor system through the air data computer (ADC) or the attitude heading reference system (AHRS) to the display system are detrimental to aircraft control. Furthermore, these latencies act cumulatively with the latencies resulting from the aircraft dynamics.

1-4244-1108-4/07/$25.00 ©2007 IEEE. 6.B.4-1

Designers of modern electronic flight information systems (EFIS) that use various sources of video raster imagery, rendered graphics, or 2D symbologies need to know what levels of latencies are permissible for specified aircraft types and regimes of flight. This study provides avionics designers with additional insight into the effects of display latency on aircraft controllability, pilot workload and performance. The human performance data resulting from this study may also serve a role in the synthetic/enhanced vision system (S/EVS) certification process. Throughout this work, we assumed that there are no significant latencies in the 2D and guidance symbology, as those components will always be generated by level-A certified hardware that is capable of minimal throughput. Latencies in this experiment were introduced into the synthetic terrain layer and the FLIR layer. Plausible reasons for latencies in operational systems may include bottlenecks in graphics rendering, database access, and image processing on raw video images.

Because there could be a prohibitively large number of combinations that would prevent systematic study with pilots in the loop, we elected to perform a two-step study methodology. The first step was a screening study to determine the major process levers that affect pilot performance the most. We refer to the first step as the breadth study because it investigated a large search space (large number of variables) but only at a small number of levels. The second step built on the results of the first step. We refer to the second step as the depth study, because it investigated a small number of highly influential variables at a larger number of levels. In the breadth study, we investigated the following levels:

1. Flight Maneuvers

A. Pathway guidance acquisition task to test the effects of SVS and FLIR latency in the presence of non-latent inner loop 2D guidance elements.

B. Circle to land task to test the effect when flying without inner loop guidance but primarily by reference to SVS terrain imagery.

C. FLIR following task to test the effects of latency in SVS and FLIR when tracking a target on the FLIR channel.

D. Low level terrain following task to test the effects of SVS and FLIR latency when flying by primary reference to SVS terrain imagery.

2. Aircraft models

A. Beechcraft King Air (BE-350) representative of a relatively small and nimble transport aircraft.

B. Boeing 737-400 (B-737) representative of a relatively large transport aircraft.

3. Latency parameters tested

A. Latent synthetic terrain (0ms, 1242ms)

B. Latent FLIR inset (0ms, 1242ms)

We selected the levels of the latency based on a small scale pilot experiment that indicated that SVS and FLIR latencies could be rather large, before they had a noticeable effect on flight technical performance. This effect is primarily thanks to the non latent 2D guidance symbology layer. We realize that 1242ms is a very large latency but in the spirit of a screening study, we wanted to have a latency that was guaranteed to generate deterioration in flight technical performance, thus allowing relative investigation of the effect of the remaining variables. The effect of latency in terrain and FLIR imagery depends, among other things, on the flight maneuvers, mission goals, and the aircraft model. During on-path operations on an instrument approach using non-latent 2D guidance cues, this study showed that synthetic or FLIR terrain depictions may be allowed to have a larger latency than when navigating solely by reference to those terrain depictions without 2D guidance cues. The flight maneuvers capture a wide range of controlling requirements. The instrument approach guidance task was primarily dependent on 2D symbology with the underlying terrain layer primarily providing situation awareness. The circle to land task involved navigation using synthetic terrain. In that task, the pilot flew a pattern around

6.B.4-2

the airport while avoiding terrain depicted on the synthetic vision system (SVS) display.

The breadth study indicated that the aircraft model had a strong effect with the more nimble model being the critical case. We also found that the circle to land task was the most demanding task as it involved all three layers of guidance including the 2D layer, SVS, and FLIR. The breadth study thus focused on the circle to land task in the BE-350 with a larger number of levels of SVS and FLIR latencies.

Measures of Effectiveness

Objective Measures of Effectiveness We devised a fairly large array of measures of

effectiveness (MOEs) to determine pilot performance as a function of display latency. Table 1 lists these objective MOEs. The first six MOEs are often referred to as flight technical errors (FTEs) because measurements of the actual aircraft performance are compared tocommanded performance, as shown in

the

ter the

ormance.

Figure 1. Deviations between actual and commanded performance indicate a continuum of loss of performance from nominal. Smaller is always betin all forms of FTEs, and we will generally use root mean squared (RMS) values of the FTEs to determine pilot perf

Table 1. Measures of Effectiveness (MOEs)

MOE Number

MOE Name

1 Cross Track Error (XTE) 2 Vertical Track Error (VTE) 3 Speed Error (SE) 4 Altitude Error (AE) 5 Heading Error 6 Heading Capture Error 7 Aileron Input Amplitude (C

Aircraft

Pilot

xsystem output

actual performance

wcommandedperformance

x - wFlight Technical Error

FTE

ZiNoise on thesystem input

ysystem input

wheel, column,pedals, throttle,

etc.

ZsNoise acting on

the system

Figure 1. Simplified Human-Aircraft Control System

Subjective Measures of Effectiveness Aircraft controllability was assessed with a

battery of subjective rating scales. The following tools were employed to collect the subjective data:

• In-flight principal investigator observation form.

• In-flight debriefing card: Used to assess Cooper-Harper, Bedford workload and Navy Pilot Induced Oscillation (PIO) scores.

• Notes taken during pilot debriefing after the study.

• Post flight (take-home) questionnaire.

Design of Experiment

Apparatus The OPL Boeing 737-800 simulator was

configured to allow independent selection of latency for the 2D symbology set, the synthetic terrain, the FLIR, and the pathway guidance. This simulator is modeled after a Boeing 737-800 (NG) flight deck. The simulator featured a 180 degrees of forward-projected outside visuals (12-ft spherical dome). The simulator used Arinc Size D EFIS displays and is also fitted with an HGS 4000 style head-up display (HUD) combiner that reflected an image generated by a CRT-based overhead unit. The Rockwell Collins Advanced Technology Center (ATC) developed a VME-based system that was able to acquire three channels of DVI input,

IA) 8 Aileron Input Frequency (CIF) 9 Roll Reversals

6.B.4-3

6.B.4-4

merge the content of the inputs using alpha masks, and apply exact latencies to each of the channels.

A total of three computers with DVI graphics cards in conjunction with a Rockwell Collins graphics engine were used to generate the PFD stimulus used in this study as follows:

1. Two-dimensional symbology (Rockwell Collins graphics engine)

A. Airspeed tape, speed bug, desired speed, ground speed.

B. Altitude tape, commanded altitude, altitude bug and MDA/DH.

C. Roll scale and roll pointer.

D. Horizon line and pitch ladder.

2. Path guidance

A. Highway in the sky (HITS).

B. Flight path marker (FPM).

C. Waterline symbol (aircraft nose).

3. Synthetic terrain depiction.

4. Forward looking infrared (FLIR).

The three channels were entered into the merge plane (MP) via DVI input boards in the VME rack. Because inputs to the VME board are limited to three, the pathway channel was blended into the 2D symbology channel, and the combined input was sent to the MP. The MP assembled the three channels into one image using an alpha layer blending methodology. Latencies could be applied to each layer separately. The resultant image was sent via DVI to the PFD in the simulator. Due to the significantly lower rendering demand when compared to the PFD, the ND was considered to be a non-latent display.

Latency Verification Latency of each channel was tested to ensure

consistency with design of experiment specifications. While the MP was capable of delay

times as minuscule as five milliseconds, our PFD only supported a refresh rate up to 60 Hz. Latency times were checked for accuracy within 60 Hz or 16.7 milliseconds. Each input channel was independently tested for accuracy. To test for latency, a stopwatch computer application was running in conjunction with MP programs necessary to run the study. Video images of the latent PFD and its non-latent counterpart were taken at 60 frames per second (fps) for the six different latency delay settings: 0 ms, 250 ms, 500 ms, 750 ms, 1000 ms, and 1250 ms. The recorded video was then analyzed by comparing the stopwatch time on the non-latent display against the stopwatch time on the latent display. Results of each channel can be seen in Table 2. As the results show, each channel of video latency was accurate within the refresh rate of the PFD.

Table 2. Latency Verification Results

Latency Setting (ms)

0 250 500 750 1000 1250

FLIR Latency 0 250 510 760 1000 1250Terrain Latency

5 260 520 760 1030 1250

2D Latency 0 250 510 760 1010 1250

Experimental Scenarios The following section details the flight

scenarios that were used in the study. It should be noted that the circle to land task was also used in the depth study as the sole flying task.

Pathway Guidance Task In the pathway guidance task, pilots were

instructed to maintain an altitude, heading, and airspeed until pathway guidance was reached (Figure 2, Segment 1). Pilots were then instructed to intercept the path (Figure 2, Segment 2) and follow the path until the airport was reached (Figure 2, Segment 3).

6.B.4-5

KMHV

End

30o

Start

12

3

Figure 2. Overhead View of Intercept Task

Circle to Land Task In the circle to land task, pilots were instructed

to maintain altitude, heading, and airspeed (Figure 3, Segment 1). While maintaining the aforementioned metrics, pilots had to scan the synthetic terrain for a specified visible landmark as their turning queue. After the pilot reached the specified landmark, they were instructed to make a level turn to a heading of 350 degrees (Figure 3, Segment 2). Again pilots were instructed to maintain airspeed, altitude, and the new heading (Figure 3, Segment 3). The next step was to locate runway 26 on the navigation display, turn at pilot discretion towards the runway, reduce airspeed to 130 knots, lower the landing gear, and proceed to land (Figure 3, Segment 4).

Start

KMHV

End

2

1

3

4

Figure 3. Overhead View of Circle to Land Task

FLIR Following Task In the FLIR following task, pilots were

instructed to initially maintain altitude, heading, and airspeed. Then they were instructed to perform a right-hand descending turn to a specified heading while maintaining airspeed (Figure 4, Segment 1). After the descending turn, pilots had to maintain airspeed and altitude at 500 ft AGL while using the FLIR inset to locate a highway (Figure 4, Segment 2). Once the highway was located, pilots were asked to track the highway using the FLIR inset while maintaining airspeed and altitude (Figure 4, segment 3).

Figure 4. Overhead View of FLIR Following Task

Terrain Following Task In the terrain following task, pilots were

instructed to maintain altitude, heading, and airspeed while flying into the mouth of a canyon (Figure 5, Segment 1). Upon reaching the canyon, pilots had to maintain 500 ft AGL by estimation based on SVS terrain imagery and maintain airspeed while attempting to keep the aircraft in the middle of the valley (Figure 5, Segment 2).

Experimental Display Configurations

The run matrix for each task in the breadth study is shown in Table 3. The configurations shown in Table 3 were administered to all pilots in the breadth study. Half of the pilots flew the BE-350 aircraft type, the other half flew the Boeing 737-400 aircraft type. Each pilot flew a total of 16 runs. Together with simulator familiarization and training, it took most pilots around 2.5 hours to complete the assignment. We felt that this was about the maximum feasible duration for a data collection session.

Figure 5. Overhead View of Terrain Following

Task

Table 3. Breadth Study Run Matrix

Run Number Run Type 2D Symbology Latency

FLIR Channel Latency

Terrain Channel Latency

1-1 Path Intercept 0 ms 0 ms 0 ms 1-2 Circle to Land 0 ms 0 ms 0 ms 1-3 FLIR Follow 0 ms 0 ms 0 ms 1-4 Terrain Follow 0 ms 0 ms 0 ms 2-1 Path Intercept 0 ms 0 ms 1242 ms 2-2 Circle to Land 0 ms 1242 ms 1242 ms 2-3 FLIR Follow 0 ms 1242 ms 0 ms 2-4 Terrain Follow 0 ms 1242 ms 1242 ms 3-1 Path Intercept 0 ms 1242 ms 0 ms 3-2 Circle to Land 0 ms 1242 ms 0 ms 3-3 FLIR Follow 0 ms 1242 ms 1242 ms 3-4 Terrain Follow 0 ms 0 ms 1242 ms 4-1 Path Intercept 0 ms 1242 ms 1242 ms 4-2 Circle to Land 0 ms 0 ms 1242 ms 4-3 FLIR Follow 0 ms 0 ms 1242 ms 4-4 Terrain Follow 0 ms 1242 ms 0 ms

Table 4. Depth Study Run Matrix The run matrix for the depth study is shown in Table 4. In the breadth study, only the circle to land task was used and the aircraft type was alwaysthe BE-350. In the depth study, the focus was measuring a larger range of latencies. A total of 16 runs were flown by all pilots in the depth study.

Terrain Channel Latency 0 ms s 133 m 266 ms 533 ms

on FLIR l

0 ms ChanneLatency

133 ms 266 ms 522 ms

6.B.4-6

Breadth Study Results The results of the breadth study were broken

down by four different flight tasks mentioned in on. Due to the two aircraft used in

the s

-

ect

oyed a

t

ld

er

n-

e entry poin le.

t

ration, as the 2

, and heading, as well as the number of

time

y ding were

o

the the experimental scenarios sectidiffering dynamics between the

tudy, flight technical performance differed accordingly. The BE-350 (King Air) is a relatively nimble aircraft and the Boeing 737-400 (B-737400) is a relatively sluggish aircraft. Pilots flying the BE-350 usually generated smaller flight technical errors than pilots flying the B737-400. However, the effect of latency had a stronger effon the BE-350 when compared to the B-737-400. The reason is that in the BE-350, pilots emplcontrolling strategy that relied on the aircraft instruments to immediately close the inner control loop. Pilots in the more sluggish B-737-400 did norely on immediate feedback from the instruments but rather trusted that their control inputs wouhave the desired effect at some time in the future. This means that pilots in the BE-350 flew with a higher gain on the inner loop control task when compared to the B-737-400 pilots. Also, the slowroll and pitch rates in the B-737-400 meant that even the latent displays were usually able to keepup with changes in attitude.

Pathway Guidance Acquisition Task The two main components of the pathway

guidance acquisition task were the off-path and opath guidance. In the initial phase of this task, the aircraft was off path about 5 miles from th

t of the pathway at a 30 degree intercept angPilots flew to the entry point by holding a heading, altitude, and speed until they were within about ¼ mile off the path. At that point, they initiated a righturn to intercept the inbound pathway.

Overall, we found that off-path, pilots showed a deterioration of the flight technical performance in the presence of SVS or FLIR latency. In contrast, when on-path, there was no such deterio

D guidance elements were non-latent and the pilots simply tried to ignore the latent FLIR and SVS.

While flying the off-path guidance portion of the pathway acquisition task, performance was measured in terms of deviation from airspeed, altitude

s the aircraft rolled through the horizontal zerodegree roll axis. Pilots maneuvering the BE-350showed significant error trends for the RMS heading metric. As shown in Figure 6, RMS heading error increased in the SVS terrain-only (“T”) and the SVS terrain and FLIR (T&F) latencconfigurations. Pilots showed the highest heaerrors when both the FLIR and terrain layers delayed by 1242 ms. Pilots also showed nearly nincrease in heading error relative to the baseline when only the FLIR (F) layer was delayed. This is because the FLIR inset was relatively small and could be ignored more readily when compared to the large field of view SVS surround.

Latency

RM

S H

dg E

rror

[deg

]

6

5

4

3

2

1

0

Boxplot of RMS Hdg Error [deg] vs LatencyAcft = BE-350

T & FTFBaseline

Figure 6. BE-350 RMS Heading Error, Pathway Guidance Task (Off-Path)

For the pilots flying the B-737-400, no

B-737-400 puts that even ith the

s

significant effect due to SVS or FLIR latency was found for RMS heading error. This is because the

reacted so slowly to aileron in the latent SVS or FLIR could keep up w

actual aircraft roll angle. The median RMS error for all run configurations differed by less than one degree. For the on-path portion of the task, we found little effect of latency. Cross track error (XTE for BE-350 pilots and B-737 pilots fell within the limits of pathway displayed. The pathway on the PFD provided a guidance path that was 50 meterwide. Both groups of pilots maneuvered their corresponding aircraft within these tight limits for all latency configurations in this task.

Circle to Land Task The circle to land task forced pilots to use a

multitude of cues for proper alignment and

6.B.4-7

guidance. For example, the turn from downwind to base n.

scale and the Cooper-Harper

ased in the presence of latency.

eed, and altitude. Roll reversals, pitch

puts.

oscillation around roll axis. A smaller number of

reve

y

d , roll

reve t

ing

was marked by a landmark on the SVS terraiUsing this turn cue required reliance on the SVS portion of the display. The turn from base to final was accomplished by anticipating the proper amount of leading distance into the turn from the MFD, which showed an outline of the approach runway. The last portion of the final approach leg required reliance on detecting the runway on the FLIR and SVS portions of the PFD. A relatively large number of significant effects were found for pilots flying the BE-350 on both subjective and objective MOEs.

Subjective Results Pilot ratings for both the Navy Pilot Induced

Oscillations (PIO)workload scale increBoth aircraft rated the baseline run with the lowestmarks and the T&F configuration with the highestmarks (higher marks are worse). Similar results can also be observed from the Cooper-Harper scale results. Again, both aircraft pilots ranked workloadsimilarly. In both aircraft scenarios, pilots ranked the T&F latency configuration significantly higher than the baseline run, indicating a significant perceived increase in workload and deterioration in controllability in the presence of extreme latencies such as those tested in the breadth study.

Objective Results For the circle to land task, performance was

expressed in terms of deviation from airspspecified headings, reversals, and aileron inputs were monitored in addition to normal FTEs. Pilots did not show a significant effect due to latencies in terms of heading and altitude maintenance. However, there were significant effects in roll reversals and aileron inThis means that the pilots were able to somewhat mask the flight technical effects of SVS and FLIR latency by working the controls more vigorously and with higher control gain. In the trimmed out condition, both aircraft were stable in the pitch axisand required little attention or correction in that axis. Pilots capitalized on this stability, resulting in a lack of a significant latency effect in the pitch axis. However, both aircraft were spirally divergent, requiring attention and correction in the roll axis.

Display latency lead to some degree of

roll reversals generally indicated better controllability of the aircraft. However, roll

rsals results should also be interpreted inconjunction with aileron inputs as it is possible forpilots to mask deteriorating performance by more vigorous manipulation of the controls.

Aileron inputs and roll reversals were onlmonitored during sections of the flight task that required maintaining constant altitude, heading, anairspeed. Therefore, in an ideal situation

rsals (Figure 7) would be zero and all aileronmovements (Figure 8) would be in a single bin, azero degrees of deflection, of the aileron histogram. Figure 8 shows the combined histogram of BE-350 pilots’ inputs for flying a constant headof 115° at 3700 ft mean sea level (MSL). The baseline run shows a small distribution with most of the aileron inputs centered at zero degrees of deflection. A large contrast can be seen in the T&F configuration that led to a significant increase in the aileron deflection.

L aten cy

Rol

l Rev

T & FTFB L

25

20

15

10

5

0

Boxplot of R oll R ev v s L atencyAcft = BE-350

Figure 7. BE-350 Roll Reversals, Circle to Land Task

Freq

uenc

y

7550250-25-50-75-100

Histogram of Aileron Inputs [BE-350]Task = Circle To Land - Hdg Hold [115]

Base T & F1600

1200

800

400

0

7550250-25-50-75-100

F T1600

1200

800

400

0

Figure 8. BE-350 Aileron Input Histogram, Circle to Land Task

6.B.4-8

The histogram of the T&F configuration showistribution of pilot aile

s a much wider d ron input. This means that pilots generate much larger inputs to the ailerons when both the SVS and FLIR were latent. A cumulative distribution graph of the same data (Figure 9) shows this effect by means of the slope of the ogives, with a shallower slope being indicative of a larger spread of aileron deflections. Other than for roll reversals, the flight technical performance in this task did not significantly deteriorate, even with the rather extreme level of latency introduced in the SVS and FLIR channel. However, our data shows that the introduced latencies did reduce aircraft controllability and required pilots to expend more controlling effort, especially in the roll axis. It should be noted that in the circle to land task, only the BE-350 showed significant results. Pilots flying the B-737-400 had large variations and overall poor flight technical performance in this task under all conditions with no particular effect due to latency.

Data

Perc

ent

100500-50-100

100

80

60

40

20

0

BaseT & FFT

CDF of Aileron Input [BE-350]Task = Hold Hdg [115]

Figure 9. CDF of BE-350 Aileron Inputs, Circle to Land Task.

FLIR Following Task The FLIR following task required pilots to

track and follow a highway at low altitude using the imagery shown on the FLIR channel.

Subjective Results Pilots rated the FLIR following task with low

overall scores. This indicates that by and large, latency did not have a strong effect on controllability. Only the PIO scale showed some trends with regard to latency.

Objective Results The FLIR following task was relatively easily

accomplished by the pilots of both aircraft types. No altitude, heading, or track related FTEs shows any significant tendencies as a function of latency. Only roll reversals in the BE-350 (Figure 10) and aileron inputs for both aircraft types showed any significant trends as a function of display latency. This task involved reference to a FLIR image and thus the effect of latency was strongest under the FLIR latency condition.

Latency

Rol

l Rev

T & FTFBase

35

30

25

20

15

10

5

0

Boxplot of Roll Rev vs LatencyAcft = BE-350

Figure 10. BE-350 Roll Reversals, FLIR Following Task

Terrain Following Task The terrain following task was difficult to fly

due to the narrowness of the valley through which the pilots had to navigate by reference to SVS and FLIR. The B-737-400 was simply too large an aircraft for this task. Thus, the flight technical errors were high, across all conditions of latency, including in the non-latent base line. The broad distribution of the flight technical errors and overall poor performance across all conditions in the B-737-400 essentially overwhelmed any effects due to latency. For the BE-350, the latent SVS condition caused substantial roll reversals. When both the FLIR and SVS channels were latent, the frequency of roll reversals went up even more. Figure 11 shows a box plot of roll reversals for BE-350 pilots in the terrain following task.

6.B.4-9

Latency

Rol

l Rev

T & FTFBaseline

40

30

20

10

0

Boxplot of Roll Rev vs LatencyAcft = BE-350

Figure 11. BE-350 Roll Reversals, TerrainFollowing Task

Figure 12 shows the histogram for the aileron inputs under the BE-350 condition. A large increasein aileron inputs was seen when the SVS channel orboth the SVS and FLIR cannels were latent.

Freq

uenc

y

7550250-25-50-75-100

1600

1200

800

400

0

7550250-25-50-75-100

1600

1200

800

400

0

Base T & F

T F

Histogram of Aileron Inputs [BE-350]Task = Terrain Follow

Figure 12. BE-350 Aileron Input Histogram,

Depth Study Results f the breadth study, we

selec

e SVS and ormation sources and latency

he

y along the downwind leg, followed by an off path base and final leg.

sion of 2.5 hours allowed for 16 ru

. elected an extreme latency

to be sure that we would see an effect. For the breadth study, we wanted to have latency numbers that were more along the lines of what might be an upper limit in avionics systems. Thus, it was expected that the flight technical effects would be a bit weaker than those seen in the breadth study.

Subjective Results After each run, pilots had to complete the

Navy PIO scale, the Cooper-Harper scale, and the Bedford workload scale. It seems that both the SVS and FLIR latency had about the same magnitude ofeffect on the subjective ratings of pilot induced oscil . Increasing latency e in all subjective ratings. However, the subjective ratings

t the pilots did not

of a

ght

l

only visible in the periphery. Thus it is expected that FLIR latency would have a greater effect than SVS latency when flying on path.

Due to the spirally divergent dynamics of the BE-350, latency produced a greater effect on cross

Terrain Following Task

Based on the results oted the circle to land task and the BE-350

aircraft type as fixed parameters in the depth study. The circle to land task involved both thFLIR channels as infin those channels seemed to have an effect on flighttechnical performance. The BE-350 seemed to be the critical aircraft type as it seemed to be more sensitive to effects of latency. We also modified ttask slightly to have a portion that required following a highway in the sk

A simulator sesns. This represented about the maximum

number of runs that we wanted to administer to a given pilot in one session. Thus, we distributed the16 runs into 4 levels of latency for the SVS and FLIR channels, respectively. The levels of the latency factor were 0ms, 133ms, 266 ms, and 533ms. It is important to note that these latencies were considerably shorter than the latency of 1242ms that was administered in the breadth studyIn the breadth study, we s

lations, aircraft controllability, and workload caused an increas

were fairly low, meaning thaperceive much of a problem due to latency.

Objective Results, On-Path Guidance In the first portion of the task, pilots were

instructed to maintain a course on the downwind path. The path provided guidance in the form50m wide path along with a moving magenta guidance cue. The vertical track error results show that only the FLIR channel latency had an appreciable effect on VTE. The reason is that since pathway guidance was used, the pilots tended to focus on the central part of the PFD where the flipath vector superimposed on the guidance cue. The FLIR channel surrounded this immediate centraarea on the PFD whereas during foveal fixations of this area the SVS channel was

6.B.4-10

track error (XTE) than it did on VTE. As shown in Figure 13, increasing latencies in either the SVS or the FLIR channel resulted in larger lateral deviations from the center of the path. It can also be seen that in all of the latencies tested, the RMS error never deviated outside the containment of the guidance path. Thus from a flight technical error point of view, it seems that the latencies we tested had very little practical effect on positional precision. Display designers should consider using these 2D plots to estimate the resulting on-path VTE and XTE for a given combination of SVS and FLIR latency.

SVS Latency [ms]

EV

S L

aten

cy [m

s]

5004003002001000

500

200

100

0

400

300

> - - - - - - - - - <

100.0 110.0110.0

90.0100.0

Contour Plot of RMS XT Terrain Latency [ms]

20.020.0 30.030.0 40.040.0 50.050.0 60.060.0 70.070.0 80.080.090.0

RMS XTE [ft[

E [ft] vs FLIR Latency [ms],

Figure 13. RMS Cross Track Error (XTE), Depth Study

Object Results, Off-Path Guidance The path ended just prior to the turn towards

the base leg. For the base turn, pilots were inmaintaining constant airspeed, altitude, and heading. Performance just after the turn was

om heading, error in

right

d

te of

with

d error is less than 4 knots, well within any practical

its. The only practically -

structed to turn to a specified heading while

assessed in terms deviation frcapturing initial heading, deviation from specified airspeed, number of roll reversals, and the number of pitch reversals. In addition, a spectral analysis was conducted on aileron input.

In contrast to the breadth study, pilots showed far fewer roll reversals. There was no effect of the latencies on pitch reversals.

Figure 14 shows RMS indicated airspeed (IAS) error as a function of terrain and FLIR latency. The maximum observed speed error occurred when only the SVS was latent (bottom

corner of graph). In that configuration, the SVS and FLIR visually separate and “swim” against each other each time the aircraft attitude changes. This appears to distract the pilots somewhat, leading to some inattention to airspeemanagement. A very interesting effect is that bysimply synchronizing the FLIR channel to the latency of the SVS, it is possible to all but eliminathe IAS error, even under the highest set-pointlatency used in the depth study. The reader may also wonder why IAS error appears to be worse at the zero latency set-point (bottom left corner ofgraph) when compared to the maximum latency set-point (upper right corner of graph). Closer inspection of the actual speed bins associated the diagonal axis from the bottom left to the top right corner indicates that the overall span of spee

test standard (PTS) limsignificant speed error effect occurs along the xaxis which signifies latencies in SVS.

SVS Latency [ms]

EV

S L

aten

cy [m

s]

5004003002001000

500

400

300

200

100

0

> - - - -

- - - <

-

5.0

2.5 3.0

1.01.0 1.51.5 2.02.0 2.5

3.0 3.53.5 4.04.0 4.54.5 5.0

[kts]IAS Error

Contour Plot of IAS Error [kts] vs FLIR Latency [ms], Terrain Latency [ms]

f

Figure 14. RMS IAS Error, Depth Study

RMS heading capture error in this portion of the study is a measure of deviation from a specified target heading when the aircraft first rolls to wings level after the base turn. Consistent with the previous flight technical errors, we see no practical effect due to FLIR latency but some effect due to SVS latency. For the latencies used in this portion of the study, the RMS heading capture error is smallby any standard. However, if possible, designers should try to minimize latency in the large field oview surround which in our case contained an SVS depiction.

While flying the off-guidance portion of this task, pilots were instructed to maintain a constant

6.B.4-11

altitude. Figure 15 shows the RMS altitude error. Both terrain and FLIR latency influenced RMS altitude error. Altitude error increased as eiterrain or FLIR latency became unsynchronized. However, as long as the two latencies remained synchronized, the resulting altitude error remained small. This characteristic was evident at all levelsof latency tested. This is an important finding. If a display element such as a large field of viewattitude depiction is latent, then the FLIR overlay should be synchronized to match that latency. Observationally, we can clearly state that unsynchronized

ther

FLIR and SVS channels are annoying during maneuvering flight and may cause difficulties in controlling behavior, result in higher workload, and cause distractions that may lead to some amount of flight technical deterioration as demonstrated by our results.

SVS Latency [ms]

EV

S L

aten

cy [m

s]

5004003002001000

500

400

300

200

100

0

> - - - - - - - - <

55.0

35.035.0 37.537.5 40.040.0 42.542.5 45.045.0 47.547.5 50.050.0 52.552.5 55.0

Alt Error [ft]

Contour Plot of Alt Error [ft] vs FLIR Latency [ms], Terrain Latency [ms]

Figure 15. RMS Altitude Error, Depth Study

task, sly

orrect t a

tegy and thus we foun ifferent pilots placed priorities on different flight technical parameters. To help gain an u

e equation below:

)

nto one

is

ppears

Due to the relatively demanding flight most of our pilots were not able to simultaneoumaintain all flight technical parameters and cfor all FTE trends. As a result, the pilots selecgeneral strategy for coping with the latent displays and large number of controllable flight technical parameters by placing a higher priority on a select number of these parameters. Pilots were not instructed to use any particular stra

d that d

nderstanding of total performance, a combined flight performance score was developed. The flightscore was calculated from th

Score = (IAS error / 2.5kts) +

(Alt error / 40ft) + (hdg error / 1.5°) +

(hdg capture error) + (roll rev / 2 ) +

(pitch rev / 1) + (XTE / 40ft) + (VTE / 40ft

This composite flight score allows for consolidation of all flight technical results ioverall graph to convey the effect of SVS and FLIRlatency. Figure 16 shows a contour plot of the combined flight score. The picture that emergesconsistent with the earlier observations. Latencies in the large surround of the SVS portion of the display have a larger effect on controllability thanlatencies in the relatively small central FLIR inset. Also, we can clearly see that unsynchronized latencies in the two channels lead to a reduction in flight technical performance. Performance ato remain high along the diagonal axis from the bottom left to the top right corner of the graph, indicating the importance of synchronizing the two channels with regard to their respective latency.

Figure 16. Combined Flight Score, Depth Study

Discussion and Inferences of Results

ected and generally remained well within limits specified by most practical test

The results of our study indicate the need to synchronize the latencies of a large field of view attitude indicator and a central inset FLIR overlay. Unsynchronized overlays cause an effect that we refer to as “swimming”, which is annoying to the pilot, leads to higher workload, and causes distractions that may lead to some amount of flight technical deterioration as demonstrated by our results. Within the limits of the latencies that were administered in this study, we found only a very small effect on actual flight technical performance. Positional accuracy and heading alignment was not strongly aff

6.B.4-12

6.B.4-13

stand

aft

f

ds and workload

than the FLIR inset. This effect occurs due to the physical size of the terrain layer relative to the rest of the display.

Our study contains data from many runs in the simulator and constitutes a sizeable database of pilot performance observations. However, we want to point out that our data should always be interpreted within the context for which it was intended. First and foremost, our data is valid only for head-down displays (HDDs). Tolerance to the effects of latency on head-up-displays (HUDs) is expected to be much smaller than that reported heapplication is both wide fie

com he facto latency. We wed that dow r aircr that

Heading 1 style. Use the Body Text style and include the acknowledgements immediately after the references.

Disclaimer If a disclaimer is necessary, include it under

the heading Disclaimer using the Heading 1 style. Use the Body Text/Normal Text style and include the disclaimer immediately after the acknowledgements.

Email Addresses

ards (PTS). Pilots tended to compensate by operating the flight controls with more gain and with larger amplitudes. This change of the controlling strategy indicates a reduction in aircrcontrollability and may not be desirable for prolonged operations.

The breadth study indicated that latency in channels that convey mission-critical information ismore detrimental than latency in channels that provide supplementary information. If the task for example requires intense use of FLIR for control othe aircraft, the tolerance to latency in the FLIR channel is smaller than if FLIR was used only occasionally or for non-critical aspects of the mission. If the task requires intense use of the SVS terrain imagery for mission critical aircraft control, the tolerance to latency is even smaller, as the large field of view of the SVS surround is a compelling cue for aircraft control. The SVS terrain layer holmore weight in pilot performance

rein as the real world scenery in a HUD ld of view and non

latent. Our data is intended as design guidance for displays that have an essentially non-latent 2D symbology (flight path vector, horizon line, pitch ladder, etc.), a large field of view SVS attitude indicator, and a central FLIR (or other sensor) overlay. Also, we stated earlier that there are essentially an unlimited number of possible

binations that could be conceived from trs that are thought to be influenced by

tested only two aircraft types and narron to one type in the depth study. There are otheaft types such as fast jets or rotary wing

were not considered in our study. Additional work is recommended but our data should serve the display design community quite well with guidance that quantifies the effects of latency on HDDs.

References [1] Schnell, T., 2004, Human Factors of Video and Data Displays, Development of a Study Plan”, Rockwell Collins Project Report, Rockwell Collins,Cedar Rapids, Iowa 52498

Acknowledgements If an acknowledgement is necessary, include it

under the heading Acknowledgements using the

Nicholas Lorch: nlorch@engineering.uiowa.edu Thomas Schnell: tschnell@engineering.uiowa.edu

mMarty Steffensmeier: mjsteffe@rockwellcollins.co

26th Digital Avionics Systems Conference October 21, 2007