EUROGRAPHICS ’0x / D. Ebert, P. Brunet, I. Navazo (Guest Editors) Volume 0 (2002), Number 0 Audio-Visual Situational Awareness for General Aviation Pilots Lilly Spirkovska NASA Ames Research Center [email protected]Suresh K. Lodha University of California, Santa Cruz [email protected]Abstract Weather is one of the major causes of general aviation accidents. Researchers are addressing thisproblem from various perspectives including improving meteorological forecasting techniques, collecting additional weather data automatically via on-board sensors and "flight" modems, and improving weather data dissemination and pre- sentation. We approach the problem from the improved presentation perspective and propose weather visualization and interaction methods tailored for general aviation pilots. Our system, Aviation Weather Data Visualization En- vironment (AWE), utilizes information visualizationtechniques, a direct manipulation graphical interface, and a speech-based interfaceto improve a pilot’s situational awareness of relevant weather data. The system design is based on a user study and feedback from pilots. 1. Introduction General aviation (GA) flight safety rests on four pillars of situational awareness: position, terrain, traffic, and weather. Loss of weather situational awareness is a main cause of GA accidents and has been attributed as a factor in over 30% of accidents and over 15% of fatal accidents 2 . One pos- sible cause is that the pilot may not absorb and retain all the weather information he/she is required to review prior to flight. Because more data is provided than is applicable to a given flight and the data is presented poorly, it requires much time and cognitive effort to develop a "big picture" view of the weather, especially in an unfamiliar area. In this work, we focus on providing weather situational awareness to gen- eral aviation pilots. There are several components of weather that a pilot needs to be aware of in order to maintain weather-related situa- tional awareness. These components include surface winds and winds aloft, cloud conditions, and visibility conditions. Important classifications of weather related data and infor- mation on weather trends to alert the pilot for impending danger or assist him in path planning or replanning is im- portant. Communication of all this information effectively so that the pilot can assimilate and act on the information is of the utmost importance. A number of researchers have addressed the poor pre- sentation problem by developing graphical displays of avi- ation weather for use prior to flight. However, most of these systems are tailored for use by commercial airline pilots 6 . The concerns of commercial airline pilots are different and have been discussed by us in our earlier work 9 . Our system AWE (Aviation Weather Data Visualization Environment) focuses on providing weather data tailored to the needs of general aviation pilots using visual displays 9 . In particu- lar, AWE focuses on graphical displays of three weather ele- ments, namely, meteorological observations (METARs), ter- minal area forecasts (TAFs), and winds aloft forecasts and maps them onto a cartographic grid specific to the pilot’s area of interest 9 . AWE was evaluated by pilots at NASA Ames Research Center, California, and found to be useful 9 . However, the pilots also gave a number of suggestions for improving the system. In this work, we describe a few important improve- submitted to EUROGRAPHICS 2002.
Transcript
EUROGRAPHICS’0x / D. Ebert,P. Brunet,I. Navazo(GuestEditors)
Volume0 (2002), Number0
Audio-Visual Situational Awareness for General AviationPilots
AbstractWeatheris oneof themajor causesof general aviationaccidents.Researchers are addressingthis problemfromvariousperspectivesincluding improving meteorological forecastingtechniques,collectingadditional weatherdataautomaticallyvia on-board sensorsand"flight" modems,andimprovingweatherdatadisseminationandpre-sentation.Weapproach theproblemfromtheimprovedpresentationperspectiveandproposeweathervisualizationandinteractionmethodstailoredfor general aviationpilots.Our system,AviationWeatherDataVisualizationEn-vironment(AWE),utilizesinformationvisualizationtechniques,a directmanipulationgraphical interface, andaspeech-basedinterfaceto improvea pilot’s situationalawarenessof relevantweatherdata.Thesystemdesignisbasedon a userstudyandfeedback frompilots.
1. Introduction
Generalaviation (GA) flight safetyrestson four pillars ofsituationalawareness:position,terrain,traffic, andweather.Lossof weathersituationalawarenessis amaincauseof GAaccidentsand hasbeenattributed as a factor in over 30%of accidentsand over 15% of fatal accidents2. One pos-sible causeis that the pilot may not absorband retain alltheweatherinformationhe/sheis requiredto review prior toflight. Becausemoredatais providedthanis applicableto agivenflight andthedatais presentedpoorly, it requiresmuchtime andcognitive effort to developa "big picture"view ofthe weather, especiallyin an unfamiliar area.In this work,wefocusonproviding weathersituationalawarenessto gen-eralaviationpilots.
Thereareseveralcomponentsof weatherthatapilot needsto be aware of in order to maintainweather-relatedsitua-tional awareness.Thesecomponentsincludesurfacewindsandwindsaloft, cloudconditions,andvisibility conditions.Importantclassificationsof weatherrelateddataandinfor-mation on weathertrendsto alert the pilot for impendingdangeror assisthim in pathplanningor replanningis im-
portant.Communicationof all this informationeffectivelysothatthepilot canassimilateandacton theinformationisof theutmostimportance.
A numberof researchershave addressedthe poor pre-sentationproblemby developinggraphicaldisplaysof avi-ationweatherfor useprior to flight. However, mostof thesesystemsaretailoredfor useby commercialairline pilots 6.Theconcernsof commercialairline pilots aredifferentandhavebeendiscussedby usin our earlierwork 9. OursystemAWE (Aviation WeatherData VisualizationEnvironment)focuseson providing weatherdatatailoredto the needsofgeneralaviation pilots using visual displays9. In particu-lar, AWE focusesongraphicaldisplaysof threeweatherele-ments,namely, meteorologicalobservations(METARs),ter-minal areaforecasts(TAFs), andwinds aloft forecastsandmapsthem onto a cartographicgrid specificto the pilot’sareaof interest9.
AWE wasevaluatedby pilots at NASA AmesResearchCenter, California,and found to be useful 9. However, thepilots alsogave a numberof suggestionsfor improving thesystem.In this work, we describea few importantimprove-
mentsover our previous work. First, we describethe en-hancementsin informationdisplaysthat provide muchim-proved situationalawareness.Second,we presenta com-pletelynew speechprocessingfunctionalitythatwill reducehead-down timefor thepilotssignificantly.
Section2 presentsa brief backgroundfollowed by ourprevious work on AWE in Section3. Section4 describesthe new informationdisplaysthat incorporatepilots’ feed-backon includingweathertrendinformation,anddisplayofmultiple forecastelementsor multiple winds aloft. Section5 presentsspeech-basedinteractionswithin AWE. We con-cludewith our futureplans.
2. Background
Significantadvanceshave beenmadein the last decadeinbothweatherforecastingandweathervisualizationfor ava-riety of audiencesincluding scientists,forecastersand thegeneralpublic 3. ProfessionalTV productionsystemssuchastheTriVis system8, interactive 3D weathervisualizationsystemssuchasVISUAL operatingin theGermanMeteoro-logicalOffice(DWD) in collaborationwith FraunhoferIGD,andpersonalizedWeather-on-Demandproductsthroughtheinternet are available. Augmentedreality weathervisual-ization systemsare alsobeing developed.Numerousavia-tion weathervisualizationefforts are underway includingthoseat theFederalAviationAdministration(FAA), theNa-tional Oceanicand AtmosphericAdministration(NOAA),theNaval ResearchLab, RockwellScienceCenter, Honey-well International,WSICorporation,BFGoodrich,MIT Lin-colnLab,EchoFlight, andJeppesenInc.
In spiteof all thisprogress,however, likeotheraudiences,pilotsarerequestingtailoredvisualizationtoolsfor theirspe-cific needs.In particular, existing and developing weatherforecastingand visualizationtechnologyneedsto be har-nessedappropriatelyfor the benefitof pilots. The goal, asoutlinedbyStough,themanagerof NASA’saviationweatherinformationsystem(AWIN) project,is to provide "weatherinformationrelativeto thepilot’sflight path,presentit to thepilot in thecockpit in aneasy-to-interpretgraphicalformat,andgive him decision-makingaidsto helphim usethat in-formation..." 4.
The Aviation Digital DataService(ADDS) system13, ajoint effort of NOAA, theNationalCenterfor AtmosphericResearch(NCAR), and the Aviation WeatherCenter, per-hapscomesclosestto fulfilling theAWIN goal.Initially re-leasedin 1997, it providesunofficial graphicaldisplaysofa variety of aviation weatherbriefing report elementsandallows pilots to zoom in to get more specific informationaboutan areaof interest.Although it presentsa greatvari-etyof weathergraphics,so-calledterminalareaforecasts(tobedescribedlater in Section3) aredisplayedonly textuallywithout any filtering. It alsodoesnot relatethe datato thepilot’s plannedpathor schedule.
3. AWE: Aviation Weather Data VisualizationEnvironment
Wenow presentanoverview of our previouswork on AWE(Aviation WeatherData VisualizationEnvironment)9. Wealsoprovidedetailsonsomespecificdisplaysin orderto ex-plainseveralimprovementswehaveincorporatedin AWE toincreasethesituationalawarenessof pilots.
In AWE, we have designedand testedthreevisual dis-playsfor communicatingcurrentandforecastconditionsatairportsincludingclouds,visibility, andsurfacewinds,andonedisplayfor communicatingwindsaloft forecasts.A pilotcanuseagraphicaluserinterfaceto interactwith theweatherdataandplanhisflightsquickly andeasily.
VFR charts as Display Background: Theweatherdatais overlayedon a VFR (VisualFlight Rules)sectionalaero-nauticalchart,asshown in Figure1. Thechartshows thelo-cationof airports(magentaor bluecirclesor shortlinesthatmimic the runway layout), airways("highways" in the skyshown aslight blue straightlines),navigation aids(mostlydepictedby a compassrose), controlled and specialuseairspace,obstructions,naturalterrainfeatures(suchaswaterandhills, depictedusingcolorcodedaltitudes),demographicfeatures(suchascities,depictedin yellow), andmaximumelevation in eacharea(depictedwith numberswith super-scripts).Although the backgroundmay look clutteredandcomplex, thechartbackgroundtexturegivespilotsafamiliarenvironmentwith which to interactandprovidesthemwithadditionalinformationfor makingtheir"go/no-go"decision.For example,windsaloft speedsof 40 knotsaremuchmoreproblematicif youwill beflying 2000feetabovemountainsthanif you’ll beflying 2000feetover flat terrain.Overlay-ing theweatheronthechartconsolidatesweatherandterrainsituationalawareness,allowing thepilot to make a decisionby lookingonly at onesource.Displayingweatherinforma-tion next to the airport it appliesto improvesa pilot’s abil-ity to perceive the big picture,especiallyin an unfamiliarareawherehe otherwiseneedsto look at a chart to deter-minewhereeachairport is locatedandhow it relatesto hisflight path.Displayingsurfacewind informationneartheair-port alsoimprovesthepilot’s ability to visually determinealandingrunway anddeterminethe amountof crosswindtoexpect. It provides information,while still away from the
airport, that is commensuratewith the information he re-ceives from the airport wind sock.We decidedon usingaVFR Sectionalchart as the backgroundafter consideringseveralalternativesasdescribedin ourpreviouswork 9. Theuseof VFR aeronauticalchartdrew unanimousenthusiasticresponsefrom thepilots.
Input: AWE beginswith anofficial briefingdownloadedfrom DUATs andextractscurrentairportconditionreports,airport forecastreports,andwinds aloft forecastreports.ADUATscurrentairportconditionsreport,shown in Figure2,provides data about surface wind velocity, cloud altitudeandcoverageamount,visibility andobstructionsto visibil-ity, temperature,dew point,andbarometricpressuresetting.A DUATs airport forecastreport (terminal areaforecast),shown in Figure3, providespredictionsaboutsurfacewindvelocity, cloudaltitudeandcoverageamount,andvisibilityandobstructionsto visibility.
Figure 2: Samplecurrent airport conditionsfile. Each linerepresentsa report for a specificairport, specifiedby theairport identifier (first element).It alsospecifiesthetimeofobservation,wind directionandwind speed,visibility, visi-bility restrictions,cloud layers (with coverage amountandaltitude),temperature, dew point,andbarometricpressure.
Visual Displays: Wenow describethethreevisualizationformatsusedin AWE. All threedisplaysusethe commonfields listed in the previous paragraph;the textual formataddsa few additional,uniquefields. For easeof interpre-tation,wechoseto usesimilardisplaymethodsfor bothcur-rentandforecastconditions.
SymbolicVisual Display: The symbolicformat is shownin Figure1.Thetoprectangleof thesymboldepictswind ve-locity. Wind directionis shown by thedirectionof thearrow,while wind speedis shown by the thicknessof the arrow.This methodallows a pilot to quickly glanceat thedisplayandvisually detectstrongwinds.The borderof the rectan-gle is alsocolor-coded(redfor verystrongwinds,yellow forsomewhatstrongwinds,andblackfor reasonablewinds) tofurtherenhancethepilot’sability to quickly determinewindconditions.
Themiddlerectangledepictscloudinformation.Therect-angle representsaltitudesfrom 0 feet (at the bottom) to
Figure 3: Sampleairport forecastfile. A forecastfor anair-port extendsfromtheairport identifierto thedot ("."). Eachforecastprovidesthetimetheforecastwascreatedfollowedbytheeffectivetimesof each forecastelement.Each elementspecifiesthewinddirectionandspeed,visibility andrestric-tions to visibility, and cloud layers (with coverage amountandaltitude).
12,000feet(at thetop,below thewind rectangle).This rect-angleis then filled with sub-rectanglesto representcloudaltitude and coverageamount.The FAA has definedfiverangesfor cloud coverage:the sky can be reportedto beclear, mayhave few clouds(FEW = lessthan1/8 of theskyis covered),scatteredclouds(SCT= between1/8and3/8ofthesky hasclouds),brokenclouds(BKN = 4/8 to 7/8 of thesky), or becompletelyovercast(OVC = 8/8of thesky). TheFAA alsoprovides a list of standardcontractionsfor visi-bility obscurations.Ourexamplesarenot for comprehensivecoverage,but ratherfor generalunderstanding,sonotall op-tions are described.Using informationvisualizationmeth-odspromotedby Tufte 10� 11 andBertin 1, we mapthesefiverangesto fivegraylevels:whitefor clearskies;progressivelydarker grayfor few, scattered,andbrokenclouds;andblackfor an overcastsky. Cloud coverageand altitudeare mea-
Figure 1: Currentcloud,visibility andwindsconditionsshownusingthesymbolicandtextual formats.Thebackgroundimageis anaeronauticalchart for theSanFranciscoBayArea.
suredby ground-basedinstrumentslocatedat theairportandcanbereportedaslayers.Thatis, cloudinformationmaybereportedasfew @ 1200ft, sct@ 3000ft, ovc @ 12,000ft.Hence,cloudcoverageis depictedby layeringappropriatelycoloredsub-rectanglesat a vertical positiondeterminedbya linearmappingfrom 0 to 12,000feet.Similar to thewindrectangle,the cloud rectangleis color-codedto easeinter-pretation.Thecolor-codingtakesthecloud layersinto con-siderationaswell astheinformationdisplayedastext in thebottomrectangle:visibility distanceandany obstructionstovisibility, suchasfog, haze,or smoke.Theborderis coloredblackif thecloudsandvisibility allow apilot to fly undervi-sualrules(lowestbrokenor overcastcloudlayer, or ceiling,atanaltitudegreaterthan3000ft andvisibility betterthan5miles),yellow if theceilingor visibility is marginal (ceilinggreaterthan1000ft but lessthan3000ft; andvisibility be-tween3 and5 miles),or redif theceilingor visibility is lowenoughto requireflight usinginstrumentflight rules(ceilinglessthan1000ft; visibility lessthan3 miles).
Both currentand forecastairport conditionscanbe dis-playedusingthesesymbols.They providemostdataneededto planflights. Otherdataavailablein currentconditionre-portsis notusuallynecessarybut is sometimesuseful.
Textual Display with Color-Coded Borders: If a pilotwantsto view the above mentioneddatatextually or if hewantsto seeprecisenumbersfor theotherelements,hecanask for a textual presentation,as shown in Figure 1. Thisformat shows all the elementsprovided.The text rectangleis color-codedusingthesamecodingweusedfor themiddlerectangleof the symbolicdisplay. That is, it representsthevisibility andcloud coveragecombinationandcanbe red,yellow, or black.
Triangular Iconic Display: The last format a pilot canrequestis the iconic format which provides an overviewof wind, visibility, cloudaltitude,andcloudcoveragedata.In the currentconditionsicon, the temperature- dew pointspread(that is, temperatureminusdew point; usefulfor de-tecting possiblefog formation) is also shown. It displays
theseelementsusingfour sub-trianglesarrangedinto a sin-gletriangle,asshown in Figure5.Eachsub-triangleis color-codedusing the colors white for good conditions,yellowfor marginal conditions,andred for adverseconditions,asdefinedfor the symbolicformat above. Becausethe airportforecastreportsdo notprovide temperatureor dew point in-formation,thecentersub-triangleis coloredgraywhendis-playingforecastdata.Theiconicdisplayis usefulin provid-ing averyquickoverview of conditions.A pilot canquicklyget thebig pictureof theweathersituation.For example,ifthebottomsectionof thetrianglesis red in a wide area,thepilot can quickly understandthat there is widespreadlowfog. On theotherhand,if ageographicareashows all whitetriangles,hecaneasilyseethatat leasttheelementsshowngraphicallyareconducive to flying.
Winds Aloft Display: Finally becausethey are comple-mentaryto current and forecastairport condition reports,wealsoprovidewindsaloft forecastsgraphically. Thewindsaloft forecastsreport,shown in Figure4,providesdataaboutthe wind velocity and temperatureat predefinedaltitudes,generallyspeakingrangingfrom 3000ft to 39,000ft in in-crementsof 3000ft. We found that pilots preferreda verysimplerepresentation- onethat doesnot requiremuchin-terpretation.Thus,we displaythewindsaloft for a selectedaltitudeasadirectionalarrow, asshown in Figure5. Thedi-rectionof thearrow specifiesthedirectionthewind is com-ing from. The wind speedis shown textually on the arrow.The pilot selectsan altitudeusinga slider provided on thegraphicaluser interface.As the pilot dragsthe slider, thewind arrows changeto reflectwindsat theselectedaltitudeusinginterpolationto computewindsat altitudesnot givenin thewindsaloft forecastreport.Thishelpsthepilot choosean appropriatecruisingaltitudeandappropriateflight pathwithouthaving to performany computation.
All of the above formatscanbe displayedfor an entireareaor just alonga pilot-selectedrouteof flight. For windsaloft displayedalonga route,we usedistance-basedinter-polationof the two closestwind reportingsitesto computewinds at selectedlocations.Similarly, if currentconditionsor forecastsare requestedfor airportsthat do not providereports,thenearestavailablereportsaredisplayed.Thispro-videsthepilot anestimateof what theweathermaybe likeat hisselectedairports.
We conducteda user study to test the usability of theabove formats.Throughinterviews anda questionnaire,wedeterminedthat pilots preferredour graphicalformatsoversevenformatsfromfour competitivesystems.Thestudyalsorevealedareasfor furtherimprovement.In thenext two sec-tions,wediscusssomeof themostimportantfeedbackfromtheuserstudy.
4. Information Displays
The focusof this work is to provide additionalinformationdisplaysto generalaviationpilotsto bettermaintainweather
Figure 5: Current weather conditions shown using theiconic format.Windsat a pilot-selectedaltitudeshownwitha cyanarrow.
In ouroriginal implementationof AWE,weprovidedasym-bolic representationof METAR/TAF display as shown inFigure1 anddiscussedearlier in Section3. In the originalversion,AWE displaysonly themostrecentreportof winds,clouds,and visibility conditions.The userstudy revealedthatthepilotswantedtheability to look backatpreviousre-portsaswell. An importantfeedbackfrom thepilots wastoprovideadisplayof weathertrendsbasedonpredictions.Animportantrelatedquestionis how do we determinewhetherthesepredictionsareaccurateor not.
In orderto addressthis feedback,we presenttwo layersof weathertrenddisplay. On the top layer, we display theforecastapplicableat time t - 2, t - 1, t, andt + 1 usingthesymbolicrepresentationasshown in Figure6.Onthebottomlayer, theactualconditionsattime t - 2, t - 1, andt areshownsymbolicallyandalignedwith thesametime stampsof thetoplayerasshown in Figure6.Thebottomlayerof informa-tion associatedwith a givenairportprovidesa pilot with in-formationonhow theweatheractuallychanged.Whencom-paredwith theforecastfor thattimeperiodon thetop layer,
thepilot candecidewhetherthecurrentconditionsin wind,cloud,or visibility displayedin thebottomlayersupportorrefutepredictionsshown in the top layer. Althoughstatisti-cal andartificial intelligencebasedapproachescanbeusedto derive statisticalmeasuresof uncertaintyof predictions,mostpilots aremorecomfortableand trustingof raw dataratherthaninterpretationsof theraw databy algorithmstheydo not understand.It is alsovery easyfor a pilot to performa visualpatterncomparison.However, if algorithmsarede-signedto deriveuncertaintiesthatgainthetrustof thepilots,suchmeasurescaneasilybe displayedwithin AWE eitheraugmentingor replacingthesuggesteddisplays.
By studyingFigure6,apilot canseethediscrepanciesbe-tweentheactualreadingsandthepredictions.Observe thatthesurfacewindsarepredictedto becalmconsistentlywhilethe actualreadingsconsistentlyhave a wind in the easterlydirectionwith stablespeed.Forvisibility conditions,thepre-dictionsarethat the fog (andassociatedlow visibility) willdissipateafter14:00andthevisibility will improveto 1 mile(from 0.25mile). Observe that in actuality, the visibility isimproving more quickly than forecastand is alreadyat 3miles.Thecloudsarealsolifting sotheforecastfor ceilingsabove 12,000ft (shown by thewhite rectangle)is plausible.
Althoughherewehavediscussedthetrendinformationtobe displayedusing symbolic representationonly, we haveimplementedthe weathertrend display using other visualmethodssuchasthetriangulariconicdisplaysdiscussedear-lier in Section3.
Therearecertainlyotherdisplaypossibilitiesfor weathertrend information suchas the direct graphicaldisplaysofthe threeelementsusing both actualand predictedcondi-tions.While sucha displaymay be moreuseful for longertime periodsbecausethey mayoccupy lessscreenarea,wepreferredto build on theoriginal displaysdesignedin AWEbecauseof two reasons.First, the trenddisplayis anexten-sion of the staticone time displayand therefore,doesnotcauseany additionalcognitive workload.Second,continu-ousgraph-baseddisplaysseemto be lesssuitablefor pick-ing out exactly wherein time the pilot is in comparisontothe discretedisplaysthat we have designed.Certainly, weneedto conductanotheruserstudyto determinetheefficacyof thenew informationdisplayspresentedin thiswork.
4.2. Multiple Forecast Element Display
In the original implementation,AWE automaticallyselectstheappropriateforecastfor thedestinationairportbasedonthepilot’sestimatedtimeof arrival, ETA. However, theuserstudyrevealedthat pilots alsowantedthe ability to seetheforecastfor nearbytimesto betterdeterminehow certaintheforecastis for their ETA. Thatis, if theweatheris predictedto change30 minutesaftertheir ETA, it maybeprudentforthe pilot to have a backupplan in casethe changeoccurs
Figure 6: Displayof trendinformationfor an airport. Thetop layer displaysthe winds, cloud, and visibility as pre-dicted;thebottomlayer displaysthewinds,cloud,andvisi-bility asmeasured.Correlationor discrepancybetweenthetwo layers allows pilots to determinehow reliable predic-tionsare likely to be.
earlier. Similarly if they are planningto arrive 20 minutesafter the fog is forecastto clear;they maywant to plan foranalternatedestinationif thefog doesnotclearonschedule.
We have incorporatedthis suggestionso that AWE nowhasthe capability to display the predictedweatherfor alltimeperiodsprovidedby theDUATsreportfor pilot-selectedairports. Rather than having AWE select the appropriateforecastfor theestimatedarrival time,AWEcannow displaythe entireseriesof forecastelements.We refer to this dis-play asmultipleforecastelementdisplay. AWE canuseanyoneof thethreevisualizationmethods– textualdisplaywithcolor-codedborders,overview triangularvisualdisplays,orsymbolicvisualdisplays– to visualizemultiple forecastsatseveralairportssimultaneously. Figure7 shows anexampleof a triangularoverview display for threeairportssimulta-neously. The fourth airport shown with a single triangle isonethepilot hasnot requestedadditionalinformationabout.Hence,only the elementapplicableto the AWE-computedarrival time is shown.
Fromthisdisplay, thepilot canseethat,althoughtherearefour elementsgiven in the TAF for oneof his selectedair-ports(OaklandInternational,left sideof the image,shownwith 4 triangles),it will haveacceptablesurfacewinds,visi-bility, andcloud conditionsduring all periods.In contrast,he can very quickly seethat for SanJoseairport (shownon the bottom of the image,with 6 elements)the windswill be moderateduringoneforecastduration,improve fora time, and thendeterioratealongwith poor visibility andloweredcloud ceiling. If the singleelementdisplayfor hisETA showed only a white triangle,he may not be able todevelop a realistic big picture of the expectedconditionsnearbyhis ETA. Notethatall elementsfor the24 hourfore-
castperiodprovidedby DUATs areshown. Thepilot is un-ableto determinewhenachangeoccurs.To getsuchdetails,heneedsto referencethesymbolicor textual formats.Usingthe overview display, he canonly determineif a changeisforecast.
Figure 7: Overview of weatherforecastsfor all periodspro-videdby DUATs at threeairports usingthe triangular dis-play. Ratherthan showingjust the forecastelementappli-cable to the AWE-computedestimatedtime of arrival (asshownfor the left-mostairport), it showsall the elementsassociatedwith a particular airport, asprovidedbyDUATs.Thiskindof displayis likely to beusefulto thegeneral avia-tion pilot to get a quick overview of forecastconditions.
If achangeis forecast,thepilot canaskfor additionalde-tails to determinehow it mayaffect his flight. He canmix-and-matchdifferentkindsof displaysto extract just the in-formationheis interestedin. For example,Figure8 shows asingleforecastelementselectedby AWE asapplicableattheflight’sETA, similarto thedisplayof Figure1, togetherwithall weatherpredictionsfor 3 airportssimultaneously, oneus-ing the textual format to get all the details,and the othertwo usingthesymbolicvisual representation.Thepilot cannow seethat Oaklandairport hasfour elements(the whitetrianglesshown in Figure 7) becausethe wind conditionsareforecastto varysignificantly. Hecanalsoseethatthede-creasedvisibilitiesattheupperright airport(SCK,Stockton)aredueto mist ("BR") andfog ("FG") andwill be a factorfrom 8:00Zulu to 17:00Zulu asfog andlow cloudsdevelopandthendissipatefrom California’s CentralValley. He canalsoseethat the conditionsat SanJoseairport (bottomoftheimage)arecausedby a front moving through,with asso-ciateddecreasingceilings.
4.3. Multiple Altitude Winds Aloft Display
AWE allows thepilot to evaluatepotentialcruisingaltitudesinteractively by displayingwinds aloft velocitiesonly oneat a time ashe manipulatesthe altitudeselectorslider. Theuserstudyrevealedthatpilotswantedto seeforecastsfor allwindsaloft altitudessimultaneouslyto formulateabetterbigpicture.To formulatethesamepicturewithoutthis function-ality, the pilot needsto watchmultiple arrows ashe movesthe sliderandrememberwinds for previous altitudes.With
Figure 8: Overview of weatherforecastsat two airportsusing the symbolicformat and one using the textual for-mat. Thedisplay for SFOairport showsonly the AWE se-lectedelementapplicableat the pilot’s ETA. The two air-portsdisplayingsymbolictrendinformationshowfog fore-cast to develop (upper right airport) and a front movingthrough (lower centerairport) with associateddecreasingvisibilitiesandceilings.Thetextualtrendinformationshowsthatalthoughconditionsat OAK will begoodthroughouttheforecastperiod, the surfacewindswill be changingsignifi-cantly. Thiskindofdisplayis likelytobeusefulto thegeneralaviationpilot in pathre-planningto decidewhetherto landat analternateairport or expediteor delaytheflight arrivaltime.
simultaneousdisplay, all theinformationwouldbeavailableconcurrentlyto allow for aquickvisualevaluation.
AWE now providesmultiple altitudewindsaloft display.DUATswindsaloft reportsprovideforecastsfor wind veloc-ities at 3000,6000,9000,12,000,18,000,24,000,30,000,34,000,and 39,000feet. Aircraft flown by the target au-diencefor AWE rarely fly above 12,000ft. The 18,000ftforecastprovidesadditionalinformationaboutthepotentialfor thunderstormactivity. Thus,we limited themultiplealti-tudewindsaloft displayto 18,000ft. An examplefor theSanFranciscoairport is shown in Figure9. Windsaredisplayedfrom 3000ft at thebottomto 18,000ft at thetop.Thewinddirectionis displayedusingadirectionalarrow andthewindspeedis displayedastext. Therectanglefor eachaltitudeiscolor-codedfor easierdetectionof strongwinds.Wechoseamodifiedstop-lightcolorcodingschemewith greenfor calmwinds( � 10kts),yellow ( � 20kts) for moderatewinds,or-ange( � 30 kts) for strongerwinds,andred ( � 30 kts) forvery strongwinds. The altitudeassociatedwith eachrect-angleis not shown in order to decreaseclutter. It’ s alwaysthesamealtitudesandthealtitudesfollow standardDUATsreports;hence,they areeasilyinternalizableby pilots.
5. Speech-based User Interface
The original implementationof AWE hasonly visual ele-ments.Oneof themostimportantfeedbacksfrom thepilotswasto incorporateaspeech-baseduserinterface.It is hoped
thatwith this interface,thepilot would not have to dedicatemuch eye time and concentrationon interpretingthe tex-tual data.In aneyes-busy, hands-busy environmentsuchasflying an aircraft, a speech-baseduserinterfacewill allowtheuseof anadditionalmodalityto interactwith thesystemdistributing thecognitive overloadbetweendifferentsenses.Experimentalevidenceindicatesthatif a useris performingmultiple tasks,performanceimprovesif thosetaskscanbemanagedover independentinput/outputchannels,or modal-ities 7. Moreover, it is expectedthataspeech-basedinterfacewouldallow easieraccesswithout thedisadvantagesassoci-atedwith WIMP (windows, icons,menus,pointingdevice)interfacesin mobileapplications.
To make AWE easierto usein flight, we have now addeda speech-baseduserinterface.Speechhasa numberof ad-vantages.Using speech,the pilot can extract informationfrom AWE without having to devote muchvisual attentionto the task.Speechhaslow panelreal-estateneeds.SmallGA cockpitsdo not allow for muchotherinput-outputpara-phernalia.Speechasaninputmodalityis quicker thaneithertypingor writing. A personcanspeakseveralhundredwordsperminute,anexcellenttypistcanproduce80-100wpm,andonecanhandwrite at lessthanhalf the typing rate.Speechalsohasdisadvantages.Listeningis slower thanreading.It’ salsotemporal,serial,"bulky", andmaynotbeprivateenoughin somesituations.Also, thetechnologyis still immature.Itworksbestwith easilydistinguishablevocabulariesandlim-itedgrammars.Last,theoutputtendsto soundunnaturalandcanbe misunderstooduntil you becomeaccustomedto itsaccent.
Theadvantagesoutweighthedisadvantagesin theGA do-main for a numberof reasons.First, the official languagefor aviation is English,even in many non-Englishspeakingcountries.However, non-Englishspeakers(pilots,air trafficcontrollers,andflight servicespecialists)mustlearnenough
Englishto transfernecessaryinformation.Second,pilotsareaccustomedto specializedvocabulariesandgrammars.Be-causeof congestedradiocommunications,conversationsareoftenterseandfollow astandardphraseology. Pilotsarealsoaccustomedto communicationusinga headsetor handheldmicrophone.Finally, pilots are accustomedto the speechsynthesizeraccentbecausesomeweatherdata is alreadysynthesized.
Given theadvantagesandlimitationsof speech,we haveimplementedaSUI (Speech-basedUserInterface)for AWE.Wedid not implementanaturallanguageunderstandingsys-tem. Rather, we focusedon a specializedvocabulary andgrammarthat generalaviation pilots often use. We usedspeechasan input andoutputmodality. Theobjective is toprovidethepilot with amechanismto expressneedsandob-tain servicesfrom AWE. We alsochosenot to implementadialog tree 5. They areuseful in situationswherethe userwill not get any pre-usetraining, suchas telephonebanktellers.PilotsusingAWE will get initial training.Also, dia-log treesdo not mimic any communicationstypically foundin GA andcanbe frustratingto use.Commandandcontrolinterfacesbettermimic otherGA communicationsandpilotsshouldfind themmorepleasantto use.
AWE wasimplementedin C++,usesOpenGLandGLUTfor the GUI, andrunsunderLinux. For the SUI, we choseIBM’ s ViaVoice for Linux productsfor speechrecognitionand speechsynthesis.To improve speechrecognitionac-curacy, we designeda grammarthat definedthe directivesthatcanbe issuedto the recognitionengine.Thechallengehereis to designthegrammarsothepilot hassufficient ex-pressive power to ask for what he needs,but the grammarremainslimited enoughto get good recognitionaccuracy.Thegrammaris written in the SpeechRecognitionControlLanguage(SRCL),which is basicallya BNF (Backus-Naurform) grammaradaptedto speechrecognition.Like a BNFgrammar, it hasasetof productionruleswith non-terminals,terminals,andsomepredefinedterminalsthatallow thede-veloperto specify optional words or repeatedwords. TheAWE grammaris constructedto constrainthe pilot to re-questcurrentairportconditions,airportforecasts,andwindsaloft forecasts,andto setprogramparametersaurally. Someexamplesentencesthegrammaracceptsinclude
� show areacurrentweatherassymbols� show routeforecastasiconsat15 30zulu� hideroutewindsaloft� saywindsatPaloAlto� sayforecastvisibility for Monterey at 18hundred� setdisplaytypeto icon� show trendinformationfor Stocktonasicon
AWEcanrespondto thepilot’sdirectivesbyeithergraphi-calfeedback(for show, hide, orsetcommands)orauralfeed-back(for saycommands).Wechoseanimplicit confirmationstrategy for auralfeedback.AWE doesnotexplicitly askthepilot "Did you say...?"andthenwait for a yesor no before
answering.Ratherit providesa succinctanswerto a querybut alsoprovidesenoughcontext so thepilot knows there-questwas understoodcorrectly. If the pilot detectsthat anincorrectquestionis beinganswered,he would reissuethequery, just ashewould if hewerespeakingto a Flight Ser-vice Stationspecialistwho misunderstandsa query issuedover theaircraftradio.Wefeel this is justifiedbecauseAWEprovidesanswersto queries.Therefore,a recognitionerrorresultsin a delayin thepilot gettingthe information,but isotherwiseharmless.
6. Summary and Future Work
AWE (Aviation WeatherData VisualizationEnvironment)hasbeentailor designedto suit the needsof generalavia-tion pilots. It focusedon the gapsfound in other aviationweathervisualizationsystemsby providing displaysrelatedto METARs, TAFs, and winds aloft display. Our previoususerstudywith pilots resultedin usefulfeedbackabouttheusabilityof AWE.
In this work, we have provided additionalfunctionalitybasedon the pilots’ feedback.In particular, we have madeseveral enhancementsto weatherinformationdisplaysandaddedaspeech-baseduserinterface.Ournext stepis to con-ductanotheruserstudyto determinewhetherour implemen-tationof theseadditionalfeaturessatisfiespilots’ needs.
Wethenplanto furtherenhanceAWE to incorporatemoresituationalawareness.In its currentimplementation,AWEreliesonthepilot todeterminewhichareaof thechartto lookat. It would be useful to automaticallyobtain an aircraft’scurrentpositionandusethatto provideonly relevantdatatothe pilot. We will accomplishthis by interfacingto a GPSunit.
Finally, to furtherdecreaseapilot’sworkload,wewill en-hanceAWE to take theinitiative in providing informationitdeterminesmay be of useto the pilot. For instance,whennew weatherupdatesarereceived,AWE canautomaticallylook for inconsistenciesbetweentheforecastsandtheevolv-ing conditions.If it detectsany unexpectedconditions,it canspontaneouslywarnthepilot. Of course,all thedatawill alsobeavailableto thepilot for hisown exploration.
AWE alreadyprovidesuseful functionality to help a pi-lot maintain an awarenessof the weathersituation.Withplannedfuture functionality, he will be able to maintainweathersituationalawarenesswith further reducedwork-load.
3. H. Haase,M. Bock, E. Hergenrother, C. Knopfle,H.-J. Koppert,F. Schroder, A. Trembilski,andJ. Weiden-hausen.Meteorologymeetscomputergraphics– alookatawiderangeof weathervisualizationsfor diverseau-diences.Computers andGraphics, 24:391–397,2000.
4. T. S.Perry. Trackingweather’s flight path. IEEESpec-trum, 37(9):38–45,September2000.
5. R. Rosenfeld,D. Olsen,andA. Rudnicky. Universalspeechinterfaces. Interactions, 8(6):34–44,Novem-ber/December2001.
6. CharlesH. Scanlon. Cockpit graphicalweatherinfor-mation shown to enhanceefficiency, safety, and sit-uationl awareness. In Proceedingsof the 39th An-nualCorporateAviationSafetySeminar(CASS),FlightSafetyFoundation, pages83–94,April 1994.
7. C. Schmandt.Voice Communicationwith Computers.VanNostrandReinhold,New York, NY, 1994.
8. F. Schroderand M. Lux. Trivis: Professionaltele-vision weather presentation. World Wide Web,October 1997. http://www.igd.fhg.de/www/igd-a4/projects/docs/trivis/trivis_e.html.
9. L. SpirkovskaandS.K. Lodha.Awe: Aviationweatherdatavisualizationenvironment.ComputersandGraph-ics, 26(1),January2002.To appear.
10. Edward R. Tufte. TheVisual Display of QuantitativeInformation. GraphicsPress,Cheshire,CT, 1983.
11. Edward R. Tufte. EnvisioningInformation. GraphicsPress,Cheshire,CT, 1990.
