Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 1 A Distributed Architecture for Intelligent Unmanned Aerial Vehicle Experimentation P. Doherty, P. Haslum, T. Merz, E. Skarman, G. Conte, S. Duranti, F. Heintz, P-O. Pettersson, T. Persson, B. Wingman Dept. of Computer and Information Science Link¨ oping University SE-58183 Link ¨ oping, Sweden [email protected]Abstract The emerging area of intelligent unmanned aerial ve- hicle (UAV) research has shown rapid development in recent years and offers a great number of research chal- lenges for artificial intelligence. In this article, a pro- totype distributed architecture for intelligent unmanned aerial vehicle experimentation is presented which sup- ports the development of intelligent capabilities and their integration in a robust, scalable, plug-and-play hardware/software architecture. The architecture itself uses real-time CORBA to support its infrastructure and it is based on a reactive concentric software control phi- losophy. A number of capabilities of the architecture are presented including a multi-mode flight control sys- tem for a Yamaha RMAX VTOL platform, an on-board path planning service and a dynamically reconfigurable image processing system. A research prototype system has been built, is operational and is being used in actual missions. In the article, we emphasize the characteris- tics of the architecture which support the integration of numerous AI technologies. Introduction The emerging area of intelligent unmanned aerial vehicle (UAV) research has shown rapid development in recent years and offers a great number of research challenges for artifi- cial intelligence. Much previous research has focused on low-level control capability with the goal of developing con- trollers which support the autonomous flight of UAVs from one way-point to another. The most common type of mis- sion scenario involves placing sensor payloads in position for data collection tasks where the data is eventually pro- cessed off-line or in real-time by ground personnel. Use of UAVs and mission tasks such as these have become in- creasingly more important in recent conflict situations and are predicted to play increasingly more important roles in any future conflicts. Intelligent UAVs will play an equally important role in civil applications. For both military and civil applications, there is a desire to develop more sophisticated UAV plat- forms where the emphasis is placed on development of in- telligent capabilities. Focus in research has moved from Copyright c 2003, American Association for Artificial Intelli- gence (www.aaai.org). All rights reserved. low-level control towards a combination of low-level and decision-level control integrated in sophisticated software architectures. These should also integrate well with larger net-centric based C I systems. Such platforms are a pre- requisite for supporting the capabilities required for the in- creasingly more complex mission tasks on the horizon and an ideal testbed for the development and integration of AI technologies. The WITAS 1 Unmanned Aerial Vehicle Project 2 (Doherty et al. 2000) is an ambitious, long-term basic research project whose main objectives are the development of an integrated hardware/software VTOL (Vertical Take-Off and Landing) platform for fully-autonomous missions and its future de- ployment in applications such as traffic monitoring and sur- veillance, emergency services assistance, photogrammetry and surveying. Basic and applied research in the project covers a wide range of topics which include both the devel- opment of traditional AI technologies, core functionalities and their pragmatic integration with other technologies in a prototype experimental UAV system. The following is a non-exclusive list of some of the activ- ities in the project: Development of a generic distributed deliberative/reactive software architecture for (aerial) robotic systems. Development of a helicopter control system with flight modes for stable hovering, takeoff and landing, trajectory following, and reactive flight modes for interception and tracking of vehicles. Development and integration of numerous AI technolo- gies. These include both path and task-based planning systems, a chronicle recognition system for identifying complex vehicular patterns on the ground, and other high- level services for reasoning about action and change. Development and integration of numerous knowledge representation technologies. These include an on-board geographical information system; a dynamic object repos- itory to anchor, manage and reason about dynamic objects 1 WITAS (pronounced vee-tas) is an acronym for the Wallen- berg Information Technology and Autonomous Systems Labora- tory at Link ¨ oping University, Sweden. 2 This work and the project is funded by a grant from the Wal- lenberg Foundation.
Transcript
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 1
A Distrib uted Ar chitecture for Intelligent UnmannedAerial VehicleExperimentation
P. Doherty, P. Haslum, T. Merz, E. Skarman,G. Conte,S.Duranti, F. Heintz, P-O. Pettersson,T. Persson,B. Wingman
The emerging areaof intelligent unmannedaerial ve-hicle (UAV) researchhasshown rapid developmentinrecentyearsandoffersa greatnumberof researchchal-lengesfor artificial intelligence. In this article, a pro-totypedistributedarchitecturefor intelligentunmannedaerialvehicleexperimentationis presentedwhich sup-ports the developmentof intelligent capabilitiesandtheir integration in a robust, scalable,plug-and-playhardware/softwarearchitecture.The architectureitselfusesreal-timeCORBA to supportits infrastructureandit is basedonareactiveconcentricsoftwarecontrolphi-losophy. A numberof capabilitiesof the architecturearepresentedincludingamulti-modeflight controlsys-temfor aYamahaRMAX VTOL platform,anon-boardpathplanningserviceanda dynamicallyreconfigurableimageprocessingsystem.A researchprototypesystemhasbeenbuilt, is operationalandis beingusedin actualmissions.In thearticle,we emphasizethecharacteris-tics of thearchitecturewhich supporttheintegrationofnumerousAI technologies.
Intr oductionThe emerging areaof intelligent unmannedaerial vehicle(UAV) researchhasshownrapiddevelopmentin recentyearsandoffers a greatnumberof researchchallengesfor artifi-cial intelligence. Much previous researchhasfocusedonlow-level controlcapabilitywith thegoalof developingcon-trollerswhich supporttheautonomousflight of UAVs fromoneway-point to another. The mostcommontype of mis-sion scenarioinvolvesplacing sensorpayloadsin positionfor datacollection taskswherethe datais eventuallypro-cessedoff-line or in real-timeby groundpersonnel. Useof UAVs andmissiontaskssuchas thesehave becomein-creasinglymoreimportantin recentconflict situationsandare predictedto play increasinglymore importantroles inany futureconflicts.
Intelligent UAVs will play an equally importantrole incivil applications.For both military andcivil applications,thereis a desireto develop more sophisticatedUAV plat-forms wherethe emphasisis placedon developmentof in-telligent capabilities. Focus in researchhas moved from
Copyright c�
2003, American Associationfor Artificial Intelli-gence(www.aaai.org). All rightsreserved.
low-level control towardsa combinationof low-level anddecision-level control integratedin sophisticatedsoftwarearchitectures.Theseshouldalsointegratewell with largernet-centricbasedC� I systems. Suchplatformsare a pre-requisitefor supportingthe capabilitiesrequiredfor the in-creasinglymorecomplex missiontaskson the horizonandan ideal testbedfor the developmentandintegrationof AItechnologies.
TheWITAS1 UnmannedAerial VehicleProject2 (Dohertyetal. 2000)is anambitious,long-termbasicresearchprojectwhosemainobjectivesarethedevelopmentof anintegratedhardware/softwareVTOL (Vertical Take-Off andLanding)platform for fully-autonomousmissionsand its future de-ploymentin applicationssuchastraffic monitoringandsur-veillance,emergency servicesassistance,photogrammetryand surveying. Basic and appliedresearchin the projectcoversa wide rangeof topicswhich includeboththedevel-opmentof traditionalAI technologies,core functionalitiesandtheir pragmaticintegrationwith othertechnologiesin aprototypeexperimentalUAV system.
Thefollowing is anon-exclusivelist of someof theactiv-ities in theproject:� Developmentof agenericdistributeddeliberative/reactive
softwarearchitecturefor (aerial)roboticsystems.� Developmentof a helicoptercontrol systemwith flight
� Developmentand integrationof numerousAI technolo-gies. Theseinclude both path and task-basedplanningsystems,a chronicle recognitionsystemfor identifyingcomplex vehicularpatternsontheground,andotherhigh-level servicesfor reasoningaboutactionandchange.
� Developmentand integration of numerousknowledgerepresentationtechnologies.Theseincludean on-boardgeographicalinformationsystem;adynamicobjectrepos-itory to anchor, manageandreasonaboutdynamicobjects
1WITAS (pronouncedvee-tas) is an acronym for the Wallen-berg Information TechnologyandAutonomousSystemsLabora-tory atLinkopingUniversity, Sweden.
2This work andtheprojectis fundedby a grantfrom theWal-lenberg Foundation.
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 2
suchasvehiclesdiscoveredduring missionexecution;aqualitativesignalprocessingframework for dynamiccon-structionof trajectoriesand historiesof world behaviorwith associatedreasoningmechanisms;anddevelopmentof knowledgestructuresfor signal-to-symbolconversionandapproximatereasoningin soft real-time.
� Developmentof multi-modal interfaces (including di-alogue) for ground operator/UAV communicationwithbothspeechgenerationandrecognitioncapability.
� Developmentof simulationenvironmentswith hardware-in-the-loopfor both testingandvisualizationof control,reactionanddeliberationfunctionalities.
Developmentof sucha complex systemwith its manycomponentsis a multi-disciplinaryeffort and is dependenton competencesfrom artificial intelligence,computersci-ence,controltheory, andsignalprocessing.Oneof themostimportantaspectsin the project is the actualdevelopmentand engineeringof an integratedhardware/softwarearchi-tecturewith a supportingdevelopmentenvironment. Thesystemmustbeableto supportbothbasicresearchendeav-ors andthe developmentandtestingof appliedresearchinboth the on-boardsystemand in simulationenvironmentson acontinualbasis.
Much progresshasbeenmadein thesedirectionsandthemajority of technologieslisted above have beentestedin aprototypeVTOL systemwith segmentsof actualmissionsflown in an interestingurban environmentpopulatedwithbuilding androadstructures.
Figure1: Aerial photooverRevinge,Sweden
Figure1 showsanaerialphotoof ourprimarytestingarealocatedin Revinge,Sweden.An emergency servicestrain-ing schoolis locatedin this areaandconsistsof a collec-tion of buildings, roadsand even makeshift car and trainaccidents.This providesan ideal testareafor experiment-ing with traffic surveillance,photogrammetricandsurvey-ing scenarios,andscenariosinvolving emergency services.Wehavealsoconstructedanaccurate3D modelfor thisareawhich hasproveninvaluablein simulationexperimentsandpartsof which havebeenusedin theon-boardGIS.
The efforts describedin this papershouldbe consideredaswork in progressandadditionaliterationswith both thesoftwarearchitectureitself andthemany technologiesusedin thearchitecturewill be requiredin orderto reacha level
of integrationandrobustnessnecessaryfor safe,productionversionsof any futuresystem.Thisbeingsaid,wedobelievethatany futureproductionversionof a UAV systemwill re-quiremany of thecorefunctionalitieswhich arepartof thecurrentprototypearchitecturein additionto the integrativeandscalablenatureof thearchitecture.
In theremainderof thepaper, we will focusprimarily ona descriptionof the engineeredon-boardsystemitself andthe integrationof several of the AI-basedservicesusedinthesoftware/hardwarearchitecture.Therearea greatmanytopics that will not be considereddue to pagelimitations,particularlyin theareaof knowledgerepresentationcapabil-ities, andin technologiessuchasdialoguemanagementforsupportof groundoperationpersonnel.
The VTOL and HardwarePlatformTheVTOL platformusedin theprojectis aslightly modifiedYamahaRMAX helicoptermanufacturedby YamahaMotorCompany. It is commerciallyavailablein Japanasa radio-controlledplatform.
The RMAX is approximately2.7 � 0.7 meterswith amain rotor 3 metersin length. It hasan empty weight of61 kg and a take-off weight of 95 kg. In addition to theRMAX sensorsincludedwith the platform, the followinghasbeenadded: a Honeywell HMR3000digital compass,a static pressuresensor, a SystronDonner digital quartzINS/GPS,temperaturesensorsfor the systemandenviron-ment,a video anda datalink to the groundstation,andadifferential GPS.The platform is evolving and additionalsensorsmaybeaddedin thefuture.
A PC104(PentiumP5,266MHz) boardusingRTLinux3
is currentlybeingusedastheheartof thecontrolsystem.Itexecutesall real-timecontroltasksinvolving helicoptercon-trol andcollectsall availablesensordata. AnotherPC104(PentiumPIII, 700MHz),with TimeSysLinux, is beingusedfor controlof thecamerasystemandall on-lineimagepro-cessingcomputation. The helicopter is equippedwith aSony
FCB-EX470LP color compositevideo camerawith apan/tilt interfacemountedon a stabilizedgimbal which at-tenuatesvibrations.A third PC104(PentiumPIII, 700MHz)board,using (TimeSys)Linux,is usedfor the executionofdeliberative andreactive componentsin the softwarearchi-tecturesuchaspathplaning,chroniclerecognition,on-boardGIS, and task procedureexecution. Figure 2 shows theexperimentalUAV platform. Figure 3 shows a high-levelschematicof the hardwareplatform that we have built andintegratedwith theRMAX platform.
Thegroundstationincludesa wirelessmodemconnectedto a groundstationcomputer(GS) which receivesteleme-try from theUAV andsendscommandsanddifferentialGPSdatato theUAV with adownlink rateof 8kBytes/sandanup-link rateof 2 kBytes/s.Thereis alsoa receiver for compos-ite video broadcastedfrom the onboardcamerawhich canbe viewed on the GS computer. A Leica referencestation
3A migration to TimeSysLinux is in progressas part of theCORBA-izationprocess.
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 3
for a differentialGPSis serially connectedto the GS com-puter. An additionallaptopis connectedto theGScomputervia anEthernetconnectionandis usedto visualizetelemetrydata. It alsoexecutesa multi-modalinterfaceanddialoguemanager. Experimentationwith dialoguemanagementis inprogresswherebothvoicecommandscanbesentto theUAVandvoiceresponsesaregeneratedby theUAV(Lemonetal.2001).
Theability to switchmodescontingentlyis afundamentalfunctionalityin thearchitectureandcanbeprogrammedintothe taskproceduresassociatedwith the reactive componentin the architecture.Exampleswill be provided later in thepaper. Wearecurrentlydevelopingandtestingthefollowingcontrolmodes:� take-off andlanding� hovering(H-Mode)
All modeshave beendeveloped,implemented,testedandusedin actualflights.
Proportionalnavigation is an interestingreactive controlmodewhichisbasedonideasfrommissileinterceptiontech-nology. It would beusedmostoftenin a scenariowherein-formationis providedaboutamoving objectandtheUAV isrequiredto continuallygenerate,updateandfollow a trajec-tory which would createan interceptsituationbetweenthemoving object and the UAV. The moving object might bea vehicledriving on a roador anotherUAV. Thebasicideafor the control law is depictedin Figure4. Given that
¬is
the angleof line of sight from a helicopterto its goal rel-ative to the North, if the changein bearing
¬is kept at ® ,
theUAV will eventuallycollidewith its target.Thebasisforthecontrol law for proportionalnavigation is that
¬canbe
controlledwith thehelicopter’saccelerationvector.
¯±°a²-³�´
µ¶¸· ¹¸º¼»¸½¾À¿ Á¸Â�Ã
Ä±Å¸Æ Ç È!É¸Ê¸Ë Å¸Ì
Í!θÏ�Ð¸Ñ¸Ò ÓÕÔ¸Ö ×¸Ø¸Ô¸ÙÚ±Û¸Ü Ü Ý Þ!Ý Û¸ß
Figure4: ProportionalNavigation
Initial testswith thePN-Modeandtheothermodeshaveshown promisingresults.In recentflight tests,a numberoffully autonomousflights have beendemonstratedsuccess-fully. Theseincludestablehovering,predefined3D trajec-tory following including 360 degreebanked turns,vehicleinterceptionandroadfollowing. Accelerationandbrakingwith noovershoothavebeentestedatspeedsof 55km/handcoordinatedbankedturnshavebeentestedwith aturnrateof20 degrees/s.We have completedautonomousmissionsupto a durationof 20 minutesusinga combinationof controlmodesin additionto interactionwith agroundoperator.
Software SystemAlthoughthemaingoalof theprojectis to dobasicresearchrelatedto thedevelopmentof corefunctionalitiesrequiredinintelligent aerialrobotic systems,the successfulevaluationof the projectrequiresthe developmentof a prototypesys-tem which canin fact usethesefunctionalitiesin real mis-sionsin an integratedmanner. Consequently, much efforthasbeenplacedonthepragmaticdevelopmentof asoftwarearchitecturecapableof meetingthis requirement.
We have chosenReal-Time CORBA (ObjectComputing,Inc. 2000)4 asa basisfor thedesignandimplementationofa loosely coupleddistributedsoftwarearchitecturefor our
4We are currently using TAO/ACE. The Ace Orb is an opensourceimplementationof CORBA 2.5.
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 4
aerialroboticsystem.Webelievethisis agoodchoicewhichenablesus to managethe complexity of a deliberative/-reactive (D/R) softwarearchitecturewith asmuchfunction-ality aswerequirefor ourapplications.It alsoensurescleanandflexible interfacingto thedeliberative andcontrolcom-ponentsin additionto thehardwareplatformvia the useofIDL (InterfaceDefinitionLanguage).
In short,CORBA (CommonObjectResourceBroker Ar-chitecture)is middlewarethatestablishesclient/server rela-tionshipsbetweenobjectsor components.A componentcanbe a complex pieceof softwaresuchasa pathplanner, orsomethinglesscomplex suchasa taskprocedurewhich isusedto interfaceto helicopteror cameracontrol.Objectsorcomponentscanmake requeststo, andreceive repliesfrom,otherobjectsor componentslocatedlocally in thesamepro-cess,in differentprocesses,or ondifferentprocessorsonthesameor separatemachines.In our case,we have threeon-boardPC104sin additionto groundstationcomputers.
Theuseof CORBA is commonlyassociatedwith provid-ing a basisfor looselycoupledscalabledistributedcompo-nentarchitectureson theInternet.Therearealsovery goodreasonsfor usingCORBA asa basisfor thetypeof deliber-ative/reactivearchitecturewe areproposinghere.� Characteristics of the D/R architecture – The archi-
tecture is necessarilya distributed, concurrent,multi-componentarchitecture.Communicationandcontrol isevent-drivenandasynchronous.Servicesrequiredynamicconfigurationof partial staterepresentationsof both theexternal environmentand of the system’s own internal
environment. This is provided in part throughCORBAevent channel servicesand event filters. Quality-of-serviceguaranteesarerequiredfor time-constrainedcom-municationbetweenvariousservicesin the architecture.Real-timeeventchannelsandschedulingmechanismsin-tegratedwith the real-timeOS’s in the architecturecon-tributeto meetingsuchguarantees.
� Scalability and abstraction – A long term goal of theprojectis thedevelopmentof ascalableandmodularsoft-ware infrastructurefor cooperative robotics where onemoves from a single helicopter/groundoperatorsystemto one with helicopterfleets, ground robots, and mul-tiple communicationcenters. In this case,abstractionmechanismsandscalabilityof the infrastructureis vital.CORBA (andsimilar middlewareideas)provide theuni-form glue to supportthe developmentof large-scalenet-centricsystems.In addition,it supportstheuseof power-ful designpatternsassociatedwith component-basedar-chitectures,for abstraction.
� Developmentand prototyping – Wehavementionedthenecessityfor many differenttypesof servicesassociatedwith deliberation,reaction,controlandsensing.Differentprogramminglanguagesaremoreor lesssuitablefor theimplementationof differenttypesof services.In ourcase,we have implementedpartsof the architectureusingC,C���
, Java, Erlang,LISP, Prolog,Perl, andXML tools.We have alsoworked with a numberof legacy systems,wherewe have tried to integrate them rather than startfrom scratch. For example,we are experimentingwithlegacy applicationsand software suchas Mnesia,Post-greSQL,Open-AgentArchitecture,IXTET, Nuance,Fes-tival, ArcInfo/Arcview. CORBA andthemiddlewareideaprovide the necessarytools to implementcomponentsindifferentlanguagesandto integratelegacy softwarein aseamlessmanner.
Although therearea greatmany goodreasonsfor usingCORBA, experiencehasshown thatthereis alsoadownside.For example, the learningcurve for CORBA is steepandreal-timeCORBA is a relatively new ideaundercontinualdevelopment.It is still unclearjusthow deepinto thecontrolsystemonewouldliketo extendtheuseof CORBA andreal-time event channels.This is, in fact, an interestingavenueof empiricalresearchfor sucharchitectures.
A greatdealof effort in theartificial intelligencecommu-nity hasrecentlybeendirectedtowardsdeliberative/reactivecontrol architecturesfor intelligent robotic systems. Twogood sourcesof referenceare (Arkin 1998) and (Ko-rtenkamp,Bonassao,& Murphy 1998). Many of thearchi-tecturesproposedcanbeviewedin a loosesenseaslayeredarchitectureswith a control, a reactive and a deliberativelayer (see(Gat 1998)). The softwarearchitecturewe havedevelopeddoeshavedeliberative,reactiveandcontrolcom-ponents,but ratherthanviewing it from theperspectiveof alayeredarchitecture,it is bestviewedasa reactiveconcen-tric architecturewhich usesservicesprovided by both thedeliberative andcontrol componentsin a highly distributedandconcurrentmanner. At any time duringtheexecutionofa mission,a numberof interactingconcurrentprocessesat
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 5
variouslevels of abstraction,rangingfrom high-level ser-vices such as path plannersto low-level servicessuchasexecutionof control laws, arebeingexecutedwith variouslatencies.
The Modular TaskAr chitecture
The conceptuallayersusedto describean architecturearelessimportantthan the actualcontrol flow and interactionbetweenprocessesinvoked at the variouslayers. What ismost importantis the ability to usedeliberative servicesina reactive or contingentmanner(which we call hybrid de-liberation) andto usetraditionalcontrol servicesin a reac-tiveor contingentmanner(whichis commonlycalledhybridcontrol). Figure7 depictssomeof theseideas.Dueto there-active concentricnatureof thearchitecture,taskproceduresandtheir executionareacentralfeatureof thearchitecture.
The modular task architecture(MTA) is a reactive sys-tem designin the procedure-basedparadigmdevelopedforlooselycoupledheterogeneoussystemssuchastheWITASaerial robotic system. A task is a behavior intendedtoachievea goalin a limited setof circumstances.A taskpro-cedure (TP) is the computationalmechanismthat achievesthis behavior. A TP is essentiallyevent-driven;it mayopenits own (CORBA) event channels,and call its own ser-vices(bothCORBA andapplication-orientedservicessuchaspathplanners);it may fail to performits taskdueto thelimited setof circumstancesin which it is specifiedto oper-ate; it may be called,spawnedor terminatedby an outsideagent,or by otherTPs;andit maybeexecutedconcurrently.
Formally, aTP is any CORBA objectthatimplementstheWitas::Idl::Tex::Taskinterfaceandadheresto thebehavioralrestrictionsof theMTA specification.5
To increasetheeaseandflexibility of specifyingandim-plementingindividual TPs,a TaskSpecificationLanguage(TSL) has beendeveloped. TSL is an XML-based codemarkuplanguageintendedto simplify the implementationof TPs.Theideais thatfor any TPthereis anapplicationde-pendentor operative partwhich canbe implementedin anyhost languagesupportedby TSL. Currently TSL supportsbothJava andC++. Theapplicationindependentpartof theTP is setupautomaticallyin a translationprocessfrom TSLto thehostlanguage.Figure6 showsaschematicof aspecialtype of TP specifiedin the TSL with someof the essentialmarkuptags,includingthosefor afinite statemachine(fsm)block.
Taskprocedurescanbeusedfor many differentpurposes.Figure7 depictsseveraltypesof usageashybriddeliberationTPs,hybridcontrolTPsor mixedTPs.
A good way to view and representa hybrid controlleris as an augmentedautomaton. We have provided struc-tural and macro tags for this purposewhich can be usedin a TP. Figure 8 shows a TSL schematicfor the finitestatemachinespecificationwhich would be includedin the����������� �!� ��"#���!���
tag block in Figure 6. Someadditionaltags not listed allow for specificationof jumps to other
5The TPEM in Figure5 representsthe setof TP CORBA ob-jectsused.Thereis nocentralizedexecutionmechanism.
An importantpropertyof D/R architecturesis theirabilityto integrateandexecuteprocessesat the deliberative, reac-tive andcontrol levels concurrentlyandto switch betweenthemseamlessly. Reactive componentsmustbe ableto in-terfacein a cleanmannerto helicopterandcameracontrolandadaptactivity to contingenciesin the environmentin atimely manner. Likewise, they mustbe ableto interfacetodeliberative servicesin a cleanmanner. Consequently, useof TPsfor hybridcontrolandhybriddeliberationis of funda-mentalimportance.An exampleis providedin thefollowingsection.
UsingTaskProceduresfor Flight ControlIn this section,theinterfacebetweenTPsfor hybrid controlandflight controlmodes,in additionto somelimited hybriddeliberation,will be described. The main ingredientsareTPsthat supportthe representationof finite statemachinesandtheuseof adeclarativeflight commandlanguage(FCL).Basedon the currentstate,a TP will continually generateflight commandswhich aresentto the flight controllerand
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 6
[]\%^ _ `\�a bc _ d%ef]gihAj�k l m�noQp�q�r s p�t
u�vxw�y z {}|~�� z w�~�y �x� z ������x��� � �Y������� � ���
continually receive eventsfrom the event channelsit sub-scribesto. Theseeventsmakeup its partialstaterepresenta-tion.
Supposea mission in Revinge has been generatedatthe deliberative level using a task-basedplannersuch asTALPlanner(Doherty& Kvarnstrom2001)to find andtrackavehiclewith aparticularsignature.A call from adelibera-tivecomponentwouldstartexecutionof aTPparameterizedwith thevehiclessignatureandaregionto look for thevehi-cle. ThisparentTPmightcall anotherTP, NavToPt, to nav-igateto a waypointin the region of interest(ROI). In orderfor NavToPt to do its job, it would requirebothdeliberativeandcontrol services.It would first call a deliberative pathplanningservice(describedlater) to provide a (segmented)trajectoryto get the UAV to theROI. If successful,the tra-jectory segmentswould be passedto the trajectoryfollow-ing flight modein the control system(via the FlyPath TPandflight commands)andflight to theregion would beini-tiated.During trajectoryfollowing, NavToPt would receiveeventsfrom variouseventchannelsmonitoringprogress.Anautomatonfor NavToPt is depictedin Figure9.
OncetheUAV arrivedat (or approached)thefinal trajec-tory endpoint,theparenttaskprocedurewould make a call
Lock UAVto position pathplanner
Wait for CallFlyPath
UAV stable path ready finished
UAV off courseno path found fail (other reson)
FAIL
EXIT
Figure9: TheNavToPt automaton
to enterhovering modeusinga flight command.The par-ent TP would thencall the imageprocessingservices(de-scribedlater) and attemptidentificationof vehiclesin thearea. Supposea vehicle matchingthe initial signatureisidentified. TheparentTP would thencall a new TP proce-dure FlyTo which would placethecontrolof theUAV intoproportionalnavigationmode.In factaniterationon FlyTowith differentparameterswould ensue. An automatonforFlyTo is shown in Figure 10. Upon execution,the FlyToTP generatesa streamof flight commandspassedto thePNflight modecontrollerandreceivesa streamof eventsfromtheflight controlsystemandimageprocessingsystem.Bothautomatadescribedarein fact implementedusingTPswithstructuresimilar to thatin thetwo TSL schemata.
FlyToaligned
Init
Align
Climb
not aligned
aligned
not aligned
not at altitude
at altitude
closingBrake
at position
not aligned
Holdexit
closing
Figure10: TheFlyTo automaton
The flight commandlanguage(FCL) is usedto bridgetheabstractiongapbetweentaskproceduresin the reactivecomponentandflight andcameramodesin thecontrolcom-ponent. Figure11 shows a numberof representative flightcommandsusedfor the proportionalnavigation flight con-trol mode.
In this casethePN modeis controlledby providing com-mandsto XY, Z andYaw channels,in additionto anadmin-istration channel.Theadministrationchannelis usedto setvariousparameters.It is alsousedto inform the PN modewhich objectto fly to (FlyObject), this maybe a waypointor a moving objecton thegroundpreviously identified,andtheobjectto look at with thecamera(LookObject). Addi-tional flight commandsareprovidedfor otherflight controlmodesandcameracontrol. In thecaseof trajectoryfollow-ing, representationsof parameterizedcurvesare passedasargumentsto flight commands,theseargumentsaregener-atedvia a taskprocedurecall to thepathplanningservice.
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 7
The use of TSL with XML markup tags and a host lan-guagetranslatorprovides a modular, extensiblemeansofconstructingandexperimentingwith TPsthathidesthesyn-tactic overheadassociatedwith initialization of CORBAevent channels,servicesand error handling code. EachTP is automaticallygeneratedasa standardCORBA serverand can be called, spawned, or terminated using theWitas::Idl::Tex::Taskinterface.We believe that thegreatestflexibility is providedif theactualsemanticcontentof a TPcanbespecifiedin a familiar languagesuchasC++ or Java.Thestructuringof codein this manneralsoprovidesacleanway to analyzethebehavior of a TP formally. Theuseof aflight commandlanguageprovidesa smoothtransitionfromdiscreteto continuouscontrolbehavior betweenthereactiveandcontrol components.New controlmodescanbeaddedin a modularmannerand interfacedby addingnew flightcommands.Theflight commandstreamsthemselvescanbegeneratedand analyzedin many different ways. The useof eventchannelsandfilters by TPsalsoprovidesa flexiblemeansof dynamicallyconstructingpartial staterepresenta-tionswhich drive thebehaviorsof TPs.
SomeDeliberativeServicesDue to pagelimitations, we will only briefly considertwohigher-level servicesdeveloped,implementedand usedinourapplications.Emphasiswill beplacedon theintegrationof theseserviceswith the softwarearchitecturepreviouslydescribed.
The Path Planner Service
Thepathplannerusedfor thehelicopteris anadaptationofprobabilisticroadmap(PR)algorithms(Kavraki etal. 1996)to our applicationdomain. The problemof finding an op-timal pathbetweentwo robotconfigurationsin a configura-tion spacesuchasRevingeis intractable.For staticconfigu-rationspaceswhereoneassumesnodynamicobjectsexceptcarson the groundanda helicopter, PR algorithmshedgethe intractability problemby outputingnon-optimalpathsandworking in two phases,oneoff-line andtheotherduringruntime. The main processingstagesfor our adaptationofthePRalgorithmareshown in figure12.
��������������! �"� �# $&% ' ()"
*&+-,�,/. 0�1 2/34&5�6 6 7 8�7 5:9-4�;</=/>7 9/?
@BADCFEG�H I/J
K�L�M N/O-PDO�Q�R S/T�O�U)O�V�WXBY:Z Z [ \�[ Y�]-XB^/_/`/a[ ]�b
In theoffline phase,a roadmapgraphis generatedfor theareaof interest(e.g.. Revinge). This processtakes a 3Dpolygonalmodelof the areaandthe helicopterkinematicsasinput. Helicopterconfigurations6 arerandomlygeneratedandchecked for collisionswith the model. An attemptisthenmadeto connectcollision-freeconfigurationsusingalocal pathplanner(seebelow) which takesinto accountthekinematicanddynamicconstraintsof thehelicopter. Eachofthe local pathsgeneratedalsohave to bechecked for colli-sions.Thecollision checker, usedto checkwhethera givencurve intersectsany obstaclein the environment, is basedon the OBBTree-algorithm(Gottschalk,Lin, & Manocha1996),thatusesatreeof orientedboundingboxesasits cen-tral component.
Therearea numberof choicesthat canbe madefor thelocal path plannerat this stagedependenton how muchor little work is chosento be doneduring run-time. Thechoice describedin the diagramis to initially ignore thenon-holonomicconstraintsof the helicopterin the off-linephaseandadd themasrefinementsto the plan in smooth-ing andcurve replacementphasesduring run-time(Sekha-vat,P. Svestka,& Overmars1998).In thiscase,theroadmapgeneratedandusedat run-timeusesstraightlines.Theotheralternative is to generatetheroadmapwith spline-curvesal-readyat this stage. We experimentwith both approaches.Using thesetechniques,roadmapscan be generatedwithhigh-levelsof coverageevenin tight spaces.Usingstraightlines,aninitial roadmapfor Revingewasgeneratedcontain-ing 5000nodeswith 97%coverage.Usingspline-curves,analternative roadmapwasgeneratedusing15000nodeswith70%coverage.
During the missionor run-timephase,the pathplanningserviceis calledwith an initial andgoal helicopterconfig-uration. An attemptis madeto connectthe two config-urationsto the previously generatedroadmapand, if suc-cessful,an A
[searchis usedon the graph to generatea
multi-segmentedtrajectory. In the diagram,the resultingstraight lines are smoothedand if possiblereplacedwithsplinecurvesusingthe local pathplanner. In the curve re-placementphase,the local pathplanneris separatedinto ahorizontalanda verticalcomponent.In theplane,two con-
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 8
figurationsare connectedto eachother by aligning the \ -axis of the coordinatesystemthroughthe two points andfinding the uniquecubic function that goesthroughthesepointswith theappropriatedirections.A changein altitudebetweentwo configurationsis currentlyexecutedin a linearmannerbut we will in the nearfuture move to true spacecurves,givenasparameterizedcubicpolynomials.7
The multi-segmentedtrajectory generatedby the pathplannerserviceis providedpiecewiseto theTF-Modein thecontrolsystemusinganappropriateTP which monitorstheflight progressionasconsideredpreviously. For many of ourmissionsin Revinge,we have usedthis serviceto generatepathplansof reasonablecomplexity in underasecond.
ImageProcessing
TheImageProcessingModule(IPM), like otherservicesinthe architectureis implementedasa CORBA server whichcanbe calledandusedby othercomponents,in particulartaskproceduresusedfor achieving tasksspecifiedatthemis-sionlevel. For example,a typical scenarioin traffic surveil-lancewouldbeto find, identify andtrackavehiclebasedoninformationaboutits signature(color, size,type,etc),whenit waslastseenandwhere.Theservicesprovidedby theIPMincludethe executionof imageprocessingalgorithmsusedin achieving tasks.Sincethegoalsandrequirementsat themissionlevel maychangeovertime,oneimportantaspectoftheIPM is thatit shouldallow for thedynamicmodificationandswitching of IP algorithmsat run-time. Internally, IPalgorithmsarerepresentedusinga DataFlow Graph(DFG)basedmodel. Nodesin the graphsrepresentIP operationsandthearcsrepresentdataon which theoperationsareper-formed.
Conceptually, the ImageProcessingModule (IPM) con-sistsof two parts,] IPAPI - An ImageProcessingApplicationProgramInter-
face– This is thedeclarative interfacethatothercompo-nentsin the D/R architecturecanuseto create,manipu-late, configureandexecutevariousimageprocessingal-gorithms.
] IPAPI Run-Time - This is the run-time componentthatmanagesthe configurationand executionof imagepro-cessingalgorithms, dynamically allocatesmemory forbuffers neededduring execution,interfacesto the imagecontrollerandothercomponentsin the architecture,andmanagesanexisting library of predefinedimageprocess-ing algorithms.8
It is importantto emphasizethat tight integration is re-quired betweenthe low-level imageprocessingcapabilityand the deliberative or reasoningcapabilitiesof the UAVwhichusequalitativemodels.Integrationandrun-timemod-ification capabilitiesareachievedin partvia theuseof twoothermoduleswhich the IPM communicatesmostcloselywith. Theseare the Task ProcedureExecution Module
7The latter alternative hasalreadybeentestedsuccessfullyinsimulation.
8For details,see(Nordberg et al. 2002).
GetROI Store2Dcontroller
Image
TrackMatch
DynamicObjectRepository
Find car object DFG
Track DFG
Figure13: DFGs for identifying and trackingcar objects.The ROI is chosendynamicallyby sendingrequeststo theImageController.
(TPEM) and the Dynamic Object Repository(DOR), de-picted in Figure 5. The central role of the TPEM in thearchitecturehasalreadybeendiscussed.
The DOR is essentiallya soft real-timedatabasewhichkeepsrecordsof information about various objects,bothstatic and dynamic, that the systemneedsto know aboutwhen achieving various tasks. This may include sightedgroundvehiclesduring a traffic surveillancescenario,butmayalsoincludeinformationaboutthehelicopteritself andthecamera.Onetaskof theIPM is to provideup-to-datein-formationabouttheobjectsit sensesandto storethis in theDOR. TPs determinewhich objectsare interestingby ac-cessinginformationin theDOR andusingdeliberative ser-vices.TPsmaythenredirectfocusof interestby communi-catingwith theIPM. TheinteractionbetweentheTPEMandtheIPM maybeviewedasa form of higher-level active vi-sionwherethe context which determinesimageprocessingpolicy is representedimplicitly in theDOR andinterpretedby theTPEM via theuseof otherdeliberativeservices.
Recall two of the high-level tasksmentionedpreviously,“find vehicle” and“track vehicle”. An exampleof how theprocessingis setupfor thesetasksis illustratedin Figure13.A typical resultof the “find vehicle” operationis thata setof potentialvehiclesare identified. The terminal nodeofthe correspondingdataflow graphthen exports the list ofobjects,actingasaCORBA client, to theDORwhich is im-plementedasa CORBA server (right partof Figure13). Atthis point, the reasoninglayersof the systemexaminethe“vision objects”,e.g.,checkif their velocitiesareconsistentwith the direction of the road, or if their positionsare ona road,or areconsistentwith predictionsfrom earlierposi-tions. If this is thecase,a hypothesiscanbemadethat thisis an“on-roadobject” or a “car object” afteradditionalrea-soningandlinkagesarecreatedandmaintainedbetweentheobjectstructures.A chroniclerecognitionservicecanthenbe called to identify variouspatternsof interest,i.e., sim-ple sequencesof eventssuchas changinglanes,stopping,turning, vehicleovertaking,etc. The linkagestructuressetup in the DOR arean importantpart of the signal to sym-bol conversionsrequiredfor groundingqualitative symbolstructuresrepresentingworld objectssuchasvehiclesto thesensorydataassociatedwith them.
RelatedWorkThereis, without a doubt,muchactivity with UAVs in themilitary sector, primarily with fixed-winghigh altitudeve-
Draft to be Submitted to 5th IFAC Symposium on Intelligent Autonomous Vehicles (IAV2004). 9
hicles.Muchof thefocusis ondesignof physicalplatforms,low-level control,andsensing(uavr 2001). Lessfocushasbeenplacedon thetypeof systemdescribedhere.This alsoappliesto the majority of commercialattemptsat market-ing UAVs, althoughhere,therehasbeenmoreactivity withVTOL platforms.Academicresearchwith UAVs is increas-ing atarapidpace.Hereagain,themajorityof projectshavefocusedonlow-level control,althoughthereis moreactivitywith softwarearchitecturesandintegrationof somedeliber-ativecapabilities.Thework closestto oursis thatbeingpur-suedat Georgia Tech(GT1). Ratherthanlist publications,we refer thereaderto thefollowing (non-exhaustive) list ofwebsitesfor information aboutinterestinguniversity UAVactivities: Georgia Tech(GT1), M.I.T(MIT1 ), Carnegie-Mellon(CM1 ), U. of C., Berkeley(USCB1), StanfordUni-versity(SU1).
ReferencesArkin, R. C. 1998.Behavior-BasedRobotics. MIT Press.
Gottschalk,S.; Lin, M.; andManocha,D. 1996. Obbtree:A hi-erarchicalstructurefor rapid interferencedetection. In Proc. ofthe23rd Int’l. Conf. on Computergraphicsandinteractivetech-niques, 171–180.
Kavraki, L. E.; Svestka, P.; Latombe, J.-C.; and Overmars,M. H. 1996. Probabilisticroadmapsfor pathplanningin high-dimensionalconfigurationspaces.IEEETransactiononRoboticsandAutomation12(4):566–580.
