Autonomous navigation for low-altitude UAVs in urban areas

Thomas Castelli12 Aidean Sharghi3 Don Harper3 Alain Tremeau2 and Mubarak Shah3

Abstractmdash In recent years consumer Unmanned AerialVehicles have become very popular everyone can buyand fly a drone without previous experience which raisesconcern in regards to regulations and public safety Inthis paper we present a novel approach towards en-abling safe operation of such vehicles in urban areasOur method uses geodetically accurate dataset images withGeographical Information System (GIS) data of road networksand buildings provided by Google Maps to compute a weightedA shortest path from start to end locations of a missionWeights represent the potential risk of injuries for individuals inall categories of land-use ie flying over buildings is consideredsafer than above roads We enable safe UAV operation inregards to 1- land-use by computing a static global path depen-dent on environmental structures and 2- avoiding flying overmoving objects such as cars and pedestrians by dynamicallyoptimizing the path locally during the flight As all inputsources are first geo-registered pixels and GPS coordinatesare equivalent it therefore allows us to generate an automatedand user-friendly mission with GPS waypoints readable byconsumer dronesrsquo autopilots We simulated 54 missions andshow significant improvement in maximizing UAVrsquos standoffdistance to moving objects with a quantified safety parameterover 40 times better than the naive straight line navigation

I INTRODUCTIONUAVs are becoming increasingly present in our everyday

lives their extensive use recently jumped from military tohobby and professional applications The consumer marketis growing and now it offers a wide range of micro and miniUAVs at affordable costs But this popularity induces somedangerous behavior most people do not realize that a simplemistake can cause severe injuries to themselves or othersIn the United-States the FAA has taken measures to informhobbyists and encourage them to follow a code of conductto prevent accidents The only form available is the advisorycircular lsquoAC 91-57 rsquofrom June 9th 1981 it advises pilotsto keep their UAVs within their line of sight below 400feet above ground level further than 5 miles from anairport (or warn them) and to avoid flying above peopleEven for the vast majority of UAV users that are responsibleand careful in their use there is no automated means to flysafely in regards to the UAVrsquos environment Our work aimsto provide such functionality to micro and mini UAVs thatare operated in urban areas

In this paper we propose a novel method for autonomousnavigation for low-altitude UAVs in urban areas For a

1 Survey Copter Airbus Defense and Space Pierrelatte FRANCE2 Hubert Curien Laboratory Saint-Etienne FRANCE

alaintremeauuniv-st-etiennefrthomascastelliuniv-st-etiennefr

3 Center for Research in Computer Vision UCF Orlando USAaideansharghiknightsucfeduharperucfedu shahcrcvucfedu

given mission our method computes safe waypoints whichdynamically adapt the flight plan to the UAVs surroundingsby avoiding objects such as cars and pedestrians We takeadvantage of satellite and georegistered data to adapt theUAVs mission layout by computing a weighted shortest pathinstead of flying in a straight line Weights in our costfunction for computing the flight path are defined using land-use summarized in three classes most dangerous areas areroads and paths where people are prone to the danger theUAV represents safest are buildings and water and the restis in between (Fig 1) For increased safety our methodalso adapts dynamically to moving objects while in flightby adding new local weight to the global weight map

In our general scenario we assume a UAV with videocamera flying over a given geographical region for whichgeodetically accurate reference image GIS data of build-ings and road networks are available Captured videos aregeoregistered with the reference image in order to transferpixel coordinates to GPS coordinates Moving objects egvehicles and pedestrians are detected and tracked from frameto frame Given the tracks GIS and reference image theoptimal UAV path is dynamically computed For simplicityin this paper we employ ground truth tracks availablefrom WPAFB and PVLabs datasets providing geo-registeredimages and ground truth for moving objects Finally wesimulate a real flight by complying with the lsquoAC 91-57rsquoformand using parameters of compatible hardware

Fig 1 Visualization of the weight map overlaid on the correspondingsatellite image for WPAFB dataset Colors represent costs in the weightmap red transparent and green respectively represent dangerous neutraland safer areas

arX

iv1

602

0814

1v1

[cs

RO

] 2

5 Fe

b 20

16

II RELATED WORK

Many different topics are studied to enhance the usabilityand to develop new functions to make drones more capableand autonomous There are several subfields which are re-lated to this work including video geo-registration detectionand tracking of moving objects in videos detection of roadsbuildings water bodies from satellite imagery and flight pathplanning

The most popular trend in UAV video analysis has beenmoving object detection and tracking from aerial imagesmany approaches have been proposed with or without usingGIS data and geo-registration steps Kimura et al [5] useepipolar constraint and flow vector bound to detect movingobjects Teutsch et al [6] employ explicit segmentation ofimages Xiao et al [7] restrain the search on the roadnetwork and Lin et al [8] use a motion model in geo-coordinates Moving object detection and tracking are mainlyused to follow targets for surveillance as Quigley et al [4]and Rafi et al [9] describe with their flight path adaptationsolutions or for consumer applications at very low-altitudeas in [18] and [19]

Another area that has been getting a lot of attention isautonomous navigation Different subproblems have beenstudied path planning in dynamic environment [10] [11]GIS-assisted and vision-based localization using either roaddetection [12] buildings layout [14] or DEM (Digital Eleva-tion Map) [13] Various methods have been proposed forUAV navigation using optical flow with [15] or withoutDEM [16] or using inertial sensors [17]Obstacle avoidance is also a big concern for automating UAVoperation but research has mostly been focused on groundrobots [20] [22] even if there has been adaptations for UAVsas Israelsen et alrsquos intuitive solution for operators [21]

The approaches for autonomously navigating UAVs havebeen studied but previous work focus on target followingor keeping the UAVrsquos integrity However in this paper wepropose an autonomous UAV navigation method in order toincrease public safety in regards to drones operation and alsoto prevent UAVs finding themselves in difficult situations

III OUR METHOD

Our contribution towards safe integration of small UAVsinto the airspace has two main steps The first step describedin section A and B takes into account the physical sur-roundings of the UAV by computing as part of the missionpreparation a global path between the user-given start andend locations This path is represented as a succession ofwaypoints exactly as users are accustomed to in missionplanner softwares Before takeoff the user is able to validatethe automated path and he can modify the waypoints ifneeded The second step described in section C runs inonline fashion during the flight and takes into account theenvironment of the UAV by dynamically adapting its behav-ior in regards to moving objects that need to be avoided

A Extracting the geo-referenced weight mapA convenient approach to gain awareness of the UAVrsquos

surroundings is to use satellite imagery and the meta-dataprovided by Google Maps DigitalGlobe Planet Labs orothers To jointly use geo-registered data and aerial imageryobtained from UAV there has to be a common representationand space Public tools providing satellite images are verypopular and well integrated in third party UAV softwaresuch as Mission Planner For simplicity and compliancethe solution is then to register video images onto a geo-registered satellite image of the area of interest UAV andworld coordinates systems are related with (1) as describedin [2]

~Xcamera = GyGzRyRxRzT ~Xworld (1)

with ~Xworld the world coordinates system T is thetranslation matrix derived from the vehiclersquos latitudelongitude and altitude and Gy Gz Ry Rx and Rz arerotation matrices regarding respectively camera elevationangle camera scan angle vehicle pitch angle vehicle rollangle and vehicle heading angle

We chose to use Google Maps API for itrsquos convenienceand for the quality of the data provided1 This free APIallows anyone to request satellite images and roadmapsdisplaying buildings and roads These three links 1 2 and 3give example commands to request satellite road map andbuilding images

We assume that flying above buildings represents less riskthan doing so above other environmental elements such asroads or crowded streets The resulting weight map for theWright-Patterson Air Force Base (WPAFB) area shown inFig 1 displays three categories

bull Red is to be avoided for roads and pathsbull Green is to be preferred for buildings and waterbull Transparent is in between for other land-use

To extract the map only two GPS locations are neededas input from the operator top left and bottom right GPScoordinates A grid of image GPS locations is then computedbased on Google Mapsrsquo camera parameters and resolutionlevel in other words the ground sampling distance (GSD)to ensure sufficient overlap between images for stitchingWe have defined the GPS grid in a way that successiveimages have pure translations between them We thus canstitch them together using straight forward normalized crosscorrelation which is a robust and fast method given that wemanipulate large images and avoid scale change and rotationThis process allows us to minimize the error while creatingthe geo-registered map As a result we obtain 3 images forany given area (Fig 2) Given the image center GPS locationLat1 and Lon1 the corresponding GPS location (Lat2 andLon2) of all other pixels location at (∆x∆y) from thecenter can be determined as follows

1The proposed method is not dependent on the source any satellite imageand data provider can be used

II RELATED WORK

Many different topics are studied to enhance the usabilityand to develop new functions to make drones more capableand autonomous There are several subfields which are re-lated to this work including video geo-registration detectionand tracking of moving objects in videos detection of roadsbuildings water bodies from satellite imagery and flight pathplanning

The most popular trend in UAV video analysis has beenmoving object detection and tracking from aerial imagesmany approaches have been proposed with or without usingGIS data and geo-registration steps Kimura et al [5] useepipolar constraint and flow vector bound to detect movingobjects Teutsch et al [6] employ explicit segmentation ofimages Xiao et al [7] restrain the search on the roadnetwork and Lin et al [8] use a motion model in geo-coordinates Moving object detection and tracking are mainlyused to follow targets for surveillance as Quigley et al [4]and Rafi et al [9] describe with their flight path adaptationsolutions or for consumer applications at very low-altitudeas in [18] and [19]

Another area that has been getting a lot of attention isautonomous navigation Different subproblems have beenstudied path planning in dynamic environment [10] [11]GIS-assisted and vision-based localization using either roaddetection [12] buildings layout [14] or DEM (Digital Eleva-tion Map) [13] Various methods have been proposed forUAV navigation using optical flow with [15] or withoutDEM [16] or using inertial sensors [17]Obstacle avoidance is also a big concern for automating UAVoperation but research has mostly been focused on groundrobots [20] [22] even if there has been adaptations for UAVsas Israelsen et alrsquos intuitive solution for operators [21]

The approaches for autonomously navigating UAVs havebeen studied but previous work focus on target followingor keeping the UAVrsquos integrity However in this paper wepropose an autonomous UAV navigation method in order toincrease public safety in regards to drones operation and alsoto prevent UAVs finding themselves in difficult situations

III OUR METHOD

Our contribution towards safe integration of small UAVsinto the airspace has two main steps The first step describedin section A and B takes into account the physical sur-roundings of the UAV by computing as part of the missionpreparation a global path between the user-given start andend locations This path is represented as a succession ofwaypoints exactly as users are accustomed to in missionplanner softwares Before takeoff the user is able to validatethe automated path and he can modify the waypoints ifneeded The second step described in section C runs inonline fashion during the flight and takes into account theenvironment of the UAV by dynamically adapting its behav-ior in regards to moving objects that need to be avoided

A Extracting the geo-referenced weight mapA convenient approach to gain awareness of the UAVrsquos

surroundings is to use satellite imagery and the meta-dataprovided by Google Maps DigitalGlobe Planet Labs orothers To jointly use geo-registered data and aerial imageryobtained from UAV there has to be a common representationand space Public tools providing satellite images are verypopular and well integrated in third party UAV softwaresuch as Mission Planner For simplicity and compliancethe solution is then to register video images onto a geo-registered satellite image of the area of interest UAV andworld coordinates systems are related with (1) as describedin [2]

~Xcamera = GyGzRyRxRzT ~Xworld (1)

with ~Xworld the world coordinates system T is thetranslation matrix derived from the vehiclersquos latitudelongitude and altitude and Gy Gz Ry Rx and Rz arerotation matrices regarding respectively camera elevationangle camera scan angle vehicle pitch angle vehicle rollangle and vehicle heading angle

We chose to use Google Maps API for itrsquos convenienceand for the quality of the data provided1 This free APIallows anyone to request satellite images and roadmapsdisplaying buildings and roads These three links 1 2 and 3give example commands to request satellite road map andbuilding images

We assume that flying above buildings represents less riskthan doing so above other environmental elements such asroads or crowded streets The resulting weight map for theWright-Patterson Air Force Base (WPAFB) area shown inFig 1 displays three categories

bull Red is to be avoided for roads and pathsbull Green is to be preferred for buildings and waterbull Transparent is in between for other land-use

To extract the map only two GPS locations are neededas input from the operator top left and bottom right GPScoordinates A grid of image GPS locations is then computedbased on Google Mapsrsquo camera parameters and resolutionlevel in other words the ground sampling distance (GSD)to ensure sufficient overlap between images for stitchingWe have defined the GPS grid in a way that successiveimages have pure translations between them We thus canstitch them together using straight forward normalized crosscorrelation which is a robust and fast method given that wemanipulate large images and avoid scale change and rotationThis process allows us to minimize the error while creatingthe geo-registered map As a result we obtain 3 images forany given area (Fig 2) Given the image center GPS locationLat1 and Lon1 the corresponding GPS location (Lat2 andLon2) of all other pixels location at (∆x∆y) from thecenter can be determined as follows

1The proposed method is not dependent on the source any satellite imageand data provider can be used

Lat2 = Lat1 minus sinminus1(r middot∆yEr

) (2)

Lon2 = Lon1 + sinminus1(r middot∆x

Er middot cos(Lat1 middot π180 )

) (3)

with Lat1 Lat2 Lon1 Lon2 representing latitudes andlongitudes of the start and end points Er the mean radius ofthe earth r the pixel ratio in meters per pixel depending onthe ground altitude and on the requested image scale andresolution ∆x and ∆y are the difference in pixels on themap between the two points

Fig 2 This figure shows from bottom to top three types of data used inthis work satellite image buildings and water map and roads map

B Global path planning

The vast majority of UAS (Unmanned Aerial Systems) canbe used with a ground control station (GCS) for exampleAPMCopter (previously known as ArduCopter) has its ownmission planner with all the necessary tools The conven-tional ways of controlling UAVs are either with a manualradio controller or by using a GCS that defines successiveGPS waypoints (specifying the GPS location altitude andvelocity) to which the UAV will fly autonomously Despitetheir efficiency and convenience there is a crucial flaw withwaypoints they are defined by the user and do not take intoaccount the surroundings of the UAV This is precisely whatwe want to tackle with our global path computation By usingthe three types of data shown in Fig 2 we define the optimalpath (example in Fig 3) between two points and thus add asafety parameter to mission planning

We find the safest route between two GPS coordinates byconverting them into image pixels and computing a weightedshortest path algorithm using A algorithm The segmentsrsquolengths between two adjacent pixels are the Euclidian dis-tance multiplied by the weight defined by the map classPixels that are in red have a weight of 100 green is at 5and the rest is at 20 Those values have been determinedempirically This process ensures that the red areas areavoided but also makes sure the UAV wouldnrsquot take a long

Fig 3 Different path planning solutions between two GPS coordinatesWhite dotted path classic straight line path used in typical systemsusing waypoints Blue path our method which determines the shortestpath by minimizing the cost function such that the resulting path avoidsflying over red areas that are dangerous and prefer green areas that are safer

Fig 4 Path computed by algorithm shown in yellow converted towaypoints and visualized in Mission Planner Green ticks are waypointslocations white dotted circles are areas where the UAV will consider havingreached the waypoint and yellow dotted line represents the simple straightpath

detour to reach its destination thus keeping the loss of flighttime to a minimum

As the map is geo-registered the outputted path can easilybe converted into GPS coordinates using (2) and (3) and putin KML an TXT files to be readable by mapping softwareand ground control stations (Fig 4)

This global weight map considers the static environmentthat the UAV will encounter such as roads and buildingsIn order to ensure a higher level of safety in all stages ofthe flight we also adapt the path locally during the flight inregards to moving objects as explained in the next section

C Local path planning

For increased safety the path needs to be adapted dynam-ically during the flight to avoid moving objects detected in

the field of view of the UAVrsquos embedded camera In order toensure a sufficient distance margin between each object andthe UAV the weight map used for shortest path is modifiedaccording to the objectsrsquo location trajectory and velocity Wecompute the new weights of the map by applying at chosenlocations a multivariate normal probability density function

The variance Σx and Σy for each distribution are depen-dent on the objectrsquos characteristics The x term is propor-tional to the width of the object in pixels and y term (4) isproportional to the objectrsquos velocity

Σy = VObj middot S (4)

where VObj is the current velocity of the object and S thesafety margin to avoid collision in seconds

The resulting distribution is normalized rotated to alignwith the objectrsquos trajectory and centered on the chosenlocation (5) The weight map is then multiplied with thedistribution instead of being swapped in order to keep theglobal environment based information

The locations where the distribution is applied to aredefined given two criteria One is whether the object collideswith the UAVrsquos path and the other is how this collisionhappens The object and UAV will take respectively tObj andtUAV seconds to the collision point if |tObjminustUAV | lt ∆ (∆is set to 5 s in our experiments) the distribution is appliedon the collision point and also on a projected location toavoid re-planing a path that will create a similar situationThe projected location (6 7) is estimated as follows thetime for the UAV to travel to the current objectrsquos locationis computed the projected location is where the object willbe at that time given constant velocity and trajectory for theobject For objects that will not collide or that do meet therequirement of δt gt ∆ the distribution is applied at the nextand projected locations

Ω = Ω middot [R T ] Φ (5)

where Ω is the weight map [R T ] the affine transforma-tion applied to the multivariate normal probability densityfunction Φ to allocate costs to Ω Φ is centered at thewanted location Lxy and oriented given the rotation matrixR dependent on the objectrsquos trajectory and

T =

LObjx +D middot sin(α)LObjy +D middot cos(α)

1

(6)

is the translation component of the affine transformationLObjx and LObjx are the image coordinates of the object andD middot sin(α) and D middot cos(α) are respectively the distances inX and Y to the desired location

D =

VObj middot dUAV minusObj

VUAVfor projected location

VObj middot 1fps for next location

(7)

with D the distance used in 6 VObj and VUAV the currentvelocities of the object and UAV dUAVminusObj the distance

between object and UAV and fps the frame-rate orcomputation time

This method will ensure that the resulting path will leavesufficient ground distance between objects and the UAV andif multiple objects are close together it will create a barrierand encourage the UAV to find a safer path thus preventingit to fly above any moving objects (Fig 5)

Fig 5 Left column shows the images and right column shows thecorresponding weight maps Objects trajectories are shown in white (inimages) and black (in weight maps) The global paths shown in red andblack dashed lines are adapted with weight adjustments to avoid flyingover objects Please note that in the bottom row the path crosses the roadperpendicularly on the right part of the images (more visible on the bottomleft image) This is due to the fact that we want to minimize crossing high-cost road pixels

IV EXPERIMENTSA Methodology

In order to simulate a real world scenario as accuratelyas possible our method uses dataset images and typicalUAVsrsquo specifications and camera parameters We made sureto comply with the latest regulations and advice regardingUAV operation and used the following flight and hardwareparameters

bull Altitude above ground level 50mbull Velocity lt 15msbull Camera Horizontal Field Of View (HFOV) 9740o 2bull Horizontal ground sampling resolution 884cmpixel

The principles used to build the simulation scheme are thefollowing

bull UAV videos are registered in the geo-referenced spacewe can thus work in pixels coordinates and convertback to GPS anytime

2HFOV for a configuration using a PointGrey Blackfly 13MP 13rdquocamera of 1288x964 resolution and a Kowa LM3PB lens

bull The datasetsrsquo ground truth gives the moving objectsrsquolocation for every frame (motion vectors in Fig 6)

bull The UAV will follow the global path (blue in Fig 6)bull For every frame the UAVrsquos displacement in the image

is dependent on itrsquos velocity and direction (8)bull The considered objects are only the ones visible in the

field of view of the embedded camera (exterior reddotted line around the UAV in Fig 6)

bull For convenience we call lsquocollision rsquothe situation wherethe UAV will fly over an object

bull A collision is detected if the direction of an objectrsquosmotion vector intersects the path in front of the UAV

bull A danger area is computed and is visible as the smallestred dotted rectangle in Fig 6 for every frame dependingon UAVrsquos velocity so that the UAV will reach theboundary in 5 seconds at current and constant velocity

∆p =VUAVFd middot rm

(8)

where ∆p is the number of pixels to advance along the pathVUAV is the velocity of the UAV Fd is the framerate of thedataset and rm represents the ground sampling distance ofthe geo-registered map

Fig 6 Small red dotted square on the top right represents the danger areathat the UAV would reach in 5 seconds at the current velocity and the largersquare shows the FOV Objects locations and their motion vectors given byground truth are shown by colored arrows At bottom right a notification isdisplayed in red if objects are present in the FOV

B Datasets

We use two datasets to run our safe navigation pipelineWright-Patterson Air Force Base (WPAFB) [1] and PVLabsThey are wide-area motion imagery (WAMI) and provideground truth for moving objects on ortho-rectified imagescaptured by UAVs Both of those datasets have been capturedat high altitude with embedded sensors and a matrix ofmultiple cameras We use the provided regions of interestoutputted by a geo-registration step described in [1] Foreach dataset we run the different steps of the pipeline Wefirst create the weight map using the process described in

section III-A Videos are then precisely geo-registered ontothe map via homography transformation The global path(Fig 7) is generated before the simulated flight and adapteddynamically on the way

V RESULTSFor both WPAFB and PVLabs datasets we defined 9

different pairs of start and end GPS coordinates (Fig 7)based on the environment and busyness of the roads tocreate challenging situations that will require global pathadaptation And each path is executed at three different UAVvelocities 5 8 and 11 ms The total traveled distance byusing the global path compared to the classic straight linepath for each dataset executed for all paths at three abovevelocities is 20 higher or 513 min longer for WAPAFBand 6 or 32 s for PVLabs making our safety increasedpath an affordable measure in term of autonomy

Fig 7 Visualization of the nine paths that have been tested (each at 58 and 11 ms) for both datasets Top WPAFB Bottom PVLabs Imagesacquired using the Google Maps API

To quantify the performance of the proposed method weintroduce a metric assimilated to safety We consider theUAV to object proximity the closer the UAV is to an objectthe more danger it represents for it we therefore compute atotal cost for each dataset as in (9)

Cg =

Objectssumi=1

α middot eminusDiou (9)

with Cg the global cost for the considered datasetDou the ground distance between the UAV and each object

the field of view of the UAVrsquos embedded camera In order toensure a sufficient distance margin between each object andthe UAV the weight map used for shortest path is modifiedaccording to the objectsrsquo location trajectory and velocity Wecompute the new weights of the map by applying at chosenlocations a multivariate normal probability density function

The variance Σx and Σy for each distribution are depen-dent on the objectrsquos characteristics The x term is propor-tional to the width of the object in pixels and y term (4) isproportional to the objectrsquos velocity

Σy = VObj middot S (4)

where VObj is the current velocity of the object and S thesafety margin to avoid collision in seconds

The resulting distribution is normalized rotated to alignwith the objectrsquos trajectory and centered on the chosenlocation (5) The weight map is then multiplied with thedistribution instead of being swapped in order to keep theglobal environment based information

The locations where the distribution is applied to aredefined given two criteria One is whether the object collideswith the UAVrsquos path and the other is how this collisionhappens The object and UAV will take respectively tObj andtUAV seconds to the collision point if |tObjminustUAV | lt ∆ (∆is set to 5 s in our experiments) the distribution is appliedon the collision point and also on a projected location toavoid re-planing a path that will create a similar situationThe projected location (6 7) is estimated as follows thetime for the UAV to travel to the current objectrsquos locationis computed the projected location is where the object willbe at that time given constant velocity and trajectory for theobject For objects that will not collide or that do meet therequirement of δt gt ∆ the distribution is applied at the nextand projected locations

Ω = Ω middot [R T ] Φ (5)

where Ω is the weight map [R T ] the affine transforma-tion applied to the multivariate normal probability densityfunction Φ to allocate costs to Ω Φ is centered at thewanted location Lxy and oriented given the rotation matrixR dependent on the objectrsquos trajectory and

T =

LObjx +D middot sin(α)LObjy +D middot cos(α)

1

(6)

is the translation component of the affine transformationLObjx and LObjx are the image coordinates of the object andD middot sin(α) and D middot cos(α) are respectively the distances inX and Y to the desired location

D =

VObj middot dUAV minusObj

VUAVfor projected location

VObj middot 1fps for next location

(7)

with D the distance used in 6 VObj and VUAV the currentvelocities of the object and UAV dUAVminusObj the distance

between object and UAV and fps the frame-rate orcomputation time

This method will ensure that the resulting path will leavesufficient ground distance between objects and the UAV andif multiple objects are close together it will create a barrierand encourage the UAV to find a safer path thus preventingit to fly above any moving objects (Fig 5)

Fig 5 Left column shows the images and right column shows thecorresponding weight maps Objects trajectories are shown in white (inimages) and black (in weight maps) The global paths shown in red andblack dashed lines are adapted with weight adjustments to avoid flyingover objects Please note that in the bottom row the path crosses the roadperpendicularly on the right part of the images (more visible on the bottomleft image) This is due to the fact that we want to minimize crossing high-cost road pixels

IV EXPERIMENTSA Methodology

In order to simulate a real world scenario as accuratelyas possible our method uses dataset images and typicalUAVsrsquo specifications and camera parameters We made sureto comply with the latest regulations and advice regardingUAV operation and used the following flight and hardwareparameters

bull Altitude above ground level 50mbull Velocity lt 15msbull Camera Horizontal Field Of View (HFOV) 9740o 2bull Horizontal ground sampling resolution 884cmpixel

The principles used to build the simulation scheme are thefollowing

bull UAV videos are registered in the geo-referenced spacewe can thus work in pixels coordinates and convertback to GPS anytime

2HFOV for a configuration using a PointGrey Blackfly 13MP 13rdquocamera of 1288x964 resolution and a Kowa LM3PB lens

bull The datasetsrsquo ground truth gives the moving objectsrsquolocation for every frame (motion vectors in Fig 6)

bull The UAV will follow the global path (blue in Fig 6)bull For every frame the UAVrsquos displacement in the image

is dependent on itrsquos velocity and direction (8)bull The considered objects are only the ones visible in the

field of view of the embedded camera (exterior reddotted line around the UAV in Fig 6)

bull For convenience we call lsquocollision rsquothe situation wherethe UAV will fly over an object

bull A collision is detected if the direction of an objectrsquosmotion vector intersects the path in front of the UAV

bull A danger area is computed and is visible as the smallestred dotted rectangle in Fig 6 for every frame dependingon UAVrsquos velocity so that the UAV will reach theboundary in 5 seconds at current and constant velocity

∆p =VUAVFd middot rm

(8)

where ∆p is the number of pixels to advance along the pathVUAV is the velocity of the UAV Fd is the framerate of thedataset and rm represents the ground sampling distance ofthe geo-registered map

Fig 6 Small red dotted square on the top right represents the danger areathat the UAV would reach in 5 seconds at the current velocity and the largersquare shows the FOV Objects locations and their motion vectors given byground truth are shown by colored arrows At bottom right a notification isdisplayed in red if objects are present in the FOV

B Datasets

We use two datasets to run our safe navigation pipelineWright-Patterson Air Force Base (WPAFB) [1] and PVLabsThey are wide-area motion imagery (WAMI) and provideground truth for moving objects on ortho-rectified imagescaptured by UAVs Both of those datasets have been capturedat high altitude with embedded sensors and a matrix ofmultiple cameras We use the provided regions of interestoutputted by a geo-registration step described in [1] Foreach dataset we run the different steps of the pipeline Wefirst create the weight map using the process described in

section III-A Videos are then precisely geo-registered ontothe map via homography transformation The global path(Fig 7) is generated before the simulated flight and adapteddynamically on the way

V RESULTSFor both WPAFB and PVLabs datasets we defined 9

different pairs of start and end GPS coordinates (Fig 7)based on the environment and busyness of the roads tocreate challenging situations that will require global pathadaptation And each path is executed at three different UAVvelocities 5 8 and 11 ms The total traveled distance byusing the global path compared to the classic straight linepath for each dataset executed for all paths at three abovevelocities is 20 higher or 513 min longer for WAPAFBand 6 or 32 s for PVLabs making our safety increasedpath an affordable measure in term of autonomy

Fig 7 Visualization of the nine paths that have been tested (each at 58 and 11 ms) for both datasets Top WPAFB Bottom PVLabs Imagesacquired using the Google Maps API

To quantify the performance of the proposed method weintroduce a metric assimilated to safety We consider theUAV to object proximity the closer the UAV is to an objectthe more danger it represents for it we therefore compute atotal cost for each dataset as in (9)

Cg =

Objectssumi=1

α middot eminusDiou (9)

with Cg the global cost for the considered datasetDou the ground distance between the UAV and each object

bull The datasetsrsquo ground truth gives the moving objectsrsquolocation for every frame (motion vectors in Fig 6)

bull The UAV will follow the global path (blue in Fig 6)bull For every frame the UAVrsquos displacement in the image

is dependent on itrsquos velocity and direction (8)bull The considered objects are only the ones visible in the

field of view of the embedded camera (exterior reddotted line around the UAV in Fig 6)

bull For convenience we call lsquocollision rsquothe situation wherethe UAV will fly over an object

bull A collision is detected if the direction of an objectrsquosmotion vector intersects the path in front of the UAV

bull A danger area is computed and is visible as the smallestred dotted rectangle in Fig 6 for every frame dependingon UAVrsquos velocity so that the UAV will reach theboundary in 5 seconds at current and constant velocity

∆p =VUAVFd middot rm

(8)

where ∆p is the number of pixels to advance along the pathVUAV is the velocity of the UAV Fd is the framerate of thedataset and rm represents the ground sampling distance ofthe geo-registered map

Fig 6 Small red dotted square on the top right represents the danger areathat the UAV would reach in 5 seconds at the current velocity and the largersquare shows the FOV Objects locations and their motion vectors given byground truth are shown by colored arrows At bottom right a notification isdisplayed in red if objects are present in the FOV

B Datasets

We use two datasets to run our safe navigation pipelineWright-Patterson Air Force Base (WPAFB) [1] and PVLabsThey are wide-area motion imagery (WAMI) and provideground truth for moving objects on ortho-rectified imagescaptured by UAVs Both of those datasets have been capturedat high altitude with embedded sensors and a matrix ofmultiple cameras We use the provided regions of interestoutputted by a geo-registration step described in [1] Foreach dataset we run the different steps of the pipeline Wefirst create the weight map using the process described in

section III-A Videos are then precisely geo-registered ontothe map via homography transformation The global path(Fig 7) is generated before the simulated flight and adapteddynamically on the way

V RESULTSFor both WPAFB and PVLabs datasets we defined 9

different pairs of start and end GPS coordinates (Fig 7)based on the environment and busyness of the roads tocreate challenging situations that will require global pathadaptation And each path is executed at three different UAVvelocities 5 8 and 11 ms The total traveled distance byusing the global path compared to the classic straight linepath for each dataset executed for all paths at three abovevelocities is 20 higher or 513 min longer for WAPAFBand 6 or 32 s for PVLabs making our safety increasedpath an affordable measure in term of autonomy

Fig 7 Visualization of the nine paths that have been tested (each at 58 and 11 ms) for both datasets Top WPAFB Bottom PVLabs Imagesacquired using the Google Maps API

To quantify the performance of the proposed method weintroduce a metric assimilated to safety We consider theUAV to object proximity the closer the UAV is to an objectthe more danger it represents for it we therefore compute atotal cost for each dataset as in (9)

Cg =

Objectssumi=1

α middot eminusDiou (9)

with Cg the global cost for the considered datasetDou the ground distance between the UAV and each object

detected in the FOV during the experiment and α a constant

Note that for the same start and end locations whendifferent paths are compared the UAV will not encounter thesame situations This is why for clarity we include with theresults in Table I the number of objects seen by the UAVrsquoscamera throughout the simulation for each dataset

TABLE ISAFETY ESTIMATION RESULTS FOR WPAFB AND PVLABS

Straight path Static path Dynamic pathWPAFB of obj 2759 4588 7597

Global WPAFB cost 2439 629 56PVLabs of obj 3600 4022 5959

Global PVLabs cost 1881 3263 98

We can clearly see in Table I that our proposed methodencounters more objects in the FOV but it has the means tokeep the UAV afar from them Objects which are over 20maway are not in danger but having a car or pedestrian closerthan 5m to the UAV represents a very concerning situationin terms of safety for people This is why we have chosento compute the global cost with a negative exponentialweight function that way the shorter the distance the morecost is applied to the global metric The proposed methodencounters over twice the amount of moving objects butsafely keeps away from them (Fig 8) making the resultingsafety parameter much better than global path and most ofall better than classic straight line path

Fig 8 Comparison of the number of detected objects in the FOV asfunction of UAV-to-objects ground distance between 0 and 10m for all ninepaths executed at 5 8 and 11 ms The perfect solution would be 0 objectsfor all distances Left WPAFB Right PVLabs

VI CONCLUSION

In this paper we introduced an environment and safetybased path planning for low altitude UAV operating in urbanareas We compute a global path for any mission given a pairof start and end GPS locations by using a weighted shortestpath The weight map is defined using ground classificationdata summarized in three classes highest cost is for roadsand paths because of the high probability of presence ofpeople for which the UAV represents a safety threat safestare buildings and water and neutral areas are the restAdditionally we included a dynamic path planning that willmodify locally the flight plan while in flight to avoid being

close to moving objects such as vehicles and pedestriansOur proposed method has been tested in simulation usinggeo-registered data and images from two WAMI datasetsWPAFB and PVLabs and it showed significant improvementcompared to the current and manual mission planning solu-tion in terms of a safety metric quantifying threat in functionof UAV-to-object distance

Our safety planning and navigation scheme can be imple-mented on-board a UAV and will consist in the followingsteps 1- before takeoff acquire necessary GIS data for themission area and generate mission waypoints using globalweighted path planning 2- during the flight geo-registerthe embedded camerarsquos images using a sensor model andgimbal readings detect moving objects (as in [3]) or anyother type of objects to avoid and generate new local pathand waypoints to stay clear of the detected objects

ACKNOWLEDGMENT

The research was supported by a DGA-MRIS scholarship

REFERENCES

[1] Cohenour et al rdquoCamera models for the wright patterson air force base2009rdquo IEEE Aerospace and Electronic Systems Magazine 2015

[2] Sheikh et al rdquoGeodetic Alignment of Aerial Video Framesrdquo in VideoRegistration Eds Boston 2003

[3] Castelli et al rdquoMoving object detection for unconstrained low-altitudeaerial videos a pose-independant detector based on Artificial FlowrdquoISPA 2015

[4] Quigley et al rdquoTarget Acquisition Localization and SurveillanceUsing a Fixed-Wing Mini-UAV and Gimbaled Camerardquo ICRA 2005

[5] Kimura et al rdquoAutomatic extraction of moving objects from UAV-bornemonocular images using multi-view geometric constraintsrdquo IMAV 2014

[6] Teutsch et al rdquoEvaluation of object segmentation to improve movingvehicle detection in aerial videosrdquo AVSS 2014

[7] Xiao et al rdquoVehicle detection and tracking in wide field-of-view aerialvideordquo CVPR 2010

[8] Lin et al rdquoEfficient detection and tracking of moving objects in geo-coordinatesrdquo Machine Vision and Applications 2011

[9] Rafi et al rdquoAutonomous target following by unmanned aerial vehiclesrdquoSPIE 6230 2006

[10] van Toll et al rdquoDynamically Pruned A for re-planning in navigationmeshesrdquo IROS 2015

[11] Xu et al rdquoReal-time 3D navigation for autonomous vision-guidedMAVsrdquo IROS 2015

[12] Dumble et al rdquoAirborne Vision-Aided Navigation Using Road Inter-section Featuresrdquo Journal of Intelligent amp Robotic Systems 2015

[13] Pritt et al rdquoGeoregistration of multiple-camera wide area motionimageryrdquo IGARSS 2012

[14] Habbecke et al rdquoAutomatic registration of oblique aerial images withcadastral mapsrdquo Trends and Topics in Computer Vision 2010

[15] Tchernykh rdquoOptical flow navigation for an outdoor UAV using a wideangle mono camera and DEM matchingrdquo IFAC 2006

[16] Hrabar et al rdquoCombined optic-flow and stereo-based navigation ofurban canyons for a UAVrdquo IROS 2005

[17] Achtelik et al rdquoOnboard IMU and monocular vision based controlfor MAVs in unknown in-and outdoor environmentsrdquo ICRA 2011

[18] Pestana et al rdquoComputer vision based general object following forGPS-denied multirotor unmanned vehiclesrdquo ACC 2014

[19] Pestana et al rdquoVision based gps-denied object tracking and followingfor unmanned aerial vehiclesrdquo SSRR 2013

[20] Ess et al rdquoObject detection and tracking for autonomous navigationin dynamic environmentsrdquo International Journal of Robotics 2010

[21] Israelsen et al rdquoAutomatic collision avoidance for manually tele-operated unmanned aerial vehiclesrdquo ICRA 2014

[22] Gonzalez et al rdquoUsing state dominance for path planning in dynamicenvironments with moving obstaclesrdquo ICRA 2012

• I INTRODUCTION
• II RELATED WORK
• III OUR METHOD
• • III-A Extracting the geo-referenced weight map
  • III-B Global path planning
  • III-C Local path planning
  • • IV EXPERIMENTS
    • • IV-A Methodology
      • IV-B Datasets
      • • V RESULTS
        • VI CONCLUSION
        • References