xView: Objects in Context in Overhead Imagery

Darius Lam1 Richard Kuzma2 Kevin McGee3 Samuel Dooley4 Michael Laielli4Matthew Klaric4 Yaroslav Bulatov5 Brendan McCord2

Abstract

We introduce a new large-scale dataset for theadvancement of object detection techniques andoverhead object detection research. This satel-lite imagery dataset enables research progress per-taining to four key computer vision frontiers.Weutilize a novel process for geospatial category de-tection and bounding box annotation with threestages of quality control. Our data is collectedfrom WorldView-3 satellites at 0.3m ground sam-ple distance, providing higher resolution imagerythan most public satellite imagery datasets. Wecompare xView to other object detection datasetsin both natural and overhead imagery domains andthen provide a baseline analysis using the Sin-gle Shot MultiBox Detector. xView is one of thelargest and most diverse publicly available object-detection datasets to date, with over 1 million ob-jects across 60 classes in over 1,400 km2 of im-agery.

1. Introduction

The abundance of overhead image data fromsatellites and the growing diversity and signifi-cance of real-world applications enabled by thatimagery provide impetus for creating more sophis-ticated and robust models and algorithms for ob-

1D. Lam is at Harvard College, in support of DefenseInnovation Unit Experimental (DIUx)

2R. Kuzma and B. McCord are at DIUx3K. McGee is at DigitalGlobe4S. Dooley, M. Laielli, and M. Klaric are at the National

Geospatial-Intelligence Agency (NGA)5Y. Bulatov is in support of DIUx

Figure 1: Four of the many views of xView. Im-agery comes from different geographical locationswith different levels of human use. Imagery in thisfigure is from DigitalGlobe.

ject detection. We hope xView will become a cen-tral resource for a broad range of research in com-puter vision and overhead object detection.

The vast majority of satellite information in thepublic domain is unlabeled. Azayev notes a lackof labeled imagery for developing deep learningmethods [2]. Hamid et al. also note that the cu-ration of high quality labeled data is central todeveloping remote sensor applications [11]. Ishiiet al., Chen et al., and Albert et al. all developmethods for detection or segmentation of build-

1

arX

iv:1

802.

0785

6v1

[cs

.CV

] 2

2 Fe

b 20

18

ings, all using different (and sometimes customcollected and labeled) datasets [13, 4, 1]. The uti-lization of different datasets makes it difficult tocompare results between different authors. We de-veloped xView as a general purpose object detec-tion dataset of satellite imagery so as to be famil-iar to the computer vision community and remotesensing community alike.

Several object detection datasets exist in thenatural imagery space, but there are few foroverhead satellite imagery. The public overheaddatasets in existence typically suffer from low classcount, poor geographic diversity, few training in-stances, or too narrow class scope. xView reme-dies these gaps through a significant labeling effortinvolving the collection of imagery from a varietyof locations and the use of an ontology of parent-and child-level classes.

We created xView with four computer visionfrontiers in mind:

Improve Minimum Resolution and Multi-Scale Recognition: Computer vision algorithmsoften struggle with low-resolution objects [9, 5].For example, the YOLO architecture limits thenumber of predictable bounding boxes within aspatial region, making it difficult to detect smalland clustered objects [24]. Detecting objects onmultiple scales is an ongoing topic of research[19, 3]. Objects in xView vary in size from 3 meters(10 pixels at 0.3 meters ground-sample distance[GSD]) to greater than 3,000 meters (10,000 pixelsat 0.3 meters GSD). The varying ground sampledistance of different satellites means that xViewhas significantly higher resolution than many pub-lic satellite imagery datasets.

Improve Learning Efficiency: In the realworld, objects are often not evenly distributedwithin images. There may be many thousandsmore cars in any given city than there are hos-pitals. Imbalanced classification and localizationon uneven datasets is important for real-world ap-plications. xView captures this property by in-cluding object classes with few instances as wellas classes with many instances (see Figure 4).

Push the Limit of Discoverable ObjectClasses: xView includes 60 classes. For refer-ence, COCO includes 91 classes and SpaceNet in-

cludes 2 classes. There is significant class diver-sity in xView, which contains both land-use andpedestrian classes with easily discretized objectssuch as cars and buildings as well object classesinvolving groupings of multiple object types suchas construction sites and vehicle lots.

Improve Detection of Fine GrainedClasses: Fine grained object detection is neces-sary for practical applications. Detecting a ’sail-boat’ gives different information than detecting an’oil tanker’, despite them both being ’maritimevessels’. Fine grained object detection is a difficulttask and an ongoing area of research [30, 27, 14].Over 80% of classes in xView are fine grained, be-longing to 7 different parent classes. For example,xView contains 8 distinct truck child classes, in-cluding ’pickup truck’, ’utility truck’, and ’cargotruck’.

To create xView we designed a substantiveannotation and quality control process. Withchipped satellite images delivered in RGB and 8-band multispectral format, annotators used QGIS(Q-Geographic Information System), an opensource tool, to load up and mark image chips.Using an in-house plugin, annotators are able tocreate axis-aligned bounding boxes for individualobjects. Our dataset includes images prepared inways that are typical for satellite images includ-ing orthorectification, pan-sharpening, and atmo-spheric correction.

In order to minimize biased image sampling,we define scene types that are relevant to manyoverhead applications and strive for a uniform dis-tribution of images across those scene types aswell as the ways in which those scenes may ap-pear. Variability in scene type can come from aplace’s function, while the visual appearance of ascene may vary according to a multitude of fac-tors. xView pulls from a wide range of geographiclocations (see Figure 11). Each location has itsown distinct features, including physical differ-ences (desert, forest, coastal, plains) and construc-tional differences (layout of houses, cities, roads).The variety of collection geometries possible withsatellite imagery produces images with multipleperspectives on objects within a given class.

The xView dataset contains 60 object categories

2

with 1 million labeled objects covering over 1,400km2 of the earth’s surface. The large chip sizesallow variability in pre-processing techniques; wediscuss several options in section 4. xView hasa similar number of instances and class counts asCOCO and substantially greater number of classesthan SpaceNet and Cars Overhead with Context[20, 21, 23].

2. Related Work

The complexity of our world, combined with dif-ferences in collection geometry from space-basedimaging platforms, makes object recognition insatellite imagery a difficult task. xView con-tributes a large, multi-class, multi-location datasetin the object detection and satellite imagery space,built with the benchmark capabilities of PASCALVOC, the quality control methodologies of COCO,and the contributions of other overhead datasetsin mind. This combination opens up opportunitiesfor applied research in census mapping (e.g., corre-lating building count and inhabitance), economicreporting (e.g., predicting income level throughvehicle density), disaster response (e.g., identify-ing damaged regions), and more.

The task of object detection is properly iden-tifying an object within an image and localizingit, either through bounding boxes or segmenta-tion. One such dataset, PASCAL VOC, has beenmaintained since 2005 and has grown to 20 objectclasses in 11,530 images containing 27,450 bound-ing boxes and 7,000 segmentations [8]. In thepast decade, the PASCAL VOC dataset has beenwidely used in the object detection space. A num-ber of object detection papers have used PASCALVOC as a benchmark [24, 20, 25, 7]. PASCALVOC, however, contained mostly “iconic view”scenes that are often non-representative of the realworld, an issue remedied by COCO. COCO con-tains 91 object classes across around 328,000 im-ages. The COCO dataset has on average morecategories per image at smaller sizes than PAS-CAL VOC [20]. The ImageNet detection datasetis a large-scale dataset containing 200 classesand around .5 million labeled instances (ILSVRC2014) [26]. Most recently, Google released Open-Images V2, a large-scale dataset containing 1.6

million images, around 4 million bounding boxes,and 600 object classes on natural imagery [15].

The Cars Overhead with Context (COWC)dataset by the Lawrence Livermore National Lab-oratory is an overhead image dataset with around32,700 labeled cars [23]. COWC uses aerial im-age capture as opposed to satellite image cap-ture, so their images are of high resolution butcapture less area. COWC includes images fromsix locations. The SpaceNet dataset focuses onobject segmentation. SpaceNet has segmentationmasks for around 5 million buildings in 5 loca-tions [21]. Recently, the SpaceNet dataset hasexpanded to include roads. Both SpaceNet andCOWC have images taken at similar times of day.Each of these datasets contains few classes in fewgeographic regions, limiting their usability for gen-eral object recognition in overhead imagery. Re-cently, IARPA released their Functional Map ofthe World (FMoW) dataset with RGB and mul-tispectral imagery for recognizing functional landuse from temporal sequences of satellite images[6]. FMoW contains around 1 million images in63 categories from over 200 countries. The FMoWdataset is designed for temporal reasoning in clas-sification of land-use subregions. FMoW classesdo not include vehicles (e.g., sailboat, fishing ves-sel, and small car) [6]. xView includes vehicles,which makes it more representative of the realworld and also better targets the multi-scale prob-lem.

Figure 2 highlights the differences betweenxView, PASCAL VOC, and COWC car classes.COWC only contains cars, while xView alsocontains other vehicles like trucks and tractors.COWC images are also only captured aerially, socars are more consistent in viewpoint and lighting.PASCAL VOC contains other non-car vehicularclasses, such as bus and train, but all images areall ground-level natural imagery. VOC cars alsooccupy a larger proportion of area than COWCand xView cars. xView cars have varied sensorelevation levels and are from more locations thanCOWC. xView cars provide a better indication ofenvironments that would be experienced in reallife: not all cars in satellite images will be imagedfrom a 90-degree elevation angle and in perfect

3

Figure 2: COWC, PASCAL VOC, and xView cars, respectively [23, 8]. COWC provides object labels assingle points corresponding to the center of each car [23]. To generate bounding boxes, we create 20x20pixel boxes around each point. COWC and xView imagery in this figure are extracted from a singlelocation. xView imagery in this figure is from DigitalGlobe.

daylight.

3. Dataset Details

xView is one of the largest and most diversepublicly available overhead imagery datasets forobject detection. Annotators used QGIS (Q-Geographic Information System), an open-sourcesatellite imagery manipulation tool, along with adeveloped in-house plugin to create axis-alignedbounding boxes for individual objects. Ourdataset includes images prepared in ways that aretypical for satellite images including orthorectifi-cation, pan-sharpening, and atmospheric correc-tion.

3.1. Image Collection

We created xView by first selecting a widegroup of object categories to be considered. Ofthis group, we down-selected to 60 classes, whichwere organized in a parent-child format where par-

ents were more general categorizations and chil-dren represented specific instances of these generalcategories. For example, the ‘engineering vehicle’parent class contained the ‘excavator’ child class.xView contains seven parent classes: ‘fixed wingaircraft’, ‘passenger vehicle’, ‘truck’, ‘railway ve-hicle’, ‘engineering vehicle’, ‘maritime vessel’, and‘building’. Not all classes are contained under aparent superset (e.g., ’helipad’).

In order to minimize biased image sampling, weselect imagery from an appropriate distribution ofareas of interest (AOIs). AOIs in xView includemines, ports, airfields, and coastal, inland, urban,and rural regions. They also exist over multiplecontinents. AOI selection can be broken down intotwo smaller steps: identifying lat-long coordinatesfor investigation and drawing polygons in largerareas surrounding those points. First, lat-long co-ordinates of mines, ports, airfields, and other loca-tions with relevant classes are collected from open

4

source databases. Next, we determine if satelliteimagery covers the lat-long coordinate and sur-rounding area. A manual search is performed toconfirm the existence of objects in the relevantclass and if present, a polygon is drawn, creatingthe AOI to be extracted from the larger satelliteimagery available in the region.

The next step is to collect 1 km2 image chipsbased on the specified AOIs. This is done in twoparts: selecting the image strips and applying a 1km2 grid of the polygons that intersect previouslycollected imagery. Some may have multiple imagestrip options and may be chosen in order to varyweather conditions (snow, clouds, etc.) or to cap-ture ephemeral construction sites. Once an imagestrip is chosen, a 1 km2 grid is applied. All gridcells that are intersected by a specified polygonare chipped out. The 1 km2 grid is a UniversalTransverse Mercator (UTM) zone derived grid toensure 1 km2 of area per grid cell, and to estab-lish a repeatable process. The grid also serves asa production tool, to keep crowd workers focusedon a small area for feature extraction, and to markareas as complete by object type to track progress.

3.2. Image Annotation

The design of a high quality annotation pipelinewas critical for xView. This pipeline includeda crowdsourcing approach that leveraged labelerswith experience across the breadth of classes inthe dataset. We achieved consistency by havingall annotation performed at a single facility, fol-lowing detailed guidelines, with output subject tomultiple quality control checks. Workers exten-sively annotated image chips with bounding boxesusing an open source tool.

The annotators were trained through multiplesessions, both online and in person, in which theysaw examples of the total 60 classes using bothcross view and overhead imagery with both cor-rect and incorrect labels. Annotators were in-structed to draw axis-aligned bounding boxes thattightly contained the object instance. The privatecrowd had direct messaging access to the datasetteam. Questions (e.g., “What differentiates a facil-ity from a building?”) could be relayed and solvedin a timely manner.

Figure 3: The QGIS annotation software. Drawnannotations are shown in red. Imagery in this fig-ure is from DigitalGlobe.

Each labeler is responsible for labeling all in-stances of one general category. If the labeler canconfidently determine an instance to be of a finegrained child category, then that labeler will clas-sify the instance at the child level. However, ifthe labeler is unable to make a confident determi-nation, then they will fall back to the more gen-eral parent category. Annotators are instructedto annotate all objects within the chip, unless theobject is approximately ≤ 20% visible. Boundingbox annotation is completed one 1 km2 chip at atime, to keep work focused on small areas, trackcompletion by object class, and provide discreteareas to the quality control reviewers to provideimmediate feedback to the labeling team. For clus-ters of objects, annotators were instructed to la-bel each object individually. For example, clustersof tightly packed buildings were present in someimages, and each building was to be individuallylabeled to avoid labeling object groups. In prac-tice, this was difficult to achieve, and groups ofobjects could be labeled as singular due to humanerror. In large-scale overhead imagery, this erroris representative of data encountered in real-world

5

Figure 4: Top: xView instance count distribution by class. Bottom: xView pixel area distribution byclass.

situations.

3.3. Quality Control and Gold Standards

Annotation quality control was conducted inthree stages: Worker, supervisory, and expert.Worker quality control involved annotators per-forming the role of quality control reviewer on arotational basis so that they could check the workof others, identify errors, and improve their ownannotation. Reviews focused on category identifi-cation, bounding box size, bounding box orienta-tion, and duplication.

The second stage, supervisory quality control,involved checks for duplicate features, invalid la-bels, invalid geometries (polygons rather thanaxis-aligned bounding boxes), non-exhaustivelylabeled image chips, features that fell outside ofimage chips, and empty tiles. The supervisorystep produced feedback in the form of workertraining sessions in addition to maintaining qual-ity.

The third stage, expert quality control, involvedcreating a gold standard dataset and analyzingworker quality by applying a precision and re-call threshold measurement between batches of

worker-produced annotations and that gold stan-dard dataset. Gold data was created by samplingand labeling six 1 km2 chips from each batch byexpert workers who were co-authors of the pa-per as well as professional imagery analysts. Thebatches represented 10/40/70/100% of the totaldataset. In order to pass expert quality con-trol, the batch was required to have a precisionof 0.75 and recall of 0.95 at 0.5 intersection overunion (IoU) when compared to the gold standard.Batches that failed expert quality control were re-mediated and resubmitted. We utilized an addi-tional crowd-sourcing platform called Tomnod forremediation of difficult chips.

3.4. Dataset Statistics

The xView dataset covers over 1,400 km2 ofthe earth’s surface, with 60 classes and approxi-mately 1 million labeled objects. The most com-mon classes are ‘building’ and ‘small car’ due totheir prevalence in densely populated areas. Fig-ure 4 shows the number of instances per class aswell as the average pixel area per class. Anal-ogous with their real-world prevalence, buildingsand small cars have the highest instance counts.

6

Figure 5: Total Instance Count versus Number ofClasses for major object detection datasets. Blueindicates overhead imagery datasets; red indicatesnatural imagery datasets.

By average pixel area, the largest objects are typ-ically land-use locations (e.g., ’construction sites’,’marinas’, ’facilities’, ’vehicle lots’).

We made three splits to the public release ofxView: train, test, and val. The total percent-ages of objects in each split are 59.2%, 20.0%,and 20.8% for train, test, and val, respectively.Each split has at least 5 instances of each category,which is especially important for categories witha small number of instances. We created splitson a per-image level, which made it difficult toproportionally split by category since xView im-ages are large and object categories are spatiallycorrelated. The average of the per-category per-centages in each split is 60.0%, 21.0%, and 19.4%for train, test, and val, respectively.

Additionally, we compare xView to other objectdetection datasets. Figure 5 shows total numberof classes versus total number of instances in sev-eral object detection datasets. The closest datasetto xView to date is the FMoW land use dataset.SpaceNet has a greater number of instances butonly for buildings and roads. xView incorporatesimages from various countries. Each location hasdistinct visual features that affects the appear-ances of objects within that location (see Figure6). This variety will drive a need for contextualunderstanding and adaptability to multiple view-types of objects.

Figure 6: Top: Extracted cars from four separategeographic locations. Bottom: Extracted planesfour geographic locations. Imagery in this figureis from DigitalGlobe.

4. Algorithmic Analysis

We conducted object detection experiments us-ing the Single Shot Multibox Detector meta-architecture (SSD). SSD extracts prediction fea-tures at multiple layers for better multi-scale de-tection [22]. We evaluated three permutationsof the dataset in order to assess xView difficultyand establish a baseline for future research. Be-cause each 1 km2 image is around 3, 0002 pixels,we first pre-processed the data by chipping it into3002 pixel non-overlapping images. The large in-put images allow for great range of chipping op-tions. For our experiments, bounding boxes par-tially overlapped with a chip were cropped at thechip edge. This infrequently resulted in crop-ping sections of large objects like facilities andstadiums. We then created a multi-resolutiondataset by chipping at different sizes (3002, 4002,

7

5002), a pre-processing step suggested by Zhanget al [29]. We also created a multi-resolution-augmented dataset via added image augmentationonto the multi-resolution dataset (shifting, rota-tion, noise, and blurring) (see Figure 7). We eval-uated the SSD method over the three datasets:vanilla, multi-resolution (multires), and multi-resolution-augmented (aug). We ran experimentson four M60 GPUs for seven days. Each datasetwas created by splitting the entire xView datasetinto 70-30% split for train and test, respectively.All three datasets were evaluated on the vanillatest dataset to maintain consistency.

Training on the multi-resolution dataset pro-duced a better model than both the vanillaand augmented datasets, by a significant mar-gin (0.2590 to 0.1456 and 0.1549 total mean av-erage precision, respectively). The best detectedclasses for the multi-resolution dataset are: ‘pas-senger/cargo plane’, ‘helicopter’, ‘shipping con-tainer lot’, ‘passenger car’, and ‘building’. Thebest detected classes for the augmented datasetare: ‘haul truck’, ‘passenger/cargo plane’, ‘smallaircraft’, ‘building’, and ‘tugboat’. The best de-tected classes for the vanilla dataset are: ‘pas-senger/cargo plane’, ‘passenger car’, ‘building’,‘small aircraft’, and ‘helicopter’. The best detectedclasses bias towards those that have large pixel ar-eas and are set in uniform backgrounds. For ex-ample, most planes are imaged on runways withonly slightly changing backgrounds (see Figure 6).Small cars, even with the second largest instancecount, were more poorly detected than rarer butlarger and more contextually easy classes. Figure8 shows extracts of the ‘small car’ class across var-ious geographical contexts. Several classes werepoorly detected across all experiments regardlessof average pixel area and instance count. The‘pickup truck’ and ‘hut/tent’ classes have over onethousand instances each and yet scored <1% mAPacross the board.

There were intra-experimental differences aswell as intra-class differences. The ‘constructionsite’ class has the largest pixel area at over 373,000pixels2. However, the vanilla dataset experimentperformed worse than the multi-resolution experi-ment (0.0172 to 0.1711 AP, respectively). This sig-

Vanilla Multires AugAircraft Hangar 0.1698 0.5270 0.3247Barge 0.1829 0.3738 0.2210Building 0.4718 0.5534 0.4451Bus 0.2949 0.3773 0.2609Cargo Truck 0.0493 0.0972 0.0445Cargo/container car 0.3659 0.4737 0.1676Cement mixer 0.0863 0.1441 0.1220Construction site 0.0172 0.1711 0.0032Container crane 0.0663 0.2879 0.1648Container ship 0.2269 0.4660 0.3400Crane Truck 0.0946 0.0838 0.0894Damaged/demolished building 0.0269 0.0785 0.0366Dump truck 0.1468 0.2275 0.0858Engineering vehicle 0.0020 0.1234 0.0357Excavator 0.3535 0.4691 0.2064Facility 0.0777 0.3750 0.1201Ferry 0.0532 0.3771 0.2197Fishing vessel 0.1768 0.1839 0.0968Fixed-wing aircraft 0.0888 0.1218 0.1042Flat car 0.0000 0.0000 0.0000Front loader/Bulldozer 0.1644 0.3220 0.1959Ground grader 0.1590 0.1910 0.0289Haul truck 0.3542 0.2109 0.6875Helicopter 0.3788 0.5800 0.2965Helipad 0.2459 0.4500 0.1889Hut/Tent 0.0004 0.0006 0.0000Locomotive 0.0760 0.1929 0.1124Maritime vessel 0.1947 0.4040 0.2884Mobile crane 0.0248 0.1375 0.0945Motorboat 0.0811 0.2488 0.1110Oil Tanker 0.0958 0.3677 0.1193Passenger Vehicle 0.4765 0.5569 0.2980Passenger car 0.0305 0.0471 0.0000Passenger/cargo plane 0.6508 0.6691 0.6104Pickup Truck 0.0011 0.0078 0.0000Pylon 0.0089 0.0011 0.0625Railway vehicle 0.0000 0.0833 0.0000Reach stacker 0.0000 0.2625 0.0000Sailboat 0.2614 0.0450 0.0453Shed 0.0071 0.3027 0.0277Shipping container 0.0283 0.3835 0.0426Shipping container lot 0.1644 0.5676 0.1890Small aircraft 0.4610 0.3771 0.4815Small car 0.3607 0.4083 0.3651Storage Tank 0.3484 0.4462 0.3700Straddle carrier 0.3045 0.4293 0.3262Tank car 0.3733 0.1123 0.2664Tower 0.0042 0.1233 0.0000Tower crane 0.0196 0.0385 0.0303Tractor 0.0000 0.1109 0.0000Trailer 0.0651 0.2151 0.0861Truck 0.1526 0.0469 0.1356Truck Tractor 0.0048 0.2129 0.0000Truck Tractor w/ Box Trailer 0.1355 0.0863 0.1188Truck Tractor w/ Flatbed Trailer 0.0322 0.1261 0.0336Truck Tractor w/ Liquid Tank 0.0180 0.4744 0.0000Tugboat 0.2044 0.0380 0.4119Utility Truck 0.0098 0.2846 0.0091Vehicle Lot 0.1105 0.4115 0.1402Yacht 0.0701 0.0701 0.1868Total mAP 0.1456 0.2590 0.1549

Table 1: Per-class average precisions for three ex-periments (Vanilla, Multi-resolution, and Augmented).Green, yellow, and red filled cells indicate highest, sec-ond highest, and lowest APs per row, respectively.

8

nificant change could be due to construction sitesbeing cropped too drastically for vanilla (3002)chips, while being more adequately detected bytraining on the multiple resolution dataset. The‘bus’ and ‘tractor’ classes each have an area of <500 pixels2, and were scored significantly higher bythe multi-resolution experiment. The overall high-performance of the multi-resolution experiment in-dicates that the multiple object scales plays animportant role in detection precision. We cappedtraining at seven days for all three experimentsto maintain consistency. The poor quality of theaugmentation experiment indicates that the aug-mentations and increased dataset size add signifi-cant regularization. With more training iterationswe would expect the augmentation experiment toimprove substantially. Both the multi-resolutionand augmented datasets showed continuous per-formance improvement while the vanilla datasetperformance plateaued.

Figure 7: Two examples of multi-resolution-augmented chips. Ground truth bounding boxesare shown in red. To prevent severe distortion,chip and bounding box rotations were boundedin both directions. Imagery in this figure is fromDigitalGlobe.

Our experiments illustrate that the xViewdataset is difficult. The SSD meta-architecturewith multi-layer feature map extraction achievesrelatively low mean average precision. While lowinstance count and average class size are not di-rectly correlated with per-class average precision,the top detected classes across all experiments biastowards larger and more contextually discernibleclasses. Our multi-resolution experiment per-formed better than both vanilla and augmenteddatasets. However, additional training time or ad-

ditional techniques such as supervised pre-trainingcould improve performance [16, 10].

5. Conclusion and Future Work

We introduced xView, an overhead object de-tection dataset with over 1 million instances in 60classes. xView was labeled through an extensiveannotation process and three-stage quality con-trol. Our classes include land-use objects suchas buildings as well as vehicles and mini-scenes.xView contains a greater variety of classes and vi-sual contexts than other overhead datasets. Wehope xView will serve as a general-purpose ob-ject detection dataset and as a unifying datasetfor overhead object detection research.

There are a variety of future research direc-tions that xView can support. Few-shot learningis one particular direction that could have broadimpacts on overhead imagery applications, e.g., byempowering disaster relief efforts with computervision tools that are quickly adaptable. The per-formance of recent few-shot learning techniques[17, 28] has typically been evaluated on “K-shot,N-way” datasets which assume even distributionsof instances over categories [18]. A more real-istic benchmark has been proposed by Harihanet al. [12], featuring a large base dataset withmany instances per category coupled with a low-instance-count target set. xView can be couchedin this realistic setting by splitting the categoriesinto high vs. low instance counts. Future exten-sions to xView could seek to bridge the divide be-tween “K-shot, N-way” classification and realisti-cally imbalanced datasets by adding many moreobject classes with low instance counts (e.g., 20 to100 instances per category).

Domain adaptation is another research direc-tion that xView is well-suited to support given thegeographic diversity of xView images. The nexthurricane, earthquake, or other natural disasteris likely to occur in an area where curated train-ing data (i.e., pre-labeled imagery) is not readilyavailable. Better domain adaptation techniquesare needed to quickly apply existing models tonew geographic areas that were not reflected inthe training set.

We hope that the release of xView and related

9

code will support future object detection researchin the overhead imagery domain as well as in re-lation to natural imagery. The RGB release ofxView is a general object detection dataset and assuch can be used as a standalone object detectionbenchmark. We also hope to see xView appliedto the intersection of satellite imagery and com-puter vision, whether for demographic studies orhumanitarian efforts.

6. Appendix Overview

In the appendix we provide additional datasetexamples, comparisons, and examples of imagesthat were re-labeled.

6.1. Appendix I: Quality Control

Figure 9 illustrates two examples of pre- andpost-quality control images. Each example is onlyof a small subset of the overall image. Images thatfailed gold-standard quality control were sent forre-labeling. The two pre-QC examples shown be-low contain poorly labeled objects or objects un-labeled altogether. The post-QC examples wereremediated.

6.2. Appendix II: Dataset Examples

Figure 10 illustrates a fully annotated image.Elongated objects such cargo ships are often un-avoidably enclosed by bounding boxes that con-tain significant background. Even seemingly rect-angular objects such as shipping crates can havethis problem due to the rotation of the objects rel-ative to the satellites and our axis-aligned bound-ing boxes.

Figure 11 shows a comparison of chips betweenxView, COWC, and SpaceNet. Cars are mostvisible in COWC images because of their aerialperspective. SpaceNet has the least visible cars.Buildings are the most prominent visual feature inSpaceNet chips. The low sun-angle in the xViewchip can be identified through the long objectshadows. Table 2 categorizes all 60 classes bytheir parent-child relationship. There are 7 parentclasses and several classes that do not belong toa parent. Not all parent classes have equal num-bers of fine grained classes. Figure 12 displaysthe geographical locations of xView, COWC, and

SpaceNet. A dot was plotted for every region datawas captured from, and dots are not scaled basedon instance count. Figure 13 displays an extrac-tion from each class in xView.

10

Figure 8: Two examples of image chips pre- and post- quality control. Objects that were not labeled orwrongly labeled were remediated.

Fixed-Wing Aircraft Passenger Vehicle Truck Railway Vehicle Maritime Vessel Engineering Vehicle Building NoneSmall Aircraft Small Car Pickup Truck Passenger Car Motoboat Tower Crane Hut/Tent HelipadCargo Plane Bus Utility Truck Cargo Car Sailboat Container Crane Shed Pylon

Cargo Truck Flat Car Tugboat Reach Stacker Aircraft Hangar Shipping ContainerTruck w/Box Tank Car Barge Straddle Carrier Damaged Building Shipping Container LotTruck Tractor Locomotive Fishing Vessel Mobile Crane Facility Storage Tank

Trailer Ferry Dump Truck Vehicle LotTruck w/Flatbed Yacht Haul Truck Construction SiteTruck w/Liquid Container Ship Scraper/Tractor Tower Structure

Oil Tanker Front Loader HelicopterExcavator

Cement MixerGround Grader

Crane Truck

Table 2: Parent and child denominations for all 60 classes. Parent classes are at the headings of eachcolumn. The only exception is the last column ‘None’, which corresponds to classes that have no parent.

11

Figure 9: A fully annotated image from xView. Classes are denoted by different bounding box shadings.All imagery in this figure is from Digital Globe.

12

Figure 10: xView, COWC, and SpaceNet chips, respectively. xView imagery in this figure is fromDigitalGlobe.

Figure 11: The geographical locations of xView, COWC, and SpaceNet. Blue, green, and purple dotsrepresent xView, SpaceNet, and COWC locations respectively. Dots are not scaled based on instancecount.

13

Figure 12: An image example per category. Class labels as well as dimensions for the correspondingimage are present under each extraction. All imagery in this figure is from Digital Globe.

14

References[1] A. Albert, J. Kaur, and M. C. Gonzalez. Using

convolutional networks and satellite imagery toidentify patterns in urban environments at a largescale. In Proceedings of the 23rd ACM SIGKDDInternational Conference on Knowledge Discov-ery and Data Mining, KDD ’17, pages 1357–1366,New York, NY, USA, 2017. ACM.

[2] T. Azayev. Object detection in high resolutionsatellite images. Faculty of Electrical Engineering;Department of Cybernetics, 2016.

[3] Z. Cai, Q. Fan, R. S. Feris, and N. Vasconcelos. Aunified multi-scale deep convolutional neural net-work for fast object detection. In Computer Vision- ECCV 2016 - 14th European Conference, Am-sterdam, The Netherlands, October 11-14, 2016,Proceedings, Part IV, pages 354–370, 2016.

[4] C. Chen, W. Gong, Y. Hu, Y. Chen, andY. Ding. Learning oriented region- based convo-lutional neural networks for building detection insatellite remote sensing images. The InternationalArchives of the Photogrammetry, Remote Sensingand Spatial Information Sciences, 42(1), 2017.

[5] M. Chevalier, N. Thome, M. Cord, J. Fournier,G. Henaff, and E. Dusch. Lr-cnn for fine-grainedclassification with varying resolution. In 2015IEEE International Conference on Image Process-ing (ICIP), pages 3101–3105, Sept 2015.

[6] G. Christie, N. Fendley, J. Wilson, andR. Mukherjee. Functional Map of the World.ArXiv e-prints, Nov. 2017.

[7] J. Dai, Y. Li, K. He, and J. Sun. R-fcn: Objectdetection via region-based fully convolutional net-works. In NIPS, 2016.

[8] M. Everingham, L. Van Gool, C. K. I. Williams,J. Winn, and A. Zisserman. The pascal visual ob-ject classes (voc) challenge. International Journalof Computer Vision, 88(2):303–338, June 2010.

[9] W. Fang, J. Chen, C. Liang, X. Wang, Y. Nan,R. Hu, and S. Luo. Object detection in low-resolution image via sparse representation. pages234–245, 2015.

[10] R. Girshick, J. Donahue, T. Darrell, and J. Ma-lik. Rich feature hierarchies for accurate objectdetection and semantic segmentation. 11 2013.

[11] R. Hamid, S. OâĂŹHara, and M. Tabb. Global-scale object detection using satellite imagery. TheInternational Archives of the Photogrammetry,Remote Sensing and Spatial Information Sciences,40(3), 2014.

[12] B. Hariharan and R. B. Girshick. Low-shot vi-sual recognition by shrinking and hallucinatingfeatures. 2017 IEEE International Conference onComputer Vision (ICCV), pages 3037–3046, 2017.

[13] T. Ishii, E. Simo-Serra, S. Iizuka, Y. Mochizuki,A. Sugimoto, H. Ishikawa, and R. Nakamura. De-tection by classification of buildings in multispec-tral satellite imagery. In 2016 23rd InternationalConference on Pattern Recognition (ICPR), pages3344–3349, Dec 2016.

[14] L. Karlinsky, J. Shtok, Y. Tzur, and A. Tzadok.Fine-grained recognition of thousands of objectcategories with single-example training. In TheIEEE Conference on Computer Vision and Pat-tern Recognition (CVPR), July 2017.

[15] I. Krasin, T. Duerig, N. Alldrin, V. Ferrari,S. Abu-El-Haija, A. Kuznetsova, H. Rom, J. Ui-jlings, S. Popov, A. Veit, S. Belongie, V. Gomes,A. Gupta, C. Sun, G. Chechik, D. Cai, Z. Feng,D. Narayanan, and K. Murphy. Openimages:A public dataset for large-scale multi-label andmulti-class image classification. Dataset availablefrom https://github.com/openimages, 2017.

[16] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Im-agenet classification with deep convolutional neu-ral networks. Commun. ACM, 60(6):84–90, May2017.

[17] B. M. Lake, R. Salakhutdinov, and J. B.Tenenbaum. Human-level concept learningthrough probabilistic program induction. Science,350(6266):1332–1338, 2015.

[18] B. M. Lake, R. Salakhutdinov, and J. B.Tenenbaum. Human-level concept learningthrough probabilistic program induction. Science,350(6266):1332–1338, 2015.

[19] H. Li, Y. Liu, W. Ouyang, and X. Wang. Zoomout-and-in network with map attention decisionfor region proposal and object detection. 09 2017.

[20] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Per-ona, D. Ramanan, P. Dollar, and C. L. Zitnick.Microsoft coco: Common objects in context. InD. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars,editors, Computer Vision – ECCV 2014, pages740–755, Cham, 2014. Springer International Pub-lishing.

[21] D. Lindenbaum. Spacenet on aws.[22] W. Liu, D. Anguelov, D. Erhan, C. Szegedy,

S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Singleshot multibox detector. 2016. To appear.

[23] T. N. Mundhenk, G. Konjevod, W. A. Sakla, andK. Boakye. A large contextual dataset for classi-fication, detection and counting of cars with deep

15

learning. In B. Leibe, J. Matas, N. Sebe, andM. Welling, editors, Computer Vision – ECCV2016, pages 785–800, Cham, 2016. Springer In-ternational Publishing.

[24] J. Redmon, S. Divvala, R. Girshick, andA. Farhadi. You only look once: Unified, real-time object detection. In 2016 IEEE Confer-ence on Computer Vision and Pattern Recognition(CVPR), pages 779–788, June 2016.

[25] S. Ren, K. He, R. Girshick, and J. Sun. Fasterr-cnn: Towards real-time object detection with re-gion proposal networks. In Proceedings of the 28thInternational Conference on Neural InformationProcessing Systems - Volume 1, NIPS’15, pages91–99, Cambridge, MA, USA, 2015. MIT Press.

[26] O. Russakovsky, J. Deng, H. Su, J. Krause,S. Satheesh, S. Ma, Z. Huang, A. Karpathy,A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual RecognitionChallenge. International Journal of Computer Vi-sion (IJCV), 115(3):211–252, 2015.

[27] J. T. Turner, K. M. Gupta, B. T. Morris, andD. W. Aha. Keypoint density-based region pro-posal for fine-grained object detection and clas-sification using regions with convolutional neuralnetwork features. CoRR, abs/1603.00502, 2015.

[28] O. Vinyals, C. Blundell, T. Lillicrap,K. Kavukcuoglu, and D. Wierstra. Match-ing networks for one shot learning. In NIPS,2016.

[29] W. Zhang, G. J. Zelinsky, and D. Samaras. Real-time accurate object detection using multiple reso-lutions. 2007 IEEE 11th International Conferenceon Computer Vision, pages 1–8, 2007.

[30] B. Zhao, J. Feng, X. Wu, and S. Yan. A sur-vey on deep learning-based fine-grained objectclassification and semantic segmentation. Inter-national Journal of Automation and Computing,14(2):119–135, Apr 2017.

16