Environmental Sensing UsingLand-Based Spectrally-SelectiveCameras and a Quadcopter

Jnaneshwar Das, William C. Evans, Michael Minnig,Alexander Bahr, Gaurav S. Sukhatme, and Alcherio Martinoli

Abstract. We investigate the reconstruction of an environmental scalar fieldusing robotic mobility and heterogeneous sensing. Using two land-based,immobile, co-located spectrally selective cameras, and a non-contact infrared-based temperature sensor on a quadcopter, we study the problem of recon-structing the surface temperature of the ground under survey. Both land units— a thermographic camera for low-resolution thermal images and a commer-cial digital camera for high resolution truecolor images — are mounted onan elevated camera rig. We explore methods for field reconstruction using acombination of the three imaging sensors. First, we show that the quadcopterdata is correlated with the synoptic snapshots obtained by the thermal imag-ing camera. Next, we demonstrate upsampling of the low-resolution thermalcamera data with truecolor images. This results in high-resolution reconstruc-tion of the temperature field. Finally, we discuss adaptive sampling techniquesthat utilize the mobility of the quadcopter to ‘fill the gaps’ in data acquiredby the thermal imaging camera. Our work experimentally demonstrates thefeasibility of heterogeneous sensing and mobility to effectively reconstructenvironmental fields.

1 Introduction

Fast sampling of terrestrial environmental fields is of importance for vari-ous studies. In this work, we address rapid sampling of environmental fields

William C. Evans · Michael Minnig · Alexander Bahr · Alcherio MartinoliDistributed Intelligent Systems and Algorithms Laboratory,École Polytechnique Fédérale de Lausanne, Switzerlande-mail: {william.evans,michael.minnig,

alexander.bahr,alcherio.martinoli}@epfl.ch

Jnaneshwar Das · Gaurav S. SukhatmeRobotic Embedded Systems Laboratory, Dept. of Computer Science,University of Southern California, Los Angeles, CA 90089 USAe-mail: {jnaneshd,gaurav}@usc.edu

J.P. Desai et al. (Eds.): Experimental Robotics, STAR 88, pp. 259–272.DOI: 10.1007/978-3-319-00065-7_19 c© Springer International Publishing Switzerland 2013

260 J. Das et al.

using static and mobile imaging sensors. Specifically, we demonstrate ef-fective reconstruction of the surface temperature field of a patch of vege-tation. Accurate monitoring of surface temperature is desirable for atmo-spheric boundary layer studies over complex terrain [1], to cite one exam-ple. We describe an experiment where the surface temperature of a regionis observed with images from a land-based thermal imaging camera aug-mented with truecolor images from a parallel-mounted commercial digitalcamera. In addition, a downward-facing non-contact infrared temperaturesensor mounted on a quadcopter serves as a mobile sensing platform. Theland-based thermal imaging camera is mounted at an elevation along withthe truecolor camera, providing snapshots of the surface temperature andhigh resolution true-color images respectively. The quadcopter serves as afast aerial observation platform, allowing rapid sampling of surface tempera-ture using its downward-facing temperature sensor. This provides both speedand flexibility compared to land-based observation platforms such as robotic

Fig. 1 Illustration of the experimentalsetup to sample the surface temperatureof a patch of land

rovers. This work has three goals:a) investigate upsampling of thethermal camera data using high res-olution truecolor images from thedigital camera, b) compare the tem-perature data acquired by the quad-copter with the synoptic thermalimage captured from the thermalcamera, and c) to explore adap-tive sampling strategies that usethe synoptic data from the thermalcamera to guide the quadcopter toregions of high prediction uncer-tainty. Our goal is to demonstratesynergistic use of mobile and staticsensors for rapid characterization ofenvironmental phenomena. Such acapability is necessary when thereare constraints on the use of land-based imaging sensors resulting insparse data. This can happen due tolong distance between test site and the land-based camera, or insufficient el-evation of the camera rig. By using mixed sensing, we can reconstruct thetemperature field at a resolution higher than that provided by the individualsensors.

The paper is organized as follows. In Section 2 we briefly describe relatedwork. In Section 3, we lay the groundwork for the analysis of the experimentaldata by describing our technical approach. In Section 4, we describe our fieldsetup followed by analysis of the data from the field trials. We conclude witha summary and discussion of future work in Section 5.

Environmental Sensing Using Land-Based Spectrally-Selective Cameras 261

2 Related Work

Ecological monitoring of large farmlands using UAVs has been studied forrapid mapping and classification of vegetation [2]. Adaptive sampling forenvironmental monitoring has been investigated in the context of intelli-gent placement of static sensor nodes [3], and informative paths for mobileaquatic platforms [4, 5]. Upsampling of multimodal remote sensing imageshas been explored to fuse low-resolution hyperspectral images with high-resolution truecolor images [6]. Our work presents an agile setup that pro-vides a quick reconstruction of the environmental field in a region by useof selective spectral-cameras operating at different resolutions, aided by themobility of a quadcopter.

3 Technical Approach

Our goal is to investigate the use of mixed sensing in the form of static land-based cameras and a quadcopter to rapidly sample the surface temperature ofa terrestrial patch of vegetation. We will first describe the land-based camerarig and the quadcopter, followed by a discussion of unwarping and correctionof the thermal and truecolor images. We then describe the three contributionsof this work for mixed sensing field reconstruction: a) upsampling of thermalimages using high-resolution truecolor images (Subsection 3.3), b) comparisonof quadcopter data with land-based camera images (Subsection 3.4), and c) anadaptive sampling scheme for the quadcopter to augment land-based sensors(Subsection 3.5).

3.1 Sensing Apparatus

Fig. 2 The camera rig consisting of athermographic camera and a digital true-color camera mounted on a pan-tilt head

The sensing apparatus consists ofthree imaging sensors, two mountedon a land-based camera rig, andone mounted on a quadcopter.The camera rig consisted of anFLIR A320 thermographic camerawith a resolution of 640x480 anda CANON 300D digital camera,both mounted on a tripod head,triggered by a computer to si-multaneously capture truecolor andthermal images of the survey site.The aerial platform was an As-cending Technologies Hummingbirdquadcopter. It used its onboardcomputer to log data from a

262 J. Das et al.

non-contact IR temperature sensor at a rate of 5 Hz along with GPS data at1 Hz. We chose an IR-based temperature sensor so that the quadcopter wouldshare a similar modality with the thermographic camera. A human operatormaneuvered the quadcopter via remote control. The IR sensor captured theemitted IR radiance from a 10 degree view of field. Hence the temperaturecaptured by the sensor was dependent on the height from the target. How-ever, in this work, we did not model the sensor properties, and ignored theheight of the quadcopter in flight.

3.2 Image Unwarping and Correction

Fig. 3 The quadcopter in-flight with thedownward looking infra red temperaturesensor

We first explore fusion of the ther-mal and truecolor images for quickinspection of the scene. Since thethermal and truecolor cameras aremounted at an elevation, generatinga perspective view of the scene, wecompute perspective transforms tounwarp the truecolor and thermalimages. First, we manually markedfour landmarks in the thermal im-age and the truecolor image. We se-lected corners of man-made structures such as metal electric poles and con-crete slabs because these were easily recognizable in both the thermal andtruecolor images.

Next, we must ensure that all data shares a common frame of reference. Weproceed by transforming all images to the Earth’s coordinate frame, with anapproach similar to that used during the image unwarping step. Landmarksin the thermal and truecolor images are used along with landmarks in asatellite truecolor image of the scene obtained from commercial map servers(e.g., Google Earth). The transformation was computed using the OpenCVlibrary. Once a perspective transformation matrix is computed, we obtainthe unwarped data points z = [x, y, t]T , where x is the longitude, y is thelatitude, and t is the color value of the pixel that was unwarped.

3.3 Upsampling

To demonstrate upsampling of sparse thermal data using dense truecolordata, we subsample a sparse set of points from the unwarped thermal cameraimage along with the corresponding truecolor pixel values1. Our goal is tolearn a model that predicts surface temperature at unobserved locations using1 Computed using a nearest neighbor search with the thermal camera data points.

Environmental Sensing Using Land-Based Spectrally-Selective Cameras 263

the truecolor data. The underlying assumption is that surface patches withsimilar color will have similar temperature.

Gaussian Process Regression

We use Gaussian process regression (GPR) [7], a nonlinear Bayesian regres-sion technique commonly used in geostatistics under the name ‘Kriging’. Itassumes that the samples from the function to be estimated are normallydistributed with the covariance between samples given by a ’kernel’ or co-variance function. We consider the case where the observations are unbiased,that is, the mean of the joint Gaussian distribution is zero. This can be sim-ply satisfied by ’demeaning’ the observed data. As a result of its formulation,GPR automatically achieves model regularization from data only, withouthaving to choose model complexity parameters a priori. Additionally, GPRis defined completely by a kernel function that controls how quickly the in-put space becomes decorrelated. This enforces smoothness constraints in thetrained function, ideal for spatial models where usually the observed valuesfor nearby input samples are more correlated than the ones farther apart.

Assume we have training data given by D = < x1, y1 >, ..., < xn, yn >,drawn from the noisy process,

yi = f(xi) + ε (1)

where ε is a Gaussian noise term.Given the training data, posterior mean and covariance for a test data

point x∗ is given by the following equations,

GPμ = k∗(K + σ2nI)

−1y (2)GPΣ = k(x∗, x∗) + k∗(K + σ2

nI)−1k∗ (3)

The kernel function k is usually chosen to be a squared-exponential functiongiven by,

k(xp, xq) = e−1

2λ2 |xp−xq|2 (4)

where λ is the decorrelation length scale. The hyperparameters for the kernelfunction can be learned using iterative methods such as conjugate gradientdescent. K is the Gram matrix with its elements given by Kpq = k(xp, xq),I is the identity matrix, and k∗ is the vector of covariances between the testdata point and the training data points.

To apply the GPR model to upsample thermal image data, let us considerthe unwarped pixels from the truecolor camera given by the vector Xtc =<Lontc, Lattc, Rtc, Gtc, Btc >. We use nearest neighbor search to obtain atraining dataset of thermal camera image data points and their correspond-ing truecolor pixel data. This is given by Xtrain =< Lon, Lat, R,G,B >,and Ytrain = T . Now, we use GPR to learn a function f that maps predic-tion points Xtest =< Lontest, Lattest, Rtest, Gtest, Btest > where Lontest and

264 J. Das et al.

Lattest are the longitude and latitude of unobserved locations, and Rtest,Gtest,and Btest are the RGB pixel values from the high-resolution imagescaptured by the truecolor camera corresponding to the query points.

3.4 Comparison of Quadcopter and Thermal CameraData

To investigate the feasibility of using a quadcopter along with a land-basedthermal camera, we need to compare the data obtained by the two sensors.Linear interpolation on the quadcopter sensor data on a regular grid im-mediately reveals visual similarity with the thermal imaging camera data.However, for a quantitative comparison of the two datasets, we compute thePearson correlation coefficient for quadcopter data and the co-located ther-mal camera data calculated using nearest-neighbor search on each quadcopterdata point. The Pearson correlation coefficient is given by,

R =

∑ni=0(xi − x)(yi − y)

√∑ni=0 (xi − x)2

∑ni=0 (yi − y)2

(5)

where x and y are the two sensor data streams being compared. Higher valuesof R indicate a stronger correlation.

The imaging sensors on the thermographic camera and the IR sensor on thequadcopter are not cross-calibrated at the outset. For calibration, we choosea 440 sample data window (90 seconds) of the quadcopter data that is highlycorrelated with the thermal camera data (R>0.8) and use it to learn a linearmapping from raw quadcopter data to corrected quacopter data, given by,tcorr = a1t+ a2, where t is raw quadcopter data point, tcorr is the correcteddata point, and a1 and a2 are regression coefficients.

3.5 Adaptive Sampling with Quadcopter

Fig. 4 illustrates the sparsity of data away from the thermal camera once theacquired image is unwarped. This effect is more pronounced when the camerarig is farther away from the test site, or not highly elevated. This scenariowill be common in unstructured environments. Also, there are sections of thethermal image (the vertical corners) without any data points. Our goal is toinvestigate field reconstruction that takes into account the uncertainty of esti-mates from the thermal camera data as a result of data sparsity. We proposegreedily collecting data from regions with high variance. We use a sparserversion of the thermal camera data to build a probabilistic spatial model ofthe temperature field using GPR as described earlier for the upsampling task.Then, we greedily add data points and analyze how many points are neededfrom the quadcopter data to reduce uncertainty in the reconstructed thermalimage.

Environmental Sensing Using Land-Based Spectrally-Selective Cameras 265

Fig. 4 Perspective corrected (top-view) thermal camera data highlighting sparsityof measurements farther away from the camera (bottom of the image), and completelack of data in the top right and left corners

4 Experiments and Results

We carried out a series of field experiments on the EPFL campus in a 25m x22m area with a significant portion being vegetation (grass), and a section ofgravel path (Fig. 5). This provided a natural environmental field for the mea-surement of surface temperature. The land based camera-rig was mounted ona tripod at an elevation of 5m and at a distance of 20m from the experimentsite. Images were captured simultaneously by both cameras every minuteduring the course of the experiment. For our analysis, we use one such con-current snapshot. The quadcopter was operated manually at a mean heightof 3.8m by a human pilot for a period of 10 minutes to capture the surfacetemperature over a lawnmower pattern as shown in Fig. 6. We show resultsfrom one of the field trials.

4.1 Image Unwarping and Correction

Fig. 7 shows the result of using visual landmarks in the truecolor and thermalimages to find the transformation between the two cameras, which was thenused to generate an overlaid image of the scene showing both truecolor andthermal images. This is useful as an initial overview of the scene and can beobtained in realtime. Next, we obtained a remotely-sensed truecolor image ofthe scene from a commercial map-server. From this, we used visual landmarksto compute a perspective transform to unwarp the images from the thermal

266 J. Das et al.

Fig. 5 The experiment site was a 25m X 22m patch of land with varying densityof grass, and a gravel path. The camera-rig was at a height of 9m from the test site,at a distance 30 m from the nearest edge of the test site.

(a) The quadcopter path(solid red path) from one ofthe experiment runs. A hu-man operator maneuveredthe quadcopter to carryout a ‘lawnmower’ pattern.

(b) Quadcopter temperature datapoints from the field trial.

Fig. 6 Data from quadcopter field trial

Environmental Sensing Using Land-Based Spectrally-Selective Cameras 267

(a) Truecolor image fromCanon 300D digital cam-era.

(b) Thermal image fromFLIR A320 thermo-graphic camera.

(c) Truecolor and ther-mal image superimposedafter image registration.

Fig. 7 Landmarks in the truecolor and thermal images used to generate an overlaidimage for quick survey of the scene

(a) Unwarped truecolorimage.

(b) Unwarped thermal camera image and linear inter-polated quadcopter temperature data.

Fig. 8 Visual comparision of corrected truecolor image, thermal image, and inter-polated quadcopter data

camera and the digital truecolor camera. Fig. 8 shows the unwarped topview of the truecolor and thermal images. We performed interpolation on theGPS-tagged surface temperature data collected by the quadcopter for initialvisual comparison of the two datasets. Fig. 8b shows this image.

4.2 Upsampling

As described in Section 3, we use GPR on thermal camera data augmentedwith truecolor data to predict temperature at unobserved locations wheretruecolor data are available. The result of the upsampling analysis is shownin Fig. 9.

268 J. Das et al.

(a) Sparse thermal camera data. (b) Sparse truecolor data correspondingto thermal camera data.

(c) Dense prediction points, with true-color data.

(d) Estimated thermal field.

Fig. 9 Upsampling thermal camera data with high-resolution truecolor data usingGaussian process regression with input augmented with RGB data

4.3 Comparision of Quadcopter and Thermal CameraData

We computed the Pearson correlation coefficient between the quadcopter datapoints and the perspective corrected data points from the thermal imagingcamera. Since the thermal camera data density is much higher than the quad-copter data, we found the Euclidean nearest-neighbor thermal data points tothe quadcopter data. Fig. 10 shows the quadcopter data alongside nearest-neighbor thermal camera data. The two vectors were of length 3607 data-points each, and showed R = 0.477, demonstrating a statistically significantcorrelation between quadcopter data and thermal camera data. Additionally,to analyze the effect of outliers in the quadcopter data (for example, due tounfavorable altitude), we computed the Pearson coefficient on a sliding win-dow of 440 data points (corresponding to 90 second of quadcopter flight time).The resulting distribution of correlation coefficient is shown in Fig. 11a.

Environmental Sensing Using Land-Based Spectrally-Selective Cameras 269

6.5658 6.566 6.566246.5191

46.5192

46.5193

46.5194

46.5195

46.5196

46.5197

46.5198

46.5199

Longitude

La

titu

de

28

30

32

34

36

38

40

42

6.5658 6.566 6.566246.5191

46.5192

46.5193

46.5194

46.5195

46.5196

46.5197

46.5198

46.5199

Longitude

La

titu

de

26

27

28

29

30

31

32

Fig. 10 Quadcopter data points (left) and nearest-neighbor thermal camera datapoints (right)

−0.4 −0.2 0 0.2 0.4 0.6 0.8 10

5

10

15

20

25

Corr2

Fre

qu

ency

(%

)

(a) Pearson correlation coefficient dis-tribution for a sliding window of440 datapoints over quadcopter andnearest-neighbor thermal camera data.

28 30 32 34 36 38 40 42 4425

26

27

28

29

30

31

32

33

Quadcopter temperature

Therm

al cam

tem

pera

ture

(b) Cross-calibration fit between quad-copter and thermal camera data.

Fig. 11 Correlation between quadcopter and thermal camera data, and plot oflinear cross-calibration fit

270 J. Das et al.

Next, we cross-calibrate the quadcopter data with the thermal camera databy finding a linear fit to a section of the quadcopter and thermal camera datawith R>0.8. Fig. 11b shows the the data points and the resulting linear fit.

4.4 Adaptive Sampling with Quadcopter

We flew the quadcopter remotely to carry out lawnmower surveys of the testarea capturing samples of temperature with the downward looking IR temper-ature sensor. From this, we can choose points to emulate adaptive collectionof data. This approach allows us to try various techniques adaptively withouthaving to perform multiple experiments. We use a subsampled version of thethermal camera image as the pilot data to learn a GPR-based probabilisticregression model of the temperature field. We then use the variance of thefield to greedily choose new sample points. As a reference, we show the re-constructed field from the quadcopter data in Fig. 12a. Fig. 12b shows theinitial reconstruction of the temperature field from the thermal camera data.The top left and right corner of the reconstructed field are regions that ex-hibit extrapolation, with high associated uncertainty, as showed in Fig. 12c.Data from the quadcopter is used to fill these gaps, and in Fig. 12d, we seetwenty additional data points added greedily to the reconstruction from thecross-calibrated quadcopter dataset. Each addition of a data point is followedby relearning of the temperature field. As seen in this figure, as a result ofaddition of the new samples, the top left corner of the field now exhibitsmoderate temperature.

5 Discussion and Conclusions

In this paper, we presented an experiment using mixed sensing to reconstructthe surface temperature of a patch of land. A land-based camera-rig consistingof a thermographic camera and a high-resolution truecolor camera was usedto generate upsampled thermal images of the experiment site. A quadcopterequipped with a downward looking IR temperature sensor measured surfacetemperature during flight. We compared the data from quadcopter with thethermal camera and found they are correlated (Pearson coefficient of 0.47).Finally, we investigated adaptive sampling strategies to fill gaps in thermalcamera data using the quadcopter. We achieved this by using data from aquadcopter run offline.

This work has limitations that merit future work. First, we have not in-cluded the quadcopter altitude in the estimation of the temperature field.Since the IR temperature sensor has a relatively large field of view, the alti-tude has an impact on the measured data. This likely this has an impact onthe correlation between the quadcopter data gathered during the field trialand the corresponding thermal camera data. Fig. 13 shows the distributionof altitude for the 10 minute flight at the experiment site. In the future, we

Environmental Sensing Using Land-Based Spectrally-Selective Cameras 271

(a) GPR reconstruction on a regulargrid for quadcopter data. Black dotsshow training data locations.

(b) GPR reconstruction on a regulargrid for thermal data. Black dots showtraining data locations.

(c) Prediction variance for recon-structed thermal image. Red regionsshow high uncertainty.

(d) GPR reconstruction on a regulargrid for thermal data after 20 adaptivesamples from quadrocopter dataset.

Fig. 12 GPR reconstructed thermal and quadcopter data

Fig. 13 The distribution of quadcopter altitude during the field trial

272 J. Das et al.

plan to model the IR-based temperature sensor as a pixel-average sensor andtake into account the effect of height while reconstructing the temperaturefield. Second, we have used RGB data from the truecolor camera to aug-ment the thermal camera image to perform upsampling. We plan to exploreother characteristics of the land patch in addition to RGB for this task. Fi-nally, we have not carried out online adaptive sampling experiments with thequadcopter. Instead, we collected data using a lawnmower survey and usedthe data offline to emulate new samples. In future, we will carry out onlineexperiments to validate our approach.

Acknowledgements. This work was supported in part by the National ScienceFoundation under award CCF-0120778 and IIS-1107011. We thank the organizers ofthe Twenty-second International Joint Conference on Artificial Intelligence (IJCAI)Doctoral Consortium, and the Swiss National Center of Competence in Researchin Robotics (NCCR Robotics) for facilitating the extended research visit by thefirst author to EPFL that made this work possible. William C. Evans and Alexan-der Bahr were partially supported by "The Swiss Experiment" of the CompetenceCenter Environment and Sustainability of the ETH Domain (CCES), and by theNCCR Transfer project "Tamperproof Monitoring Solution for Weather Risk Man-agement" sponsored by the Swiss National Science Foundation and managed by theNational Center of Competence in Research in Mobile Information and Commu-nication Systems (NCCR-MICS). We thank Holly Oldroyd and Daniel Nadeau ofthe Environmental Fluid Mechanics Laboratory at EPFL for lending us the ther-mographic camera and helping us with the science motivation.

References

1. Nadeau, D., Parlange, M.: Atmospheric Boundary Layer Dynamics of Transi-tional Flows over Complex Terrain. Ph.D. dissertation, EPFL (2011)

2. Bryson, M., Reid, A., Hung, C., Ramos, F., Sukkarieh, S.: Cost-Effective Map-ping Using Unmanned Aerial Vehicles in Ecology Monitoring Applications. In:Khatib, O., Kumar, V., Sukhatme, G. (eds.) Experimental Robotics. STAR,vol. 79, pp. 509–523. Springer, Heidelberg (2012)

3. Krause, A., Singh, A., Guestrin, C.: Near-Optimal Sensor Placements in Gaus-sian Processes: Theory, Efficient Algorithms and Empirical Studies. J. Mach.Learn. Res. 9, 235–284 (2008)

4. Zhang, B., Sukhatme, G.S.: Adaptive Sampling for Estimating a Scalar Fieldusing a Robotic Boat and a Sensor Network. In: IEEE International Conferenceon Robotics and Automation, pp. 3673–3680 (2007)

5. Binney, J., Krause, A., Sukhatme, G.S.: Informative Path Planning for an Au-tonomous Underwater Vehicle. In: IEEE International Conference on Roboticsand Automation, pp. 4791–4796 (2010)

6. Thomas, C., Ranchin, T., Wald, L., Chanussot, J.: Synthesis of MultispectralImages to High Spatial Resolution: A Critical Review of Fusion Methods Basedon Remote Sensing Physics. IEEE T. Geoscience and Remote Sensing 46(5),1301–1312 (2008)

7. Rasmussen, C.E.: Gaussian Processes for Machine Learning. MIT Press (2006)