Characterizing the temporal and spatial variability oflongwave infrared spectral images of targets and backgrounds

Nirmalan Jeganathana, John Kerekesb, and Dalton Rosarioc

a,bRochester Institute of Technology, Chester F. Carlson Center for Imaging Science, DigitalImaging and Remote Sensing, 54 Lomb Memorial Drive, Rochester, NY, USAcUS Army Research Laboratory, 2800 Powder Mill Rd, Adelphi, MD, USA

ABSTRACT

Following the public release of the Spectral and Polarimetric Imagery Collection Experiment (SPICE) dataset, apersistent imaging experiment dataset collected by the Army Research Laboratory (ARL), the data were analyzedand materials in the scene characterized temporally and spatially using radiance data. The noise equivalentspectral radiance provided by the sensor manufacturer was compared with instrument noise calculated from in-scene information, and found to be comparable given differences in laboratory setting and real-life conditions. Theprocessed dataset have regular “inconsistent cubes,” specifically for data collected immediately after blackbodymeasurements, which were automatically executed approximately at each hour mark. Omitting these erroneousdata, three target detection algorithms (adaptive coherent/cosine estimator, spectral angle mapper, and spectralmatched filter) were tested on the temporal data using two target spectra (noon and midnight). The spectralmatched filter produced the best detection rate for both noon and midnight target spectra for a 24-hrs period.

Keywords: Diurnal variation, spatial variation, temporal variation, long wave infrared, SPICE

1. INTRODUCTION AND OBJECTIVES

The Spectral and Polarimetric Imagery Collection Experiment (SPICE) was a collaborative effort between theUS Army Research Laboratory (ARL), US Army Armament Research, Develepment and Engineering Center(ARDEC) and the US Air Force Institute of Technology (AFIT) focused on collecting and exploiting long waveinfrared (LWIR) hyperspectral and polarimetric imagery. SPICE autonomously collected an expansive dataset ofhyperspectral and polarimetric modalities spanning multiple years in a wide range of meteorological conditions.1

The SPICE dataset has been studied and analyzed by ARL for the last few years. The hyperspectral sensorused was a lightweight and compact imaging radiometric spectrometer manufactured by Telops. Accordingto the manufacturer, the sensor was employed during SPICE in a setting of continuous and autonomous datacollection for the very first time in this product’s history. Therefore, one of the initial assessments done byARL was to characterize the quality of the data obtained by the sensor. The SPICE data quality was comparedto a similar sensor but with higher spectral resolution and a human operator monitoring the collection,2,3 aswell as with MODerate TRANsmission (MODTRAN) models.2 An algorithm test carried out on this datasetwas the anomaly detection algorithm, notably the Range-Invariant Anomaly Detection (RIAD) and Reed-Xiaolianomaly detection (RXD) algorithms.4,5 Target detection of tank paint was performed using two methods:repeated sampling trial single-class SVM and the longitudinal data model.6 These previous studies concludedthat this dataset was rich in spectral content and temporal variation, despite being noisier than other datasetsobtained using far more expensive sensors.7

The objective of this paper is to further analyze and characterize the temporal and spatial variability of theSPICE data for better understanding. An autonomous collection often has imperfections, including erroneousdata. Locating and removing these imperfections improves the quality of the dataset. Some deficiencies, such assensor noise, cannot be completely removed; however, the noise can be characterized to help explain observations.

Further author information: (Send correspondence to A.,B.)A..: E-mail: nj1520@rit.edu, Telephone: 1 585 414 1637B.: E-mail: kerekes@cis.rit.edu , Telephone: 1 585 475 6996

While consisting of only a single scene, the observed area contained multiple materials, both natural and man-made. Characterizing the various materials temporally and spatially can shed light on the behaviors of selectmaterials compared to others.

The paper is organized as follows: Section 2 describes the location, sensor and the targets of the collectionexperiment, Section 3 provides a brief summary of the available dataset; Section 4 reports the usable datasetfrom Section 3; Section 5 characterizes the instrument noise; Section 6 and Section 7 describe the temporal andspatial variation of the data; and finally Section 8 will conclude the paper.

2. SPICE DATA COLLECTION OVERVIEW

2.1 Location

The Precision Armaments Laboratory (PAL), located at ARDEC, Picatinny Arsenal, New Jersey (40o55’40.8”N74o34’52.0”W) is specialized in testing sensors under adverse weather conditions. The sensors were placed atopthe 65-m PAL tower (effective height of 126 m since the tower is positioned atop a 61-m ridge) and the targetsite area is 549 m from base of the tower (as seen in Figure 1). An automated meteorological instrumentationsite is located close to the tower, and its measurement instrumentation include wind speed, wind direction,temperature, humidity and barometric pressure.8 A detailed list of PAL basic meteorological instrumentationtypes is provided in a previous publication.9

Figure 1. The PAL tower, where the sensors are placed, and the target site.10

2.2 Sensor

The Telops Hyper-Cam Long-Wave (LW) was used for data collection. It was a commercially available lightweightFourier-transform spectrometer LWIR imager which incorporated a 320 by 256 photovoltaic mercury cadmiumtelluride (PV MCT) focal plane array (FPA).1 Sensor specifications are provided in Table 1.8

Table 1. Hyper-Cam LW sensor specificationRegion of spectrum (µm) 7.7 to 11.5Focal length (mm) 86Cooled FPA 320 by 256Pixel size (µm) 30Instantaneous FOV (mrad) 0.35Black body Internal (2)Spectral resolution (cm-1) 0.25 to 150Typical NESR (nW/cm2sr·cm-1) <20

2.3 Targets

The target site consisted of man-made objects surrounded by natural vegetation. The man-made targets includedthree surrogate Russian 2S3 howitzers (tanks) oriented in aspect angles of 0o, 90o, and 135o (counterclockwise)with respect to the sensor, a white panel (skyplate/aluminum panel) and a black canvas as shown in Figure 2.1

Figure 2. Area imaged by Telops Hyper-Cam with a selection of man-made objects labeled, adapted from Roasario et al.1

3. SPICE DATA SUMMARY

ARL conducted SPICE spanning two years, 2012 and 2013. The Telops LW Hyper-Cam imaged the targetsite approximately every five minutes 24 hours per day. The full SPICE dataset contains over 25 000 LWIRhyperspectral data cubes. The meteorological instruments collected data approximately every 2 seconds. Themeteorological data were matched with the corresponding Hyper-Cam data in accordance with time. All thecollected data were processed and delivered in HDF5 file format. Data provided to Rochester Institute ofTechnology (RIT) for this study included 5855 LWIR HDF5 files, encompassing 18 days from 2012 and 8 daysfrom 2013.

The data in each HDF5 file provided by ARL have, among other data, three hyperspectral cubes: calibratedradiance, brightness temperature and relative emissivity. The radiance is provided in W/m2sr·µm and thebrightness temperature is in oC. The brightness temperature was obtained by using the inverse Planck equation,and the relative emissivity was retrieved using the QTES5 algorithm.

Each 2012 data cube has 256 × 320 pixels × 165 bands. The 2013 data had its spatial window size alteredand its dimensions were 224 × 300 pixels × 165 bands. The 165 bands corresponded to wavelengths between7.4081 µm and 12.4493 µm. Due to sensor limitations, only 105 bands were used in this analysis by omitting thefirst 30 and the last 30 bands. The 105 bands corresponded to wavelengths between 8.00075 µm and 11.0712 µm.After an initial analysis, it was discovered that the 2012 data were collected while experimenting with sensorparameters, and thus was not suitable for further analysis. Results presented in this paper only used data from2013. Figure 3 displays radiance, brightness temperature and relative emissivity images captured on 1 May 2013,09:38 at wavelength 8.21995µm. A histogram equalized relative emissivity image is also displayed in Figure 3for better visualization.

Figure 4 shows the pixel masks for ten different materials in the scene. They include three tanks (Tank0,Tank90, Tank135), two panels (black canvas, aluminum panel (sky-plate)), gravel and four types of vegetation

Figure 3. Top left: Radiance image (W/m2sr·µm). Top right: Brightness temperature image (oC). Bottom left: Relativeemissivity image. Bottom right: Histogram equalized relative emissivity image for better visualization of the scene.

(near-trees, rear-trees, bushes and grass). Figure 5 shows pixel mask subsets for each material, where a subsetof each material is included to minimize the adjacent material effects.

Figure 4. Material masks (Three tanks, black canvas,aluminum panel, gravel, near-trees, rear-trees, bushesand grass).

Figure 5. Subset of materials as defined in Figure 4while minimizing edge effects.

4. DATA SCREENING

While studying the data-set, it was observed that some data had inconsistent behavior. An example inconsistentresult was a plot of diurnal brightness temperature variation of the data. As observed in Figure 6, there are

“periodic” data spikes occurring throughout the plot. The following steps were carried out to explore thisobservation further:

1. Plot a brightness temperature pixel of one tank for a selected band for all cubes in one day (see Figure 6).

2. Determine the “inconsistent” cubes using the plot (i.e. cubes producing “spikes” are inconsistent).

3. Individually verify each inconsistent cube to confirm if the whole cube is inconsistent (the inconsisten-cies were determined by comparing the brightness temperature of vegetation pixels of said cube with itsneighboring cubes). It was discovered that all image pixels generated the spikes for certain cubes and thisbehavior was not limited to the selected tank pixel.

4. Repeat for all available data days.

5. Tabulate inconsistent cubes.

The data cubes collected immediately after the automatic hourly blackbody measurements were known tohave a low probability of being reliable data. These inconsistent data cubes corresponded to such measurementtimes. A total of 339 cubes of the dataset studied were identified as inconsistent, and these cubes were removedfrom further analysis for this paper.

Figure 6. 01 May 2013 diurnal brightness temperature of a tank pixel. Y-axis correspond to brightness temperature (oC),x-axis correspond to the 24-hrs duration of a day and the multiple plots correspond to 105 spectral bands. It is notedthat all bands spike for certain time cubes.

5. NOISE CHARACTERIZATION

The data provided by ARL were processed data cubes, and previous studies concluded that intrinsic systemnoise in the sensor was a minor to moderate concern compared to the atmospheric and environmental variationeffects.2 We characterized the instrument noise by calculating difference standard deviation of a uniform object.For this method, we assumed the black panel to be perfectly uniform with negligible surrounding contribution.

5.1 Difference Standard Deviation

When a sensor is imaging a perfectly uniform area, it should result in the same value for all the pixels. If asubset of such uniform scene is subtracted from the same subset translated by one pixel to the right, the resultshould be zero, and any deviation may be attributed to sensor noise. The standard deviation of the differenceshould approximate the instrument noise.11 Therefore, taking the standard deviation of the subset difference ofthe black panel should approximate the instrument noise.

Figures 7 and 8 provide the average black panel radiance provided by ARL, along with noise estimated by theabove procedure. The sensor specification quotes that the typical NESR is <20 nW/cm2sr·cm-1. The horizontalline is the specified NESR in the same units as the calculated noise.

Figure 7. 01 May 2013, 11:58 black panel average radiancewith 5.1 noise and provided NESR.

Figure 8. 01 May 2013, 23:58 black panel average radiancewith 5.1 noise and provided NESR.

5.2 Noise Correlation Coefficient Matrix

Depending on operating conditions, the noise associated with an imaging spectrometer can change drastically.For this reason, the noise is often characterized with a specific image acquisition (a dark-field image is usuallytaken before and after each acquisition).11 The noise correlation coefficient matrix, the covariance betweentwo bands normalized by the standard deviations in the two bands, of the dark-field image is often used forvisualization. Figure 9 depicts three correlation coefficient matrices for a difference subset in the black panelfor three times on the same day. The difference subset image was obtained by taking the difference between asubset of the black panel and the same subset translated by one pixel to the right. It could be seen that evenfor a five minute acquisition time difference using the same sensor, the correlation matrices differ slightly.

Figure 9. 01 May 2013, correlation coefficient matrix for 00:13 (left), 12:13 (center) and 12:18 (right)

6. TEMPORAL VARIATION

6.1 Diurnal Data

The temporal variation of eight materials is shown in Figure 10 (Tank0, Gravel, White Panel, Black Panel, Bush,Grass, Near-trees and Rear-trees). For these materials, the average radiance at 10.0681µm for each material asper the masks in Figure 4 was plotted. It could be noted that day and night period are easily distinguishableeven if an x-axis labeling the time of day was not present. During nighttime, all materials have low radianceand are very similar to each other. However, during daytime, select materials heat up and emit higher radiance(notably the tank and the black panel). The aluminum panel behaves as expected, having very small variationfor the course of the day. The slight increase during the middle of the day could be explained by the sky-radiance(the air temperature ranged from -1.9oC to 21oC). An interesting phenomena was observed for the vegetation.The bush and the trees had similar trend during daytime, as one would expect due to them being all vegetation.However, the grass’s trend was closer to the gravel than the vegetation. This could be due to soil effects (whichindicate that each mask is not definitively one material). This was even further observed during nighttime, whenthe radiance got much lower than any other vegetation.

6.2 Standard Deviation Over Time

Even with a uniform material, the spatial variation may be influenced by surrounding materials causing edgeeffects. In order to minimize this influence, a smaller subsection of each material (as per Figure 5) was usedto calculate the radiance standard deviation. Note that the variation observed is a combination of instrumentnoise and spatial variability. Figure 11 shows that the white panel has minimal standard deviation. The truestandard deviation of the white panel is expected to be smaller than what is shown in Figure 11 when we considerthat the instrument noise is also a factor. The black panel follows the white panel, however, unlike the whitepanel, the temperature of the day does affect the standard deviation of the black panel. The material with thegreatest standard deviation are the trees, which is expected since it is the only “moving” material relative to theremaining “stationary” materials. In addition, the trees are not completely solid material. Despite only havinga subset of near-trees and rear-trees, there are “air gaps” that allow “contamination” of other materials as well,which increases the standard deviation.

6.3 Detection

We explored a hypothesis that using a target radiance spectrum at one observation time, the target could bedetected at other times, including different days. This hypothesis was tested using three signature-matchedtarget detection algorithms. Adaptive Coherence/Cosine Estimator (ACE),12 Spectral Angle Mapper (SAM),12

and Spectral Matched Filter (SMF)12 were implemented for two scenarios to detect a target, and the area underthe receiver operating characteristic (ROC) curve was used to quantify the rate of detection. For both scenarios,the average of Tank0 was used as the target spectra and all three tanks were identified as “true” targets. Only 50bands ( 9-10.5µm) out of 165 bands were used for target detection since this subset behaved with more stability.For scenario one, a target spectrum were obtained from a noon cube (11:58) and a midnight cube (23:58) from1 May 2013. They were used to detect the tanks from the same day data cubes. The results are displayed inthe first two rows of Figure 12. For scenario two, the same target spectrum from 1 May were used to detect thetanks on 2 May 2013 data cubes. The results for scenario two are displayed in the last two rows of Figure 12.

If we assume that a threshold higher than 0.8 area under the ROC curve is a good indicator of target detection,SMF seem to produce better results for both noon and midnight target spectra. The percentage of detection ina day for both time-frames by each method is detailed in Table 2.

Table 2. Time index percentage of area under the curve greater than 0.8 (noon/midnight).

ACE SAM SMF1 May 2013 0.0% / 13.3% 0.0% / 32.5% 52.1% / 40.0%2 May 2013 4.2% / 25.6% 0.0% / 35.8% 52.9% / 39.6%

It is also noticeable that the first two rows’ plots in Figure 12 are very similar to the last two rows’ plotsin Figure 12. This indicates that targets could be detected on data spanning multiple days for time frames

Figure 10. Average radiance of eight materials for 24-hrs period using the band corresponding to 10.0681µm.

Figure 11. Standard deviation of eight materials for 24-hrs using the band corresponding to 10.0681µm.

approximately similar to when the target spectra was obtained (given that overall meteorological conditionswere comparable).

7. SPATIAL VARIATION

The spatial variation was investigated using the same materials. We compared the spatial variation of the twopanels, the three tanks and the grass, for two distinct time frames. The mask size for the black panel and the

Figure 12. Noon (11:58) and midnight (23:58) target spectra from 1 May testing ACE, SAM, and SMF algorithms on 1and 2 May 24-hrs data. The area under the ROC curve for the three detection algorithms is plotted against the time ofday. First row: Results for 1 May using the noon spectrum. Second row: Results for 1 May using the midnight spectrum.Third row: Results for 2 May using the noon spectrum. Fourth row: Results for 2 May using the midninght spectrum.

white panel are 8×13 and 7×11 pixels respectively. However, for this analysis, a subset of the mask (6×11 pixelsfor black panel, and 5×9 pixels for white panel) was used to limit adjacency effects. Figure 13 show the spatialvariation of two uniform targets (panels) and two semi-uniform targets (tanks and grass) for midnight time-frame.We expect minimal spatial variation within the two panels. The low variation (0.21 W/m2sr·µm) in the blackpanel cannot totally be considered spatial variation. It is the result of instrument noise as discussed in Section5, combined with environment/adjacent material effects. The adjacency effect is better observed in the whitepanel. There is a larger variation in the white panel (0.35 W/m2sr·µm). This is after we omit the edge pixelscontributing to adjacent material effects (very low emissive aluminum surrounded by large emissive vegetation),yet the adjacency effect was still observed in the lower left of the panel. For a large surface area covering threetanks, the variation is only 0.9 W/m2sr·µm. A larger variation is observed in the grass (4 W/m2sr·µm).

Figure 14 shows the spatial variation of the same targets as for Figure 13 but for noon time frame. Therewas greater variation within the uniform targets compared to Figure 13. This is expected since in reality, thepanels did not heat-up uniformly (the upper portion of the black canvas absorbed more heat). The edge effects

Figure 13. Spatial variation during midnight. Figure 14. Spatial variation during noon.

were more prominent in the white panel where the center of the panel is somewhat uniform while the top pixelsand the left pixels have significant influence from the surrounding vegetation. We would expect greater spatialvariation for the semi-uniform targets during noon since all parts of the tanks will not heat up equally. Thisis seen in lower-left portion of Figure 14, where the variation is 2.4 W/m2sr·µm. Despite best effort, the tankmask does incorporate pixels that could be categorized as other materials due to “air gaps”. As expected, thegrass has the highest variation for this time-frame as well with 4.8 W/m2sr·µm.

The range of variation in the black panel is 0.9 W/m2sr·µm for noon and 0.21 W/m2sr·µm for midnight, whilethe range in the white panel is 0.56 W/m2sr·µm for noon and 0.35 W/m2sr·µm for midnight. This reflects ourintuition that the black black panel heats up with time, but not necessarily uniformly. Whereas the aluminumpanel is almost a perfect reflector and reflects the sky-radiance.

8. SUMMARY AND CONCLUSION

There is an extensive amount of SPICE data available but this paper focused on a subset from a short time periodin 2013. Before performing the characterization studies, we screened the data for obvious artifacts and foundperiodic “erroneous” data cubes corresponding to ones collected immediately following an hourly blackbodycalibration. These cubes were removed from further analysis.

Obvious and expected characteristics were observed in the temporal plots. The highly reflective aluminumpanel remained constant with time, while highly emissive tanks and black panel produced much higher radianceduring daytime. On the other hand, an unexpected phenomenon observed was that the vegetation closer to theground (grass) behaved more similar to gravel than the vegetation surrounded by air and other vegetation. It wasalso noted that conventional target detection methods can be successfully employed to detect targets temporally,as long as a target spectra for similar time-frame are available. For such a case, SMF yielded the best result.

Spatial variability was as expected. There was a higher variation in the black panel during the day thanin the night, while the spatial variation in the white panel was somewhat consistent during the course of onediurnal cycle.

In future, these characteristics could be analyzed for different times of the year to see if they are comparable.In addition, exploring the target detections for winter months would be interesting with the addition of snowand the impact of lowered temperature.

REFERENCES

[1] Rosario, D., Romano, J., and Borel-Donohue, C., “Spectral and Polarimetric Imagery Collection Experiment(SPICE) Longwave Infrared Spectral Dataset.” Army Research Lab Technical Report 7051. 2014.

[2] Rosario, D., Romano, J., and Borel, C., “First observations using spice hyperspectral datasset,” in [Algo-rithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XX ], Velez-Reyes, M.and Kruse, F., eds., Proc. SPIE 9088 (2014).

[3] Rosario, D., Borel, C., and Romano, J., “Against conventional wisdom: Longitudinal inference for patternrecognition in remote sensing,” in [Applied Imagery Pattern Recognition Workshop (AIPR), 2014 IEEE ],1–10 (2014).

[4] Borel, C., Rosario, D., and Romano, J., “Range-invariant anomaly detection applied to imaging fouriertransform spectrometry data,” in [Imaging Spectrometry XVII ], Shen, S. and Lewis, P., eds., Proc. SPIE8515 (2012).

[5] Borel, C., Rosario, D., and Romano, J., “Data processing and temperature-emissivity separation for tower-based imaging Fourier transform spectrometer data,” International Journal of Remote Sensing 36(19-20),4779–4792 (2015).

[6] Rosario, D., Borel, C., and Romano, J., “Solid target spectral variability in lwir,” in [Algorithms and Tech-nologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXII ], Velez-Reyes, M. and Messinger,D., eds., Proc. SPIE 9840 (2016).

[7] Rosario, D., Romano, J., and Borel, C., “Pattern recognition in hyperspectral persistent imaging,” in[Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXI ], Velez-Reyes,M. and Kruse, F., eds., Proc. SPIE 9472 (2015).

[8] Romano, J., Rosario, D., Farley, V., Chenault, D., and Sohr, B., “Spectral and Polarimetric ImageryCollection Experiment.” Army Research Lab Technical Report ARMET-TR-11027. 2011.

[9] Rosario, J. R. D., Farley, V., and Sohr, B., “Spectral imagery collection experiment,” in [Algorithms andTechnologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XVI ], Shen, S. and Lewis, P., eds.,Proc. SPIE 7695 (2010).

[10] “Google maps - targets and tower.” https://www.google.com/maps/place/40%C2%B055’49.

5%22N+74%C2%B035’13.4%22W/@40.930834,-74.6013787,5250m/data=!3m1!1e3!4m5!3m4!1s0x0:

0x0!8m2!3d40.930414!4d-74.587059. Accessed: 2017-01-16.

[11] Schott, J. R., [Remote Sensing: The Image Chain Approach ], Oxford University Press, 198 Madison Avenue,NY, second ed. (2007).

[12] Eismann, M. T., [Hyperspectral Remote Sensing ], SPIE Press, Bellingham, Washington, first ed. (2012).