Development of an Experimental PrototypeMulti-Modal Netted Sensor Fence forHomeland Defense and Border Integrity

Weiqun Shi, Gus Arabadjis, Brett Bishop, Peter Hill, and Rich Plasse, The MITRE Corporation,Bedford, MA, USA, wshiomitre.org

methodology for performing key 24/7 sentry functions thatAbstract- Potential terrorists/adversaries can exploit a can be used for the protection and the surveillance of critical

wide range of airborne threats against civilian and military infrastructure from airborne threats. The methodology istargets. Currently there is no effective, low-cost solution to based on joint multi-sensor fusion technology. It consists of arobustly and reliably detect and identify small, low-flying forward-based fence comprised of a mixture of selected lowairborne vehicles such as fixed-wing aircraft or unmanned cost, low power, netted sensors including a simple radar,aerial vehicles (UA Vs) that might be carrying out chemical, acoustic microphone array and optical (Infrared and visible)biological or nuclear attacks, or smuggling drugs or illegal cameras to detect, track and discriminate potential airborneimmigrants across the border. This paper presents a low-cost targets. Our early approach [1] studies the concept ofand low-power methodolog for performing key 24/7 sentry operation, assessment of detection and detectability for eachfunctions that can be used for the protection and the modality, and the target signature phenomenology. In thesurveillance of critical infrastructure from airborne threats. '

fuin present work, we focus on the development of an experimentalThe methodology is based on joint multi-sensor prototype end-to-end proof of concept system with deployabletechnology. It consists ofaforward-basedfence comprised of software, hardware and connectivity.a mixture of selected low cost, low power, netted sensors

including a simple radar, acoustic microphone array andoptical (Infrared and visible) cameras to detect, track and The rest of this paper is organized as follows. In section II,discriminate potential airborne targets. An experimental we introducethe general system concept, the prototype sensorprototype end-to-end proof of concept system with deployable system components and configuration. In section III wesoftware, hardware and connectivity is built using COTS describe the design and development of a real time, multi-component. Multi-modal senor fusion algorithms employing modal kinematic tracker employing Kalman filter. The trackerKalman filter for target tracking and acoustic and image is used to fuse asynchronous radar and acoustic data to predictrecognition algorithm for target classification are kinematic properties and the flight trajectory of the airborneimplemented Results from field tests reveal reasonable target, and then cue a camera to aim and photograph the target.detection and discrimination among candidate aircraft. The resulted target images, combined with information from

the target kinematic properties, target acoustic signature, arethen used to achieve a final target classification. We presentthe classification work flow, procedure and results in section

I. INTRODUCTION IV, followed by a brief conclusion.potential terrorists/adversaries can exploit a wide range of

airborne vehicles to effectively deliver weapons (nuclear, II. SYSTEM CONCEPT AND COMPONENTchemical and biological) against civilian and military targets.Candidate airborne threats may include small, light weight The forward-based multi-modal netted sensor fence systemtargets such as small civilian aircraft that tend to fly low to and associated sensors is designed to be low cost, low power,elude conventional search radar detection. The large volume and portable. A conceptual illustration of the multi-modalof airspace that needs to be surveyed and the vast number of netted sensor fence is shown in Figure 1. The primarypotential targets in the US makes the detection and detection component in the system is a radar fence positioneddiscrimination of such airborne threats a key air defense and remotely from a central location. The radar fence is designedsurveillance challenge. Existing airborne and ground-based to detect approaching targets and provide a cue to the acousticsystems are inadequate due to economical and technical and infrared sensors that perform the discrimination task. Thechallenging. radar fence consists of multiple, low power radar with a radar-

In this paper we present a low-cost and low-power to-radar separation of approximately 4 km. The radarsmeasure target range and report these values to a centralprocessing station that cues the acoustic and infrared sensors

1-4244-1053-3/07/$25.OO ©2007 IEEE. 221

(if a target report is issued), and then fuses the reports from all rectangularly-arranged acoustic array (B&K microphones),sensors (radars, acoustic, etc.) to form a target track and alert and a mini-computer (Slim Pro PC) that performs target rangerear-area weapons systems or potential interceptors so that and angle detection and reports those results to the centraldefensive action can be taken. node. A first order classification based on target acoustic

signature is also performed at each remote node and the resultAcoustic microphone arrays are used as the second sensor is reported back to the central node. The central node contains

modality in the system to detect broadband acoustic emissions an JR camera (uncooled BAE Micro JR sensitive to the 8-12from approaching targets. Acoustic sensors are non-line-of- um waveband), a Pelco pan and tilt controller device mountedsight, passive, low-cost, and portable sensors that can be on a small tower, and the central computer that performs dataeffectively deployed in wide areas. Primary objectives of fusion and final target classification. The connectivity isacoustic sensors in this sensor fence system are: 1) to provide provided through a simple point-to-point 802.11 wirelesstarget direction of arrival (DOA) estimates that will then be communications network consisting of signal boosters andfused with radar measurements to form a target track; 2) to omni-directional antenna located at the remote node and aprovide a means for target identification and classification; Yagi-type directional antenna located at the central node. Thisand 3) to mitigate false alarms. The system is designed to modular and compact system allows for rapid and inexpensivecontain several equally spaced, diagonally-arranged production of nodes and rapid deployment of the netted sensormicrophone arrays. fence system.

The third sensor modality in the fence is an optical systemwhich is cued by the radar and/or acoustic sensors and slewsin angle to acquire track and identify the potential airbornethreat. The system is designed to contain an uncooled infrareddetector sensitive to the 8-12 um waveband to provide day andnight time operation. The uncooled JR detector array usesonly several Watts of power. A boresighted visible camera isalso used for improved target resolution during the daytime.Visible cameras are inexpensive and have improved resolutioncompared to the infrared detector array. Figure 2, Remote node (left) and the central node (right)

..................................... ............X-Band....Horn...Antenna.... 4-BelementAtenacoustict aouarrayraat............

......................................Nobel.e.....reakout-...............................................N....(T dpod Mount)Iadar Bean............... ..t.ae.Bx.................................................. .........M ini...PC

.............................. ..................... ........................ -- ----------------------~ ~~~~thene

..........................Char e Comm.............................R gulator... Powe.Networ

Cooling~~~~~~............................

~~~~ R mot N de En lo urFan...

NobeiTe rea~~raer CT

'-pe"kh"tr seam~~~ ~ ~ ~ ~ ~ ~ ~ ~ ~~~~~~~~~Ntwr Digtier.onroler Caer~~~~~~~ ~ ~~~~ ~ R

~~~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ CetalP Itrfc Plo Z........arget.WLAN.Sation......Cnverter.R.4.

Classiicatio.Centrl.Node....Eternet...........~t

IR&Visible~~~~~~~~~~~~~~~~~~~~~.... Coordinates...Figure.3,.System..hardware..diagramFigure1, A conceptual illustration of the multi-modal netted sensor fence. III. MULTI-MODAL.......................KINEMATIC...TRACKER.......A.majr.challenge.of.the.projet.has.been.the.design.anAnexperimental prototype consisting of three remote development of sensor fusion.........techniques..................which..combine...data.sensornodes and a central processing node has been from multi-modality and multi-sensor nodes to.....................................achieve..developed and built using COTs components. Figure 2 and 3 improved accuracy in target detection, tracking,..........................andshowthe photos of the actual system and the system hardware classification. A multi-modal kinematic tracker....employing.adiagram, respectively. Each remote sensor node contains a Kalman filter [2, 3] has been implemented.......in.the.prototype

lowcost, low-power range only radar sensor (NobelTec 1R2 system. The objective.. of................................the...tracker....is..to..fuse..asynchronous....

radar and acoustic data to predict kinematic properties such as x(t)=[x(t) y(t) z(t) (t) (t) (t)]the location, the speed, the heading, and the flight trajectory ofthe target. This prediction is then used to automatically aim a Let Z, (t) be the measurement vector from remote node i atcamera to the predicated point and photograph the target. time t and Ri the corresponding covariance matrix. For the

A. Tracker requirements, Functional Capability, and radar range measurement,Restrictions Z, (t) = [r, (t)], R, =CR]The tracker is located at the central node. The remote nodes Where r, (t) is the range measurement value from remote

transmit detections of an aircraft target (time, range, azimuth,* ~~~~~~~~~~~~~nodei and a is the standard deviation of the covarianceelevation) to the central node. Each transmitted signal packet R

contains all the detections (there may be none) accumulated matrix. In parallel, for the acoustic angle measurementsince the previous transmission. The data is time-stamped at AzFA; (t)] _ o]each remote site with the time at which the signal arrives to l-L(t)] R - L 0the sensor. For such multi-modal sensor system the tracker W t a th a a EI

must be designed to accommodate asynchronous data streams l lfrom multiple remote sites. The tracker is also required to angle. Due to significant propagation time differences betweenallow for an arbitrary number of remote sites, remote site the radar and the acoustic data (e.g. at a range of 4 km the

in the data, the correction acoustic sensor data corresponds to a point on the flight pathdorpropo atdriong run, fal rorts that is 4000/340 = 12 sec in the past, while the radar data isfor propagation tme delay of the acoustic sgl and the virtually instantaneous), it is necessary to correct the

prsobedict oferatgth trget ltionminethe.future.Thetrackerms timestamp of the acoustic sensor data to correspond to thetime the signal left the target.

These requirements in turn imply that the tracker must have Initializethe following functional capabilities: T .

* Track initialization, i.e., the capability to start a Nodtrack from the data R,

* Rejection gates to eliminate false reports N Initial Propagation Time

* Data association, i.e., the capability to associate anew report with a track for track updating t3As, Az3, El3

* Kalman filter remote site data fusion for minimumvariance state estimate

* Dynamic status tables identifying which sites areactive or inactive

Time management ' coordination ' to allow Wait No Data YsGet Next* Time management and coordination logic to allow for More Present Measure- Kalman FilterData From All ment Processfor asynchronous data streams and propagation Nodes9./

time delays of acoustic data.Figure 4 Tracker processing flow for the input to Kalman filter.

The development of such a complex tracker involvesconsiderable effort. In order to keep the tracker development C. Acoustic time Delay Correctioneffort within manageable bounds, several key restrictions and Assume at time t, r [x y z] is the position andsimplifications were imposed on the tracker capability. These

i

include a single target track, and a flat earth model assumption r j z] is the velocity of the target. The acousticmeasurement received at time t should correspond to an earlier

B. Tracker Processing Flowtime to at which the position of the target is r0, as shown in

The multi-modal kinematic tracker employs a Kalman filterto update the data measurement. Figure 4 shows the tracker Figure 5 . For a target traveling at a constant speed v (O<v<c,processing flow for the data input to the Kalman filter. At where c is the spend of sound in the air) between locations of

each cycle, the tracker corrects the acoustic data timestamp for ro and r, through simple geometrical derivations, it is easy topropagation delay, attempts to initialize a track if none exists, obtain,performs the association function, discards false reports, and ca2+ C2-V2)r2- acperforms the Kalman filter update of the time sequenced data. ro = C2 v2 0 < v < c

WhereAssume at time t the state vector of the track is a six

dimensional vector of target position (x(t), y(t), z(t)) and v x 2+2=r vvelocity (x(t),yj(t),z±(t)) in Cartesian coordinates relative Anto an east-north-up topocentric coordinate system with origin a = rr = xx + .yyi +zzat the central node,

223

timet between the measurement and the expected target location(r, Az,El=timeto t - rolc begins by creating an expected state vector for time t where tis the time of the measurement, and then convert the expected

state vector to the measurement vector. The differencesbetween the measurement and the expected value is denoted

Lsensor as, £= Z(t) -Z(t), where Z(t) is the measurement at time t,Figure 5, Diagram of acoustic travel time correction and z(t) is the expected measurement converted from the

D. Tracker Initialization expected state vector. A set of typical range, azimuth and

The Kalman filter is a recursive algorithm which starts with an elevation gate GR, Gang is defined in Table 1. The errorestimated state vector based on past data and updates it with & must be within those gates in order for a data measurementnew data. In order to start the algorithm it is necessary to have to associate.an initial estimate of the state. This is done by generating aninitial guess of a constant-velocity path via a least-squares fit Table 1. Association Gatesto the batch of collected data. This method allows us to RangeGate G =150mseparate a reasonable collection of false alarms from real data Rwithout relying on prior knowledge of the target's position. Angle Gate G = 20'The least-square fit minimizes the following objectivefunction for a measurement Z, that has an expected value of Ei,

(El Z )2 F. Kalman Filter Data Fusion- 2 Kalman filter is employed for data update and predictions. It is

This formula is modified slightly for acoustic measurements in assumed that the flight path is a constant velocity plus a

that it combines azimuth and elevation measurements into a Gaussian white noise acceleration term (plant noise). This

single term using the law of cosines to calculate the great implies that the state obeys the linear difference equationcircle distance on a unit circle: x(t + r) = FD(r)x(t) + w(t, t + ),

Where @D(r) is the transition matrix,Ei -Zi = arccos(sin(E/l) sin(EE/) + cos(El ) cos(EEI) cos(EAZ - Azi)) 1 0 0 r 0 0

While minimizing Q is the core function ofthe initialization 0 1 0 0 z 0tracker, it is not nearly adequate for consistently producing O(Dl) _ O 1 0 0 zreliable tracks. Steps must be taken to both minimize the 0 0 0 1 0 0number of false tracks and maximize the chances of 0 0 0 0 1 0generating acceptable tracks when real targets are present.These steps include windowing, setting a minimum number of 0 0 0 0 0 1data points, placing bounds on target speed for acceptable and w(t, t + r) is a random zero mean Gaussian plant noisetracks, discarding outliers, selecting the best of multipleindependently generated tracks, and setting maximum values qp2c3/3 0c q2 o a2/2a 0for Q such that a track is still valid. rOq2333 3 0 q r 2j2/0 0

0 q2r3J3 0 0 qr2z2/2 0E. Data Association 0 0 q23/3 0 0 q2T2/2

Data arrives to the tracker from various remote nodes. The q2r2/2 0 0 q2r 0 0data from each node consists of either radar range data alone, 0 q2 '2/2 0 0 q2r 0acoustic sensor azimuth and elevation data alone, or both radar 0 0 q222/2 ° ° q2Tand acoustic sensor data. At each time point the acoustic Where q. , , q, are plant noise intensities in x, y, z directions.sensor outputs only one detection (or possibly none) and thisprobably corresponds to the loudest source. By contrast the Let i(t t0) be the Kalman filter optimum estimate ofthe stateradar can output any number of detections at each time pointas many as cross the detection threshold. The tracker must x(t) at time t based on data taken up to and including time to,allow that many of these detections could be false alarms, P(t to) is the covariance matrix of the errors in this estimate ofarising from random noise and clutter. They may also be the state, the recursive Kalman filter estimate of the updateddetections of other real targets within the range of the sensors.The association process is an attempt to weed out the tracker from time to to time t = + T is thus written,irrelevant detections so as not to corrupt the updating of the x(tlt)= 4D(t - t0)i(t0)track. P(t to ) = 1)(t - to )P(to )q)T (t - to ) + Q(t - to)

This association is performed by comparing each new Transform the predicted track i(t t0) to the measurement themeasurement to some previously generated expectation of thetarget's location. This previous expectation is generally the measurement variables 2, (t t0)n Form the innovation, which isresult of the most recent tracker output. The comparison

224

the difference between the actual measurement z, (t) and the via comparisons of the tracker results with the ground truthpredicted measurement t -2 't t

recorded by an on board GPS. In general the tracker resultsZT the ti show good agreement with the GPS ground truth data.

The covariance matrix ofthe innovation is

Si (ttj) = Hi (t)P(tHT)(t)+TRWhere H11(t) is the matrix of partial derivatives of themeasurement variables of node i with respect to the statevariables at time t. Thus for the radar measurement

Hi_)-a' t____ (t) ap,(t) 0 0 OJax y az]

and for the acoustic measurement(t) _Az _t_aA;. 0_

a( a.y , zHIirt E t aEl (t) attEl,(t) 0 0

L~~~~ 0 01

a~~~~~~~~~~~~~~~~~~sSXrWWMx ay i e

At this point the tracker makes a test to assure that thelgtPt

innovation is consistent with its covariance matrix. The Flight PathMahalonobis distance d between the measurement and itspredicted value is

d = ,kT (t t )Sy' (t t, )2,k(tt0).Where d is a chi-squared distributed variable with n Figure 6. Sensor configuration and layout in the field test.

(radar) or n 2 (acoustic sensor) degrees of freedom. If I tI .vsI n v Tmd <T for a given threshold T, then the data is accepted. If......

d>Tthen the Mahalonobis distance is too large and the.......data is rejected. The threshold has been chosen at the 100%level, i.e., the probability that d is larger than T is one in ten. 0 4L ------- -- ---- -The value of the threshold is obtained from tables of the chi- . .squared distribution, as follows 4

[2.7055 n= 1 (radar) 1g152 A3 a IO 30 40*04.6052 n =2 (acoustic sensor)

The Kalman filter then updates the state and the associated -0----------------3--------------------------covariance matrix, 0 Azimuth vs Time Elevation vs. Time

'.\~~~~~~~~~~.........-------£(t)=i(t~to)+K, (t~toj)k(tto) X~~~~~~~~~~~~~4 ..*-.*... L.... E

/ \ / / I \ / \\ / I \ W i'''j'I~~~~~~~~~~~~~N\(t (I KI (t to)I\(t/Pt t oi

Where K (t tj) is the Kalman gain matrix given by,i V | ° J g g Y | ~~~~~~~~~~~~NIl . l ml M' 4a ..K1 (t"t) = p(t{to)HT (t)S 1'(ttj)

C. Tracker Field Test Figure 7 Comparisons oftracker results with the ground truth GPS recordingas a function of time including plots of. the target position (upper left, note:

Field tests of the netted sensor fence have been conducted at GPS recordings are denoted in circle dots, whereas the tracker results areNashua Municipal Airport, Nashua, New Hampshire. Typical denoted by linesn); the range (upper right); the azimuth (lower left), and theNashua

,Muiia ipr, Naha Ne Haphie Tyia elvtoloe gtexperiment layout and sensor array positions are shown in elevation (lower right)

Figure 6. The sensor suite is positioned near the end of the IV. TARGET CLASSIFICATIONrunway. The test aircraft are flying at a flight test matrix withmultiple combinations of altitude and engine RPM. GPS data Target classification is performed as part of the sensor

recording systems are mounted on the aircraft so the ground fusion. Once the target track is established from fusing thetruth information can be transmitted in real time to the central radar range detections and the acoustic angle detections, thenode for target validation. The target aircraft used in one of kinematic properties of the approaching targets such as targetthe most recent tests was a Beech BE-76 Duchess. The remote velocity, range and location can be extracted from the trackersensor nodes node 1 & 2 were placed at the end of the runway to give an initial classification of target types. This trackerso that planes taking off and those flying parallel to the result is also used to automatically aim a camera to therunway would cross the fence. Due to space constraints, the predicated point and photograph the target. Acousticremote nodes were placed in a T configuration with spacing of measurements can be used to further divide the target groupsapproximately 200 meters. The central node was collocated based on Harmonic Line Association (HILA) method bywith one of the remote nodes (remote node 3) at the base of extracting a set of feature vectors from acoustic spectrogramsthe T. Figure 7 shows plots depicting the tracker performance adcmaigte gis h cutctre aaae[]

225

Therefore targets such as typical false alarms (e.g., birds, confidence level of the classification results.. An imageducks, etc.), propeller driven aircraft (civilian small aircraft), classification example is shown in Figure 9. In this case, ahelicopters, and jets can be classified. twin engine Beech BE-76 Duchess was correctly identified

..................dPERSPECTIVEANGLE 3D MODEL PROJECTIONCALCULATED FROM .J..TRACKER OUTPUT i:S

JR MAe frI IR Image fra|1~~~~~~W 31 dd. dd0 w d m m age

PREDICTED DIAGE

IRIMAGE DATA i__AFTER, 100

PREPROCESSNG 1GBMAIENGi,

USING MOMENT "E E .INVARTANTMETHOD fiNDNEARESTNEIGEBORCLASSIIER

4

ai. 2Wi S g U 1ItOU 10 301 400 500 E,0Figure 8, Processing flow ofthe target image classification

Figure 9, An image classification example shows a pair of collected IRimage frames (upper), the extracted target silhouette (lower left) and the

A further classification can be achieved via image classification result (lower right).classification. Figure 8 depicts the processing flow of theimage classification. In order to compare the candidate target V. CONCLUSIONtemplates to the observed images, the corresponding three-dimensional models must be projected into a two-dimensional Small, low-flying airborne vehicles may pose an imminentimage plane with the appropriate azimuth and elevation angles threat to homeland security and border integrity. Using acorresponding to the camera viewing angles. Such azimuth forward-based fence that contains a mix of low cost, lowand angle information can be determined from the output of power radar, acoustic and optical (Infrared and visible) sensorsthe kinematic tracker using simple geometrical translation. by appropriate sensor fusion methodologies it is feasible toAfter 2-D image projection, the Moment Invariant method is detect, track and discriminate small, low flying airbornethen applied to extract a set of numerical attributes - the targets and provide 24/7 sentry functions to protect criticalmoment feature vectors which uniquely characterize the shape civilian and military infrastructure. We have demonstrated the

of an object and yet have the desired property of invariance technical feasibility of the netted sensor fence approach. Aproof-of-concept initial experimental prototype has been built

under imagetanslatinandrtatiand tested using COTs components. The technology is highlymodular by modality, and adaptable to potential customerThe mathematical foundation of Moment Invariants for needs and requirements.

two-dimensional shape recognition was first introduced by Hu[5], in which a set of shape descriptor values were computedfrom central moments through order three that are independent REFERENCESto object translation, scale and orientation. Translationinvariance is achieved by computing moments that are [1] Weiqun Shi, Ronald Fante, John Yoder, Gregory Crawford, "Multi-

Modal Netted Sensor for Homeland Security", Proceedings of SPIE Vol.normalized with respect to the centre of gravty so that the 5796, p416-427, 2005centre of mass of the distribution is at the origin (central [2] Anderson, Brian D.O. and Moore, John B., Optimal Filtering, Prentice-moments). Size invariant moments are derived from Hall, Inc., Englewood Cliffs, NJ, 1979.introducing a simple size normalization factor. From the [3] Grewal, Mohinder S., and Andrews, Angus P., Kalman Filtering: Theory

and Practice Using MATLAB, John Wiley & Sons, Inc., 2001.second and third order values of the normalized central [4] S. A. Dudani, K. J. Breeding, and R. B Mcghee, "Aircraft Identificationmoments a set of invariant moments can be computed which by Moments Invariants", IEEE Transactions on Computers, Vol. C-26,are independent of rotation. In our investigation six invariant No. 1 1977

[5] M. K. Hu, "Visual Pattern Recognition by Moment Invariants", IREmoment functions that appear to be suitable for the present Trans. Inform. Theory, Vol. IT-8, 179-187, Feb. 1962.problem are selected. Readers can refer to detailedmathematical representations of the invariant moment vectorsin previous publications [1, 4].

A nearest neighbor classifier is then applied to train theextracted features and the final classification results can thenbe obtained. Euclidean distances between the moment vectorsextracted from the observed images and those from thesuspected 3D numerical models can be used to measure the

226