Hard Sensor Fusion: LIDAR and IR
David HallRick TutwilerJeff Rimland
1
2
Objectives:• Perform pre-processing of hard sensor data• Perform identification, localization, and tracking of people, vehicles and artifacts
• Represent data in OGC-compliant formats for compatibility with fusion processesDoD Benefit:• Extract/represent data that will be fused with soft data
Scientific/Technical Approach• Select sensor suite to emulate military grade sensors
for COIN environment• Begin fusion processing at data level and proceed to
feature and state vector levels• Process both synthetic and real sensor data (and
correlate with soft data opportunities)
Accomplishments• Selected and deployed a sensor suite • Developed 3-D LIDAR and MWIR data level fusion
techniques that out-perform conventional α-blending• Developed algorithms for stereoscopic 3-D estimation
from multiple camera suite• Initiated development of feature level and state vector
level algorithms (including neural net methods for feature detection)
Challenges• Challenging observational environments • Correlation of hard and soft data 2
Penn State UniversityHard Sensor Processing Overview
Anticipated Year‐3 Focus and Option Years
• Year 3: Fusion of hard sensor data: Continue algorithm development and evaluation for fusion of hard sensor data
• Continue development of hard and soft fusion algorithms • Use point cloud data to develop shape descriptors for 3D object recognition• 3D object identification via PCA anomaly detection (level set procedure to compute geometric features of each object)• Fuse bore‐sighted HSI sensor with point cloud dataset
• Extend target tracking and ID beyond image sensors to include acoustic sensors• Develop and evaluate methods for association/correlation of data related to moving targets in complex urban and non‐urban environments (e.g., tracking individuals in crowds and targets in on‐road/off road conditions)
• Develop framework for JDL Level 4 hard/soft resource allocation
•Years 4 and 5: Extend modalities of data fusion, implement Level 4 processes, address distributed end‐to‐end hard and soft fusion
• Automated computer tasking of soft sensing (viz., humans as observing resource for on‐going computational processes)• 3D smart pixel/point vector representation (size, shape, range, material signature) from fused LIDAR/HSI sensor products• Hybrid fusion processing to include data, feature and state‐vector levels
• Utilize manifold representation of smart pixel set• Implement theory of joint manifolds for hybrid fusion
• Introduction of semantic labeling/indexing methods for image & signal data for semantic‐level fusion and context‐based reasoning• Explore hybrid hard/soft data from mobile devices (e.g., dynamic human annotation of hard data with automated geo‐tagging)• Processing of human in the loop experimental data• Design and implementation of an integration and transition environment• Exploration of hybrid environments (e.g., location‐based tasking for observing and cognition)• Develop semantic context‐based scene labeler for fused data
3
Sensor Selection
Selection Criteria• Sensors that are representative of tactically deployed sensors
• Sensors that provide informational “value added” to the inference process on our selected targets
•A suite that can be utilized in real demonstrations and campus‐based experiments
•At least one sensor that allows innovations for the hard sensor processing flow
Selected Sensors• LIDAR• Short‐Wavelength Infra‐red (SWIR)• Long Wavelength Infra‐red (LWIR)• Visual Video•Acoustic Sensors
4
3‐D Po
int C
loud
Noise Processing
Ortho‐rectify 3‐D image to 2D imageFlash LIDAR
Visual(2‐D)
Video
PixelLevelFusion
• Color Intensity Mapping
Range‐Intensity Fusion
PCA Shape Decomposition
• Point Cloud Classification
• 3D Descriptors
XI/DAtt.
2‐D Im
age Processin
g
Feature Extraction
• Ransac• HOG• SIFT• Hough ‐
Visual(2‐D)
•Pattern recognition• Jones & Rehg’sGaussian mixture models Da
ta Associatio
n via multi‐
camera ho
mograph
y&
geom
etry‐based
metho
ds Target
Tracking• K. Filter• P. Filter• LOB
X
Target ID• Neural nets
• Bayes
I/D
Repo
rt (state vector) level Association/Co
rrelation
Repo
rt (state vector) level ta
rget tracking
TML Transform
Hard Sensor Processing Flow
• SWIR•VNIR
•LWIR
Pixel‐Level FusionPoint‐Cloud range estimates mapped to 2‐D camera components
Year 1 Year 2 Year 3+
5
Hard/Soft Tasking Framework
Hard Sensor Processing
Soft Sensor Processing
JDL 4 Tasking(Multi‐agent,
capability‐based)
Hard / Soft
Fusion Engine
Human Tasking
Sensor Tasking
HCI for hybrid processing/cognit
ion
Automated Algorithm Selection
6
LIDAR Image Fusion and Cross‐Modality Representation
Rick Tutwiler
7
Event Summary
• Scenario/Theme: IED Attack / Coordinated Sniper Fire Participants: 21 (PSU+TSU)
• Vehicles: 3• Event Days: 3• Sensors: 9+ Cameras, 1 Flash LIDAR, 2+ KINECT Devices• Mobile Devices: GeoSuite Mobile App on Android tablet• Event/Activity Synchronization: auditory and visual cues
8
Hard Sensor Experiment Description
Observation Post
Building # 2
G1 B1 B2
9
Sensor Locations
IST Camera SuiteLocation: Roof Building 2( Camera 1) Camera Type: Sony HDR‐FX1( Camera 2) Camera Type: Sony HDR‐HC5Location: Insurgent Car Bomber( Camera 3) Camera Type: Panasonic PV‐GS80ARL Camera SuiteLocation: 3rd Floor Building 2( Camera 4) Camera Type: Panasonic MiniDVLocation: Observation Post( Camera 5 Bonnie Stereo Pair) Camera Type: Cannon HV40( Camera 6 Clyde Stereo Pair) Camera Type: Cannon HV40( Camera 7 Dylan) Camera Type: Cannon HV40( Camera 8 Mounted Suite ) Camera Type: Usoria ½” CMOS color camera( Camera 9 Mounted Suite ) Camera Type: ASC Flash Lidar
Observation Post
Building # 2
Camera 8Camera 9 Camera 7
Camera 5
Camera 1
Camera 2
Camera 3
Camera 6
Camera 4
• Action is monitored by three cameras • Two cameras have overlapping field of views, and are calibrated as a stereo pair• Calibration involves;
• Computing the extrinsic camera parameters & determining if the translations (physical offsets between the cameras on the ground‐plane) match up with distances computed from these field measurements.
• Accounting for camera positional changes ‐ since the extrinsic parameters will be with respect to one of the cameras imaging axis, and not necessarily the orthogonal axes shown in this drawing, use the absolute distance for this “reality check”
10
Example: Coordinated Text Message Generation
Urban Action Scene
Time:
Total Time: 1560 seconds = 26 min.
Definitions:
B(i,j) = Red Team, j = Team Member, i = Vehicle Number
C (k, l) = Blue Team l = Team Member, k = Vehicle Number/Group Association
G(i,j) = Green Team j = Team Member, i = Vehicle Number
G(1,1) = Dignitary
G(1,2) = Body Guard
P(m,n) = Green Team n = Team Member, m = Group Association
Pre-Event Messages
1. 01/24/2010: U.S. Embassy in Baghdad issues report stating: “The Iraqi Election Council (IEC) releases list of banned candidates from March’s upcoming Iraqi Parliamentary Election. The banned candidates are composed of mostly Sunni-Iraqis with ties to Saddam Hussein’s regime. Sectarian tensions are expected to rise in coming weeks as the election approaches.”
2. 01/25/2010: In response to U.S. Embassy Report of 01/24/2010, U.S. Force commanders conduct assessment of probable hot-spots and order increased patrol coverage, additional checkpoints at Sunni-Shi’a fault lines in Greater Baghdad and step up communications monitoring of political party headquarters and key politicians.
3. 01/27/2010: U.S. analysts monitoring websites of Iraqi political parties report that Mohammad Omar Salah, a former Saddam Hussein aid and the current leader of Sunni political party, The National Front for Iraqi Dialogue (NFID), has called for a protest against the Iraqi Government at Al-Anbia Mosque in Adhamiya on 1/31/2010, following mid-day prayers.
4. 01/28/2010: Communication monitoring of NFID reveals call for “strong” presence at planned 31 Jan protest of ICE, stemming from “Banned Candidate List.”
5. 01/28/2010: Analysts assess call for “strong” presence at Al-Anbia Mosque in Adhamiya by NFID slated for 1/31/2010 is a likely call for violence. 11
Example: Hard Sensor Data
12
3D LIDAR Images
• Short‐wave Infrared (SWIR)– Intensity– Range
• Representation?– 2D or 3D?– Orthorectified?– Range or Intensity?
Range
SWIR
OrthorectifiedRaw
13
2D/3D Fusion Concept
• Dimension Reduction– High Dimensional Information
– Low Dimensional Representation
• Choice of Representation– Max information content– Color, Intensity, and Saturation are orthogonal image measures
• Define sliding scales in each dimension (Intensity and Hue Scales shown)
• Assign the scales to a physical quantity to encode information in RGB
HUEINTENSITY
INTENSITY HUE
14
• R = Range Image– Orthorectified– Registered to thermal image
• I = Video Image• MR = Hue map• MI = Intensity Map• F = Fused image
3:)( pxpRN RMR ℜ→ℜ=
pkIIR ℜ∈,,
ppIk IMI ℜ→ℜ= :)(
Nkkkkn RIIIF o][, =
Discretize Range into N color bins. The three columns refer to Red, Green, and Blue components, respectively
For p pixels
Discretize Thermal intensity into k bins in the range [0,1]
Direct multiply the intensity values to scale colors in RN 15
2D/3D Fusion Algorithm
Example: Fusion Results
• The results of mapping thermal data to brightness and range to hue• Data can be artificially rotated to get a further sense of the 3D
context• Thermal features stand out and can be spatially referenced
16
Example Fusion Results:Mounted Suite
VNIR
FLASH LIDAR
17
Lidar Range Image' Frame Number:1
100 200 300 400
100
200
300
400
500
600 20
30
40
50
60
70
80
90
100
110
120
130Lidar Intensity Image' Frame Number:1
100 200 300 400
100
200
300
400
500
600
0
500
1000
1500
2000
2500
3000RGB CompositeImage' Frame Number:1
100 200 300 400
100
200
300
400
500
600
Example Fusion Results:Alignment and Registration
18
Example Fusion Results:Fused Lidar and VNIR Video
19
Example Fusion Results:Mounted Suite
VNIR
FLASH LIDAR
2020
Example Fusion Results:Fused Lidar and VNIR Video
21
Hotter colors designate nearer objects in this range image.
Hotter colors designate stronger light reflection.
Example Fusion Results:Persistent Surveillance
22
2D/3D Fusion Approach
– Multi‐Camera Handoff for Occlusion Detection• Joint Manifold representation of Composite Track Files
– 2D/3D Fusion• Can Multiple Platforms be Fused using the same process
– Lidar/VNIR» Range/3D‐2D Shape/Color
– Lidar/VNIR/MWIR/LWIR» Range/3D‐2D Shape/Color/∆T
– Lidar/HSI» Range/3D‐2D Shape/Spectral Signatures Material
– Joint Manifold Fusion• Can Manifold Representation of Composite/Disparate Sensor Suite’s be utilized to Semantically Label Scene Attributes
• Can the Joint Manifold Relationship be utilized to ingest both Hard and Soft Data Sets
23
• Since we know how spatial sample rate changes as a function of range we can define a spherical region centered on every sample in the cloud which is a constant number of sample intervals in diameter.
• This means that the spherical regions are large in the “back” of the cloud and smaller in the “front” of the data.
• What this gives us is a relatively consistent sample population in each neighborhood*
Decomposition of the Image into Shape Structural Classes• We then perform PCA on each sample population.
• Since the sample populations are rarely perfect geometric arrangements we threshold the singular values.
• What is returned is a local shape classification for each sample in the cloud.
• We can subdivide the cloud according to these classes.
• We can compute this for different sized neighborhoods (where size is specified in sample intervals)
These are our shape classes:
24
Image Shape Decomposition into Structural Classes
Multi‐Resolution Shape‐Class Flow
• Sometimes a particular sample in the cloud will change classes as neighborhood sizes are changes.
• As in the Plane to Volume, this can happen when the neighborhood grows to encompass a small object. (lots of other examples)
• What Can this tell us about the geometry around the sample?
25
Multi‐Resolution Shape‐Class Flow
26
Implementation of Hard and Soft Data Collection, Processing and
Interchange
Jeff Rimland
27
Infrastructure & Transition Approach
• Challenges– Multi‐faculty, multi‐performer
research environment– Heterogeneous software
environment– Diversity of information processing
types (text, signals, images, etc.)– Challenging knowledge
representation– Transition to U. S. Army
environment
• Approach– Utilize Service Oriented
Architecture (SOA/ESB)• Standardized TML for report‐level
data representation/interchange• ROS for inter‐process
communication and efficient hardware utilization
– Create integration/evaluation environment to provide distributed & federated access
– Utilize OGC standards and best practices for data representation
– Integrate with the GeoSuite analysis and mobile applications for HCI
– Utilize multi‐agent approach to JDL level 4 tasking.
28
GeoSuite Integration
ROS/TML Architecture
29
GeoSuite Integration
Web Tools
Mobile App
Extreme Events Lab
• SYNCOIN DATA• Ontologies
SSL
ROS Nodes
• Top‐level nodes communicate via TML for collaboration, mobile access, or additional processing capability
• Mid‐level nodes can communicate with lower‐level nodes as well as process data and create/transmit TML over Wi‐Fi or 3G/4G.
• Some nodes communicate only via low‐level protocols (e.g. UDP) or analog (e.g. 900mhz, 2.4ghz)
TML SOA
Uses IPC to leverage distributed processing power of mobile sensing devices
Local Video I/O
Local Sensor I/O
Integration / Transition via GeoSuite
• Civilian version of General Dynamics Command Post of the Future (CPOF)
• Enables shared, distributed collaboration
• Web and mobile (Android) apps for remote data entry
• Advanced geo‐spatial search and analysis capabilities
30
Transducer Markup Language (TML)
• TML provides the ideal protocol for transmitting feature‐level sensor data with metadata.
• OGC standard
• Transition‐compatible via GeoSuite, FEF, etc.
• Ideal for transfer over SOA/ESB
• Demonstrated utility for hybrid simulated/actual sensor data
31
Hard/Soft Tasking Framework
Hard Sensor Processing
Soft Sensor Processing
JDL 4 Tasking(Multi‐agent,
capability‐based)
Hard / Soft
Fusion Engine
Human Tasking
Sensor Tasking
HCI for hybrid processing/cognit
ion
Automated Algorithm Selection
32
Multi‐agent approach
Critic‐adjudicated, agent‐based hybrid sensing / hybrid cognition model
Hybrid‐agent architecture
• System maintains ontology of human/machine analyst/observer capabilities and availability.
• Enterprise Service Bus (ESB) facilitates secure distributed tasking, alerting, and solicitation for data, analysis, or adjudication.
• A task may require multiple iterations through human/machine analysts/observers.
Visualization / HCI methods support human cognition
33
Anticipated Year‐3 Focus and Option Years
• Year 3: Fusion of hard sensor data: Continue algorithm development and evaluation for fusion of hard sensor data
• Continue development of hard and soft fusion algorithms • Use point cloud data to develop shape descriptors for 3D object recognition• 3D object identification via PCA anomaly detection (level set procedure to compute geometric features of each object)• Fuse bore‐sighted HSI sensor with point cloud dataset
• Extend target tracking and ID beyond image sensors to include acoustic sensors• Develop and evaluate methods for association/correlation of data related to moving targets in complex urban and non‐urban environments (e.g., tracking individuals in crowds and targets in on‐road/off road conditions)
• Develop framework for JDL Level 4 hard/soft resource allocation
•Years 4 and 5: Extend modalities of data fusion, implement Level 4 processes, address distributed end‐to‐end hard and soft fusion
• Automated computer tasking of soft sensing (viz., humans as observing resource for on‐going computational processes)• 3D smart pixel/point vector representation (size, shape, range, material signature) from fused LIDAR/HSI sensor products• Hybrid fusion processing to include data, feature and state‐vector levels
• Utilize manifold representation of smart pixel set• Implement theory of joint manifolds for hybrid fusion
• Introduction of semantic labeling/indexing methods for image & signal data for semantic‐level fusion and context‐based reasoning• Explore hybrid hard/soft data from mobile devices (e.g., dynamic human annotation of hard data with automated geo‐tagging)• Processing of human in the loop experimental data• Design and implementation of an integration and transition environment• Exploration of hybrid environments (e.g., location‐based tasking for observing and cognition)• Develop semantic context‐based scene labeler for fused data
34