47
On Biometrics Aly A. Farag University of Louisville Department of Electrical & Computer Engineering CVIP Laboratory August 26, 2008
On Biometrics
Aly A. FaragUniversity of Louisville
Department of Electrical & Computer Engineering CVIP Laboratory
August 26, 2008
Outline• Sensors
o Camerao LIDARo LADARo Scanners and Flasho Reflective and Thermal IR Imagery
Human Signatureso Biometricso Reflective Signatures (Video + Near IR)o Thermal Signatures (Thermal IR)
CVIP Lab Setupso 3D Reconstruction Setupo CardEye
• 3D Reconstruction Techniqueso Stereoo SFSo Space Carvingo SFM
Sensors
Non-Optical
ScannersFlash
A Taxonomy
of Range Imaging Sensors:
Shape From-X
Space Carving ShadingStereo
Shape Acquisition
Optical
Active Passive
Imaging Radars Triangulation
Range Imaging SensorA
combination of hardware and software elements capable to produce a depth image of a world scene:
• Resolution: Smallest change in depth that sensor can report
• Repeatability: Statistical variations among repeated measurements of the exact
same distance over time
• Accuracy: Statistical variations among repeated measurements of known value
19th century studio camera
Camera obscura
Kodak No. 2 Brownie box camera, circa 1910
Light Behavior through Pinhole
Biometrics:2D face detection & recognition3D To reconstruct the face from:
Shading (1 view + 1 image) Photometric Stereo (1 view + N images)Stereo (2 views+ 2 images)Space Carving ( N views + N Images)Motion (N views + N Images)
CCD “Charged Coupled Device”: A photoelectronic imaging device (1.5 cm square).It has at least 250,000 individual light-sensitive picture elements (pixels). Each pixel, less than 0.03 mm in size, is capable of storing charges created by absorption of light.Camera Calibration: is the process of estimating two sets of parameters: the intrinsic parameters and the extrinsic parameters.
yw
x
w
z
w
O
Pw
[R t ]K
First accurate description was in early 11th century Egypt “Ibn Al-Haitham”
Ibn Al-Haitham established that “Light travels in a straight line and when some of the rays reflected from a bright subject pass through the small hole in thin material they do not scatter but cross and reform as an upside down image on a flat white surface held parallel to the hole. The smaller is the hole the clearer is the picture”
First permanent photograph was in Paris 1826 by Niépce using a sliding wooden box camera.
The use of photographic film was pioneered by Eastman 1885 , His first camera, the "Kodak”
Camera
LIDAR is a "radar" system utilizing electromagnetic radiation at optical frequencies.
LIDAR operate in the ultraviolet, visible and infrared region of the electromagnetic spectrum.
LIDAR Light Detection and Ranging; or Laser Imaging Detection and Ranging “airborne-based” applications
The range to an object is determined by measuring the time of flightApplications
LIDAR technology has application in geology, engineering geology, seismology, remote sensing and atmospheric physics.
FALCON II is An airborne LIDAR system for the acquisition of digital elevation models and ortho images.
SPARCLE Space Readiness Coherent Lidarfor wind measurement .
LIDAR Result “Healy, Alaska”
LIDAR
Biometrics Applications
???
LIDAR SENSOR SYSTEM (FALCON II)
LADAR A device consisting of a photon source , a photon detection system, a timing circuit, and optics for both the source and receiver. Distance from the device to targets is measured by the time-of-flight.
LADAR Laser Detection and Ranging, or laser radar is often used in military context. “ground-based ” applications
The device could be a single shot measurement system (range only), “laser rangefinders”
LADAR, generates a 3-D Range Image.
ApplicationsReverse engineering (3D models), automated
process control, target recognition, and autonomous machinery guidance
Robotics: Five LADAR units were used for short range detection on the autonomous car that won the 2005 DARPA Grand Challenge.
LADARSystems
Optech, IncorporatedWavelength of illumination source 1540 nmMaximum Range 1500 mRange resolution (depth) 3 mm Range accuracy 10 mmRetail cost $150,000
Sick, Inc.Line Scanner 1DMaximum Range 30 mRange resolution 10 mm Range accuracy 15 mmRetail cost $4,000
M.I.T. Lincoln LaboratoryWavelength of illumination 780, 532 nmMaximum Range 1200 mRange resolution (depth) 150 mm Range accuracy 150 mm
Biometrics Applications
Can be use to build 3D model for the face
3D image generated by LDARMaximum Range 100 mRange resolution 20cm Distance is related to color of
the surface
Short-range scanners operate under triangulation principle
Long-range scanners works under the TOF/Continuous principles
Typical Range Resolutions for Scanners:
0.05 mm @ 0.6m –
2.5m0.10 mm @ 3.0m –
12.6m 0.5-2.0 mm @ 15m –
100m10 mm @ 100m –
800m
Imaging Radars: Scanners and Flash
Range Image is constructed by scanning each available scene
segment (Point/Line) one by one
Range Image acquired in one Burst at each Laser Pulse with Video
Frame Rate or Higher
Flash Range Imaging System comprised of a broad field illumination Laser Source and Focal Plane Array (FPA) detector
For each laser pulse the complete Range Image Frame is acquired
Operate under Continuous Wave Principle
Short range System Versions are commercially available with:1mm –
5mm @ 0.5m –
15m
Disadvantage: • Small FPA: 256x256 Commercially Available• Current Commercial Systems are Range Limited
Disadvantage: • Slow Frame Rate and not favorable in
dynamic applications
•
Pulsed, time-of-flight radars: emit energy in short pulses, wait for an echo, use the transit time to determine range:
•
Continuous-wave (CW) radars:emit energy continuously duringthe time that range is beingmeasured. Amplitude modulated(measure phase difference)
Range from the phase difference
Depth of field from range ambiguity interval
Imaging Radars: Principle Of Operation
illuminationSource
DetectedSignal
b
a
cbca
abc
cab )sin()sin()sin(
==
Active Triangulation: The Principle
Drawbacks:->Missing part problem & ->Slow Data Acquisition to be applied for Moving Objects
EM Spectrum
Ultra Violet Radio
Reflective & Thermal
Infra Red (IR) Sensors
Reflective Band:Reflective Band:Associated with
Reflected ambient radiation.
Visible0.4-0.7 mμ
Short IR0.9-2.4 mμ
Thermal Band:Associated with
Emitted Radiation
Mid/Long IR3-5, 8-14 mμ
Reflective IR Sensors
Track at 1km, at night, during thunderstorm
Advantages:• Can capture Clear Images at distances ~1km at Night, Fog, Dust etc.
• Invisible and eye safe
• Can see through a window into a building or a car
• Cope with Light Variability Problem encountered in Visible Face Recognition Task
• Human skin has different unique reflectance in different parts of Reflective-IR band what simplify the face detection problem significantly
Active Eye Safe, SWIR illuminators applied (bulbs,
LED, Lasers)
PassiveUse reflected Ambient SWIR radiation from the
Sun, the Moon, the Skies …
Thermal IR Sensors
Advantages:• Less scattered and absorbed by Fog, Smoke, and Dust than Video and Reflective IR
• With proper lens, a clear Human Face Thermal Image can be captured at distances ~1km
• In Face Recognition, the Within Class Variability Significantly lower than that in Video
• Thermal Imaging is Absolutely Passive and requires NO Illumination
• In many cases Thermal have advantage over Video Imagery in face detection task
The sensor MEASURES object’s temperature distribution by the amount of received IR energy:
Temperature F°
Thermal
Radiation
Thermal IR Camera
Biometric Signatures
2D & 3D Face Recognition
Gait Analysis
IRIS
Fingerprint
Voice
Biometric Signatures
contact Near Far
Camera √ √
Microphone √
Fingerprint reader √
IRIS sensor √
Anthropometric Analysis
Reflective Face Signatures: Video & Near-IR•
Non-intrusive and Simple
• Can Be captured at FAR • Biometric data is readable By Human • Well Public acceptance• Video Data is available police Data BasesPROBLEMS:PROBLEMS:•
Differences in Pose –
Facial orientation
•
Partial Occlusion •
Facial Expression
•
Can be easily fooled by a simple plastic surgery
Variations in Lighting encountered in Video Imagery Successfully can be solved in Near-IR
which requires NO LIGHTINGand uses particular wavelenght illumination source
Range image Signatures
Advantages over 2D Video:• Invariance to lighting conditions• Invariance to head orientation (face model can be rotated in 3D)
• Invariance to Camera distance (3D models captured to scale)
• Cope with changes in expression, glasses, beards …
• But Much more expensive equipment
Thermal Face Signatures
Physiological Change due to
Health, Emotion,Environment etc.
10/2003
04/2004
Thermal image Vascular Network
Vascular NetworkStay
Remarkablythe Same
-> Unique (Accuracy is Questionable)
-> Requires No Lighting (Sense Emitted Radiation, not Reflected Light)
-> Immunity from Forgery (even with plastic surgery)
-> Overall Thermal Face Scan, depends on factors like: view Angle, subject’s Emotion, body Temperature
2D Face Database (Gallery)Enrollment
Feature Extraction
Learn distinguishing features (PCA, LDA, ICA)
Probe Image
Feature Extraction
Face Detection and Alignment
Feature Matching
Identity
2D Face Recognition (Video)
(PCA, LDA, ICA)
PCA –
Principal Component AnalysisLDA –
Linear Discriminant AnalysisICA –
Independent Component Analysis
3D Face Recognition
3D Face Database
Surface Matching (Elastic Image Registration,
Mutual Information)
Test Scan
Identity
CVIP Lab Setups
3D Reconstruction SetupWhat we have?(A) 3d Laser Scanner 3D (Cyberware
3030)-
acquires 3d shape + texture-
used as ground truth 3D(B) CCD Camera (Sony Donpisha
XC-003)-
capture sequence of images(C) A Workstation loaded by CVIP lab
algorithms for 3d reconstructions using the captured images.
(B)(A)
What we can do?•
Using the Laser Scanner to build a database for 3D faces
•
capture sequence of images for each subject and utilize these images for recognition based on both the 2D poses and the 3D information.
β Rotatingbackground
scanner head
CCD camera
scanner base
support
β
Top view
CardEye
System Geometry
A robot-controlled, trinocular multi sensor, active vision system.
It has 3 cameras, pan, tilt, focus, zoom in its vision module.
The target is contained in a virtual sphere with radius R and distance d from the cameras
CardEye
System Setup System GUI
Biometrics:CardEye is controlled till the face within the virtual sphere of its FOVReconstruct the 3D face , to recognize the person
d [m]
Rmax[m]
Size range for R [m]
1.2 0.33 0.2-0.3
1.9 0.53 0.3-0.5
2.7 0.73 0.5-0.7
3.4 0.93 0.7-0.9
4.1 1.13 0.9-1.0
Functionality for 3D Reconstruction
Face image
Reconstruction
CardEyeSensor Planning: “active sensing”
The parameters are controlled so that object features are within the field of view and are in focus
Maximize the effectiveness of 3D reconstruction algorithm
3D Reconstruction:Multiple recovery techniques are
used to reconstruct 3D objects.
2D Face Recognition Setup
Canon Digital Rebel XT
System Components•
Canon EOS Digital Rebel XT (High-
performance digital SLR with 8.0 Megapixel CMOS Sensor) –
Stereo pair•
Canon EF-S (18-55mm) lens•
Workstation
Features•
2D Face Verification•
2D Face Identification
2D Face Database (Gallery)Enrollment
Feature Extraction
Learn distinguishing features (PCA, LDA, ICA)
Probe Image
Feature Extraction
Face Detection and Alignment
Feature Matching
Identity
2D Face Recognition (Video)
(PCA, LDA, ICA)
PCA –
Principal Component AnalysisLDA –
Linear Discriminant AnalysisICA –
Independent Component Analysis
3D Reconstruction Techniques
Stereo Vision: Basic Idea
The human vision is a stereo vision system
• Machine stereo vision is inspired from the human vision system.
• The origin of the word “stereo” is the Greek word “stereos” which means firm or solid, i.e. with stereo vision, the objects are seen solid in 3D.
• In stereo vision, the same seen is captured using two sensors from two different angles. The captured two images have a lot of similarities and smaller number of differences.
• In human perception, the brain combines the captured to images together by matching the similarities and integrating the differences to get a 3D model for the seen objects.
• In machine vision, the 3D model for the captured objects is obtained finding the similarities between the stereo images and using projective geometry to process these matches.
Stereo Vision: Basic Math
Cl Cr
plpr
Depth is calculated from a stereo correspondence using triangulation
1
( ) ( ( ))( )ˆ2
L L l
R s L s
T TL s R s L s R s
TL s R s
L
q K pq R q t
q R q t q R q tq R q tP
α β γα β
−== +
− − = × −
+ −=
Lqα
LqRqRqβ
L̂P
[ , ]s sR t
• Camera Calibration Camera parameters affect directly the elements of the both the projection and fundamental matrix. Hence, the accuracy of 3D reconstruction is highly influenced by the accuracy of the camera calibration method.
• Correspondence Matching False matches result in false 3D points. Due to the different appearance of same points when imaged from different angles, the matching is considered as a challenging point in stereo vision.
• Self Occlusion Self occlusion produces unmatched points, or even false matches.
• Textureless Region Matching Point-based correspondence matching in textureless regions is almost impossible. This is why stereo approaches usually fails to get 3D models for textureless objects.
Stereo Vision: Issues Involved
SFS is a classic problem in computer vision. It uses the brightness variation in a single image to compute the three dimensional shape of a surface [Horn 70].
Shape from shading (SFS)
SFS
?
)cos(),( iyxI θ=
||||||||.),(
LnLnyxI rr
rr
=
||||||||.)(),(
LnLnnRyxI rr
rrr==
iθ
nrLr
(x, y)
Solving the SFS requires two tasks:1.
to formulate an imaging model that describes the relation between the surface shape and the image brightness. (camera
+ light
+ surface)
2.
to develop a numerical algorithm to reconstruct the shape from the given image.
I: is the image brightness (is known)n: is the normal vector at the surface point (our unknown)L: is the vector of the light direction (given or can be estimated)θi
: the incident angle
•
Abdelrehim Ahmed and Aly Farag, "A New Formulation for Shape from Shading for Non-Lambertian Surfaces," Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR'06), New York, NY, USA June 17-22, 2006
•
M. G. Mostafa, Sameh
M. Yamany
and Aly A. Farag, “Integrating Shape From Shading and Range Data,”
IEEE International Conference on Computer Vision and Pattern Recognition (CVPR’99), Fort Collins, Colorado, pp. 15-20, June 1999.
•
M. G. Mostafa, S. M. Yamany
and Aly A. Farag, ``Integrating Stereo and Shape from Shading,'' IEEE International Conference on Image Processing (ICIP'99), Kobe, Japan, Vol. 3, pp. 130-134, October, 1999.
•
M. G. Mostafa, S. M. Yamany
and Aly A. Farag, ``Data fusion for 3D object reconstruction,'' Proceedings of SPIE, vol. 3523, p.88-99, Sensor Fusion and Decentralized Control in Robotic Systems, Paul S. Schenker; Gerard T. McKee; Eds., Oct 1998.
•
………….
CVIP Lab. Contributions (related to Shape from Shading)
SFS features •
Shape from Shading (SFS) is the only method that uses single image to get the 3D information.
•
Recently researchers utilized shape from shading for face recognition. SFS helps in developing a face recognition system that is robust to changes in illumination conditions.
•
SFS can work for texture-less surfaces (Stereo and Space curving can’t)
Drawbacks •
Since the reconstruction accuracy of the SFS is less than other methods such as Stereo and Space carving (use more images), SFS gives low recognition rate if it used alone.
Fusion between SFS+ Stereo+ Space carving for acceptable recognition rate
SFS as a Tool for Face Recognition
•The environment is represented as a discretized
set of voxels. •Each voxel is projected to the input images using their respective projection matrices.•Consistent voxel is assigned the color of its projections, and inconsistent voxel is removed from the volume.•This process is repeated until the final 3-D shape agrees with all the input images.
Space Carving
a consistent voxel, has thesame color in all projections=> Keep it
an inconsistent voxel, does nothave the same color => Remove it
S={ } /* initial set of colored voxels is empty
for i = 1 to r do /* traverse each of r layers
for each V in the ith layer of voxels do
project V into all images where V is visible
if sufficient correlation of the pixel colors
then add V to S
Space Carving Algorithm
Input View 2 output views
•
A. Eid
and A. A. Farag, “A Silhouette-Contour Based 3-D Registration Methodology As a Pre-Evaluation Step Of 3-D Reconstruction Techniques,”
Proc. of IEEE International Conference on Image Processing (ICIP), Genova, Italy, September 11-14, 2005
•
A. Eid
and A.A. Farag, “Design of an experimental setup for performance evaluation of 3-D reconstruction techniques from sequence of images,”
Proc. of Eighth European Conference on Computer Vision (ECCV'04), Workshop on Applications of Computer Vision, Prague, Czech Republic, May 2004, pp. 69-77
•
A. Eid
and A.A. Farag, “A unified framework for performance evaluation of 3-D reconstruction techniques,”
Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR'04), Workshop on Real-time 3-D Sensors and their Use, Washington, DC, June 27-July 2, 2004, pp. 41-48.
•
A. Eid
and A. A. Farag, “On The Performance Characterization of Stereo and Space Carving,”
Advanced Concepts for Intelligent Vision Systems Conference, Ghent, Belgium, September 2-5, 2003, pp. 291-296.
•
A. Farag and A. Eid, “On the Performance Evaluation of 3-D Reconstruction Techniques from a Sequence of Images,”
European Journal of Applied Signal Processing (EURASIP), Vol. 13(2005), pp. 1948-1955 .
•
………….
CVIP Lab. Contributions (related to Space Carving)
Space Carving Features •
Good results on difficult real-world scenes (such as Faces..!)•
Coarse-to-fine Reconstruction•
By using Hardware-Acceleration the algorithm can process voxels of an entire plane at a time
Drawbacks •
Need to acquire calibrated images•
Restriction to simple radiance models (Lambertian) •
Bias toward maximal (fat) reconstructions
Fusion between SFS+ Stereo+ Space carving for acceptable recognition rate
Space Carving as a Tool for Face Recognition
SFM “Shape from Motion” or “Structure from Motion”Extracting the shape of a scene from the spatial and temporal changes occurring in an image sequence. Given a set of image feature trajectories over time, solve for their 3D positions and camera motion. This includes estimation of
Camera's internal geometry “ Focal length”Relative 3D motion between the camera and the scene The structure “the depth, or 3D Shape”
Structure from MotionStructure from Motion
Firstsecond
third
Structure from MotionStructure from Motion
2
21 1
1( , ) ( , )2
i
ik
Km
ik i ji k
E X M u h m xσ = =
= −∑∑
Basic Math
Given the 2D measurements uik, and correspondence indicators jik
Minimize sum of squared re-projection errors
σ standard deviation of measurements noise
projection of 3D feature on camera mi( , )
iki jh m xikj
x
An example with 4 features seen in 2 images. The 7 measurements uik are assigned to the individual features by means of the assignment variables jik .
ikjx
Applications:
3D Model Reconstruction ,
3D Motion Matching drive 3D models for animation purposes
Self Camera Calibration
3D Vision computational sub-component helps in tracking, recognition and modeling
Robotics 3D scene structure is an important intermediate step in navigation and obstacle detection
3D Coding of Image Sequences compactly encode information about the scene.
2
21 1
1( , ) ( , )2
i
ik
Km
ik i ji k
E X M u h m xσ = =
= −∑∑
Structure from MotionStructure from MotionBiometrics Applications
Find the face, locating eyes, nose and mouth coordinates.
The features are fed into the SfM algorithm resulting in the recovery of 3D points.
The SFM output can be fed forward into a module where a 3D shape estimator can be used to compute a full 3D model
Sensors Fusion
CardEye-LASER Fusion for 3D Face Reconstruction
LASER
Sensor Planning
Fusion
Range-Video-IR Fusion: Example
Register and Align Multimodal Data Sets
Range Video IR
LWIR Camera3D/Video Scanner
Each pair of Sensors create its own Facial Range Image Using Stereo, Shape From Shade and Space Carving
The Range Images are Combined to Make More Accurate 3D Face Map
Video, Near-IR
& Thermal
Textures are mapped onto the 3D Model
Remote, Covert and Robust Identification
500 yardsDay Or Night, Fog or Sunny
Near IR Laser Light (1.4 -1.5 )Laser: Near
IRμ
Near-IR
Near-IR
Video
Video
Thermal-IR
Thermal-IR
Ster
eo P
airs
Thank You
