Seeing the Invisibles:Seeing the Invisibles:Recent Progress in Info. ForensicsRecent Progress in Info. Forensics
Min Wu
Media and Security Team (MAST)Media and Security Team (MAST) ECE Department / UMIACS
University of Maryland, College Park
Include joint work with Wei-Hong Chuang, Ravi Garg, Hongmei Gou,
http://www.ece.umd.edu/~minwu/research.html
Adi Hajj-Ahmad, K.J. Ray Liu, Hui Su, Ashwin Swaminathan, and Avinash Varna.
Questions about Digital Images Questions about Digital Images and Video …and Video …
Photographed? Scanned? Or computer generated?
Tampered? When and where captured? Who/where’s the leak? Post on YouTube or not?
Min Wu (UMD): Seeing Invisibles - Spring 2013 2Iranian Missile: Illustration by The New York Times; photo via Agence France-Presse
Many Forms of “Digital Fingerprints”Many Forms of “Digital Fingerprints”
Many types of fingerprints for multimedia protection & management
I C EI C EI. C. E.I. C. E.
Embedded FingerprintEmbed unique ID/signal as digital fingerprints to track individual copy and trace unauthorized use
C t t b d Fi i tContent-based FingerprintCompact content signature for content identification, and also useful for watermarking and content authentication
Intrinsic FingerprintExamine inherent traces left on multimedia by device or processing –Provide non intrusive forensics to determine origin integrity etc
Min Wu (UMD): Seeing Invisibles - Spring 2013 3
Provide non-intrusive forensics to determine origin, integrity, etc.
Many Forms of “Digital Fingerprints”Many Forms of “Digital Fingerprints”
Many types of fingerprints for multimedia protection & management
I C EI C EI. C. E.I. C. E.
Embedded FingerprintEmbed unique ID/signal as digital fingerprints to track individual copy and trace unauthorized use
C t t b d Fi i tContent-based FingerprintCompact content signature for content identification, and also useful for watermarking and content authentication
Whi HAlice
w1 LeakLeak
Intrinsic FingerprintExamine inherent traces left on multimedia by device or processing –Provide non intrusive forensics to determine origin integrity etc
White House
Satellite Image
Bobw2
LeakLeak
Min Wu (UMD): Seeing Invisibles - Spring 2013 4
Provide non-intrusive forensics to determine origin, integrity, etc.g
Carl
w3
Many Forms of “Digital Fingerprints”Many Forms of “Digital Fingerprints”
Many types of fingerprints for multimedia protection & management
I C EI C EI. C. E.I. C. E.
Embedded FingerprintShazam app
Embed unique ID/signal as digital fingerprints to track individual copy and trace unauthorized use
C t t b d Fi i t
Shazam app for iPhone
Content-based FingerprintCompact content signature for content identification, and also useful for watermarking and content authentication
Intrinsic FingerprintExamine inherent traces left on multimedia by device or processing –Provide non intrusive forensics to determine origin integrity etc
Min Wu (UMD): Seeing Invisibles - Spring 2013 5
Provide non-intrusive forensics to determine origin, integrity, etc.
Many Forms of “Digital Fingerprints”Many Forms of “Digital Fingerprints”
Many types of fingerprints for multimedia protection & management
I C EI C EI. C. E.I. C. E.
Embedded FingerprintEmbed unique ID/signal as digital fingerprints to track individual copy and trace unauthorized use
C t t b d Fi i tContent-based FingerprintCompact content signature for content identification, and also useful for watermarking and content authentication
Intrinsic FingerprintExamine inherent traces left on multimedia by device or processing –Provide non intrusive forensics to determine origin integrity etc
Min Wu (UMD): Seeing Invisibles - Spring 2013 6
Provide non-intrusive forensics to determine origin, integrity, etc.
Intrinsic Traces in Images and VideoIntrinsic Traces in Images and Video
Represent group properties– “Digital / software” components of device or processing system
f– Ensemble properties of analog components: e.g. statistical noise profile of sensors
R ?R ?? ? ? ?? ?
R ?
? ?
R ?
? ?
? ?? ?
CandidateCFA pattern CFA InterpolationCFA pattern
Fitting errorFitting error
7Digital photograph Scanner model 1 Scanner model 2
Intrinsic Traces in Images and VideoIntrinsic Traces in Images and Video
Represent group properties– “Digital / software” components of device or processing system
f– Ensemble properties of analog components: e.g. statistical noise profile of sensors
Represent individuality of capturing device or environmentRepresent individuality of capturing device or environment– “Unreproducible / unclonable” individual properties
~ e.g. individual variation from “analog” part of sensors due to manufacturing variabilitymanufacturing variability
Min Wu (UMD): Seeing Invisibles - Spring 2013
8Samsung i760Apple iPhone 3G
Intrinsic Traces in Images and VideoIntrinsic Traces in Images and Video
Represent group properties– “Digital / software” components of device or processing system
f– Ensemble properties of analog components: e.g. statistical noise profile of sensors
Represent individuality of capturing device or environmentRepresent individuality of capturing device or environment– “Unreproducible / unclonable” individual properties
~ e.g. individual variation from “analog” part of sensors due to manufacturing variabilitymanufacturing variability
– Unique time-varying location-dependent conditions during capture
D t ti C t d t ti Detection vs. Counter-detection– Determine integrity, origin, time/location, processing history. etc.– Remove detectable/inferable traces for privacy sanitization
Min Wu (UMD): Seeing Invisibles - Spring 2013 9
Forensic Forensic Questions Questions on “Time” and “Place”on “Time” and “Place”
500
600
-90
-80
300
400
500
me
(in s
econ
ds)
-120
-110
-100
90
9 6 10 10 4 10 8
100
200
Tim
-150
-140
-130
When was the video actually shot? And where? Was the sound track captured at the same time as the
9.6 10 10.4 10.8Frequency (in Hz)
Was the sound track captured at the same time as the picture? Or super-imposed afterward?
Explore fingerprint influenced by power grid onto sensorExplore fingerprint influenced by power grid onto sensor recordings
10Min Wu (UMD): Seeing Invisibles - Spring 2013
Ubiquitous Forensic Fingerprints from Power GridUbiquitous Forensic Fingerprints from Power Grid
400
500
onds
) -40
-20
0.7
0.8
0.9
effic
ient
400
500
600
nds) -100
-90
-80
100
200
300
Tim
e (in
sec
o
-100
-80
-60
30 20 10 0 10 20 30
0.3
0.4
0.5
0.6
Cor
rela
tion
co
100
200
300
Tim
e (in
sec
o
-150
-140
-130
-120
-110
49.5 50 50.5 51 51.5Frequency (in Hz)
-30 -20 -10 0 10 20 30Time frame lag
ENF matching result demonstrating similar variations in the ENF
Video ENF signal Power ENF signal Normalized correlation9.6 10 10.4 10.8
Frequency (in Hz)
Electric Network Frequency (ENF): 50/60 Hz nominal
g gsignal extracted from video and from power signal recorded in India
Varies slightly over time; main trends consistent in same grid Can be “seen” or “heard” in sensor recordingsHelp determine recording time/location detect tampering etc
Min Wu (UMD): Seeing Invisibles - Spring 2013
Help determine recording time/location, detect tampering, etc.
Ref: Ravi-Varna-Wu paper in ACM Multimedia 2011 11
Tampering DetectionTampering Detection
ENF signal from Video
ENF matching result demonstrating the detection of video tampering based on the ENF traces
10
10.1
10.2
10.3
eque
ncy
(in H
z)
Insertedli
160 320 480 640 800 96010
Time (in seconds)
Fre
50.2n H
z)
Ground truth ENF signal
clip
160 320 480 640 80049.9
5050.1
Ti (i d )
Freq
uenc
y (in
Adding a clip between the original video leads to discontinuity in the ENF signal extracted from videoCli i ti l b d t t d b i th id ENF
Time (in seconds)
Clip insertion can also be detected by comparing the video ENF signal with the power ENF signal at corresponding time
12Min Wu (UMD): Seeing Invisibles - Spring 2013
Forensic Binding of Audio and Visual TracksForensic Binding of Audio and Visual Tracks ENFs in audio and video tracks captured at the same time have
high correlation
Research questions ahead: (1) How to accurately estimate and match weak and noisy ENF? (2) Can ENF be removed? Tampered? (3) How to prevent anti forenics on ENF?be removed? Tampered? (3) How to prevent anti‐forenics on ENF? (4) New applications: smart grid monitoring; ……
13Min Wu (UMD): Seeing Invisibles - Spring 2013Ref: Chuang-Ravi-Wu paper in ACM CCS 2012
Explore Machine Learning to Infer LocationExplore Machine Learning to Infer Location Inter-Grid location-of-recording estimation from sensing
signals containing ENF traces– Preliminarily identified useful features for average 85% accuracy
mat
es (H
z)
mat
es (H
z)
mat
es (H
z)
requ
ency
Est
im
requ
ency
Est
im
eque
ncy
Estim
FrFr
Time (secs) Time (secs) Time (secs)
Fre
LEBANON INDIA Eastern US
14Min Wu (UMD): Seeing Invisibles - Spring 2013
Can ENF Pinpoint to Locations Within a Grid?Can ENF Pinpoint to Locations Within a Grid? Main trend of ENF is known to be same in a grid “Microscopic” traces due to localized effect
– Small variations aren’t felt between places far apart– Dynamic distributed control to stabilize power grid has a response
propagation speed of about 500 miles per secondp p g p p
Our multi-location studies in U.S. east and west grids
Min Wu (UMD): Seeing Invisibles - Spring 2013 15(a) ENF signals from different locations
of US Eastern grid(b) Correlation between ENF signals
after high-pass filtering
Include joint work with Wei-Hong Chuang, Ravi Garg, Hongmei Gou, Adi Hajj-Ahmad, K.J. Ray Liu, Hui Su, Ashwin Swaminathan, and Avinash Varna.
Min Wu (UMD): Seeing Invisibles - Spring 2013 17