Jitter Camera: High Resolution Video from a Low

Resolution Detector

Moshe Ben-Ezra, Assaf Zomet and Shree K. Nayar

IEEE CVPR ConferenceJune 2004, Washington DC, USA

Video Resolution

MiniDV Camera. Resolution: 720x480

Digital Camera. Resolution: 2592x1944

Plasma DisplayResolution 1366x768

Fundamental Resolution Tradeoff

Spatial resolution (pixels)

Tem

pora

l res

olut

ion

(fps

)

30

330K720x480

Conventional video camera

130

3

3M2048x1536

Hi-resolution still Camera

Super-Resolution

Sequence taken by a moving camera

High-Resolution computed image

Super-Resolution

Shechtman, Caspi, and Irani ECCV2002Zomet and S. Peleg. ICIP2000Baker and Kanade. CVPR2000Chiang and Boult, IVC2000

Capeland, Zisserman ICPR2000Elad and Feuer IP1997 Irani and Peleg GMIP1996

Super Resolution

All SampledImages

y = (D G)x + z

Decimation

Blurring Op.

Hi Res. Image

Noise

Motion Blur Hurts Us Again!

Capture Images without Motion Blur

Effect of Motion Blur on Super-Resolution

Input: No Motion Blur Super-Resolution Result

Input : With Motion Blur (known) Super-Resolution Result

Quantifying The Affect of Motion Blur

Empirical tests: RMS error. Volume of Solutions (Linear Model):

y Ax z x A 1y A 1z

High-ResolutionImage

Noise (Quantization)

InputImages

Volume of Solutions 1/det(A)

Blur &Decimation

Baker and Kanade

How Bad is Motion Blur for Super-Resolution?

Space of Super-Resolution Solutions

0 1 2 3 4 5

Motion blur in pixels

RMS Error After Super-Resolution

0 1 2 3 4 5

18

Motion blur in pixels

Avoid Motion Blur using Jitter Sampling

Conventional Sampling

Time

Space

Spatial Jitter Sampling

Time

Space

The Jitter CameraLens Detector

Micro-actuator

The Jitter CameraLens Detector

Micro-actuator

Detector is a light weight device!

Jitter is instantaneous and synchronous

Computer ControlledX Micro-Actuator

Computer ControlledY Micro-Actuator

Board Camera

Lens

1μm X Pixels

Y P

ixel

s

Jitter Mechanism Accuracy

Desired locations. Actual locations.

Result: Resolution Chart

Super-Resolution ImageFour Images fromthe Jitter Camera

Result: Color DeMosaicing and Super-Resolution

1 (out of 4) Jitter camera Image Super-Resolution

Jitter Video (Stabilized)

How can we handle dynamic scenes?

Adaptive Super-Resolution for Dynamic Scenes

Static blocks: 4 frames used.

Occlusions: 1 frame used.

Moving object: 2 - 4 frames used

I-3 I-2 I-1 I I+1 I+2 I+3

Adaptive Super-Resolution Algorithm

1. Estimate the aliasing error ‘’ (stdv) for each block Ik in I.2. Compute robust block matching between all pairs {I}{I1,2,3}. Use

‘’ as a scale factor for an M-Estimator error function.

3. For each block Ik try to find 3 matching blocks {Ix}k, s.t. : a) SSD(Ik, {Ix}k)-0.5 < 3b) {Ix}k are temporally closest to Ik (smallest x)

4. Apply super-resolution to the selected blocks.

The algorithm degrades gradually from 4-frames super-resolution to a single frame interpolation and deblurring.

Scale EstimateMean 6.4, Stdv 14

Mean 10.5, Stdv 27

Mean 7.5, Stdv 16

Mean 15.2, Stdv 30

Mean 8.6, Stdv 17

Mean 17.7, Stdv 33

Low Res - Hi-ResAliasing Error(Simulated)

Low Res2nd derivative(Simulated)