Super-Resolution

Dr. Yossi Rubneryossi@rubner.co.il

Many slides from Miki Elad - Technion

1

Example - Video

53 images, ratio 1:42

40 images ratio 1:4

Example – Surveillance

3

Example – Enhance Mosaics

4

5

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution• SR in time

6

D

For a given band-limited image, the Nyquist sampling theorem states that if a uniform sampling is fine enough (D), perfect reconstruction is possible.

D

Intuition

7

Due to our limited camera resolution, we sample using an insufficient 2D grid

2D

2D

Intuition

8

However, if we take a second picture, shifting the camera ‘slightly to the right’ we obtain:

2D

2D

Intuition

9

Similarly, by shifting down we get a third image:

2D

2D

Intuition

10

And finally, by shifting down and to the right we get the fourth image:

2D

2D

Intuition

11

It is trivial to see that interlacing the four images, we get that the desired resolution is obtained, and thus perfect reconstruction is guaranteed.

Intuition

12

What if the camera displacement is Arbitrary ? What if the camera rotates? Gets closer to the object (zoom)?

Rotation/Scale/Disp.

13

There is no sampling theorem covering this case

Rotation/Scale/Disp.

14

15

3:1 scale-up in each axis using 9 images, with pure global translation between them

A Small Example

Further Complications

• Complicated motion– perspective, local motion, …

• Blur– sampling is not a point operation– Spatially variant blur– Temporally variant blur

• Noise

• Changes in the scene

16

Super-Resolution - Agenda

• The basic idea

• Image formation process• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution• SR in time

17

Image Formation

Scene Noise

HR LR

Can we write these steps as linear operators?

Geometrictransformation

kF

OpticalBlur

kH

Sampling

kD

HRLR kkk FHD 18

Geometric Transformation

• Any appropriate motion model• Every frame has different transformation • Usually found by a separate registration algorithm

Scene Geometrictransformation

kF

19

Geometric Transformation

Can be modeled as a linear operation XkF

kF

X

=

kF XXkF

20

Optical Blur

• Due to the lens PSF and pixel integration• Usually

Geometrictransformation

OpticalBlur

kH

HH k

21

H

PSF PIXEL H* =

22

Optical Blur

Can be modeled as a linear operation XH

H

X

=

XHXH

23

Sampling

• Pixel operation consists of area integration followed by decimation• D is the decimation only• Usually

Optical Blur Sampling

kD

DD k

24

Decimation

Can be modeled as a linear operation XD

D

X

=

X

01

01...

01

01

01 o

o

D

1

XD

25

Super-Resolution - Agenda

• The basic idea• Image formation process

• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution• SR in time

26

Super-Resolution - Model

N

k

nkkkkkk VVXY1

2,0~ ,

NFHD

X

High-Resolution

ImageH

H

Blur

1

N

F =I1

FN

Geometric warp

D

D1

N

Decimation

V1

VN

Additive Noise

Y1

YN

Low-ResolutionExposures

27

Simplified Model

N

k

nkkkk VVXY1

2,0~ ,

NDHF

X

High-Resolution

ImageH

H

Blur

F =I1

FN

Geometric warp

D

D

Decimation

V1

VN

Additive Noise

Y1

YN

Low-ResolutionExposures

28

The Super-Resolution Problem

• Given

Yk – The measured images (noisy, blurry, down-sampled ..)

H – The blur can be extracted from the camera characteristics

D – The decimation is dictated by the required resolution ratio

Fk – The warp can be estimated using motion estimation

n – The noise can be extracted from the camera / image

• RecoverX – HR image

2,0~ , nkkkk VVXY NDHF

29

VX

V

V

V

X

Y

Y

Y

Y

NNNNN

G

FHD

FHD

FHD

2

1

222

111

2

1

The Model as One Equation

1 size of ,

size of

1 size of

22

222

2

MrVX

MrNM

NMY

G

r = resolution factorMXM = size of the framesN = number of frames

r = resolution factor = 4MXM = size of the frames = 1000X1000N = number of frames = 10

=[10M×1]=[10M×16M]=[16M×1] Linear algebra notation is

intended only to develop algorithm 30

SR - Solutions• Maximum Likelihood (ML):

N

kkk

XYXX

1

2minarg DHF

Smoothness constraintregularization

Often ill posed problem!

XAYXXN

kkk

X

1

2 minarg DHF

• Maximum Aposteriori Probability (MAP)

31

ML Reconstruction (LS)

N

kkkML YXX

1

22 DHFMinimize:

0ˆ21

2

N

kkk

TTTk

ML YXX

XDHFDHF

Thus, require:

k

N

k

TTTk

N

kk

TTTk YX

11

ˆ DHFDHFDHF

A B

BA X̂32

LS - Iterative Solution

• Steepest descent

N

kknk

TTTknn YXXX

11

ˆˆˆ DHFDHF

Simulated error

Back projection

All the above operations can be interpreted as operations performed on images.

There is no actual need to use the Matrix-Vector notations as shown here.

33

LS - Iterative Solution

• Steepest descent

N

kknk

TTTknn YXXX

11

ˆˆˆ DHFDHF

nX̂

1ˆ

nX

geometrywrap

convolvewith H

downsample

upsample

convolvewith HT

inversegeometry

wrap

kY

-

-

kF H DTD TH T

kF

For k=1..N

34

Example

Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

HR image Least squaresLR + noiseX4

35

Robust Reconstruction

• Cases of measurements outlier:– Some of the images are irrelevant – Error in motion estimation– Error in the blur function– General model mismatch

36

Robust Reconstruction

N

kkk YXX

1

2 1

DHFMinimize:

N

kknk

TTTknn YXXX

11

ˆsignˆˆ DHFDHF

37

Robust Reconstruction

• Steepest descent

N

kknk

TTTknn YXXX

11

ˆsignˆˆ DHFDHF

sign

For k=1..N

nX̂

1ˆ

nX

geometrywrap

convolvewith H

downsample

upsample

convolvewith HT

inversegeometry

wrap

kY

-

-

kF H DTD TH T

kF

38

Example - Outliers

Simulated example from Farisu at al. IEEE trans. On Image Processing, 04

HR image LR + noiseX4

Least squares

Robust Reconstruction39

20 images, ratio 1:4

L2 norm based

Example – Registration Error

L1 norm based

40

MAP Reconstruction

• Regularization term:

– Tikhonov cost function

– Total variation

– Bilateral filter

XAYXXN

kkkMAP

1

22 DHF

2XXAT

1

XXATV

P

Pl

P

Pm

my

lx

mlB XSSXXA

1 41

Robust Estimation + Regularization

P

Pl

P

Pm

my

lx

mlN

kkk XSSXYXX

11

1

2 DHF Minimize:

P

Pl

P

Pmn

my

lxn

my

lx

ml

N

kknk

TTTknn

XSSXSSI

YXXX

ˆˆsign

ˆsignˆˆ1

1

DHFDHF

42

Robust Estimation + Regularization

1ˆ

nX

geometrywrap

convolvewith H

downsample

upsample

convolvewith HT

inversegeometry

wrap

kY

sign

-

P

Pl

P

Pmn

my

lxn

my

lx

mlN

kknk

TTTknn XSSXSSIYXXX ˆˆsignˆsignˆˆ

11 DHFDHF

nX̂

horizontalshift l

verticalshift m

horizontalshift -l

verticalshift -m

sign

-

-

From Farisu at al. IEEE trans. On Image Processing, 04

- lm

For k=1..N

For l,m=-P..P

43

Example

• 8 frames

• Resolution factor of 4

From Farisu at al. IEEE trans. On Image Processing, 0444

Example

Images from Vigilant Ltd.45

Handling Color in SR

XAYXXN

kkkMAP

1

22 DHF

Handling color: the classic approach is to convert the measurements to YCbCr, apply the SR on the Y and use trivial interpolation on the Cb and Cr.

Better treatment can be obtained if the statistical dependencies between the color layers are taken into account (i.e. forming a prior for color images).

In case of mosaiced measurements, demosaicing followed by SR is sub-optimal. An algorithm that directly fuse the mosaic information to the SR is better.

46

20 images, ratio 1:4

SR for Full Color

47

20 images, ratio 1:4

Mosaiced input

Mosaicing and then SR Combined treatment

SR+Demosaicing

48

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution

• Special cases and related problems• Limitations of Super-Resolution• SR in time

49

Special Case – Translational Motion

• In this case H and F commute:

• SR is decomposed into 2 steps1. Find blur HR image from LR images non-iterative2. Deconvolve the result using H iterative

XZVZ

VX

VXY

kk

kk

kkk

HDF

HDF

DHF

TTk

Tk

Tkk HFFHHFHF

50

IntuitionXZVZY kkk HDF

X PSF*X Z=PIXEL*PSF*X

• Using the samples can, at most, reconstruct Z• To recover X, need to deconvolve Z

51

Step I – Find Blurred HR

• L2 For all frames, copy registered pixels to HR grid and average [Elad & Hel-Or, 01]

• L1 For all frames, copy registered pixels to HR grid and use median [Farisu, 04]

N

kkkML YZZ

1

22 DF Minimize:

52

Solution for L2

N

kkkML YZZ

1

22 DF Minimize:

N

kk

TTk

N

kk

TTk

YP1

1

DF

DFDFR

PZˆR

0

2

Z

ZMLThus, require:

Sum of HR grid

Diagonal, number Of occurrences

per HR grid

53

Step II - Deblur

XAZXX 1

2 H Minimize:

n

n

knT

nn XAX

ZXXX ˆˆ

ˆsignˆˆ1 HH

54

Example

From Pham at al. Proc. Of SPIE-IS&T, 05. Simulated.

64X64 LR 256X256Before deblur

256X256After deblur

55

Related Problems

• Denoising (multiple frames)

• Denoising (single frame)

• Deblurring

• Interpolation – “single-image super-resolution”

2,0~ nVVXY NDH ,

2,0~ nkkk VVXY N ,

2,0~ nVVXY NH ,

2,0~ nVVXY N ,

56

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems

• Limitations of Super-Resolution• SR in time

57

Limiting Factors

• Main factors– SNR– PSF (optical+pixel)– Number of inputs – Pixel size (sampling rate)– Fill factor– Image content

58

Limiting Factors

Registration DeblurringFusion

LR1

…

LRN

HR

SNR # of inputs PSF

Pixel sizeFill factor

Image content

59

SR Limits Analysis

• Noise– Registration noise– Fusion noise

• SR factor– Point-Spread-Function (PSF)

• Optical Transfer Function (OTF)

– Sensor pixel size• Sensor Transfer Function (STF)

60

Registration Noise

• Using Cramer Rao Lower Bound (CRLB)– Lower bounds for shift estimation:

• Better registration accuracy by:– Less noise– Higher derivatives in image– Bigger registration area– Narrow PSF

222

2222

222

22

222

22

var

var

yxyx

yxnreg

yxyx

xn

yxyx

yn

v

u

IIII

II

IIII

I

IIII

I

61

Fusion Noise

• r = super-resolution factor

• N = number of images

22

2nfusion N

r

62

Registration + Fusion Noise

• If N∞ then– Fusion error vanishes– Registration error is equivalent to Gaussian

blur

2

22

222

222

nn

yxyx

yxtotal N

r

IIII

II

63

Super-Resolution - Agenda

• The basic idea• Image formation process• Formulation and solution• Special cases and related problems• Limitations of Super-Resolution

• SR in time

Work and slides by Michal Irani & Yaron Caspi (ECCV’02)

64

“Classical” Image Super-Resolution

Low-resolution images:

High-resolution image:

Scene:

65

time

time

time

Space-Time Super-Resolution

Super-resolution in space and in time.

time

High space-time resolution sequence: time

Low-resolution imagesvideo sequences:

66

What is Super-Resolution in Time?

Observing events “faster” than frame-rate.

• Handles:

(1) Motion aliasing (2) Motion blur

• Application areas: - sports scenes- scientific imaging- etc...

67

(1) Motion AliasingThe “Wagon wheel” effect: Slow-motion:

time

Continuous signal

time

Sub-sampled in time

time

“Slow motion”68

(2) Motion Blur

69

lnS

lS1

Sh(xh,yh,th)

Space-Time Super-Resolution

x

y t

y

x

t

Blur kernel:

PSF

Exposure time

T

71

Input 1 Input 2

Input 3 Input 4

Example: Motion-Aliasing

25 [frames/sec]72

Input sequence in slow-motion (x3):

75 [frames/sec]

Super-resolutionSuper-resolution in time (x3):

75 [frames/sec]

Example: Motion-Aliasing

73

Output trajectory:

Without estimating motion of the ball!

Output sequence:

(x15 frame-rate)

Deblurring:

Input:

Output:

3 out of 18 low-resolution input sequences (frame overlays; trajectories):

74

Frames at collision:

4 input sequences:

Output frame at collision:

Video 1

Video 3

Video 2

Video 4

Example: Motion-Blur (real)

75