1
Motion Compensated SNR and DR Enhancement With Motion Blur Prevention Using Multicapture
Ali Ercan & Ulrich Barnhoefer
EE392J Project Ali Ercan & Ulrich Barnhoefer 2
Introduction & Motivation
Single exposure trade-off High noise if short
exposure time Motion blur if long
exposure time
EE392J Project Ali Ercan & Ulrich Barnhoefer 3
Introduction & Motivation DR is another problem
Short exposure: Dark areas in the scene cannot be seen
Long exposure: Bright areas saturate If both high DR scene and motion,
with single capture Motion blur free, but noisy and non-
visible dark areas image Less noisy, but motion blurred and
saturated image
EE392J Project Ali Ercan & Ulrich Barnhoefer 4
Introduction & Motivation Our approach to solve these
problems: Use of multicapture combined with
motion estimation High speed is definitely needed Normal video mode can be used –
poorer results due to noise adding CMOS imagers suitable For a better understanding, let us
introduce a simple model of CMOS imagers and describe multicapture
EE392J Project Ali Ercan & Ulrich Barnhoefer 5
Sensor Model Charge Integration
Light on photodiode generates charges
Saturation when well capacity is reached
Noise sources (Reset noise) Shot noise UT Read noise VT,Vo (Dark current) (Fixed pattern noise)
0VVUTiQ TTph
EE392J Project Ali Ercan & Ulrich Barnhoefer 6
Multicapture Nondestructive
multiple readout – Single integration
Less noise per capture compared to conventional video sensor – readout noise
k
loklphk VVUkiQ
1
EE392J Project Ali Ercan & Ulrich Barnhoefer 7
Implemented AlgorithmSCENE
CAMERASIMULATOR
MOTIONESTIMATOR
PHOTO-CURRENTESTIMATOR
FINALIMAGE
EE392J Project Ali Ercan & Ulrich Barnhoefer 8
Camera Simulator Multicapture, noise, ADC
implemented – pixel values out
EE392J Project Ali Ercan & Ulrich Barnhoefer 9
Motion Estimator Block based motion estimation on
difference frames Search range ±1 and block size 3x3 Fast imager (e.g. 10,000 fps available) Search range and block size can be increased
in expense of computational load Noise suppression
Known noise levels – characterized CMOS sensor
Error = SSD + xDistance is proportional to noise Thanks to Sebe!
EE392J Project Ali Ercan & Ulrich Barnhoefer 10
Motion Estimator Estimated and perfect motion vectors
EE392J Project Ali Ercan & Ulrich Barnhoefer 11
Photocurrent EstimatorA
B
C
k
l
Co
Ckl
Ck
k
l
Bo
Bkl
Bk
k
l
Ao
Akl
Ak
VVUikQ
VVUikQ
VVUikQ
1
1
1
CCC
BBB
QQQ
QQQ
56'
1
34'
1
CCCC
BBBB
VVUiQ
VVUiQ
566'
1
344'
1
Ao
AAAA
Ao
AAA
VVUUiQ
VVUiQ
2212
111
2
A
CBA
QQQQ
2
~
2
'1
'11
~
1 )(31
EE392J Project Ali Ercan & Ulrich Barnhoefer 12
Photocurrent Estimator
j
l
n
l
Xj
jj Qn
Q1
~ 1
k
okk
ll
kk n
VVU
nikQ
1
~ 1
knVV
kn
Ui
kQ
Ik
ok
k
k
l lkk
1
~~
N
jjj IaI
1
~^
2^
2 )( iIE
iIE )(^
11
N
jja
N
j j
vjN
jl l
lu nj
a
lna
1
222
22 2
N
l l
jj
j
l l
l
v
ujj
j
jjj
b
ba
nlb
jnbnn
jj
nnjb
b
1
1
12
2
111
1
21
1
EE392J Project Ali Ercan & Ulrich Barnhoefer 13
Results
EE392J Project Ali Ercan & Ulrich Barnhoefer 14
Results
EE392J Project Ali Ercan & Ulrich Barnhoefer 15
Results
EE392J Project Ali Ercan & Ulrich Barnhoefer 16
ResultsIMAGE ERRORS
(STD OF ERROR IMAGE)CHECKE
RLENA CAMERAMAN
10 ms image 100.9 69.43 71.31
160 ms image 70.79 33.84 37.41
Const. with est. motion vectors 2.587 21.28 12.05
Const. with perfect motion vectors
2.576 17.22 3.092
EE392J Project Ali Ercan & Ulrich Barnhoefer 17
Conclusion
Promising results achieved with this preliminary analysis Motion blur reduced Noise reduced DR increased in dark end and in
bright end in special cases
EE392J Project Ali Ercan & Ulrich Barnhoefer 18
Conclusion Lots of more things to do
Use more sophisticated motion estimation algorithms
Separate motion detection from motion estimation and do estimation when detection occurs
Include extension of DR with sensor saturation
Handle the occlusions
EE392J Project Ali Ercan & Ulrich Barnhoefer 19
Questions