114R.I. Hornsey, University of Waterloo
NoiseinImage Sensors
115R.I. Hornsey, University of Waterloo
Introduction• We have seen how pixels are designed to
maximise the sensitivity to illumination
• However, this is only part of the story
• The overall performance of the sensor isultimately limited by the noise that is added bythe system to the signal
• In this sense, the noise figure of the detectorsystem is a measure of its “perfection”
• Noise comes from numerous sources and itsminimisation requires optimisation of manyindividual parts of the system
• Our discussion will not consider external noisesources, such as electrical pick-up
» the only “external” noise we will include is noise in theoptical signal itself
• The treatment of noise is a complex subject, andit is even harder to measure the individualcomponents accurately
» the theoretical treatment is important, however, as adesign tool for optimising the performance of specificstages in the system
116R.I. Hornsey, University of Waterloo
Types of Noise
• “Noise” in image sensors is typically separatedinto two categories
» random noise
» pattern noise
• Random noise is what you might call “real”noise
» it is temporally random and is not constant from frameto frame in the image
» hence, it can be reduced by averaging successiveframes
» and is described by statistical distributions
• Pattern noise is effectively a spatial noise asseen by the observer of the image
» it does not change significantly from frame to frame
» and so cannot be reduced by frame averaging
• Pattern noise is divided into two components
» fixed pattern noise (FPN)
» photo-response non-uniformity (PRNU)
117R.I. Hornsey, University of Waterloo
Pattern Noise
• FPN is the component of pattern noisemeasured in the absence of illumination
• It is mainly due to variations in
» detector dimensions
» doping concentrations
» contamination during fabrication
» characteristics of MOSFETs (VT, gain, W, L, etc.)
• PRNU is the component of pattern noise thatdepends on the illumination
• PRNU depends on
» detector dimensions
» doping concentrations
» thicknesses of overlayers
» wavelength of illumination (spectral response)
• Historically, pattern noise (FPN in particular) hasbeen the factor limiting the acceptability ofCMOS imagers
» PRNU is not often mentioned ...
» shortly, we will see how FPN can be reduced
118R.I. Hornsey, University of Waterloo
Describing Noise
• Pattern noise is usually specified in terms of thevariation in the signals from individual pixelsunder uniform illumination
» usually as a percentage of the saturation output
• Random noise is expressed in terms ofparameters which describe the statisticaldistribution of voltage or current
• If there are n samples of the signal
» x1, x2, x3, . . . xn
• then the mean is x = (x1 + x2 + x3 + . . . xn) / n
• However, the mean for many noise sources iszero
» leaving the DC level of the signal unaffected
• So a more useful description of the noise iseither the variance (<x2>) or the standarddeviation (√<x2>, in rms units)
» which measures the scatter of the data points aboutthe mean
x2 = 1n
x j − x( )j=1
n∑
2
119R.I. Hornsey, University of Waterloo
• To sum noise sources, we have to add thevariances
• or the standard deviation is given by
x = x12 + x2
2 + x32 + ... xn
2
x2 = x12 + x2
2 + x32 + ... xn
2
120R.I. Hornsey, University of Waterloo
Importance of Noise
• We can illustrate the importance of the noise onthe overall sensor performance as follows
• Dynamic range = (saturation signal / rms noiselevel)
» saturation ≈ 200,000 e-, noise ≈ 40 e- rms
» typical value is 5,000 for a PD (~75dB)
» assuming dark current is not the limiting factor
102
103
104
105
106
# electrons
Light intensity(µW/cm2)
sensor + amplifier noise
saturation level
slope
= re
sponsi
vity
0.01 1010.1
saturation equivalentexposure
minimum resolvable signal
121R.I. Hornsey, University of Waterloo
• Responsivity = (# electrons / light intensity)
» in linear portion of the curve (electrons.cm2/µW)
• Provided that the dark current is small, theminimum resolvable signal is determined by thenoise in the system
• Hence, a good responsivity is not enough toensure a good signal at low light levels
» a low “noise floor” is also required
• In a convenient model, the rms system noise is
» where the floor is determined by the amplifier noise,the reset noise, and the analog-to-digital converternoise
» the noise floor is often referred to as the read noise
• The other noise included above is called theshot noise, which arises because of thestatistical arrival of electrons
» due to the photo-generation of the electrons
» and the thermal generation of electrons
• We will now examine some of the noise sourcespresent in image sensors
nsys = nshot2 + nfloor
2 + npattern2
122R.I. Hornsey, University of Waterloo
Thermal Noise
• Thermal noise is a white noise
» the noise power is constant over all frequencies
• For a resistor, the thermal noise root meansquare voltage is given by
» where R is the resistance, and B is the noiseequivalent bandwidth
• Since the thermal noise covers the entirefrequency range, the bandwidth determines theactual amount measured
• So the open circuit equivalent circuit is
• Alternatively
• However, an important factor is the noiseequivalent bandwidth for use in the calculation
vth = 4kTBR
ith =4kTB
R
RnoisyRnoiseless
<vth>=
123R.I. Hornsey, University of Waterloo
Noise Equivalent Bandwidth• This is defined as the voltage-gain-squared
bandwidth of the circuit
• The ideal case is that the (gain)2 is constant at avalue of A0
2 up to the bandwidth (A0 = voltagegain)
• But, the behaviour of a real circuit is not abrupt
• The NEB is defined as the point at which the twoshaded areas equal
(gain)2
A02
Bfrequency
(gain)2
A02
Bfrequency
ideal response
real response
124R.I. Hornsey, University of Waterloo
• Mathematically, this is given by
• So in the ideal case
• If we take the example of an RC low pass filter
• Calculating the transfer function
B = 1
A02 A f( )2 df
0
∞∫
bandwidth = 1
A02 A
0
∞∫
2
df =A0
2B
A02 = B
Vin
R
CVout
A ω( ) = vout
vin=
1jωC1
jωC+ R
= 11 + 2πfRC
since ω = 2πf
=f0
jf + f0 where f0 = 1
2πRC
125R.I. Hornsey, University of Waterloo
• At f = 0, A(f) = A0 = 1 for this circuit
• Now we can calculate the noise equivalentbandwidth, using A0 = 1
• The reason for choosing this example is that itis directly applicable to the resetting ofphotodiodes and the output nodes of CCD andphotogate pixels
B =f0
f 2 + f02
0
∞∫
2
df
= f02 f0
2 + f 2( )0
∞∫
−1
df
= π2
f0
126R.I. Hornsey, University of Waterloo
Reset Noise
• If we consider a diffusion (either a floatingdiffusion or a photodiode) being reset through aMOSFET
• Effectively, this is a capacitance being chargedthrough the resistance of the MOSFET channel
• So the ac-equivalent circuit is
Vreset
Vout
FETCVreset
R
R C Vout = 4kTBR
127R.I. Hornsey, University of Waterloo
• From before, the bandwidth is
• So we find the rms noise voltage
• Usually, the noise voltages is expressed interms of electrons, in order to compare directlywith the electrons in the well
• In which case the reset noise on the capacitor iscalculated from Q = nq = Cvout, and the rmsnoise electrons is given by
• This noise is generally called “kTC noise” or, inthis case, reset noise
• Calculating this out at room temperature gives
• For a floating diffusion C ~ 20fF, so nkTC = 55 e-
• For a (10µm)2 photodiode, C ~ 60pF, so nkTC =100 e-
» currently, reset noise limits the read noise in PDs
B = π2
f0 = 14RC
vout = kTC
ne = Cq
kTC
= kTCq
nkTC,RT = 400 C pF( )
128R.I. Hornsey, University of Waterloo
Shot Noise
• Shot noise is another white noise that arisesfrom the discrete nature of the electronsthemselves
» i.e. the random arrival of particles of charge
• This is the result of the random generation ofcarriers
» either by thermal generation within a depletion region(i.e. shot noise of the dark current)
» or by the random generation of photo-electrons,caused in turn by the random arrival of photons
• The rms signal is given by
• If the noise statistical distribution is describedby a Poisson distribution
» the variance is equal to the mean
» so <i2> = i
• So, if electrons are generated with a currentdensity, Jdark,in a sensor of area, A, over anintegration time, tint, the shot noise variance is
i = 2qIdcB
ndark2 = ndark =
JdarkAt intq
129R.I. Hornsey, University of Waterloo
• Similarly, the photo-electron shot noise varianceis given by
» where I0 is the photon flux (photons/cm2s) and η is thequantum efficiency
• So the total rms shot noise contribution fromthe sensor is
• For example, with
» Jdark = 200nA/cm2
» A = (10µm)2
» tint = 30ms
» I0 = 1013 photons/cm2s
» and η = 0.5
• we find <nshot> = √(37,500dark + 150,000pe) = 430 e-
npe2 = npe = ηI0At int
nshot = ndark2 + npe
2 = ndark + npe
=JdarkAtint
q+ ηI0At int
130R.I. Hornsey, University of Waterloo
Flicker (1/f) Noise
• At any junction, including metal-to-metal, metal-to-semiconductor, and semiconductor-to-semiconductor, conductivity fluctuations occur
» the causes of these are still not completely understood
• The rms 1/f noise current is given by
• 1/f noise arises mainly in amplifier circuitswhere there are numerous such contacts
• At low frequencies, 1/f noise can be thedominant component
» but, at higher frequencies, the 1/f noise drops belowthe thermal noise
» the frequency at which this happens depends on thesituation
i1 / f ∝ IdcBf
log fthermal
1/f
log <i>
131R.I. Hornsey, University of Waterloo
Array Noise Componentsreset
sensor
in-pixelamp
columncircuit
on-chipo/p amps
A-D conversion
•reset (kTC) noise•FPN (from VT)
•photon shot•dark shot
•FPN•PRNU
•thermal•1/f
•FPN•(PRNU?)
•thermal•1/f
•FPN•kTC
•thermal•1/f
•quantisation noise
to noisy off-chip amps!
++
++
++
++
++
++
NO
T P
RE
SE
NT
IN
CC
D
132R.I. Hornsey, University of Waterloo
“Referred” Noise Figures
• Conventionally, the noise figures are referredeither to the final output or to the output of theoptical detector
» i.e. to be compared directly with the number ofelectrons generated by the detector
» called input referred noise
• For input referred noise, the noise of laterstages must be divided by the gains of theintermediate stages
• Or vice versa for output referred noise
• Usually, authors in CMOS circles use the inputreferred figure
» but this is tough to obtain for intermediate stages inthe circuit owing to uncertainties in the gains of eachstage
» only the overall figure in electrons is practicallyfeasible because the appropriate inverse-conversionefficiency (e- per µV) is only known for the entireoutput circuit
133R.I. Hornsey, University of Waterloo
Typical Noise Figures
• From Mendis, the calculated and measuredinput referred noises for a 128x128 elementphotogate array are
• Mendis also reported a photodiode read noise of~80 e- rms
• Typical read noises for CMOS sensors
• Remember that this does not include shot noiseor pattern noise
Noise source Calculatedrms
Measuredrms
kTC from reset of FD negligible negligibleIn-pixel amp. 1/f 111µVkTC from columnsample & hold
93µV
Column sourcefollower 1/f
46µV
Total column noise 86µV 120µVTotal noise 152µV 151µVTotal noise electrons 41 e - 41 e -
Technology RMS read noisePhotodiode APS 50 - 80 e -
Photogate APS 20 - 40 e -
Logarithmic APS 700 e -
Passive pixels 200 - 300 e -
134R.I. Hornsey, University of Waterloo
Fixed Pattern Noise
• Fixed Pattern Noise is due to pixel-to-pixelvariations in the absence of illumination
• The main cause of FPN in CMOS imagers isvariations in VT
» between reset and buffer MOSFETs in the pixel
» and between MOSFETs in the column circuits
• FPN can also arise from repeating irregularitiesin the array clocking
» allowing small variations in integration time etc.
• In very large arrays, resistive drops in resetbuses may lead to a “droop” in the voltage towhich the pixels are reset,
» but this is not usually significant in CMOS imagers
• FPN is just as valid as a “noise” as the temporalvariety
» both affect the actual output voltage that the pixelproduces
» in a way that is not directly related to the illuminationto be measured
135R.I. Hornsey, University of Waterloo
PRNU
• The issue of photo-response non-uniformity hasnot historically received much attention in theCMOS imager community
» although there is now some occasional mention of“gain nonuniformity”
• Like FPN, PRNU is essentially time-independent,but it is signal-dependent
• Both types of pattern noise can be specified interms of either an rms or a peak-to-peak value,referenced to an average value
» e.g. the full-well capacity
• A histogram of output signals is built up in thedark or light, as appropriate
» PNrms = rms of distribution / average value
» PNp-p = peak-to-peak variation / average value
• Since PRNU is signal dependent, it is oftenexpressed as a multiplier of the number ofphotons
» <nPRNU> = Unpe
136R.I. Hornsey, University of Waterloo
Minimum Noise
• In principle, the noise floor and dark current canbe reduced so that the system is photon shotnoise limited
» this approximation is sometimes used to calculate thepixel sensitivity (µV/e-)
• But there will never be zero PRNU, so a moreachievable value would be
» The worst case when npe = nfull-well
• If we plot out this limiting noise as a function ofPRNU, it looks like
nsys = npe = npe
nsys = npe + Unpe( )2
100
1000
10000
0.01 0.1 1 10
photon shotnoise
PRNU
rms n
ois
e e
-
PRNU (%)
“knee” is optimum cost/performancevalue for PRNU, hereU ≈ 1/√npe npe = 50,000 e-
npe = 100,000 e-
137R.I. Hornsey, University of Waterloo
Noise ReductionTechniques• Having seen some of the common sources of
noise in CMOS imaging systems, how might wego about reducing them?
• Essentially, there are three classes of noise
» those we can do nothing about, such as photon shotnoise
» those we can reduce by careful design of circuitcomponents, such as thermal noise
» those we can reduce by circuit design, such as FPN
• These techniques are inter-dependent
» we shall see that adding extra circuitry to reduce FPNalso introduces extra 1/f and kTC noise
» so the optimisation of noise is a system issue, not justa question of optimising each element individually
• We will look at the general techniques forreducing noise in electronic devices, as well ascircuit techniques for pattern noise etc
• The study of noise is a specialised topic, and wewill only look at the essentials
138R.I. Hornsey, University of Waterloo
Shot Noise
• As we remarked earlier, photon shot noise isdependent on the illumination level, and there isnot much we can do about it
» except reduce the QE of the detector, which we don’twant to do!
• Shot noise also arises from the pixel darkcurrent
» which we can alter
• By changing doping levels, we can reduce thedark current
» but, in a regular photodiode, at the expense of QE
• And by removing the collection area away fromthe surface
» this is another advantage of the pinned photodiode
• The magnitude of the dark current is, of course,dependent on the pixel area
» so the shot noise will be smaller for smaller pixels
» although the perimeter component of the dark currentmeans that S/N still gets worse as pixel dimensionsare reduced
• But shot noise is not usually the limiting factor
139R.I. Hornsey, University of Waterloo
Thermal Noise
• Thermal noise is important mainly in the inputstages of amplifiers
» because of the √(4kTBRchannel) from the MOSFETs
• In general, the power spectrum of the thermalnoise will be proportional to (W/L)-1
• But it is also dependent on the current throughthe devices
• A common way of expressing thermal noise isthe noise electron density (NED)
» where en(f) is the total equivalent noise voltage at theoutput stage (e.g. a floating diffusion)
» and Ct is the total capacitance present at the input,including diffusion capacitance, gate capacitance, andeverything else (to convert to electrons)
• en(f) represents the device noise, referred to theinput
» and so includes all factors such as the transistorgeometry, iDS, device area etc. that affect the gain
NED f( ) = en f( )Ctq
2
140R.I. Hornsey, University of Waterloo
• NED, expressed in electrons2/Hz, is sketchedbelow for a bias current of 100µA
• Increasing W, for a fixed L, increases Ct becauseof the device area
• But increased W decreases en(f)
» because the current iDS, and the gain of the circuit, areincreased
» therefore reducing the input referred noise
• So the optimum condition is to keep W ≈ 15µmin these transistors
» too small a W reduces the gain
» and too large a W increases the C
• Also the NED decreases with L, which makessmaller devices advantageous
Transistor width, W (µm)
10 20 30 40 50
100
80
60
40
20
NE
D
(ele
ctr
on
2/M
Hz)
L = 3.2µm
L = 8.7µm
141R.I. Hornsey, University of Waterloo
1/f Noise
• 1/f noise arises mainly from trapping de-trapping of electrons at the Si-SiO2 interface
• We can do some things to minimise 1/f noise
» reduce device W and increase device L
» use a buried channel device to separate the channelfrom the interface
• While in standard CMOS we cannot do muchabout the second option, we can change W, L
» but the gain of the amplifier is dependent on W/L
» so increasing both W and L is the best choice!
• In the pixel, this is a bad thing since we want tominimise the areas of the transistors toachieve a high fill factor
» and care is needed to ensure that the pixelsource follower can adequately drive the columncapacitance (i.e. enough W/L)
• Note also that we need the noise added at earlystages of the process to be minimised
» since this is amplified at all subsequent stages
142R.I. Hornsey, University of Waterloo
Correlated Double Sampling
• Reset noise is difficult to design out of thesystem
» since the properties of the transistor cancel out
» although reducing the capacitance of the node isuseful for both kTC and conversion efficiency
• So the most common solution is to measure thevalue of the reset noise and then to subtract itfrom the signal
• A generic circuit for achieving this in a CCD orfloating gate APS would be
• During the sample and hold period, theappropriate switches are pulsed on & off
» to leave the voltages stored on the capacitors
+
–S&Hreset
S&Hsignal
FD
RST
143R.I. Hornsey, University of Waterloo
• The sequence of events for a CCD or photogatewould be
• Here, the reset signal is given by
» Vreset = [VDD -(VT ± ∆VT)] ± (VkTC) ± (Vpart)
• Here the ∆VT is the component of FPN arisingfrom mismatches between the reset transistors
» and is approximately the same for each frame
• VkTC is the reset noise
» and is different from frame to frame
• Note that we are considering voltages (notelectrons) at this stage
» so the “kTC” noise is given by √(kT/C), and istherefore reduced for larger C
Subtract two stored valuesoutput = (signal + reset) - (reset) = signal
Sample & hold signal charge
Transfer signal charge(charge = signal + reset)
Sample & hold value of reset signal(charge = reset)
Reset pixel
144R.I. Hornsey, University of Waterloo
• Vpart refers to what is called the partition noise
» when the reset FET turns off, the channel chargemoves either to the source (= FD) or to the drain
» difficult to determine how much goes to each
• This type of sample-and-hold technique isknown as correlated double sampling (CDS)
• The “correlated” part comes about because thenoise component of the two signals iscorrelated, and can therefore be subtracted out
• In a CCD, a single CDS circuit is neededbecause there is only one floating diffusionoutput node
• In CMOS APS, there is an output node per pixel
» but practically, we need only one CDS circuit percolumn of the array
» and the S&H is carried out for all columns in parallel
CDS
activerow
outputs
145R.I. Hornsey, University of Waterloo
CDS for Photodiode APS
• This form of CDS works very well for pixels witha floating diffusion output node
» photogate, and pinned photodiode with transfer gate
• Indeed, the main advantage of using thephotogate structure is to facilitate the removalof reset noise
» since the improvement in conversion efficiency isoffset by the lower QE
• In photodiode designs, double sampling canonly remove the FPN that results frommismatches
» this is because the double sampling is not correlated
• In FD designs, the signal was added to theexisting (and stored) reset value
» so the subtraction was of exactly the same noisesignals
• In the photodiode, there is no separate outputnode, so the signal must be read out first
» and this signal includes the original reset voltage onthe photodiode
» which in turn includes FPN and kTC noise
146R.I. Hornsey, University of Waterloo
• We can now reset the pixel again and subtractthis value
» the FPN will be much the same as that which wasincluded when we sampled the signal
» but the kTC noise will be different, i.e. not correlated
» remember kTC is the rms value of a distribution
• So now the sequence of events is
• This would be better termed pixel doublesampling
• Or, alternatively, a graphical representation is asfollows ...
Subtract two valuesoutput = (new signal + old reset) - new reset
Sample & hold reset valuecharge = new reset value
Reset pixel
Sample & hold signal valuecharge = new signal + old reset value
147R.I. Hornsey, University of Waterloo
• For example, if we read out a single pixel overseveral integration periods
» assuming constant illumination (I.e. slopes parallel)
tint 2tint 3tint
idealsignal
measured signalVreset - Vsignal
VDD
VDD - VT
tint 2tint 3tint
range dueto kTC
S&H resetS&H signal
Vreset - Vsignal
148R.I. Hornsey, University of Waterloo
• This is the reason why reset noise is now thelimiting noise source in photodiode circuits
• Note that even this noise reduction isunavailable to the logarithmic pixels, hence theirpoor FPN characteristics
• The conventional CDS circuit used in CMOSsensors is shown below
• Of course, the additional circuitry required forthe CDS implementation adds further noise tothe signal
» kTC from the sample-and-hold capacitors
» 1/f and thermal noise from the transistors
• But usually in CMOS sensors, the FPN is themore critical issue
VDD
load
load
columnload
ref. select
signalselect
SHR
SHS
VDD
VSS
VSS
VOUT
VREF
from column
1 pF
1 pF
149R.I. Hornsey, University of Waterloo
Column FPN
• The other issue with using column-wise CDS isthat FPN is then added by the CDS circuitsthemselves
» appearing as vertical streaks in the image
• This can be removed by storing and subtractingcolumn reference signals off chip
• Alternatively a second stage of double samplingis performed
» where, after the readout of the differential signal, theS&H capacitors are shorted together
» this results in a differential output that is a measure ofthe mismatches between the two sets of output stages
» Mendis calls this a “crowbar” circuit and the processdelta difference sampling
+
–
S&Hreset
S&Hsignal
FD
RST
“crowbar”switch
150R.I. Hornsey, University of Waterloo
Typical Figures
• Typical figures for FPN are hard to definebecause it depends so much on the preciseprocess used
• For photogates with a 2µm CMOS process,Mendis reported a p-p FPN of 1% – 2.5%saturation with the CDS circuit
» falling to ~0.1% sat. with the DDS as well
• A photodiode fabricated similarly showed a p-pFPN of ~0.5% sat. after CDS, and ~0.1% afterCDS + DDS
» typical raw data are about 2 - 3% p-p sat.
• For a 0.35µm process, the raw FPN for a PGarray was 6% sat.
» reducing to 0.4% after off-chip correction
• Mansoorian et al. give a final FPN of 0.6% sat. p-p for both PG and PD using a 0.55µm process
» using a similar DDS technique
• For logarithmic pixels, IMEC report a raw FPN of~100% of the useable signal range!
151R.I. Hornsey, University of Waterloo
Patent Issues
• Other methods of reducing noise are possible,although probably not so good
• Hitachi have several patents which cover theidea of active pixel sensors and the use of CDSin these devices
» in their CMOS digital still camera, VLSI Vision use amechanical shutter in order to measure a true darkimage for subsequent subtraction
• One possibility is to smooth out large signalvariations between neighbouring pixels
» a smooth curve is fitted through points either side ofthe test point, and the test point moved to fit that curve
» the smoothing is improved if the number of neighboursis increased, but the “sharpness” of the image islowered
“corrected” value of test
real value of test
pixel count
signal
neighbouring test pixels
152R.I. Hornsey, University of Waterloo
Feedthrough & Crosstalk
• We have considered here “natural” sources ofnoise such as 1/f, thermal, and shot noise
• And technological noise, such as FPN andPRNU
• In addition to these, there can be unwantedsignals in one part of the circuit due to theoperation of another part
» these can be addressed in the design of the array andcircuits
» although some sources are not easy to eliminated
• Feedthrough of digital signals from control linesinto the analog parts of the circuit can be aproblem
» analog and digital sections of the chip can beseparated to some extent
» but the array itself, and much of the analog signalprocessing is intrinsically both analog and digital
• The minimisation of these effects requirescareful layout
» and mixed signal design is currently a hot topic inmany areas, such as A-D & D-A conversion, DSP etc.
153R.I. Hornsey, University of Waterloo
References – Part III
» H.W. Ott (1988), “Noise reduction techniques inelectronic systems”, Wiley
» G.C. Holst (1996), “CCD Arrays, cameras anddisplays”, SPIE Press
» T.E. Jenkins (1987), “Optical sensing techniques andsignal processing”, Prentice Hall
» S.K. Mendis (1995), “CMOS active pixel imagesensors with on-chip analog-to-digital conversion”,PhD Thesis, Columbia University, USA
» B. Mansoorian et al. (1997), “Megapixel CMOS APSwith analog and digital outputs”, IEEE CCD and AISWorkshop, Bruges, Belgium, June 5 - 7, 1997
» S.K. Mendis et al. (1997), “Active pixel image sensorsin 0.35µm CMOS technology”, IEEE CCD and AISWorkshop, Bruges, Belgium, June 5 - 7, 1997
» O. Yadid-Pecht et al. (1997), “Wide intrascenedynamic range CMOS APS using dual sampling”,IEEE CCD and AIS Workshop, Bruges, Belgium, June5 - 7, 1997
» B. Dierickx et al. (1997), “Offset-free offset correctionfor active pixel sensors”, IEEE CCD and AISWorkshop, Bruges, Belgium, June 5 - 7, 1997
156R.I. Hornsey, University of Waterloo
CMOS Arrays
• The basic scanning of the array requires severalsets of inputs
» bias voltages
» clocking signals
» reset signals
• Bias voltages are required by the column sourcefollowers, and the S&H source followers
» and possibly by subsequent amplifier stages
• While bias voltages can be simply supplied by aresistive potential divider, this is not usuallyperformed on-chip because
» accurate resistors are hard to fabricate in digitalCMOS
» potential dividers dissipate power continuously,especially because large resistors are not feasible
» and we may well wish to adjust the biases, perhaps tochange the operating range of the sensor
• Usually, some externally adjustable voltages arerequired to program an internal active referencevoltage generator unit
» based on a bandgap reference source for stability
157R.I. Hornsey, University of Waterloo
• An alternative approach is being pursued byJPL for low-power, digitally programmable stillcamera chip
• Here, four 5-bit digital-to-analog converters areused to program the DC reference voltages
» the digital input signal to each DAC can be latchedwith little power consumption
» and the DACs are powered down when their output isnot required to reduce power consumption
• This highlights an important area ofdevelopment, for battery-powered still camerasin particular
» power conservation by intelligent on-chip powercontrol
» e.g. standby modes (40µW for the JPL camera,~10mW during operation) by turning off analogcircuitry
» e.g. column-parallel ADCs only operated when columnis read out (plus maybe one warming up)
158R.I. Hornsey, University of Waterloo
Timing• How the timing of the array readout is controlled
depends on the application
• In many cases, adherence to a standard videooutput format is not required
» and there is more flexibility in the control methodology
• If, however, the camera has to output signals instandard TV formats, for example, the frame rateand signal format are precisely defined
• The longest integration time allowable for thesensor is 1/(frame rate)
» so, 1/30 s for standard NTSC (National TelevisionSystem Committee) TV rates
• Video standards determine every aspect of thesignal output
» levels and timing of line, frame, blanking etc. pulses
» black levels and white levels
» compression of information – gamma correction
» frame rates
• There are two major standards, CCIR & EIA
» International Radio Consultative Committee (Europe)
» Electronic Industries Association (North America)
159R.I. Hornsey, University of Waterloo
• And in each standard, there are two majorscanning modes
» progressive scan and interlaced scan
• The progressive scan achieves a high resolutionby displaying every line of the image
» e.g. computer monitor
• The interlaced scan achieves the same apparentresolution, but with a reduced bandwidth, bydisplaying every other line, and then going backto fill in the gaps
» e.g. all TV sets
» originally, the bandwidth of the radio transmission wasa major factor in adopting interlaced TV broadcasting
• Each scan is called a field, and each full pictureis a frame
» so 1 frame = 1 field for progressive
» and 1 frame = 2 fields for interlaced
1234567
1526374
160R.I. Hornsey, University of Waterloo
• The frame rates are specified by the standardfield frequency
» CCIR: 50 Hz
» EIA: 60 Hz
• So an interlaced EIA system displays 30 frames(i.e. full pictures) per second
• In a CMOS imager, the easiest way to achieveinterlacing is simply to read out alternate rowsof the imager
» so there is effectively an independent array for eachfield
» [this does not work so well for CCDs, since the twofields cannot be reset separately; hence the fieldshave overlapping integration times, causing oddeffects for moving objects]
• After all the image processing, cameras forstandard video formats may include somedeliberately non-linear weighting
» known properly as “inverse-gamma correction”
» but often called simply “gamma” or “gammacorrection”
161R.I. Hornsey, University of Waterloo
Gamma Correction
• The need for gamma correction arose becausethe inherent behaviour of CRT monitors is non-linear
» the brightness of the monitor is related to theaccelerating voltage of the electrons by
• So, if the camera were to have the inverserelationship, the properties of the scene wouldbe maintained
• Of course, this only works provided the correctmatch of camera and monitor is used
» although image processing systems, includingcomputers, usually offer a choice of gamma correctionparameters
• Because of the non-linearity, image processingshould be performed on the linearized image
» even if the signal is gamma corrected againafterwards
Ldisplay = K Vacc( )γ
Ldisplay = K Vacc( )γ = K Vscene
1γ
γ
= KVscene
162R.I. Hornsey, University of Waterloo
• The precise value of gamma depends on theCRT manufacturer, but standard values arerecommended
» NTSC has γ = 2.2
» or an inverse γ of 0.45
• For low levels of light intensity, the effect ofgamma correction is very large, so there isusually some cut-off point below which gammais not applied
• In the camera, the gamma transformation isoften performed using a look-up table
» to convert the digital signals appropriately
» and in some cases, manufacturers may use their owncompression system to enhance contrast or signal-to-noise ratios
163R.I. Hornsey, University of Waterloo
Exposure and Gain Control
• Exposure control refers to the amount of lightthat is being sensed by the camera
» primarily controlled by the integration time
• The simplest way of defining the integrationtime is to make it equal to the frame time
» i.e. the time between reading the pixel once, andreading it again on the next frame
• Alternatively, for non-TV applications, anelectronic shutter can be used to define theexposure time
• An integrated camera must be able to adjust itsexposure to the prevailing conditions byadapting its integration time
» but it can also adjust the gain of subsequentamplification stages to achieve the optimumcombination of detector sensitivity and output voltageswing
164R.I. Hornsey, University of Waterloo
• In the VLSI Vision 5400 series monochromecamera chips, the gain and integration time arecontrolled together
» to maintain a relatively constant average output signal
• The number of pixels in a frame whose signal isabove some predefined threshold value iscounted
• If, for example, there are too few bright pixels
» first the integration time is increased
» if the integration time is near the maximum value, thegain is increased and the integration time reduced
» to allow room for further increase in the integrationtime
» the total range of exposure control (from min. tint &min. gain, to max. tint & max. gain) for the VVL 5400chips is about 100,000:1
• The same chips also allow for longer exposuresin part of the image in a “backlit mode”
» for dark objects on a bright background (snow, sand,sky etc)
» by applying a higher threshold in the central part of thearray
» this is a simple version of the methods used toincrease the dynamic range (see later)
165R.I. Hornsey, University of Waterloo
Clocking and Speed
• The way in which the array is scanned verymuch depends on the application
» modifications must be made to accommodateinterlacing, variations in integration time, electronicshuttering etc.
• In the basic system of photodiode readout, wemust supply clocks to operate row and columnscanners, the two sample and holds, and thepixel reset
row advance
S&H signal
S&H reset
reset
column advance1 2 3 n
add delay here for longer tint
166R.I. Hornsey, University of Waterloo
• Most camera chips generated these clocks on-chip from an externally regulated master clock
» usually a crystal oscillator like a computer to achievefrequency stability
» although on-chip oscillators are also used formaximum integration
• As with all circuits, the readout has a maximumfrequency of operation, determined by
» the fabrication technology
» the detailed design, both low-level and overall circuitfunctionality
• In particular, it is hard to drive large arrays fast,because of the capacitance of the address lines
» i = C dV/dt
» so larger C requires that the drivers of the addresslines must be able to source more current
» this requires larger transistors, and multiple stages(e.g. cascaded inverters), which also add delays
• It also makes the use of smaller pixelsadvantageous
» to reduce the array physical dimensions (hencereduce C) while maintaining resolution
» also reduces cost
167R.I. Hornsey, University of Waterloo
• CMOS addressing is typically less capacitativethan for CCDs, but the on-chip drive circuitrycannot be too large
» because of the power dissipation
• For large arrays, such as the VVL 800K-pixelsensor, readout rates are 5 - 10 MPPS
» mega-pixels per second
» which is about typical
» or a few fps for megapixel arrays, and 30 - 100 fps forVGA (640 x 480) or less
• For specialised applications, such as industrialinspection, high frame rates may be required
» the Integrate Vision Products (IVP) MAPP2200 cancapture 4000 fps with a 256 x 256 array
» an impressive 260 MPPS
• But, in general, limiting the integration time tothe frame time results in tint being too long
» so how can we make a widely variable tint?
• This is also linked to the issue of reducingimage blur for rapidly moving objects
» e.g. scissor blades passing at 400 per minute
• The answer is an electronic shutter, analogousto that found in a mechanical camera
168R.I. Hornsey, University of Waterloo
Electronic Shutter
• In a regular film camera, the shutter consists ofa slit made of two parts
» the first edge moves across rapidly, exposing the filmto the light
» the second edge follows the first, at a distancedetermined by the desired exposure time, cutting offthe illumination
» the exposure time is d/vslit, where d is the slit widthand vslit is the velocity
» this method, called a rolling shutter, is used to obtainimproved high shutter speeds
• The same method is also used to expose CMOScamera chips
» except that opening the “shutter” refers to selectiveactivation of rows of pixels
» rather than to a mechanical shutter
vslit
d
169R.I. Hornsey, University of Waterloo
• Thus the readout is implemented as follows
• The integration time is thus variable from (1/m)to (1) times the frame time
» where there are m rows
» 70µs - 33ms for a VGA at 30fps
• A derivative of this technique has also beenused to increase the dynamic range of thecamera
» by allowing two integration times
scan
integratepointer
readpointer
integration period
inte
gra
te s
can
ner
read
/reset
scan
ner
de-activated pixelsare kept “reset” to
avoid blooming
170R.I. Hornsey, University of Waterloo
WIDyR Sensor
• The Wide Intrascene DYnamic Range (WIDyR)sensor employs the same principle as the rollingshutter
» but has an additional column readout system
• The integration times can be controlled
» but not independently
» and the longest time depends on dark current
read &reset 1
read &reset 2
tshorttlong
output of integration
between reset 2 and read 1
output of integration
between reset 1 and read 2
171R.I. Hornsey, University of Waterloo
Selective Spatial Scanning
• Many of the non-video-standard cameras offertwo modes of selective spatial scanning
» windowing
» sub-sampling
• Windowing refers to the selective scanning of arestricted area of the array
• Sub-sampling is the readout of every nth pixel
• The idea of both schemes is to get informationabout the scene without collecting all the data
» usually to enable faster scanning
window
172R.I. Hornsey, University of Waterloo
• From the previous points, the scanning circuitshave to perform several functions
» sequentially scan out the columns and rows
» offer enough drive capability for the address lines
» be programmable for start, stop, and step
• There are two major candidates for the scanningcircuits
» shift register and counter/decoder
• Shift registers are simpler and more expandable
» every two clock periods, the signal on o/p1 istransferred to o/p2
» so whatever sequence we input at one end is passedalong to the array
Shift Registers
φ
φ
φ
φ
o/p 1 o/p 2
173R.I. Hornsey, University of Waterloo
• So, to read out all the columns, we put a “1”followed by (n - 1) “0”s into the shift register,and this sequentially activates the columnreadouts
• The real advantage of the shift register is that itcan be made infinitely long without any changein architecture
» just by adding on extra cells
» the input circuitry etc all remains the same
• It can also handle two pulses for the shutter andthe WIDyR sensor, although it may well be betterto use two separate registers
• However, the shift register does not easily allowanything more complex than sequential readout
» random addressing is impossible; a “1” has topropagate all the way from the input before it readsout a column at the other end
» windowing & sub-sampling; how to determine startand stop? How to skip columns?
• So shift registers are usually used in caseswhere unconventional readout is not required
174R.I. Hornsey, University of Waterloo
Decoder Scanning
• Decoder scanning has more in common with theaddressing of computer memory
» the decoder takes a binary input and selects theappropriate sequential output
• This allows any output to be selected at random
• Or, to scan the array, a binary counter canprovide the input
» this counter can have programmable counting limitsfor windowing
» and programmable counting steps for sub-sampling
• The main disadvantage is that the decodingcircuitry is more cumbersome and lessstraightforward to expand
» although modern layout tools minimise this problem
0 15...
0123binary input
175R.I. Hornsey, University of Waterloo
Analog-to-DigitalConversion• The preceding sections have covered the main
building blocks required to operate anintegrated camera
• However, there is an increasing demand fordirect digital signal output, especially forcomputer interfacing such as
» digital still cameras
» machine vision & industrial inspection
» interne-based video telephony
• But there are many possible ways of includingADCs onto the chip
» one ADC per chip
» one ADC per column
» one ADC per pixel
• The choice depends on a number of factors,such as
» speed of conversion (1 - 50MPPS for video rates)
» space (area of layout, column pitch)
» number of bits (at least 8 for most applications)
» power consumption (<100mW to avoid local heating)
176R.I. Hornsey, University of Waterloo
General Considerations
A/D
A/D A/D A/D A/D
A/D A/D A/D A/D
A/D A/D A/D A/D
A/D A/D A/D A/D
A/D A/D A/D A/D
A/D A/D A/D
A/D
In pixel
Per column
Per chip
Factor Inpixel
Percolumn
Perchip
Speed l £ ¡S/N l £ ¡FPN ¡ £ lAccuracy ¡ £ lFill factor /resolution
¡ l l
Powerconsumption†
¡ £ l
l good £ medium ¡ poor† Power consumption depends on how the system is operated.
177R.I. Hornsey, University of Waterloo
Flash ADCs
• Flash ADCs, sometimes also called parallelADCs are the fastest converters, owing to theirparallel operation
• All comparators whose reference is below theinput voltage gives an output “1”
» and the priority encoder outputs the binary numbercorresponding to the highest active comparator
R
R
R
R
R
+
+
+
+
+
Vin Vref
digitaloutput
priorityencoder
smallestreferencevoltage
largestreferencevoltage
178R.I. Hornsey, University of Waterloo
• The flash ADC is fast
» limited only by the delays in the comparators and theencoder
• But it requires 2N - 1 comparators (N = # bits)
» so the number ~ doubles for each extra bit
» and this requires a large area on the chip
• Moreover, the resistor chain imposes additionallimitations
• Firstly, there is a continuous current flowthrough the resistors
» Rs can be made large to reduce this, at the expenseof real estate
• Secondly, the accuracy of the ADC depends onaccuracy to which the resistors can befabricated
» limited to 8 - 10 bits without special “trimming”
179R.I. Hornsey, University of Waterloo
Successive ApproximationADCs• These ADCs converge on the correct digital
word by comparing the input voltage with theoutput of a DAC corresponding to that word
» and adjusting the digital word to bring the DAC outputcloser to the analog input
• The converter first tests the most significant bitof the digital word
» and turns it on or off according to whether theresulting DAC-ed voltage is greater or less than theinput
» then each lesser significant bit is tested until the bestdigital equivalent is reached
• Thus the successive approximation ADCrequires N clock cycles to perform theconversion
DAC
Successiveapproximation
register
clock
output
Vinput
180R.I. Hornsey, University of Waterloo
Single-slope ADC
• Also known as the Counting ADC, this is similarto, but simpler than, the successiveapproximation type
• Now, the binary counter is incremented (ordecremented) until the output from the DAC justexceeds (or falls below) the analog input
» when the comparator inhibits the counter
» and the output of the binary counter is the best digitalapproximation to the analog voltage
• The speed and accuracy of the conversion arenot good
» the speed depends on the voltage to be converted
» but the area and the power consumption are relativelylow
DAC
Binarycounter
clock
output
Vinput
stop
181R.I. Hornsey, University of Waterloo
Dual-Slope ADC• The dual-slope, or ratiometric, ADC is another
counting converter, but is more sophisticatedand robust than the single slope method
• Here, the analog input it integrated for a fixedperiod of time
• And then the time required for a knownreference voltage to achieve the same finalvalue is measured by a binary counter
• This is achieved by using an integrator with ananalog switch to determine the input
-Vintint
tint
t
Vint int
Vref
182R.I. Hornsey, University of Waterloo
• The accuracy of this technique can be very high,but at the expense of a long integration time
• Moreover, the effects of digital noise can beremoved by ensuring that the integration periodis an integral multiple of the interfering period
• But the power consumption and area are stilllow
–
+
Vref
(- ve)
Vin
–
+
binarycounter
clock
inhibit
analogswitch
183R.I. Hornsey, University of Waterloo
Sigma-Delta ADC
• Sigma-Delta (∑-∆) converters offer an excellentimmunity to process variations
» whereas previous types rely on high accuracycomponents
• This is because it functions on an averagingbasis
» in its simplest form the ∑-∆ converter block diagram isas follows
• During each clock cycle, the integratoraccumulates multiples of the input voltage
» until the total voltage exceeds the reference voltage,Vmax, and the output, q, goes to “1”
» at which point, the feedback loop subtracts Vmax again
integrator(summer)
–
+Vin
(< Vmax)
Vmax
+
–counter
clock
qv
184R.I. Hornsey, University of Waterloo
• The process repeats and, after another fewperiods, q → 1
• So, by counting the number of “1”s over 2N
clock periods (N = # bits), a binary counter willgive the analog output
• More complex versions of the ∑-∆ converter(second order etc) are possible
» they reduce the number of clock cycles for a givennumber of bits
» but at the expense of extra feedback loops, withhigher real estate costs
0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 1 0
Vmax
v
Vin
185R.I. Hornsey, University of Waterloo
Summary
• Following Mendis, the merits of the variousADCs can be summarized as follows
• Here, robustness refers to the immunity of thetechnique to the process variations inevitable inthe use of standard CMOS technology
• It is generally considered that flash ADCs aretoo large for other than one per chip
» but that the others would be suitable for one percolumn
» and one per pixel is not really viable
Type Resol-ution
Speed Power Area Robust-ness
Sigma-Delta
High Slow Low Low High
Succ.Approx.
Med. Fast High Med. Med.
Single/dualslope
High Slow Low Low Low
Flash Low Fast Veryhigh
High Med.
188R.I. Hornsey, University of Waterloo
Colour Processing
• For most consumer applications, colour imageshave become mandatory
• While the technology to sense colour has beenavailable for a long time, the on-chip colourprocessing and video encoding is only nowbecoming feasible
• One method for obtaining very high qualitycolour images is to use three separate sensors
» and to split the light into primary colours (red, green,blue) by a prism
» or to use a single sensor and a colour filter wheel
• The principle of operation is that additive mixingcan reproduce the entire spectrum of colours
• An important factor when designing colourimaging systems is that the human observer ispart of the system
» and so human perception must be considered whendetermining the performance of the sensor
» the human colour perception has been standardizedsince 1931!
189R.I. Hornsey, University of Waterloo
Colour Filter Arrays• For single-chip colour sensing, individual pixels
in the array are covered with coloured filters
» to form the colour filtered array (CFA)
• The spectral responses of conventional R, G, Bfilters are sketched below
• An alternative to the RGB system is to use thecomplementary colours, which have a higherpercentage transmission
» yellow (Ye) = R + G = W - B
» magenta (Mg) = R + B = W - G
» cyan (Cy) = G + B = W - R
» where W = white (transparent)
B G R
400 500 600 700Wavelength (nm)
1.0
0.8
0.6
0.4
0.2
Tra
nsm
issio
n
190R.I. Hornsey, University of Waterloo
CFA Arrangement
• Coloured filters are typically arranged either instripes or in mosaics (checkerboards)
» the mosaic is often called a Bayer pattern after itsinventor (and 1976 patent holder)
• While the stripes have some advantages interms of simplicity of fabrication, the horizontalresolution suffers seriously
» and all CMOS imagers use the Bayer patterns
» recall that the VVL 800K-pixel colour sensor usesRGB and “teal” (blue/green) to achieve a propercolour balance with their process
R
R
R
R
G
G
G
G
B
B
B
B
R
R
R
R
G
G
G
G
B
B
B
B
G
G
G
G
Ye
Ye
Ye
Ye
G
G
G
G
Cy
Cy
Cy
Cy
G
G
G
G
Stripes
Mosaics
R
G
R
G
G
B
G
B
R
G
R
G
G
B
G
B
Mg
Ye
Ye Ye
G
G
Cy Cy
Cy Cy
G
G
Mg
Ye
Mg Mg
(Bayer pattern)
R
G
R
G
G
B
G
B
R
G
R
G
G
B
G
B(modified Bayer)
191R.I. Hornsey, University of Waterloo
Matching Human Response
• The human eye derives most of its spatialinformation from the green part of the spectrum
• So the standard reponse calls for 59% of theluminance (brightness) signal to come fromgreen, 30% red, and 11% blue
• But, if only 11% of pixels detect blue light, thespatial resolution is too low, so the compromisesolution is the Bayer pattern with 50-25-25
» the modified Bayer pattern is sometimes used tofacilitate the interlaced scanning by repeating thepattern on adjacent rows
• The signals are then processed to achieve thecorrect colour balance
» in NTSC this is known as matrixing
• A selection of matrices may be available for useunder various illumination conditions (scenecolour temperatures)
» e.g. tungsten lighting, twilight, full sun
R2
G2
B2
=k11 k12 k13
k 21 k22 k23
k 31 k32 k33
R1
G1
B1
192R.I. Hornsey, University of Waterloo
• The image processing required to NTSC colourprocessing is quite sophisticated, which is whya single chip version is only now available
» from VVL (ISSCC98)
• Another potential difficulty that arises from theuse of CFAs is the low resolution of eachcoloured “array”
» leading to the possibility of aliasing
» this can be corrected with off-chip filtering, at theexpense of further image processing
• Once the NTSC colour balance has taken place,the signals are then gamma corrected
• So the whole system may look as follows
array
S/H
S/H
S/H
R
B
G
colourcorrec-
tionmatrix
γ
γ
γ
NTSCencoder
193R.I. Hornsey, University of Waterloo
Advanced focal-planeprocessing• One of the most fascinating aspects of image
sensor design is the constant battle to imitatenature
• It is a pretty good rule of thumb that if you wantto find any engineering solution, look first athow nature does the job and try to imitate it
» mechanics, structures, sensors
• Of course, there are things that artificialconstructions can do that nature cannot, andvice-versa
» and nature often has the edge by combining physics,chemistry and engineering at microscopic level
• Because of the importance of sight to humans,visual perception has always held a particularinterest
• So it is not surprising that many attempts havebeen made to implement biologically-inspiredvision systems in VLSI
» these run the range from attempts to verify models ofperception, through to engineers trying to satisfy anapplication
194R.I. Hornsey, University of Waterloo
Human Eye vs. Silicon(B. Dierickx, IMEC) criterion eye CCD CMOS APS film
spectral
response
400-700 nm
peaked at 555
400-1000 nm
smooth
400-1000 nm 300-700 nm
peak quantum
efficiency
<20% >50% >50%
dynamic range 1e6 logarithmic
1e2 linear
1e4 linear 1e4 non-linear both
dark limit 0.001 lux
1E-6 W/m2
typ: 0.1 lux
<0.0001
possible
typ: 1 lux
0.001 possible
virtually zero
noise photons
(*)
10 10 100 100
integration time
(room
temperature)
0.3 s 40 ms typical
5 minutes
possible
40 ms typical
+10 seconds
possible
virtually
unlimited
max. frame rate ca. 15 Hz 10 kHz >> 10 kHz 1 shot only
(*) noise photons is defined as: noise photons = noise electrons / (quantumefficiency * fill factor)
resolution
criterion eye CCD CMOS APS film
#pixels 120M typical: 800Krecord: 60M
typical: 800Krecord: 4M
typical: 6e6grainsrecord: papersize
pixel pitch 2-3 µm 5-10 µm 5-10 µm typ 10 µm grainpattern
focal planesize
3 cm 1 mm ...11cm
1 mm ... 2cm
only limited byfilm size
195R.I. Hornsey, University of Waterloo
operating conditions
eye CCD CMOS APS film
radiation hardness 1 mrad 10 krad 1 Mrad
operating Temp 36 C -200 C...+200 C 0 K ... +200 C 0K ... 100 C
power dissipation < 1mW 500 mW typ 50 mW typ nihil
on-board image processing
criterion eye CCD CMOS APS film
cosmetic quality perfect very good worse as good asperfect
color ideal poor (onlyRGB)
poor poor or printquality
absolutephotometry
impossible easy easy possible
focal planeprocessing
extensive none typically none none
access method data driven(focus ofattention)
serial only serial, randomaccess, ...
optical only
datapath 5M nerves 8-10 bits 8 bits none
logistics
criterion eye CCD CMOS APS film
price invaluable typ 10000 BEF typ 1000 BEF typ 10 BEF 2nd source none few many fewtechnologydevelopmentcycle
500 Myears 5 years 2 years 20 years
fabs 3E9 15 1000 10
196R.I. Hornsey, University of Waterloo
Artificial Eye?
• Imaging chips made with standard techniquesmust be planar
» so detector and processor must be side-by-side
» to maintain resolution and sensitivity, we usuallydivide the chip into sensor area and processing areas
» which limits the speed and parallelism of the process
• The maximum area for our system is also limitedby the fabrication technology
» which will restrict the total amount of processingpower available to us
» if we could include a state-of-the-art microprocessorand memory on-chip, the camera would appear muchmore intelligent! The low-level technology is there, butnot the system level.
• So there are two extreme approaches, and asyet only a few combined approaches
» complex in- and inter-pixel processing at lowresolution, with much internal control
» higher resolution, but less autonomous, peripheralprocessor-type devices that we have seen before
197R.I. Hornsey, University of Waterloo
• So, in this section, we will look at some of themore advanced “engineering” image sensors
» these typically follow the peripheral processorarchitecture, but with adaptive or programmablecapability
» but there are not too many of these
• We will look at some of the things that it wouldbe useful to do and how they are implementedcurrently
• Then we will look into the world of biologically-inspired image sensors
» many designs have been suggested to accomplishboth spatial and spatio-temporal image processingfollowing “retinal” lines
» more than 50 of these have been collected andreviewed by A. Moini from University of Adelaide,Australia (see references for web page)
198R.I. Hornsey, University of Waterloo
Increased Dynamic Range
• We saw in the previous chapter that amodification of the electronic shutter idea couldgive a sensor with two integration times
» the WiDyR sensor from JPL
» the images could be recombined to offer a sensor witha wider dynamic range
» a similar concept has been used with CCDs (calledthe Hyper-D CCD) to give two exposures of differentlengths, which are later combined
• And one of the main reasons for tolerating thedisadvantages of logarithmic pixels was the 106
dynamic range that could be achieved
• One of the potential problems with approachessuch as the WiDyR is that of blooming
» in the long tint section, charge from bright pixels willspill over into neighbouring pixels
» making their signals indecipherable
• So apparently the only other viable approach toincreasing the dynamic range – by a circuitmethod – is
» to control the integration time of each pixel individually
199R.I. Hornsey, University of Waterloo
Individual Pixel Reset
• Usually, photodiode pixels are reset by rowsafter the data has been read out
» hence the natural idea of the electronic shutter
• But the addition of another reset transistor canmake the reset signal X-Y addressable too
» this is achieved with a column reset (CRST) as well asa row reset (RRST)
• The circuit above is the most obvious method(e.g. Ginosar & Gnusin)
» but this introduces reset anomalies
» when CRST is pulsed, node N charges to VDD
» this charge is transferred to the diode when RRST ispulsed, even though CRST & RRST were never onsimultaneously
VDD
CRST
RRST
RS
200R.I. Hornsey, University of Waterloo
• To remedy this reset problem, the circuit wasmodified as above
» for use as a star tracker by JPL (Yadid-Pecht)
• So now the idea is that we can control theintegration time of each pixel
» to allow poorly illuminated pixels more time
» and to read out and reset brightly lit pixels, therebyavoiding the blooming problem
» in this work, the integration time was controlledexternally
• This approach has several importantimplications for the sensor design
• Dark current becomes more of a limitation onthe dynamic range than saturation, determiningthe maximum tint
VDD
CRST
RRST
RS
201R.I. Hornsey, University of Waterloo
• The more profound departure from normalpractice is that the time of the readout is nowimportant, as well as the signal level
» because two pixels may achieve the same signallevel, but after different integration times
» this makes reconstruction of the image more tricky
• While JPL’s approach was to control the tint
externally, several of the “biological” designsallow the pixel to control its own tint
» and maybe those of adjacent pixels as well, to achievea spatial image processing function rather than awider dynamic range per se
• Ginosar & Gnusin proposed a complicatedversion of the dual-tint approach
» where a latch in each pixel informed control circuitswhich tint applied to that pixel
» and this was adjusted for the next frame according tothe value of the present frame relative to a threshold
» but the fill factor of a 60 x 70µm pixel was only 13%
• Other attempts have led to similarly large pixelsizes
» e.g. > 100 x 100µm by Miyagawa & Kanade
202R.I. Hornsey, University of Waterloo
Image Compression• The quantity of data contained in images is so
large that most transmission systems requirethe image to be compressed beforetransmission
» 1000 x 1000 pixels gives 1MByte if 256 grey levels
» times however many frames per second
» a 38.8k bits/s modem can transmit one uncompressed(100 pixel)2 8-bit image every ~ two seconds
• There are many algorithms for compression
• Some are standard formats
» JPEG (Joint Photographic Experts Group) for stillpictures
» MPEG (Motion Picture Experts Group) for movingpictures
» H.263 low bit-rate standard, e.g. for videotransmission over a telephone line
• Some are not-yet-standard mathematical tools
» wavelet compression, fractal-based compression
» e.g. Microsoft Encarta cd-rom encyclopaedia
• And others are “natural” methods, such as
» motion detection, foveation, data reduction
203R.I. Hornsey, University of Waterloo
Mathematical Compression
• To date, there have been few reports of complexcompression techniques being performed on-chip
» JPEG encoding is possible in hardware, but has notyet been integrated with an image sensor
» MPEG encoding is hard to do on-chip, and theresulting chip would be large and have a high powerconsumption
• One of the most recent reports by Kawahito etal. in Dec. 1997 has been of an imager with two-dimensional discrete cosine transformcompression, in the analog domain
» a 128 x 128 passive pixel array was read out in blocksof 8 x 8 pixels
» the DCT operation is performed by two successiveweighted summations, each carried out in parallel, 8-bits at a time
» with intermediate results stored in an 8 x 8 memory
• While the analog processing gave theadvantages of speed and small silicon area
» about 1mm2 for the DCT processor in 0.35µm CMOS
» the precision of the method was degraded bymismatches between storage capacitors etc