Progress in sensor performance testing, modeling and range ...€¦ · Progress in sensor...

Progress in sensor performance testing, modeling and range prediction

using the TOD method: an overview

Piet Bijl, Maarten A. Hogervorst & Alexander Toet

TNO, Kampweg 5, 3769 DE, The Netherlands

ABSTRACT

The Triangle Orientation Discrimination (TOD) methodology includes i) a widely applicable, accurate end-to-end EO/IR

sensor test, ii) an image-based sensor system model and iii) a Target Acquisition (TA) range model. The method has

been extensively validated against TA field performance for a wide variety of well- and under-sampled imagers, systems

with advanced image processing techniques such as dynamic super resolution and local adaptive contrast enhancement,

and sensors showing smear or noise drift, for both static and dynamic test stimuli and as a function of target contrast.

Recently, significant progress has been made in various directions. Dedicated visual and NIR test charts for lab and field

testing are available and thermal test benches are on the market. Automated sensor testing using an objective synthetic

human observer is within reach. Both an analytical and an image-based TOD model have recently been developed and

are being implemented in the European Target Acquisition model ECOMOS and in the EOSTAR TDA. Further, the

methodology is being applied for design optimization of high-end security camera systems. Finally, results from a recent

perception study suggest that DRI ranges for real targets can be predicted by replacing the relevant distinctive target

features by TOD test patterns of the same characteristic size and contrast, enabling a new TA modeling approach. This

paper provides an overview.

Keywords: TOD, sensor test, sensor model, Target Acquisition, EO, IR, ECOMOS, EOSTAR

1. INTRODUCTION

Theoretical models and sensor tests that are able to supply Target Acquisition (TA) range performance with camera

systems are both essential elements in procurement processes and acceptance testing. TA models are used to

theoretically predict whether a certain camera system will meet the requirements. They calculate the expected TA

performance on the basis of the physical parameters of the sensor system. Sensor performance tests measure the

performance of a sensor system under well-defined (laboratory) circumstances and using well-defined stimuli, often with

a human-in-the-loop. They are required for (lab and field) acceptance of the delivered systems and their maintenance.

Ideally, a sensor performance model and a sensor performance test are two complementary parts of a single methodology

that are strongly connected and predict the same thing: field performance.

Current popular models and tests are:

• The Triangle Orientation Discrimination (TOD) model, based on the TOD test1.

• Thermal Range Model (TRM4), based on the Minimum Temperature Difference Perceived (MTDP)2 test

developed at FGAN-FOM (currently Fraunhofer) in Germany

• NV-IPM, based on the Targeting Task Performance (TTP) metric3,4, developed at NVESD in the US.

All three models are candidates to be included in an updated STANAG for nominal range performance prediction with

thermal imagers. The original STANAG 4347:19955 is cancelled. The update is currently under development by NATO

JCGISR Team of Experts Electro Optics and will include IR sensor modeling and testing. The TOD and MTDP tests are

candidates to be included as end-to-end thermal imager performance tests.

While the TTP approach is focused on modeling and also TRM4 contains a very detailed (analytical) sensor model, the

TOD was initially developed as an end-to-end sensor performance test1,6,7 combined with a range model8. A logical

choice of the (non-periodical) test pattern that represents target features, a robust bias-free forced-choice test and a

statistically sound analysis procedure form the basis of this very accurate and time-efficient test method. Ironically, these

differences with the standard EO/IR testing approach has hindered a wide acceptance and the development of test

equipment for a long time. Nevertheless, over the years the TOD test has been applied to a wide variety of image

forming systems including CCD cameras, thermal imagers, low light level cameras, image intensifiers, LADAR, X-ray

baggage screening systems, panoramic imagers, Helmet Mounted Displays and flight simulators. It has been applied to

static and dynamic conditions, to a variety of signal processing techniques, image compression, to sensors showing

smear and noise drift and even to characterize the effect of laser damage on performance9-34.

Validation studies of advanced use include: the effect of target contrast6,35, target motion, dynamic super resolution, local

adaptive contrast enhancement, sensor smear, combinations of these, and boost. The prediction error is always within

10% or within the experimental error of the validation data. In all TOD validation studies so far6,9,10,11,15,16,18,35, the

assumption that the TOD test is representative for TA performance has not been violated.

At the same time, several image-based TOD prediction models were developed: a receptor-based model, a neurologically

inspired model, and a simple correlation model36-38. Predictions were good but so far a standard model was never chosen.

A large advantage of an image-based approach over analytical modeling such as the TTP or TRM4 is that -in principle- it

allows performance characterization of non-linear systems with advanced image enhancement and or compression

techniques and may be extended to dynamic sensor performance and non-uniform scenes.

Recently, significant progress of the TOD methodology has been booked in all directions. First, interest in the test is

growing and the availability of test equipment is improving. Second, both an analytical and an image-based model have

been developed and implementation39-41 is under development. The models are based upon a systematically developed

TOD perception database. A key component of the image-based model is a Human Visual System (HVS) model that

mimics the human response in a TOD test task (i.e. judging the orientation of a set of triangle test pattern imaged by a

sensor under test). Finally, a recent experiment on the perception of ship targets35 has led to a new approach in TA range

modeling. Current TA models actually all make a range predictions for an ensemble of targets using set-dependent

criteria such as N50 (Johnson), V50 (TTP) and M75 (TOD model). Target sets are represented by a box of certain size and

contrast and without additional target shape or feature information. The new data suggest that ship detection and the

classification of ship features can be predicted by replacing those features by a TOD pattern of the same characteristic

size and (local) contrast. If this assumption is confirmed in validation studies, this may lead to DRI range predictions

based on individual target characteristics providing new opportunities for example in signature management.

This paper provides an overview of the TOD progress, bringing together results from recent papers. Section 2 addresses

the availability of EO/IR test equipment. In Section 3, the systematic approach of the development and validation of an

analytical and an image-based TOD model is described. Section 4 discusses the new approach in Target Acquisition

(TA) range modeling, and Section 5 shows recent developments in applications such as the European Target Acquisition

model ECOMOS40 and in the EOSTAR TDA41 and the use of image-based end-to-end modeling in system design. In

Section 6, a conversion from MTDP to TOD is deduced. Discussion and Conclusions are provided in Section 7.

2. TEST METHOD AND EQUIPMENT

The Triangle Orientation Discrimination (TOD)1,6,7 test was originally published in 1998 and 1999 as an alternative to

the MRTD and MRC and the procedure has essentially remained unchanged since except for some extensions and

adaptations for special cases such as: target motion, systems with noise drift or complex non-linear processing17,21,30.

Basically, a human observer judges the orientation (apex Up, Down, Left or Right) of equilateral triangle test patterns of

various sizes and contrasts using the sensor under test. See Figure 1. The threshold is defined at the 75% correct contrast

at a given angular size or the 75% correct angular size at a given contrast. A thorough statistical test has been

implemented to convert the observer response collection into the TOD curve: a threshold curve of contrast versus

reciprocal triangle angular size S-1 (see e.g. Figure 6, right picture). This threshold curve characterizes the quality of the

imaging system.

Over the years, a variety of TOD test equipment has been developed. These include: thermal test equipment13,30,34,42,43,

visual test charts and computer controlled test equipment8,21, an I2 test facility20, and an X-ray test31. Currently, the

demand for commercially available test equipment is growing and these are entering the market now.

Figure 1 The TOD test pattern: an equilateral triangle with apex Up, Down, Right or Left

2.1 Visual and NIR test charts

At TNO, high quality test charts for inside and outside use of camera testing in the visual and NIR spectral range were

developed in house. Both Visual Acuity (VA; effective resolution) and Contrast Sensitivity (CS) test charts exist with

positive and negative contrast test patterns (C = 1 for VA and C = 1 – 0 for CS). Background is grey with a reflectivity of

0.25 (near the average background on earth44) in the visible and near-infrared spectral region (see Figure 2). The charts

meet high requirements regarding 1) uniformity of test patterns and background, 2) the realization of high and low

contrast levels and 3) the edges of the targets. For image intensifier testing, additional requirements count for the spectral

properties in the visual and Near Infrared (NIR). They are calibrated for different standard light sources (D65, illuminant

A) and devices with different spectral sensitivity (visual, Gen II and Gen III image intensifiers). In addition, white charts

with black test patterns (C = -1) exist in accordance with the ECE-46 standard45 for testing visual devices in vehicles and

other applications.

Although the charts are primarily produced for own use, they can now be ordered by interested companies in limited

amounts (standard or custom made, see Figure 2, right picture).

Figure 2 TOD sensor test charts for lab and field testing; Left and middle picture: test chart for the measurement of

Visual Acuity (VA) and Contrast Sensitivity (CS) in the field. Right picture: production of test charts.

2.2 Thermal test bench

The first thermal test bench with TOD on the market is manufactured by HGH infrarouge46. A view on their user

interface is shown in Figure 3.

Figure 3. user interface of the TOD test in a commercially available IR test bench (image provided by HGH

Infrarouge)

2.3 Objective testing

Automated sensor testing using a validated objective synthetic human observer model is within reach. The HVS model

developed for TA modeling (see Section 3.3) can be applied to replace the human observer in an end-to-end system

performance test such as the TOD test in Section 2.2. This also allows more advanced applications such as the automatic

optimization of signal enhancement parameters36.

3. TOD MODELING

3.1 Standard perception database

For the purpose of model development, a standard (and extendable) TOD perception database has been developed for a

set of simulated sensor systems in which the primary sensor parameters (i.e. blur, sampling and noise) are systematically

varied. Such a set allows good control over the accuracy of the model predictions. Models that are based on perception

data for arbitrarily available combinations of parameter values may be tuned well to certain sensor systems (i.e. certain

positions in parameter space) or groups of sensor systems but there is always uncertainty about the predictive value for

other systems.

The current database47 consists of TOD thresholds for i) the unaided human eye and ii) a set of 24 simulated sensor

systems. The sensor systems have a fixed diffraction blur and a pixel pitch that is down-sampled by a decimation factor

of 2, 3, 4 and 8, respectively. With these decimation factors, the sensors in the set change from well-sampled to under-

sampled: (F·λ/d), i.e. a measure for the amount of under-sampling48, varies from 2 to 0.5. Further, static Gaussian noise

is added with σ = 0.001, 0.02 and 0.25. Up-sampling occurs in two different ways: pixel replication and bilinear

interpolation. In a recent study47, experiments were run at different observer viewing distances and display luminance

levels. For more details, we refer to Bijl et al. (2016, 2014)47,49. An extension of the database (e.g. towards dynamic

imaging and moving targets) is planned.

Example images of TOD test patterns imaged by the 24 sensors are shown in Figure 4 (pixel replication) and Figure 5

(bilinear interpolation). The TOD curves for part of this standard set (at the original display luminance L = 50 cd/m2 and

viewing distance) are given in Figure 6 for the unaided eye, and in Figure 7 for the simulated sensors. The top line

graphs in Figure 7 represent the low noise systems (0.001), middle line graphs represent medium noise (0.02) and the

bottom line shows the high noise (0.25) conditions. From left to right, sample size increases: decimation factor = 2, 3, 4

and 8 respectively. The results for different interpolation types (nearest neighbor: blue dots and bicubic: red dots) are

plotted together. A thorough analysis of the data is provided earlier (2014). In general, the high-contrast cut-off decreases

with increasing pixel pitch (as expected), contrast threshold increases with noise level (as expected), but the effect of

interpolation type is negligible although the images are very different (compare Figure 4 and Figure 5).

In Figure 8, all data from the 24 sensors are replotted together as threshold contrast /noise (= SNR, dimensionless) as a

function of pixel size d/test pattern size S (dimensionless) on a log-log scale. Thresholds for low noise sensors (n =

0.001) are plotted with open symbols, thresholds for medium noise in grey, and high noise (n = 0.25) in black symbols.

Different symbols indicate different decimation factors. Thresholds for nearest neighbor and bicubic interpolation are

plotted with identical symbols. The black line with slope 1 marks the lower visual threshold boundary induced by static

system noise. A slope of 1 is an interesting finding meaning that the total noise over the area of the test pattern is

independent of the size of the test pattern (expressed in pixels). Thresholds for the same noise level but different

decimation factor all fall on the same curve. The black symbols are purely noise limited, the grey symbols are limited by

both noise and human contrast sensitivity, and the open symbols are purely limited by human contrast sensitivity. The

vertical black line indicates the limit induced by sampling. This limit (d/S = 0.6) has been found in many TOD studies.

Figure 4 Example images of the TOD test patterns with the simulated sensors described in 3.1. Decimation factor

increases to the right (2,3,4, and 8). Top row: noise level 0.001; medium row: noise level 0.02; bottom row: noise level

0.25. Interpolation type: nearest neighbor (pixel replication).

Figure 5 Example images of the TOD test patterns with the simulated sensors described in 3.1. Decimation factor

increases to the right (2,3,4, and 8). Top row: noise level 0.001; medium row: noise level 0.02; bottom row: noise level

0.25. Interpolation type: bilinear.

0.001

0.01

0.1

1

0 0.5 1 1.5 2 2.5

Co

ntr

ast

1/S (mrad -1)

TOD naked eye, individual observers

FK

JA

MH

PB

WV

average

0.001

0.01

0.1

1

0 0.5 1 1.5 2 2.5

Co

ntr

ast

1/S (mrad -1)

TOD naked eye, L = 50 cd/m2

average

fit 50 cd/m2

Figure 6 TOD curves for the unaided human eye at a background luminance L = 50 cd/m2. Plotted is threshold contrast

against the reciprocal test pattern size S-1 or Visual Acuity (VA) in mrad-1. Left graph: the curves for the five observers

are plotted separately and the logarithmic mean over the observers (orange circle). Right: average over the observers and

a linear fit through the data.

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)

decimation: 2; noise: 0.001

nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.001

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.001

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

Figure 7 TOD curves for the 24 simulated sensors. Blue dots interconnected by blue lines: nearest neighbor interpolation;

red dots interconnected by red lines: bicubic interpolation. The top line graphs: low noise (0.001); middle line graphs:

medium noise (0.02); bottom line graphs: high noise (0.25) conditions. From left to right: decimation factor = 2, 3, 4, 8.

Note that the contrast scale in the two top middle graphs is different from the others.

0.1

1

10

100

1000

0.01 0.1 1

Co

ntr

ast/

no

ise

= SN

R

pixel size/ triangle size

all sensors

dec = 2, n = 0.001dec = 3, n = 0.001dec = 4, n = 0.001dec = 8, n = 0.001dec = 2, n = 0.02dec = 3, n = 0.02dec = 4, n = 0.02dec = 8, n = 0.02dec = 2, n = 0.25dec = 3, n = 0.25dec = 4, n = 0.25dec = 8, n = 0.25Series13Series14Series15Series16Series17Series18Series19

rc = 1

Figure 8 TOD data for the 24 sensors from Figure 7 all together in a single plot, replotted as threshold contrast /noise

(dimensionless) as a function of pixel size d/test pattern size S (dimensionless) on a log-log scale. Open symbols: n =

0.001; grey symbols: n = 0.02 and black symbols: n = 0.25. Diamonds: decimation factor = 2; squares: decimation 3;

triangles: decimation 4, and circles: decimation 8. Thresholds for nearest neighbor and bicubic interpolation are plotted

with identical symbols. The black line with slope 1 marks the lower visual threshold boundary induced by static system

noise; the vertical black line indicates the limit induced by sampling. This limit (d/S = 0.6) has been found in many

TOD studies. See text for further details

3.2 Analytical TOD model

The analytical model presented here is an improvement of the earlier version derived by Bijl et al. (2016)47. A paper

describing the model in detail is in preparation50. Overall system performance is limited by four components:

(diffraction) blur, sampling, noise, and the TOD of the human observer viewing a TOD test pattern on the display.

All four are expressed as an equation of VA = S-1 (in mrad-1) as a function of test pattern contrast Cfield. The total

performance is the inverse Pythagorean sum of the components. The equations are:

VAhuman eye (Cfield) = Msys ∙[ a0 ∙ (1 + a1 ∙ Log [(Cfield∙α∙(naveragehuman /nhuman))])] (VA ≥ 0) (1)

VAblur (Cfield) = b0∙ (D/λ) ∙ (1+(a1∙Log[(Cfield∙α∙(naveragehuman /nhuman))])] (VA ≥ 0) (2)

VAsampling (Cfield) = (h/d) (3)

VAnoise (Cfield) = (h/d)• d1 • (Cfield/nsystem) (4)

VAtotal = [VAhuman eye-β + VAblur

-β + VAsampling-β + VAnoise-β] -1/β (5)

For a thermal imager, system noise is determined by :

nsystem (K) = √(IETD2 + (tframe/tint,eye) ∙ NETD2) (6)

All model parameters and values for the constants (valid for a display luminance L = 50 cd/m2) are listed in Table 1.

Table 1 TOD analytical model parameters (left table) and constants (right table)

parameter parameter name constant value

VA (mrad-1) Visual Acuity a0 (mrad-1) 1.48

Cfield Inherent target field contrast a1 , b1 0.52

Msys System angular magnification b0 1.9

α (in K-1 for thermal) System contrast multiplication d1 0.12

D (mm) Optics diameter h 0.65

λ(μm) Optics diffraction wavelength β 2

d (mrad) Angular detector pitch (IFOV) naveragehuman 0.04

nhuman 0.04 (default)

tint,eye (ms) 100

The prediction for the unaided eye is given by the orange line in Figure 6 (right-hand side picture), and Figure 9 gives

the predictions for the 24 sensor systems together with the data from Figure 7.

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.001

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.001

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

0.01

0.1

1

0 0.2 0.4 0.6 0.8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

model

Figure 9 TOD curves for the 24 simulated sensors measured at Lbackground = 50 cd/m2 and at the original viewing

distance. Plotted together are: i) the data from Figure 7 (blue and red dots) and ii) predictions with the first order

analytical model of section 3.2 (black lines).

Using equation (5) for the system noise, predictions can be made for thermal imaging systems. In

Figure 10, model predictions are made for historical data published in earlier papers. Left graph: TOD for the simulated

‘star’ staring MWIR sensor system (from Hogervorst et al., 2001)37. Note that we discovered an error in the calculation

of ΔT: the factor k in equation 4 of the original paper should be 1/1.75 instead of 1.75. If we apply this factor, the

agreement is excellent. Right graph: TOD for the Indigo Omega uncooled microbolometer measured by Kostrzewa et al

(2003). Again, the data are predicted well.

0.01

0.1

1

0 5 10 15 20

∆T

(K)

1/size (mrad-1)

STAR sensor

data MAH

data PB

VA total (mrad-1)

VA human eye

VA diffraction

VA s + n

FNyquist

0,1

1

0 0,1 0,2 0,3 0,4

∆T

(K)

1/size (mrad-1)

Indigo Omega (2003)

VA total (mrad-1)

VA human eye

VA diffraction

VA s + n

FNyquist

data

Figure 10 Comparison of the analytical model predictions with historical data. Left graph. TOD for the simulated

‘star’ staring MWIR sensor system (from Hogervorst et al., 2001)37. Open and filled symbols: observer data. Blue

line: overall prediction, based on the component VAhuman eye (red), VAdiffraction (green) and VAsampling+noise (purple).

The agreement is excellent. Right graph. TOD for the Indigo Omega uncooled microbolometer measured by

Kostrzewa et al (2003)30. Dash-dot green line: replot from their Figure 10. Blue curve: model prediction. Again,

the data are predicted well.

3.3 Image-based Human Visual System model for TOD applications

We developed a practical static Human Visual System (HVS) model that can be applied to simulated imagery (for TA

modeling) or to the images from a sensor under test (for objective sensor testing). A third application is the optimum

system design trade-off between Size, Weight, Power and Cost (SWPaC) on the one hand, and performance on the other

The model consists of several parts: i) an algorithm to find the center of an object in an image, ii) a model to find the

most probable orientation by correlating the sensor output image with input images containing a test target with one of

the four possible test pattern orientations, and iii) the probability versus size and contrast function for a human observer

viewing the test pattern image on a display. In this way, the sensor-, display- and human-related factors are included such

as optics blur, sampling, noise, e-zoom, signal processing, display size, luminance level, viewing distance, observer

acuity and contrast sensitivity limitations.

Figure 11 shows the predicted model thresholds (blue dots) together with the human perception data (red dots) for the 12

sensors from the standard database (section 3.1) with nearest neighbor interpolation. The filled blue dots are valid

threshold estimates (maximum likelihood contrast thresholds fall within the range of presented stimuli and are approved

on the basis of a chi-square statistic), the open blue circles show the thresholds that are not statistically sound or fall

outside the range of available thresholds. The model shows a good overall agreement and there are no systematic

deviations on the basis of the amount of under-sampling (from left to right) or the amount of system noise (from top to

bottom).

Total Error=2.1718

Error3=1.3499

Error2=0.47488

Error1=0.347

Threshold=invweighted-0.0075

2: Human

1: correlation model

o: threshold OoB

interpolation: nearest

0 0.2 0.4 0.6 0.8 110

-4

10-2

100

Decimation: 2, Noise: 0.001, P-effect: -16 -4 -6 0

1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100

Decimation: 3, Noise: 0.02, P-effect: -3 0 0 0

1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100

Decimation: 4, Noise: 0.02, P-effect: -2 0 0 0

1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100

Decimation: 8, Noise: 0.02, P-effect: 0 0 0 0

1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

0 0.2 0.4 0.6 0.8 110

-4

10-2

100


1/Size (1/mrad)

Contr

ast

Figure 11 TOD data for the 12 simulated sensors with nearest neighbor interpolation (see section 3.1). Red dots:

observer data. Blue dots: Predictions using the Human Visual System (HVS) model described in section 3.3. The

agreement is good for all types of sensors in the set. See text for details.

4. TA RANGE MODELING

4.1 Conventional TA range modeling

The equations for conventional modeling are provided in Bijl & de Vries (2010)8. Basically, this is the ACQUIRE51

model in which the original Johnson criterion N50 has been replaced by a M75, i.e. the ratio of the 75% correct

characteristic size of the object set and that of the TOD test pattern which is associated with task difficulty. In 2011, a

comparison study (Bijl 2011) on the perception data underlying the TTP metric and the TOD has led to a conversion

factor between V50 and M75, enabling the exchange of US and NL target perception studies for the two models.

4.2 New approach

In a recent study35, threshold ranges for ship detection and for classification of characteristic ship parts (accommodation,

hull, mast, funnel, hat) were determined with the unaided human eye under a wide variety of conditions with different

sun directions, cloudiness, amount of atmospheric reduction and sea reflection. A few conditions are presented in Figure

12. In total, 1584 ranges were measured. Next, the ship contour, ship parts and corresponding TA ranges were analyzed

applying several feature size and luminance contrast measures. The size and contrast measure yielding the lowest

residual variance in the contrast versus threshold angular size relationship were identified as the best characteristic

Target Acquisition size and contrast measure. Finally, the TOD curve for the unaided eye was plotted together with the

contrast versus characteristic size data.

The most important results were:

The best agreement between data across target features was found when we defined the long axis of the feature as

the characteristic size of the feature and local contrast as the characteristic contrast.

The TOD curve practically coincides with the ship detection and feature classification data, i.e. M75 = 1 for ship

detection and feature classification

The results suggest that ship detection and the classification of ship features can be predicted by replacing those features

by a TOD pattern of the same characteristic size and local contrast. If the hypothesis is true, the results may directly be

applied to assess the effect on DRI ranges of features that distinct a target from potential alternatives. It also brings the

second stage of TA models to the next level where spatial target information is included in the range modeling.

Implementation into the existing TA models is simple: V50 or M75 for any target set may be assessed by indicating the

features that distinct one target from the others

Figure 12 Some examples of the images used in the ship detection and classification experiment (from Bijl et al,

2016).

5. APPLICATIONS

5.1 European Target Acquisition model ECOMOS

Both the analytical model (section 3.2) and an image-based TOD model are currently being implemented in the

European Target Acquisition Model ECOMOS40. ECOMOS is a multinational effort within the framework of an EDA

Project Arrangement. Its aim is to provide a generally accepted and harmonized European computer model for

computing nominal Target Acquisition (TA) ranges of optronic imagers operating in the Visible or thermal Infrared (IR).

The project involves close co-operation of Defence and security industry and public research institutes from France,

Germany, Italy, The Netherlands and Sweden. ECOMOS uses and combines well-accepted existing European tools to

build up a strong competitive position. This includes two TA models: the analytical TRM4 model and the image-based

TOD model. In addition, it uses the atmosphere model MATISSE.

The TOD branch consists of the following elements:

The analytical model in order to get a first order estimate of the expected performance and do an efficient Monte

Carlo simulation in the image-based calculation.

A simulation of the test procedure including the generation of TOD test patterns.

A simulation of the sensor system.

A plug-in option to include advanced signal processing if desired.

A simulation of the display.

The HVS model (see section 3.3).

The TOD TA range prediction model (as described in section 4.1).

An extensive description of ECOMOS is provided is a separate paper by Repasi et al. (2017)40.

5.2 Implementation in EOSTAR TDA

The Electro-Optical Signal Transmission and Ranging (EOSTAR) model is an image-based Tactical Decision Aid

(TDA) for thermal imaging systems (MWIR/LWIR) developed for a sea and littoral environment with an extensive

atmosphere model. In a current effort41, the TOD method is implemented in EOSTAR. Ship representative TOD test

patterns are placed at the position of the real target; the combined effects of the environment (atmosphere, background,

etc.), sensor and signal processing on the image are calculated using EOSTAR (see Figure 13 for an example), and the

results are judged using the HVS model. The thresholds are then converted into Detection-Classification-Identification

(DCI) ranges of the real target. This allows the prediction of Target Acquisition (TA) performance over the exact path

from scene to observer.

Figure 13 TOD simulation in EOSTAR, allowing the prediction of Target Acquisition (TA) performance over the exact

path from scene to observer. Ship representative TOD test patterns of different contrasts and orientations are positioned

at different ranges in the scene. The combined effects of the environment and the sensor are calculated using EOSTAR

and performance is assessed using the HVS and TOD TA range model.

6. CONVERSIONS BETWEEN TOD, TRM4 AND TTP

In earlier studies45,52,53 we showed that there are systematic system-dependent differences in the TA predictions made by

different models including the TOD model, TRM4 and TTP metric. For example, the MTDP cut-off crosses the TOD

cut-off when the sensor system changes from under-sampled to well-sampled45. Understanding the differences is

important for instance for the STANAG 4347 update that contains all three models and two tests. The TOD perception

dataset and the analytical model presented in Chapter 3 allow a systematic investigation of these differences.

In Figure 14, we plot MTDP curves calculated with TRM4 together with the perception data and the analytical model

predictions from Figure 9. In these plots, bar pattern spatial frequency in cy/mrad is plotted as 1/size, i.e. 1 cycle

corresponds with the TOD triangle square-root area (in mrad).

At low spatial frequencies, the MTDP contrast thresholds (i.e. thermal contrasts in this case) are much lower than the

TOD, especially for the low noise conditions (top graphs). In addition, the ratio between MTDP and TOD cut-off

systematically increases with decimation factor, i.e. with the amount of under-sampling in the system.

Studying the differences between the TRM4/MTDP and TOD, we made the following observations:

TRM4 does not include a human contrast sensitivity limiting model and assumes maximum contrast of the test

patterns at low spatial frequencies (i.e. always system noise limited). When measuring the MTDP, it is allowed to

adjust display level with the effect that contrast to the observer is increased and noise becomes the limiting factor.

For the same spatial frequency, the size of the four bars in the bar test pattern is 7 times larger than the area of the

TOD triangle. In a SNR limited region, this will lead to a (see Figure 8) threshold reduction by a factor of √7.

Note that these findings only hold for conditions where observer viewing distance plays an important role.

We take these factors into account by i) modeling the TOD for slightly adapted sensor systems with α = ΔTfield-1, leading

to a test pattern that always has a contrast Cdisplay = 1 at the display, and ii) correction of the TRM4 contrast thresholds by

a factor of √7. Replotting the data in terms of SNR as a function of d/size (similar to Figure 8), all on same scales, results

in Figure 15. We see that, in this representation:

TOD and MTDP thresholds at low spatial frequencies are very close to each other. Minor residual differences may

be ascribed to differences in modeling and measurement procedures, the particular choice of SNR in TRM4, etc.

The differences at low spatial frequencies are independent of noise level (from top to bottom graphs) and of

decimation factor (from left to right).

When the thresholds are plotted as SNR, the graphs for all noise levels (from top to bottom) fall exactly on top of

each other. We conclude that SNR, and not contrast, is the primary factor

There is a residual effect of (F·λ/d): for well-sampled systems the predicted MTDP cuts off before the TOD, there is

a cross-over at the transition area and for under-sampled systems the MTDP cuts off above the TOD. This is exactly

the effect described earlier by Bijl et al. (2003, 2016)45,52.

Summarizing

If human Contrast Sensitivity plays no role, then the ratio between limiting VA and limiting MTDP spatial

frequency is a function of both (F·λ/d) and SNR. This is probably a sigmoidal function: the ratio is constant at well-

sampled systems and for highly under-sampled systems and varies near the transition area45. Of course, this is a first

order approximation because only the major system parameters are taken into account.

Conversion from MTDP to TOD can be done as follows: MTDP contrast thresholds are multiplied by a factor of √7.

Then, the conversion factor f [(F·λ/d), SNR] is applied to MTDP spatial frequency. If the contrast of the target on

the display is lower than 1, then a correction for human-limited CS on VA needs to be performed using equation (1).

Finally, for range calculation, M75 should be applied instead of N50.

A more detailed analysis and validation is possible when ECOMOS becomes available. Similar comparison studies can

be performed between TTP and TOD and/or TTP and TRM4.

0,0001

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,0001

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,0001

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,0001

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1C

on

tras

t1/size (mrad-1)


nearest

bicubic

TOD

TRM4

0,001

0,01

0,1

1

10

0 0,2 0,4 0,6 0,8 1

Co

ntr

ast

1/size (mrad-1)


nearest

bicubic

TOD

TRM4

Figure 14 Measured (blue and red symbols) and modeled TOD curves (black lines) for the simulated sensors

replotted from Figure 9, together with the MTDP as predicted by TRM4 (green lines). At low spatial frequencies,

the MTDP contrast thresholds (i.e. thermal contrasts in this case) are much (5-100x) lower than the TOD,

especially for the low noise conditions (top graphs). In addition, the ratio between MTDP and TOD cut-off

systematically increases with decimation factor, i.e. with the amount of under-sampling in the system.

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

0,1

1

10

100

1000

10000

0 0,2 0,4 0,6 0,8 1

SNR

d/size


TOD

TRM4

Figure 15 TOD curves for the simulated sensors (black lines), but now predicted for noise limited

conditions. This was achieved by modeling a test pattern contrast of 1 on the observer display. TRM4

contrasts are shifted upwards by a factor of √7 to account for the difference in test pattern area. The data are

plotted as SNR as a function d/size. See text for further details.

7. CLOSING REMARKS

In the nineties of the last century, when it became clear that the standard MRTD/MRC and Johnson-based approach did

not apply to the new generation of staring imaging systems, a variety of alternatives appeared54-56. Three approaches

survived to date and are included in the draft updated STANAG for IR systems performance characterization.

Of these three approaches, the MTDP was a minor adaptation of the MRTD, allowing the characterization of staring and

conventional imaging systems. Since then, effort has been spent in order to formulate the sensor performance model and

to develop the extensive and robust TRM4 model software.

The TTP evolved from a coarse model (and software: NVTherm, NVThermIP) based on the perception of sinusoidal test

patterns and has continuously been improved and tuned to perception data up to a relatively stable model and robust and

modern software package called NV-IPM. A corresponding end-to-end sensor test has not been developed.

The TOD was built up from scratch getting the basics right and thus maximizing the chance that it is representing sensor

performance: a test that is close to the real acquisition task, a test pattern representing features of the targets that need to

be distinguished, a bias-free forced-choice observer task, and a solid statistical procedure. Next, the method has been

validated in many studies. Over the years, the TOD proved to be robust against the increasing system complexity.

Further, there is no fundamental limit to extend the method towards more complex and/or natural situations, e.g. using

motion, a complex background or a contrast higher than 1. With the ever increasing complexity of imaging systems,

image-based modeling has the future. It is expected that imaging simulation in general opens the field for a much wider

application of sensor and signal processing assessment techniques, and the TOD method forms a firm starting point.

REFERENCES

[1] Bijl, P.& Valeton, J.M. (1998). TOD, the alternative to MRTD and MRC. Optical Engineering 37, 7, 1976 - 1983.

[2] Wittenstein, W. (1999). Minimum temperature difference perceived – a new approach to assess under-sampled

thermal imagers. Optical Engineering 38, 5, 773 – 781.

[3] Vollmerhausen, R.H., Jacobs, E., Hixson, J. & Friedman, M. (2006). The Targeting Task Performance (TTP)

Metric: A New Model for Predicting Target Acquisition Performance. Technical Report AMSEL-NV-TR-230,

NVESD, Fort Belvoir, VA 22060 (January 2006).

[4] Vollmerhausen, Richard H., Eddie Jacobs, and Ronald Driggers (2004), New metric for predicting target acquisition

performance. Optical Engineering, Vol.43, No.11, pp 2806-2818.

[5] STANAG 4347:1995. Definition Of Nominal Static Range Performance For Thermal Imaging Systems. NATO

Standard Agreement (cancelled)

[6] Bijl, P.& Valeton, J.M. (1998). Validation of the new TOD method and ACQUIRE model predictions using

observer performance data for ship targets. Optical Engineering 37, 7, 1984 - 1994.

[7] Bijl, P.& Valeton, J.M. (1999). Guidelines for accurate TOD measurement. SPIE Proceedings, Vol. 3701 14 - 25.

[8] Bijl, P. & Vries, S.C. de, (2010). Visual Acuity, Contrast Sensitivity and Range Performance with compressed

motion video, Optical Engineering, Vol. 49, 103203, doi:10.1117/1.3503950.

[9] Bijl, P., Reynolds, J.P., Vos, W., Hogervorst, M.A. & Fanning, J.D. (2011). TOD to TTP calibration. In: Infrared

Imaging Systems: Design, Analysis, Modeling, and Testing XXII, 8014, 80140L; doi:10.1117/12.887219.

[10] Bijl, P., Hogervorst, M.A. & Vos, W. (2011). Direct comparison of TOD, TTP, and tactical vehicle identification

performance data underlying the TTP metric. (Report TNO-DV 2011 C486). Soesterberg, The Netherlands: TNO

Behavioural and Societal Sciences.

[11] Bijl, P. (2001). Validation of the TOD and MTDP Sensor Performance Measures for staring and scanning thermal

imagers (Report TNO-TM-01-A020). Soesterberg, The Netherlands: TNO Human Factors.

[12] Krapels, K., Driggers, R.G. & Teaney, B. (2005). Target-acquisition performance in under-sampled infrared

imagers: static imagery to motion video. Applied Optics, 44 (33), 7055-7061.

[13] Driggers, R.G., Krapels, K., Murrill, S., Young, S.S., Thielke, M. & Schuler, J. (2004). Superresolution performance

for undersampled imagers. Optical Engineering 44, 1.

[14] Jonathan Fanning, Justin Miller, Jennifer Park, Gene Tener, Joseph Reynolds, Patrick O’Shea, Carl Halford, Ron

Driggers (2007). IR system field performance with super-resolution. Proc. SPIE, Vol. 6543.

[15] Bijl, P., Beintema J. A., Dijk, J. & Leden, N. van der. (2010). Effectiveness assessment of signal processing in the

presence of smear. In: Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXI, 7662, 76620E-

76620E-10.

[16] Bijl, P. (2010). Visual Image Quality Assessment with sensor motion: Effect of Recording and Presentation

Velocity. Applied Optics, Vol. 49, Issue 3, pp. 343-349.

[17] Bijl, P., Schutte, K. & Hogervorst, M.A. (2006). Applicability of TOD, MRT, DMRT and MTDP for dynamic

image enhancement techniques. SPIE Proceedings 6207, 154-165.

[18] Bijl, P., Valeton, J.M. & de Jong, A.N. (2000). TOD predicts target acquisition performance for staring and

scanning thermal imagers, SPIE Proceeding Vol. 4030, 96-103.

[19] Bijl, P., Hogervorst, M.A. & Toet, A. (2004). Identification of military targets and simple laboratory test patterns in

band-limited noise. SPIE Proceedings Vol. 5407, 15, 104-115.

[20] Laurent, N., Bijl, P. & Deltel, G. (2015). Performance characterization of night vision equipment based on the

Triangle Orientation Discrimination (TOD) methodology. Optical Engineering 54(2), 023014.

[21] Gosselink, G.A.B., Anbeek, H., Bijl, P. & Hogervorst, M.A. (2013). TOD Characterization of the Gatekeeper

Electro Optical Security System. In: Proc. SPIE 8706, Infrared Imaging Systems: Design, Analysis, Modeling, and

Testing XXIV, 87060I; doi:10.1117/12.2016589.

[22] Dijk, J., Eekeren, A.W.M. van, Toet, A., Hollander, R.J.M. den, Schutte, K., Heijningen, A.W.P. van, Bijl, P.

(2013). Evaluation of intensified image enhancement through conspicuity and TOD measures. Opt. Eng. 52 (4),

041105 (December 04, 2012); doi: 10.1117/1.OE.52.4.041105.

[23] Desaulniers, P. & Thibault, S. (2011). Performance evaluation of panoramic electro-optic imagers using the TOD

method. In: Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXII, 8014-801409;

doi:10.1117/12.883718

[24] Gaska, J.P., Winterbottom, M., Sweet, W., Rader, J. (2010). Pixel size requirements for eye-limited flight

simulation. Image 2010 conference, July 2010, Scottsdale, Arizona.

[25] Dijk, J., Bijl, P., Eekeren, A.W.M. van. (2010). Performance evaluation of image enhancement techniques on an

EMCCD camera. In: Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXI, 7662, 766207-

766207-12.

[26] Eekeren, W.M. van, Schutte, K., Oudegeest, O.R. & Vliet, L.J. van (2007). Performance Evaluation of Super-

Resolution Reconstruction Methods on Real-World Data. EURASIP Journal on Advances in Signal Processing, vol.

2007, Article ID 43953.

[27] S. Hu, S. S. Young, T. Hong, J.P. Reynolds, K. Krapels, B. Miller, J. Thomas, and O. Nguyen (2010). Super-

resolution for flash ladar imagery. Applied Optics, Vol. 49, Issue 5, pp. 772-780

[28] Keith Krapels, Ronald Driggers, Eddie Jacobs, Stephen Burks, and Susan Young (2007). Characteristics of Infrared

Imaging Systems which benefit from Super-Resolution Reconstruction. Applied Optics, Vol. 46, Issue 21, pp. 4594-

4603.

[29] Kooi, F.L., Bijl, P., & Padmos, P (2006). Stereo Acuity and Visual Acuity in Head Mounted Displays. Proc. Of the

Human Factors and Ergonomics Society, 2693-2696.

[30] Kostrzewa, J., Long, J., Graff, J. & Vincent, J.D. (2003). TOD versus MRT when evaluating thermal imagers that

exhibit dynamic performance. SPIE Proceedings Vol. 5076, 220 – 232.

[31] Bijl, P., Hogervorst, M.A., Valeton, J.M. & Ruiter, C.J. de (2003). BAXSTER: An Image Quality Tester for X-ray

Baggage Screening Systems. SPIE Proceedings Vol. 5071, 341-352.

[32] Gunnar Ritt ; Michael Koerber ; Daniel Forster ; Bernd Eberle (2015). Protection performance evaluation regarding

imaging sensors hardened against laser dazzling. Opt. Eng. 54(5), 053106 (May 08, 2015).

doi:10.1117/1.OE.54.5.053106.

[33] Alan R. Pinkus, A.R., Dommetta, D.W. & Task, H.L (2012). A Comparison of Landolt C and triangle resolution

targets using the synthetic observer approach to sensor resolution assessment. In: Proc. SPIE 8355, Infrared

Imaging Systems: Design, Analysis, Modeling, and Testing XXIII, 835515 (May 18, 2012); doi:10.1117/12.919225.

[34] Weiß, A. R., Adomeit, U., Chevalier, Ph., Landeau, S., Bijl, P., Champagnat, F., Dijk, J., Göhler, B., Landini, S.,

Reynolds, J.P., Smith, L. (2012). A standard data set for performance analysis of advanced IR image processing

techniques. In: Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXIII, Volume 8355, pp.

835512-835512-10.

[35] Bijl, P., Toet, A. & Kooi, F.L. (2016). Feature long axis size and local luminance contrast determine ship Target

Acquisition performance: strong evidence for the TOD case. Proc. SPIE 9987, Electro-Optical and Infrared Systems:

Technology and Applications XIII, 99870M (October 21, 2016); doi:10.1117/12.2241805

[36] De Lange, D.J., Valeton, J.M. & Bijl, P. (2000). Automatic characterization of electro-optical sensors with image-

processing, using the Triangle Orientation Discrimination (TOD) method. SPIE Proceedings, Vol. 3701, 104-111.

[37] Hogervorst, M.A., Bijl, P. & Valeton, J.M. (2001). Capturing the sampling effects: a TOD sensor performance

model. SPIE Proceedings Vol. 4372, 62-73.

[38] Bijl, P., Hogervorst, M.A. & Vos, W. (2008). Modular Target Acquisition model & visualization tool. In: Infrared

Imaging Systems: Design, Analysis, Modeling, and Testing XIX, 6941, 69410E.

[39] Labarre, L., Bijl, P., Repasi, E., Wittenstein, W. & Bürsing, H. (2016). The European COmputer Model for

Optronic System performance prediction (ECOMOS). OPTRO-2016-046 (France).

[40] Repasi, E., Bijl, P., Labarre, L., Wittenstein, W. & Bürsing, H. (2017). The European Computer Model for

Optronic System Performance Prediction (ECOMOS): Overview and State of Work. SPIE paper 10178-23 (in

press).

[41] Iersel, M. van., Bijl, P., Hove, R.J.M. ten., &Oppeneer, M., (2017). Target Acquisition modeling over the exact

optical path: Extending the EOSTAR TDA with the TOD sensor performance model. Abstract SPIE Security +

Defence (Europe) 2017.

[42] Valeton, J.M., Bijl, P., Agterhuis, E. & Kriekaard, S. (2000). T-CAT, a new Thermal Camera Acuity Tester. SPIE

Proceedings Vol. 4030, 232 – 238.

[43] Wang, C., Guo, X., Ren, T. & Zhang, Z. (2014). Performance evaluation of infrared imaging system in field test ",

Proc. SPIE 9300, International Symposium on Optoelectronic Technology and Application 2014: Infrared

Technology and Applications, 93002J (November 20, 2014); doi:10.1117/12.2074332;

http://dx.doi.org/10.1117/12.2074332.

[44] Bijl, P. (2016). How to pass a sensor acceptance test: using the gap between acceptance criteria and operational

performance. Proc. SPIE 9987, Electro-Optical and Infrared Systems: Technology and Applications XIII, 99870K

(October 21, 2016); doi:10.1117/12.2241792

[45] UN regulation ECE-46 (2012). Uniform provisions concerning the approval of devices for indirect vision and of

motor vehicles with regard to the installation of these devices (UN publication, 3 October 2012)

[46] HGH Infrared Systems. www.hgh.fr.

[47] Bijl, P. & Hogervorst, M.A. (2016). A first order analytical TOD sensor performance model. Proc. SPIE 9987,

Electro-Optical and Infrared Systems: Technology and Applications XIII, 99870P; doi: 10.1117/12.2242262

[48] G.C. Holst (2007). Imaging system performance based upon Fλ/d. Opt. Eng. 46(10), 103204 (October 03, 2007).

doi:10.1117/1.2790066.

[49] Bijl, P., Hogervorst, M.A. & Kooi, F.L. (2014). A TOD dataset to validate human observer models for target

acquisition modeling and objective sensor performance testing. Proc. SPIE 9249, Electro-Optical and Infrared

Systems: Technology and Applications XI, 92490R (October 20, 2014); doi:10.1117/12.2067454.

[50] Bijl, P. & Hogervorst, M.A. (2017). Analytical TOD sensor performance model. In preparation.

[51] Scott, L.B. & and Tomkinson, D.M. (1991). Update on the c2nveo FLIR90 and ACQUIRE sensor performance

models, 1991.

[52] Bijl, P., Hogervorst, M.A. & Valeton, J.M. (2002). TOD, NVTherm and TRM3 model calculations: a comparison.

SPIE Proceedings Vol. 4719, 51-62.

[53] Bijl, P. & Hogervorst, M.A.(2007). NVThermIP vs TOD: matching the Target Acquisition range criteria. SPIE

Proceedings 6543, pp. 65430C.

[54] Webb, C.M. (1994). "Minimum resolvable temperature difference--how far can we stretch it?", Proc. SPIE 2224,

Infrared Imaging Systems: Design, Analysis, Modeling, and Testing V, 297 (July 8, 1994); doi:10.1117/12.180068;

[55] Webb, C.M. &Halford, C.E. (1999). Dynamic minimum resolvable temperature difference testing for staring array

imagers. Opt. Eng., 38 (5) (1999), pp. 845–851

[56] Jong, A.N. de, Franken, E.M. & Winkel, H. (2003). Alternative measurement techniques for infrared sensor

performance. Opt. Eng. 42(3) 712–724.

Date post:	24-Jun-2020
Category:	Documents
Upload:	others
View:	23 times
Download:	0 times

Progress in sensor performance testing, modeling and range ...€¦ · Progress in sensor...

Documents