Vehicle Detection Even in Poor VisibilityConditions Using Infrared ThermalImages and Its Application to RoadTraffic Flow Monitoring

Yoichiro Iwasaki, Shinya Kawata and Toshiyuki Nakamiya

Abstract We propose an algorithm for detecting vehicle positions and theirmovements by using thermal images obtained through an infrared thermographycamera. The proposed algorithm specifies the area of moving vehicles based on thevariations of pixel values, i.e. the standard deviations of pixel values along thetime direction of spatio-temporal images. It also specifies vehicle positions byapplying the pattern recognition algorithm which uses Haar-like features per frameof the images. Moreover, to increase the accuracy of vehicle detection, correctionprocedures for misrecognition of vehicles are employed. The results of ourexperiments show that the information about both vehicle positions and theirmovements can be obtained by combining those two kinds of detection, and thevehicle detection accuracy is 96.3 %. As an application of the algorithm, we alsopropose a method for estimating traffic flow conditions based on the resultsobtained by the algorithm. By use of the method for estimating traffic flowconditions, automatic traffic flow monitoring can be achieved. In addition, there isa possibility that traffic accidents, vehicle troubles, and illegal parking can bedetected with the proposed method. By using the traffic information obtained fromthe proposed method, we also expect to realize an optimized traffic signal controlaround the clock even in changeable weather.

1 Introduction

It is a pressing matter how to develop the vision-based traffic measurementsystems in the field of ITS (Intelligent Transportation Systems). They have theadvantage of measuring what could not be measured by conventional vehicle

Y. Iwasaki (&) � S. Kawata � T. NakamiyaDepartment of Electronics and Intelligent Systems Engineering, Faculty of IndustrialEngineering, Tokai University, 9-1-1, Toroku, Kumamoto, 862-8652, Japane-mail: yiwasaki@ktmail.tokai-u.jp

T. Sobh and K. Elleithy (eds.), Emerging Trends in Computing, Informatics,Systems Sciences, and Engineering, Lecture Notes in Electrical Engineering 151,DOI: 10.1007/978-1-4614-3558-7_85, � Springer Science+Business Media New York 2013

997

detectors: vehicle positions, vehicle lengths, and vehicle tracking on multilane.Therefore, by use of the vision-based traffic measurement systems, we can makeup both an automatic traffic monitoring system designed to find traffic incidentswithout time delay and a new traffic control strategy designed to reduce trafficjams.

The present research level of the vision-based traffic measurement systems ishigh enough to detect vehicles robustly around the clock [1, 2]. However, there aresome defects as follows.

Many of the conventional methods detect the bodies of vehicles only in thedaytime, and nighttime detection is adopted in much fewer cases. Yoneyama et al.[2] pointed out that most of the daytime detection methods lose their accuracywhen they are directly applied to nighttime detection. Therefore, generally adoptedmethods have been those of detecting the headlights or the taillights of vehicles atnighttime and of preparing two algorithms separately for daytime and nighttimedetection [1, 2]. It is generally difficult to measure the vehicle sizes at nighttime. Inthe conventional method of detecting taillights [2], the vehicle lengths can bemeasured only at the limited camera angles because the vehicle length is estimatedby the triangle of a pair of taillights and the vehicle front edge detected as the graylevel difference between the head of the vehicle and the headlight reflection areaon the road. What is the most important for both traffic signal control and trafficsimulation is to be able to measure the vehicle sizes and to specify the number oflarge-sized vehicles in the detected area around the clock. It is especially truebecause the saturation flow rate used as one of the fundamental values for them islargely influenced by the rate of large-sized vehicle mixing.

Vehicle cast shadows at daytime impair the vehicle detection of the adjoininglane. Therefore, a method of eliminating vehicle cast shadows has already beenproposed [3]. However, it has the disadvantage of restricting the elimination of thevehicle cast shadows to limited camera angles.

In the conventional methods of vehicle detection with visible light cameras, it isdifficult to detect vehicles with high accuracy in poor visibility conditions such asfog, snow, heavy rain, and darkness. However, traffic accidents and traffic jams aremost likely to happen under such circumstances. Therefore, it is a pressing matterfor us to develop a method designed to detect vehicles with high accuracy under allcircumstances.

In this paper, we propose a method for detecting vehicle positions and theirmovements by using thermal images obtained through an infrared thermographycamera instead of a visible light camera. The thermal images are expected to detectvehicles robustly regardless of changing environments around the clock. Recently,a few vehicle detection algorithms using infrared images have been proposed [4, 5].However, these algorithms cannot detect the positions of many vehicles in queuesunder heavy traffic. Therefore, these algorithms are not useful for automatic trafficmonitoring and adaptive traffic control systems.

First, we explain our observations using an infrared thermography camera inSect. 2, in which we show that clear visions of vehicles in snowy and deep foggyweather have been obtained. Second, the proposed vehicle detection algorithm is

998 Y. Iwasaki et al.

explained in Sect. 3. Third, our experiments at daytime with vehicle cast shadowsare explained in Sect. 4. Then we refer to the experimental results which show thatour method offers reliable information about both vehicle positions and theirmovements without the influence of the vehicle cast shadows. A method forestimating traffic flow conditions based on the vehicle detection results is alsoexplained. Moreover, it is shown that the vehicle detection results are useful for anautomatic monitoring of road traffic flows. Finally, we have concluded in Sect. 5.

2 Advantages of Thermal Images in the Vehicle DetectionUnder Poor Visibility Conditions

The infrared thermography camera used in our detection is TVS-200 [6]. Theframes of the infrared thermal images are transmitted to a notebook personalcomputer with the 1/60 s interval through the IEEE1394 interface. The infraredthermography camera and the notebook personal computer are shown in Fig. 1.

We have confirmed through our observations that the obtained thermal imagesare contrasted highly enough to detect the shapes of vehicles even in poor visibilityconditions.

We have obtained both thermal images and visible light images in snowy anddeep foggy weather on a mountain road of Aso in Kumamoto, Japan. Images ofmoving vehicles taken from the infrared thermography camera are shown inFig. 2a, b and an image taken from a visible light camera is shown in Fig. 2c.

Notebook personal computer

IEEE1394 cable

Infrared thermography camera

Fig. 1 The infraredthermography camera and thenotebook personal computeron a pedestrian bridge

Vehicle Detection Even in Poor Visibility Conditions 999

The outlines of the vehicles are clearly seen in Fig. 2a, b while only the position ofthe fog lamps is seen in Fig. 2c. The bars of the right sides in Fig. 2a, b show thetemperature scale.

The infrared thermography camera, therefore, can be effectively used forvehicle detection under poor visibility conditions like snowy and deep foggyweather.

3 A Method for Detecting Vehicle Positions and TheirMovements

The proposed method is applied to the detection of vehicles from the trafficthermal images obtained through the infrared thermography camera set up at theheight of a pedestrian bridge. Figure 3 shows one frame of thermal images.

Fig. 2 Thermal images and avisible light image in snowyand deep foggy weather

1000 Y. Iwasaki et al.

3.1 Spatio-Temporal Image Processing

Figure 4 shows an example of spatio-temporal images with the inscription ofspace-axes xy and time-axis t. The standard deviations of pixel values for n framesin the past are calculated from the thermal images as follows.

rðx; y; rtcÞ ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi

P

n�1

k¼0ðf ðx; y; rtc � kÞ � lðx; y; rtcÞÞ2

n

v

u

u

u

t

; ð1Þ

where r(x, y, rtc) is the standard deviation at coordinates (x, y) for n frames in thepast from the frame rtc at the current time, f(x, y, rtc-k) is the pixel value atcoordinates (x, y) in the frame rtc-k, and l(x, y, rtc) is the mean value at coor-dinates (x, y) for n frames in the past from the frame rtc.

The standard deviations indicate the variations of pixel values along the timedirection. By computing the standard deviations of all pixels in the frame, we candistinguish between the area of moving vehicles and that of the background orstopped vehicles based on n frames in the past. In other words, if the standarddeviation is not zero, it means that the pixel value is changed by a moving vehicle.Conversely, if the standard deviation is zero, it means that the pixel value main-tains the same value in the area of the background or stopped vehicles. When thestandard deviation is more than or equal to sdt, the pixel is actually assumed to bein the area of moving vehicles because each pixel value includes a noise.

3.2 Vehicle Pattern Recognition Using Haar-Like Features

The pattern recognition algorithm using Haar-like features was proposed byP. Viola et al. [7]. P. Viola et al. proved in their paper [7] the effectiveness of thealgorithm concerning the experiments of face detection. By changing the object ofpattern recognition, this algorithm can be applied to the detection of other objectslike vehicles. To detect vehicles in images, we have used two types of images: thepositive samples including a vehicle and the negative samples including no

Fig. 3 A frame of thermalimages

Vehicle Detection Even in Poor Visibility Conditions 1001

vehicle. Then, the machine learning has been done by using them, and a multistagecascade of classifiers has been constructed. We can execute a pattern recognitionby using the obtained multistage cascade of classifiers.

The images we have collected contain the windshield and its surroundingswhich clearly show the front view of a vehicle as positive samples while theycontain no vehicle as negative samples. Figure 5 shows some examples of thepositive sample images, and Fig. 6 shows some examples of the negative sampleimages. By using the upper part of the vehicle such as the windshield and itssurroundings as the target of pattern recognition, we can make a robust detectionof vehicles even when they are stopped one after another with a short distance. Wehave conducted training with these sample images to obtain a multistage cascadeof classifiers, and finally managed to make the pattern recognition of vehicles withthe obtained multistage cascade of classifiers. In our vehicle pattern recognition,we have used the extended algorithm proposed by R. Lienhart et al. [8] for thepaper [7]. In cases that two vehicles in a queue with shorter distance or a car isfollowing a large truck, the target area is disappeared. In such cases, the proposedmethod cannot of course detect the vehicle.

3.3 Correction Procedures for Misrecognition of Vehicles

In order to increase vehicle detection accuracy, the following three correctionprocedures are applied to vehicle pattern recognition results.

When the omission of vehicle detection is occurred in the vehicle patternrecognition, the omitted vehicle position is searched from the vehicle position inthe previous frame by using pattern matching as shown in Fig. 7. Figure 7 alsoshows the number of horizontal and vertical pixels in the searched area. When thesearched positions are out of the frame, the searched area is restricted to within theframe. In order to match the template against the traffic image with high accuracy,

Fig. 4 An example ofspatio-temporal images

1002 Y. Iwasaki et al.

we have examined several matching methods. As the result, we have selected thenormalized correlation coefficient matching method [9].

The size of windshield is bigger as the vehicle comes nearer to the infraredthermography camera as shown in Fig. 8. So, we have done the regression analysison the relationship between y positions (independent variable) and the sizes of therecognition-target areas (dependent variable). The size of the recognition-targetarea is calculated from the obtained regression equation after substitution ofy position. If the size of detected object is less than St % of the calculated size, thedetected object is deleted as a non-recognition target.

If two rectangles are overlapped on a vehicle as shown in Fig. 9, the smallerrectangle is deleted.

3.4 Combination of Two Kinds of Information: VehiclePositions and Their Movements

By combining the two kinds of processing (spatio-temporal image processingdescribed in the Sect. 3.1, and vehicle pattern recognition with the correctionprocedures described in the Sects. 3.2 and 3.3) in the same frame of images, theposition of each vehicle can be specified, and its movement can be classified, too.

Fig. 5 Examples of positive sample images

Fig. 6 Examples of negative sample images

Fig. 7 The area searched bytemplate matching

Vehicle Detection Even in Poor Visibility Conditions 1003

Each vehicle speed can be classified based on the ratio of the area of the movingvehicle in the rectangle which shows the windshield and its surroundings ofvehicle. In our method, we have classified the three categories: If the ratio is lessthan a1 %, the vehicle is assumed to be stopped. If the ratio is more than or equalto a1 % and less than a2 %, the vehicle is assumed to be running at low speed. Ifthe ratio is more than or equal to a2 %, the vehicle is assumed to be running at highspeed.

4 Experimental Results

4.1 Vehicle Detection Experiments

We have developed our algorithm with Visual C++ 2008 and the computer visionlibrary OpenCV [9].

The frame size of collected images is 320 9 240 pixels. The number of positivesample images and negative ones used in our experiments are 20,984 and 9,500,

Fig. 8 The correctionprocedure using a regressionequation

Fig. 9 The correctionprocedure for two overlappedrectangles

1004 Y. Iwasaki et al.

respectively. In order to secure a long measurement section on roads, the wind-shield sizes of the vehicles locating at a long distance from the infrared ther-mography camera are small. For that purpose we have used the smallest rectanglesize of 12 9 8 pixels for a recognition-target area. Each positive sample image isresized to the 12 9 8 pixels for the training. The number of the stages of theclassifiers obtained through the training is 14. We have assumed in the experi-ments that n, sdt, St, a1, and a2 are 30, 3.0, 40.0, 10.0, and 40.0, respectively.

By the regression analysis between y positions and the sizes of the recognition-target areas (S), we obtained the regression equation: S = 15.429y ? 96.0. Whencalculating the regression equation, we fixed the intercept value as 96.0 which isthe minimum value of detected rectangle area.

Figure 10 shows some results of the detections. The three images of Figs. 10 a–cshow the interim results we have got by combining the two kinds of processing. Theareas of gray pixels show those of moving vehicles specified by the processingdescribed in the Sect. 3.1, and the white rectangles show the outlines of vehiclesspecified by the pattern recognition with the correction procedures described in theSects. 3.2 and 3.3. The three images of Fig. 10d–f show the final results of the detectionwhich have enabled us to specify each vehicle position and its movement. The threeimages of Fig. 10 a–c correspond to those of Fig. 10 d–f, respectively. The dotted lines,the thin lines, and the bold lines in the final results in Fig. 10 show three categories ofvehicles: stopped ones, slowly moving ones, and fast moving ones, respectively.

In the interim results of Fig. 10, there is no connected components of graypixels for stopped vehicles because the standard deviations of pixel values aredistributed less than sdt. When a vehicle begins to move at low speed, the pixelswith the standard deviations which are more than or equal to sdt, begin to appear inthe images. Consequently, a part of the rectangle which shows the result of vehiclepattern recognition with the correction procedures contains the connectedcomponents of gray pixels. In addition, when the vehicle runs at high speed, thearea of gray pixels spread more widely, and most of the rectangle area becomesgray pixels.

By combining the results of spatio-temporal image processing and the patternrecognition with the correction procedures, we can classify the speed of eachvehicle. In our experiments, we classify vehicle speed into three categories asdescribed above.

Figure 10d shows the time soon after the green lights are turned on. Vehiclesbegin to move and increase their speed gradually from the head to the back of thelines. Figure 10e and f show the passage of time after the green lights are turnedon. Figure 10f shows the vehicular queue on the right-turn lane is made longer. Asshown in Fig. 10, each vehicle position and its movement can be detected in realtime with our algorithm.

Figure 11 shows a visible light image taken at the same time. The vehicle castshadows extend to the adjoining lane in Fig. 11 while there is no influence ofvehicle cast shadows on the detection of vehicles in Fig. 10.

In the experiments, we used 64 images at 60 frames interval (1 s interval) takenfrom the start of the green lights. The proposed algorithm detects 574 vehicles

Vehicle Detection Even in Poor Visibility Conditions 1005

(96.3 %) of 596 vehicles contained in the 64 images. On the other hand, thenumber of false detection was eight, which contains detection of two locations on avehicle and detection of non-recognition targets.

4.2 Automatic Monitoring of Traffic Flow Conditions

As an application of proposed vehicle detection algorithm, we propose a methodfor estimating traffic flow conditions based on the results obtained by thealgorithm. The measurement targets are inflow traffic into an intersection.

Fig. 10 Thermal Image detection results. a A result of both spatio-temporal processing andpattern recognition with the correction procedures. b A result of both spatio-temporal processingand pattern recognition with the correction procedures, c A result of both spatio-temporalprocessing and pattern recognition with the correction procedures, d A detection result for vehiclepositions and their movementsm, e A detection result for vehicle positions and their movements,f A detection result for vehicle positions and their movements

Fig. 11 A frame of visiblelight images

1006 Y. Iwasaki et al.

The number of vehicles in the measurement area is nv, and the degree ofmovement of each vehicle is mi. The value of mi takes 0, 1 and 2. The 0, 1 and 2mean stopped vehicle, low-speed vehicle, and high-speed vehicle, respectively.The nv represents the degree of spatial congestion which is proportional to thetraffic density.

Sm ¼X

nv

i¼1

mi : ð2Þ

mm ¼ Sm=nv : ð3Þ

The Sm calculated by equation (2) indicates the instantaneous value which isproportional to traffic volume. The mm calculated by equation (3) indicates thevalue which is proportional to the mean speed per a vehicle.

Figure 12 shows fluctuations of these three variables. As shown in Fig. 12a,when nv maintains a high value and the value of Sm (or mm) is close to zero, vehiclequeues are composed in whole measurement area. If these three variables indicatesuch the values when displaying green lights, the occurrence of traffic jams can bedetected. As shown in Fig. 12b, when Sm maintains a high value, traffic flows aresmooth and the traffic volume is high. When nv becomes a low value while thevalue of mm maintains near 2.0 as shown in Fig. 12c, it indicates that the trafficgreen lights may be turned off.

By monitoring the trends and relationships of these three variables, automatictraffic flow monitoring can be achieved. These spatial variables are also useful forthe information to control appropriately traffic signal lights.

Fig. 12 Fluctuations of the three variables: nv, Sm, and mm

Vehicle Detection Even in Poor Visibility Conditions 1007

5 Conclusions

First, we have confirmed through our observations that the thermal imagesobtained through an infrared thermography camera offer the images of vehiclesclear enough to detect their shapes even under poor visibility condition like snowyand deep foggy weather.

Then, we have proposed an algorithm for detecting vehicle positions and theirmovements by using thermal images. The proposed algorithm specifies the area ofmoving vehicles based on the standard deviations of pixel values along the timedirection of spatio-temporal images. It also specifies vehicle positions by applyingthe pattern recognition algorithm which uses Haar-like features per frame of theimages. Moreover, to increase the accuracy of vehicle detection, correctionprocedures for misrecognition of vehicles are employed. The results of ourexperiments show that the information about both vehicle positions and theirmovements can be obtained by combining those two kinds of detection, and thevehicle detection accuracy is 96.3 %. As an application of the algorithm, we alsopropose a method for estimating traffic flow conditions based on the resultsobtained by the algorithm. By use of the method for estimating traffic flow con-ditions, automatic traffic flow monitoring can be achieved. In addition, there is apossibility that traffic accidents, vehicle troubles, and illegal parking can bedetected with the proposed method. By using the traffic information obtained fromthe proposed method, we also expect to realize an optimized traffic signal controlaround the clock even in changeable weather.

Acknowledgments This study was supported in part by Research and Study Project of TokaiUniversity Educational System General Research Organization.

References

1. Cucchiara R, Piccardi M, Mello P (2000) Image analysis and rule-based reasoning for a trafficmonitoring system. IEEE Transac Intell Transp Syst, June 2000 1(2):119–130

2. Yoneyama A, Yeh CH, Kuo C- CJ (2005) Robust vehicle and traffic information extraction forhighway surveillance. EURASIP J Appl Sig Process Vol. 2005, Issue 14, pp. 2305-2321, 2005.

3. Yoneyama A, Yeh CH, Kuo C- CJ (2003) Moving cast shadow elimination for robust vehicleextraction based on 2D joint vehicle/shadow models. In: Proceedings of the IEEE conferenceon advanced video and signal based surveillance, July 2003, pp 229–236

4. El Maadi A, Maldague X (2007) Outdoor infrared video surveillance: a novel dynamictechnique for the subtraction of a changing background of ir images. Infrared Phys Technol49:261–265

5. Chen Y, Liu X, Huang Q (2008) Real-time detection of rapid moving infrared target onvariation background. Infrared Phys Technol 51:146–151 http://www.avio.co.jp/english/products/tvs/pdf/tvs200_4p_e_0520.pdf

6. http://www.avio.co.jp/english/products/tvs/pdf/tvs200_4p_e_0520.pdf

1008 Y. Iwasaki et al.

7. Viola P, Jones M (2001) Rapid Object Detection using a Boosted Cascade of Simple Features.In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision andPattern Recognition, December 2001, 1:511–518

8. Lienhart R, Maydt J (2002) An extended set of haar-like features for rapid object detection. In:Proceedings of the IEEE 2002 international conference on image processing, September 2002,1:900–903

9. Bradski G, Kaehler A (2008) Learning OpenCV. O’Reilly Media, Sebastopol, September 2008

Vehicle Detection Even in Poor Visibility Conditions 1009