Real Time Visual Traffic Lights Recognition Basedon Spot Light Detection and Adaptive Traffic Lights Templates

Raoul de Charette, Fawzi Nashashibi, Member, IEEE

Abstract-This paper introduces a new real-time traffic lightrecognition system for on-vehicle camera applications. Thisapproach has been tested with good results in urban scenes.Thanks to the use of our generic "Adaptive Templates" itwould be possible to recognize different kinds of traffic lightsfrom various countries.

Our approach is mainly based on a spot detection algorithmtherefore able to detect lights from a high distance with themain advantage of being not so sensitive to motion blur andillumination variations. The detected spots together with othershape analysis form strong hypothesis we feed our AdaptiveTemplates Matcher with.

Even though it is still in progress, our system was validatedin real conditions in our prototype vehicle and also usingregistered video sequences. We noticed a high rate of correctlyrecognized traffic lights and very few false alarms. Processingis performed in real-time on 640x480 images using a 2.9GHzsingle core desktop computer.

I. INTRODUCTION

SINCE traffic scenes are complex and include lots ofinformation, keeping constant attention on the traffic

signs is not an easy task for drivers. Therefore some trafficdata (signs) can be missed for several causes such asthe complexity of the road scene, the high number of visualinformation, or even the driver's stress or visual fatigue [1].

In order to assist this task, several driver assistant systemshave been suggested in the past years using either databaseinformation (Le. learned Geographic Information Systems)or on-vehicle sensors (Le. laser, camera, etc.) to providevarious environment information such as traffic signs, speedlimits, traffic lights, crosswalks, ... -, or any otherinformation like pedestrian or obstacles. The specificfunctionality of traffic lights detection shall be very usefulsince traffic lights position and state (go, stop or caution)provide good knowledge of the traffic environment such ashigh probability of crossroads/crosswalk, dangerous area,etc. Furthermore detecting traffic lights with an on-vehiclecamera could also be used to improve fusion of GPS andcamera visual data in order to make visual projection of roadinformation on the windshield.

Because all (or the upmost of) the previous works were

Manuscript received January 10,2009.R. C. is from the Centre de Robotic CAOR, of Ecole des Mines de Paris,

60 boulevard Saint-Michel F-75272, Paris cedex 06 FRANCE, (phone:(+33) 1-40-51-94-54; fax: (+33) 1-43-26-10-51; e-mail:raoul.de charette@ensmp.fr)

F.N. is both from the Centre de Robotic CAOR, of Ecole des Mines deParis and from INRIA, Imara Team, BP 105, 78153 Le Chesnay Cedex,FRANCE. (phone: (33) 1-39-63-52-56, e-mail: fawzi.nashashibi@ensmp.fr)

only applied for suspended traffic lights recognition, wepropose in this paper a generic modular real-time algorithmfor traffic light recognition. That is to say, a method whichwill handle supported traffic light recognition as well assuspended traffic light. Furthermore in contrary to most ofthe existing works our algorithm not only works in rural orsemi urban but also in full urban environment.

To introduce our system, paper outline is as follows.Section II is dedicated to the state of the art, we will describesome of the previous works published on traffic lightsrecognition. A system overview is presented in Section IIIand, in Section IV main steps of our recognition system aredetailed. Finally, results on urban sequence will be shownand commented in Section V, with a quick overview on thein-progress optimizations and improvements.

Fig. 1. 1st and 2nd pictures are two French supported traffic lights.3rd picture is a Belgium supported traffic light. 4th picture is aU.S.A. suspended traffic light.

II. RELATED WORK

As shown in Fig 1, traffic lights are very different acrossthe world. However we can still distinguish two main typesof traffic lights: suspended traffic lights (4th of Fig. 1), andsupported traffic light (the fITst three pictures of Fig. 1).

Almost all the previous researches have been applied onlyon suspended traffic lights and use color sensor in allrelevant cases. Indeed, recognition of suspended trafficlights is much easier since we could guess that thebackground is almost static and generally include a sky area.First attempts of traffic lights recognition were used either innon-real-time applications as presented in [2], or in real-timebut with a fixed camera as in [3]-[4]. Thus, Tae-Hyun H. etal. proposed in 2006 an approach to detect traffic lights [5]which consists of a color thresholding on the upper part ofthe image completed by a Gaussian convolution on the resultmask, in order to detect light emission of the traffic signals.In 2007, Kim Y.K. et al. [6] also showed that suspendedtraffic lights can be detected with an overall color-basedthresholding and segmentation. But even if good results

978-1-4244-3504-3/09/$25.00 ©2009 IEEE 358

Fig. 2. Representation of TLR layout.

Fig. 3. Source image which will be used as reference for furtherdemonstrations. Traffic lights facing the camera are emphasized onlyto be more easily visible by the readers.

Fig. 4. Response to a morphological Top-Hat operator using a llxllrectangular structuring element.

IV. RECOGNITION IMPLEMENTATION

A. Spot Light Detection (SLD)

Despite the fact that traffic lights can be very different, allshare the common property to emit light. Therefore a robustSLD appears to be the best base for TLR. The aim of thisSLD step is to miss the fewest sought lights as possible.However, complexity of the detection grows when you try toachieve it in urban scenes (due to the complexity of thebackground) and grows even more when you are trying todetect small spots light (in order to detect far traffic lights).Furthermore, it's important to notice that because of thepossible color variation we decided to use only gray-levelimages for the proposed SLD.

Finding Bright Areas At this step, we can define a spotlight in a grayscale image as a bright area surrounded by adarker one. In order to isolate such areas, the White Top-Hat[8] morphology operator was used. This operator consists ofthe reduction to zero of all the slow trends of the image. Ithas already been fully detailed in the literature [8]-[9] andit's most important property for us was the ability of beingnot so sensitive to illumination variations as it is appliedlocally.

As illustrated in Fig. 4, all lights emitted by traffic lightswere emphasized by the Top-Hat. Yet, plenty of unwantedareas are also visible.

Filtering Inconsistent Areas Using known properties ofthe searched lights we are able to filter drastically the resultof the Top-Hat operation. To do so, we fITst apply somebasic image processing algorithms on the resultant image inorder to remove the inconsistent areas. This fITst filtering hasthe only aim to separate the connected elements.

Insofar as we have distinguished connected component,we are then able to extract blobs using an 8-connected tunedcontours extractor algorithm. Note that because of blobextraction computation time, it is usually not applied as afITst step. Thus, using the properties retrieved from the blobextraction (area, perimeter, orientation, etc.) we are able toreject some candidates. For example, dimension ratio - asone of the needed condition to be accepted - could bewritten as follows:

MAX(width, height) < 2 * MIN(width, height)

First Step First a robust Spot Light Detection (SLD) isexecuted on the whole grayscale source image in order todetect all visible spots lights.

Second Step Creation of the candidates region accordingto the previously detected spots. Then, our AdaptiveTemplate Matcher (ATM) - which contains geometric andalgorithmic templates - evaluates matching confidence forevery template candidates.

Third Step Finally, a simple validation step is used tofilter the candidates marked by the ATM._I

~~"~

l-"..

III. SYSTEM OVERVIEW

Our goal was to design a modular algorithm for TrafficLights Recognition (TLR) that is able to detect supportedand suspended traffic lights in a dynamic urbanenvironment. Regarding the previous works, we decided tobase our algorithm on the light property emission of trafficslights. Therefore, the layout of our TLR algorithm consistsmainly of three steps as illustrated in Fig. 2.

were achieved, this method is not suitable in our casemostly because of the constraints due to supported traffi~lights and urban environment.

Various algorithms were previously used to attempt torecognize traffics lights: from Hidden Markov Models [2] tocolor segmentation [5] [6] but almost all previous attemptswere done on suspended traffic lights in rural or semi urbanenvironment. Yet, Lindner F. et al. present in their paper"Robust Recognition of Traffic Signals" [7] a comparison ofdifferent methods in order to detect and recognize Germantraffic lights in an urban environment. Various detectors arepresented, such as a color detector, a shaped based detectorwith GPS, or even an AdaBoost learning method. But incontrast to our method the only detector which meets real­time constraints in their research is a combination of "costlycolor sensor" and differential GPS map.

In our system, in order to be more flexible and lessdependent on the sensor quality and therefore, meet globalcost requirements, we decided not to use color in thedetection step.

359

Table I shows examples of the filtering criteria applied toblobs samples and Fig. 5 is the output of this filter appliedon the previous image.

Extractedblob

c

TABLE ICRITERIA USED FOR BLOB FILTERING

Samples criteriaDimension Holes Convex Filter result

ratio free approximation

X Rejected

c@ X Rejected

@ @ X Rejected

@ @ e Accepted

filter is run once more on the new blobs (with slightlydifferent criteria), resulting in only 5 blobs acceptedwith all the lamps of the traffic lights included as illustratedin Fig. 7.

Examples of the criteria used for discrimination between valid andinvalid blobs.

Fig. 5. Result of the filter based on blobs geometries properties.(-70 blobs).

Correlation With Source Image Using the previouslyvalidated blobs, others filter are then applied.

Since all spots lights should be separated from theirbackground we are consequently able to check which blobsactually verify correctly this property. This is done byfinding the corresponding extrema on the source image ofevery current validated blob and then, applying our tunedregion growing algorithm on the source image using foundextrema as seed points. Consequently, if the current testedblob is a spot light, therefore, the blob extracted from theregion growing should be very similar (as the complete lightshould have been emphasized by the Top-Hat process).Fig. 6 illustrates the process performed. Here, let us discussa critical example: unlike spot light, if a region belonging tothe sky passed the tests of the previous filters, it would berejected at this step since the region growing algorithm willreturn a too large blob comparing to the one extractedbefore.

The application of this filter on the whole image wouldreject large amount of blobs. In comparison to the 70previously validated blobs in the reference image this filteraccepted 25 blobs. And since we now use blobs extractedwith the region growing, the previously detailed geometric

360

(c) (d)Fig. 6. Overview of the whole filtering process: (a) source image (focusof the reference image used previously), (b) blob accepted by the firstfilters even though it is not a spot light, (c) local extrema of the foundblob in the source image ; (d) result of the region growing using theextrema as seed point. As a result of this filter, the found blob in (b) willbe rejected.

Fig. 7. Final result of the SLD where found spot has been projected onthe source image. As we can see even if there are still false alarms, nospots light were missed. (5 blobs remaining).

It is important to denote that false alarms are not yet aproblem as the next steps of our TLR will be able to rejectthem later. Conversely, any missed lights by the SLDprevent the detection of the associated traffic light.Regarding these constraints the SLD presented can be saidas robust, since more than 90% of the lights were detected.

B. Adaptive Template Matcher (ATM)

As mentioned above, one of our constraints was to have afully generic TLR. Therefore, and in order to be able toadapt our algorithm to different types of traffic lights, wedesigned Adaptive Templates. Those templates are evaluatedwith our Adaptive Template Matcher (ATM).

Template Matching was used previously in differentrecognition process [10]-[2]. This technique is usually slowwhen applied on the whole image. However, we use thepreviously detected spots as hypothesis for templatematching. Hence, we create candidates only where spotswere detected. Those candidates will then be evaluated and

Averageintensitv

~POSXPos Y

PosZ

Oplntensity

Element 1~ Op3DProj

Element 1

ElementN

.--------,• EISpot •

'--- ----,

Fig. 9. Illustration of the element types classes relations. All elementtypes derivate from the base class Element. Container type allowselement to contain children elements. Spot is a special type whichderivates from the Circle type and can be linked with a spot objectpreviously returned by the Spot Light Detector.

Operator Definition In addition to geometry defmition,Adaptive Templates contain also operators which can belinked to one or more elements. Those operators arealgorithmic processes which take elements as inputs andreturn output values depending on their inner behavior.Therefore, adding operators to an adaptive template leads tomore complex matching and consequently more efficientmatching.

Fig. 10. Scheme of two different operators. Operator Intensity outputthe average intensity of the first input relatively to the others inputs.Operator 3DProj takes only one input, and outputs position of thiselement in the 3D world, according to its metric properties andaccording to camera calibration data.

When defming an adaptive template, valid range, weight,or confidence threshold can be defmed for each operator andeven for each operators output. Using those properties, whenoperators are evaluated, a confidence value is set for eachoperators output depending on the ability of the value outputto fit in the valid range previously defined in the template. Ifthe confidence value of an operator output is below itsconfidence threshold, the operator it belongs to will be set toinvalid, except if the output was defmed as nondiscriminatory. Likewise, if an operator is set to invalid or, ifits confidence value does not reach the confidence threshold,the element it belongs to will be set to invalid except if theoperator was defined as non discriminatory.

Template Candidate In order to evaluate a templatematching hypothesis the ATM uses templates candidates.Those candidates inherit all their properties from theirtemplate. Using the known geometric relations betweenelements inside each template, candidate elements andoperators are evaluated hierarchically. Depending on the

uP

BoTToI

~

o~o

•ToP

DoWN

Fig. 8. Simple representation of top-down and bottom-up approaches.

Adaptive templates contain elements which describe thevisual appearance of the real object corresponding tothe template. Those elements can be instance of varioustypes - such as Circle, Square, Rectangle, Spot, Container,etc. -, and each type can have specific behavior. Fig. 9illustrates relations between the elements types.

Both element position and size are expressed in the parentreferential taking into account that the parent element widthis normalized. For instance, a child element which width, isone quarter of its parent width and which is also rightaligned in its parent, will have a 0.25 width, and its x-axisposition will be set to 0.75 (that is to say, 1.0-width).Conversely, height is not expressed explicitly but awidth/height ratio has to be defined for each element.

According to those simple rules, we can easily describefully scalable templates geometry. Furthermore,programmers can easily add new element types with specialbehavior, like we did for Spot type.

either accepted or rejected according to their matchingconfidence value. The fact that we use the templatematching only where spots were previously found, make it alot faster (as detailed in the Results section).

An Adaptive Template can be defmed as a combination ofthe 2D visual shape representations of the 3D elementswhich form the real object. In addition, templates also definealgorithmic operators linked to one or more elements andwhich will be evaluated at run time. Both elements andoperators can have different weights and matchingconfidence thresholds, or even be set to non discriminatory.Therefore, if any non discriminatory element or operatorfailed, it prevents the candidate from being rejected.

The matching process is the recursive evaluation of atemplate and its hierarchy, until all its elements and linkedoperators have been evaluated. The confidence value of anelement is computed according to weight and confidencevalue of each child element or linked operator.

Geometry Definition ATM uses the common top-downapproach to defme template geometry. This simpleapproach, shown in Fig. 8, involves decomposing a real 3Dobject (traffic light in our case) into 2D visual shapes.

361

confidence value each element and operator returned, acandidate confidence value is computed. Yet, as soon as anydiscriminatory operator/operators output/element is set toinvalid during the evaluation process, the candidate isimmediately rejected. Since some elements are not alwaysvisible, they can be set to non discriminatory. Therefore, itcould increase the confidence value in case a nondiscriminatory element is visible.

For traffic light recognition, we defmed three differenttemplates; one for each state such as go, caution or stop. Torepresent a French traffic light with the top-down approach,we use three different elements: a Spot element for the lamp,a Background element for traffic light body, and a Rectanglefor the pole. Since the pole can be hidden (because ofvehicles, traffic signs, hanging traffic light, etc), this elementis set to non discriminatory. Furthermore, in case ofsuspended traffic lights the pole is nonexistent. Fig. 11illustrates the geometry of the red traffic light template whileFig. 12 is the representation of the operators used.

BG: "body"

width 1.0w/h ratio 0.3 - .8pos x=O.O,y=O.O[... ]

The strength of our ATM is its fully generic nature. Sincethe whole class architecture was kept modular, programmerscan easily add new element types or operators. Furthermore,defming new adaptive templates can be done very easily fornon-programmer users also, and could even be done usingXML format. Future tests should be done using adaptivetemplates with traffic lights from various countries.

v. RESULTS AND FUTURE IMPROVEMENTS

A. Results

In order to evaluate our Traffic Light Recognizer werecorded several urban sequences from an on-vehiclecamera. Then, we used a ground truth editor to draw the realtraffic lights on each frame. And finally, we executed ourTLR in real time on the sequences and compared thedetected traffic lights with the ground truth.

Thus far, the tests were performed using a video databaseconsisting of 17 minutes of "useful" urban scenes sequence.Processing was performed in real-time (--20FPS) on a2.9GHz single core computer with 1GB RAM. Detailedresults of the tests are shown in Fig. 13, Fig. 14. Table II isthe result of the temporal matching according to thealgorithm results.

Note that one of the improvements in progress will enabledefming states inside templates; therefore, we will only haveone template for the traffic light.

Fig. 13. Result of the whole Traffic Light Recognition process on animage extracted from Sequence 1 (2 green traffic lights recognized).

w= 1.0

SPOT: "lamp"

parent bodywidth 0.8w/h ratio 1.0pos x=0.1,y=0.05[... ]

IIII

'" -RECT:-':~~k:'- -,~ :I I I

: width 0.4 : ~

: w/h ratio 0.1 ~w=-02------- :: pos x=0.3,y=1.1: :

:J~~. ~ ~: ~II

Fig. 11. Scheme representing adaptive template geometry of the"French red traffic light". Boxes on the left show some propertiesof the elements being pointed at. Label on the arrow is the weightof the element. As shown with dashed contours, pole is set tofacultative.

Lamp ~ 0 HSV ~AVg.H. [0.02;0.25.]p Avg S [0.53;1.0]======:::=: Avg V [not llsed]

Lamp~ OpGanssian rGauSSian approx. [0.5;1.0]

Pole~ Oplntensity rAVg intensity [0.01;0.2]

Template~ • ~pos X [-10;10]Op3DProJ Pos Y [1;5]

Pos Z [not used]

~:.::~ ~ Oplntensity rAVg intensity [20;100]

B d i------------~

o y t> OpEntropy I>Entropy [0.02;0.25]I IL l

Fig. 12. Scheme representing operators used in adaptive template of"French red traffic light". Grey objects are set to non discriminatorysuch as operator Entropy or PosZ output of the 3DPosWorid operator.

Fig. 14. Result of TLR on Sequence 2 (3 red traffic lightsrecognized).

362

The Precision and Recall used in Table II are computed asfollows:

The results presented here are obtained with temporal matching.According to the ground truth, a traffic light is counted correctlydetected if it was detected at least once during its timeline.

Whole Traffic Light recognizer computation time performed on a2.9GHz single core desktop computer on video sequences. Averageduration is obtained using computation time of the last 50 frames.

truePositivesPrecision = ----.-.-------­

truePosztzves + f alsePositives

truePositivesRecall = ..

truePosztzves + f alseNegatives

REFERENCES

[I] Kimura F.; Takahashi T., Mekada, Y., Ide I., Murase H., Miyahara T.,Tamatsu Y.: "Measurement of Visibility Conditions toward SmartDriver Assistance for Traffic Signals" Intelligent Vehicles Symposium,2007 IEEE.

[2] Zhuowen T., Ron L.: "Automatic recognition of civil infrastructureobjects in mobile object mapping imagery using a markov randomfield model" ISPRS, 2000 Amsterdam.

[3] Chung Y-C., Wang J-M., Chen S-W.: "A Vision-Based Traffic LightSystem at Intersections", 2002.

[4] Lai A-H-S., Yung N-H-C.: "A Video-Based System Methodology forDetecting Red Light Runners", MVA 98, Workshop on Machine VisionApplications.

[5] Tae-Hyun H., In-Hak 1., Seong-lk C.: "Detection of Traffic Lights forVision-Based Car Navigation System", 2006 PSIVT.

[6] Kim Y.K., Kim K.W., Xiali Y.: "Real Time Traffic Light RecognitionSystem for Color Vision Deficiencies" International Conference onMechatronics and Automation, 2007 IEEE.

[7] Lindner F., Kressel D., and Kaelberer S.: "Robust Recognition ofTraffic Signals" Intelligent Vehicles Symposium, 2004 IEEE.

[8] Meyer F.: "Contrast Features Extraction" Proceedings, 2nd EuropeanSymposium on Quantitative Analysis of Microstructures in MaterialSciences, Biology and Medicine, Caen (pages 374-380),1977.

[9] Serra J.: "Image analysis and mathematical morphology", Ac. Press,Vol. I, 1982.

[10] Jizeng W., Hongmei Y.: "Face Detection Based on TemplateMatching and 2DPCA Algorithm", 2008 Congress on Image andSignal Processing, 1982

C. Improvements

Since the suggested system is still a work in-progress weshall improve the results in the future. Furthermore, since itshows yet good recognition rates for far traffic lights; furthertests will be performed to measure precisely the accuracy ofour TLR according to the distance of the traffic lights.Another issue under study is the positioning of the trafficlight with respect to a 2D vehicle-based reference frame.This is necessary to decide if the probe vehicle is concernedby the traffic light especially in the case of crossroads. Thisrequires a precise calibration procedure of the camera anduses the hypothesis of a 2D flat road. Improving such asystem could be done by performing a fusion with an on­board GIS system.

D. Conclusion

We showed in this paper, that Spot Light Detection is agood base for a real time Traffic Light Recognizer.Furthermore, we proposed a new approach for detecting spotlights in images. Until now, the previous researches whichuse light detection for the traffic light detection were allbased on Gaussian convolution. Using the SLD we are ableto detect spot lights as soon as their width is equal or largerthan 4 pixels. Moreover, our adaptive templates can be veryuseful to easily define new traffic lights templates. But onceagain, further tests are needed to detail this point.

Finally, the whole system we presented has been provedon urban sequences which are usually difficult sequencessince the urban scenes are unceasingly moving.

2.1 ms

1.1 ms

34.2 ms

Duration(ms)

1.6ms

1.1 ms

0.1 ms

2.0ms

10.4 ms

22.2 ms

Sub stepsName Duration (ms)

TOTAL 37.4 ms

Blob extractionBlob filtering

Templatematching

Bright areaanalysis

Candidatescreation accordingto SLD result

Candidateselection

TABLE IIICOMPUTATION TIME DETAILS

Validation

Spot LightDetection

Main steps

TABLE IIWHOLE SYSTEM STATISTICS

SequenceProperties Temporal matching

Country Length Camera Precision Recall

Urban MarlinSequence 1 France 7'30"

F-046C97.77% 97.72%

On-vehicle

Urban MarlinSequence 2 France 8'48"

F-046C90.48% 95.00%

On-vehicle

ALLFrance 16'18"

Marlin95.38%) 98.41°1c»

SEQUENCES F-046C

Adaptive TemplateMatching

Several causes lead to false negatives. The main cause isusually the color distortion perceived by the camera, whichis mainly due to the camera auto exposure time. The othermain cause is when the light emitted from a traffic lightcannot be separated from the background. This is the case inFig. 14 for the suspended red traffic light. Of course, thistraffic light is detected and recognized few frames after. Dueto the fact that the vehicle is closer tothe suspended traffic light, the light emitted is separatedfrom its background and is therefore detected by the spotlight detector.

B. Computation Time

Table III details computation time step by step of thewhole TLR system.

Note that spot light detection is the most time-consumingstep. Conversely, the adaptive template matching is almostinsignificant because the template matching is only appliedon the spots found with the SLD.

363