Recognition of Traffic Lights in Live Video Streams on Mobile

Devices

Jan Roters

Xiaoyi Jiang

Kai Rothaus

2011 IEEE Transactions on CSVT

Outline

IntroductionProblemsSystem Architecture

IdentificationClassificationVideo AnalysisTime-Based Verification

Experiment ResultsEvaluationsConclusion

Introduction

People with visual disabilities are limited in mobility.

Orientate pedestrians with zebra crossings at intersections

Portable PC with a digital camera and a pair of auricular stereo

Present a system for mobile devices to help sightless people cross roads.

Problems

Program usage

Real world conditionsCamera resolutionDifferent appearances

Problems

The scale of traffic lightsMany traffic lightsOccludedIlluminationRotation

Pedestrian Lights in Germany

1) Installation

2) Shape

3) Color arrangement

4) Circuitry

5) Background

Mobile Device & Databases

Nokia N95330MHZ ARM processor18Mb RAM320240

2 publicly available databaseGround truth segmentation was made manually

System Architecture

1.

2.

3.

4.

1. Localization

Red and Green Color Filter(1/3)

1. Analyze the data

Red and Green Color Filter(2/3)

2. Design the filter rules (ex : red traffic light)

The Gaussian distribution of the red cluster is defined by its mean color = (0.48,0.06,0.07) and has three eigenvectors

A color c = (r, g, b) is a red traffic light color when

Red and Green Color Filter(3/3)

3. Optimize parameters different parameter settings for each color Use 300 images to train Measure the quality of each setting by TP, FP, FN

Recall = , Precision =

Size/Circuitry Filter

Assume the traffic light is 4 to 24 meters awayFixed camera focal length and possible aspect

ratios

1. Filter out regions that are too small or too large

2. Vertical neighbor should not have different color

Background Color Filter

Inspect the region under a red light candidate or above a green light candidate

If there are no dark pixels within search region, refuse this candidate

Search region

Search region

Validation of Localization

Validate the localization results with 201 images

Optimal Validation

recall precision recall precision

Red 76% 89.5% 71.8% 87%

green 85% 98.5% 83.3% 92.6%

Error = 33.7%

2. Classification

TLC is the broadestTLC has the smallest distance to the top of imageNo other traffic light has similar height with TLC

Performance of Classification

Red Green

Recall 86.3% 86.3%

Precision 97.4% 98.1%

3. Video Analysis(1/2)

Temporary OcclusionFalsified ColorsContradictory SceneRepeating Results

3. Video Analysis(2/2)

Find the motion vector between two framesUse KLT tracker to track feature pointsOnly search in a small area around crucial traffic light

candidate (30 pixels in each direction)Correlate the features by using SAD

Search region

Crucial traffic light

Candidate region

Feature point

𝑡𝑖 −1 𝑡𝑖

4. Time-Based Verification

Reduce the false positive detections by comparing 2 kinds of results

Use state queue with 4 scenarios1) Identification and video analysis are both successful

and the locations match with each other.

2) Identification and video analysis are successful but the locations are different.

3) Video analysis succeeds but identification fails.

4) Video analysis fails but identification succeeds.

Experiment Results

and Compute at least 5 frames per secondAt least consecutive correct detection with the

same color

Experiment Results

Evaluations

ReliabilityPrevent false positive green light detection

Evaluations

InteractivityTemporal analysis reduce the interactivityThe feedback is normally given within 2 seconds

Conclusion

The system can be helpful on driver assistance systems

Limited computational power on mobile devicesThe verification ideas can be improved