A Physical Approach to Moving Cast ShadowDetection

Jia-Bin Huang and Chu-Song Chen

jbhuang0604@gmail.com, song@iis.sinica.edu.tw

Institute of Information ScienceAcademia Sinica, Taipei, Taiwan

April 23, 2009

1 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

2 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

3 / 26

Introduction

Motivation

Moving object detection is one of the most important taskin low-level vision.

Detecting moving cast shadows is one of the mostchallenging problems for accurate object detection in videostreams since shadow points are often misclassified asobject points.

Without careful consideration, cast shadows may introducesignificant error in segmentation, tracking, and recognition.

4 / 26

Introduction

The Cause of Cast Shadows

Light sources are partially or totally blocked by the foregroundobjects.

Why Detecting Cast Shadows Is Difficult?

1 Shadow points are detectable as foreground points andtypically differ significantly from the background.

2 Cast shadows have the same motion as the objectscasting them.

3 Shaded regions are usually connected with the foregroundobjects.

5 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

6 / 26

Related Works (1/2)

Previous Works (before 2003)

A Survey paper: [Prati et al. PAMI 2003]

Statistical parametric: [Mikic et al. ICPR 2000]

Statistical nonparametric: [Horprasert et al. ICCVWorkshop 1999]

Deterministic model-based: [Onoguchi ICPR 1998]

Deterministic nonmodel-based: [Cucchiara et al. PAMI2001]

Major Drawbacks

Need to explicitly tune the parameters for each scene.

Hard to adapt to the illumination conditions andenvironment changes.

7 / 26

Related Works (2/2)

Learning-based Approaches

Basic Idea: Learn cast shadow model from video sequences.

Shadow Flow, [Porikli et al. ICCV 2005]

Gaussian Mixture Shadow Modeling, [Martel-Brisson et al.PAMI 2007]

Combining Local and Global Features, [Liu et al. CVPR 07]

Learning Physical Model of Light Sources and Surfaces[Martel-Brisson et al. CVPR 2008]

Drawbacks

Most of them assume shadow values will attenuate linearlyalong the line between the value of the correspondingbackground and the origin.

Pixel-based models may suffer from slow learning due tothe lack of sufficient samples.

8 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

9 / 26

Main Idea

A general physics-based shadow modelDecompose light incident at the background surface intotwo classes

Direct light sources (e.g., sun)Ambient illumination (e.g., light scattered by the sky,colored light from nearby surfaces (color bleeding))

Suppose we have N light sources and M ambientillumination, then the intensity function of light:

E(λ) =

N∑n

Eincident,n(λ) +

M∑m

Eambient,m(λ).

10 / 26

Ambient Illuminations and Direct Light Sources

Lambertian model: camerasensor response gk(p) at point p

gk(p) =

∫E(λ, p)ρ(λ, p)Sk(λ)dλ.

E(λ, p) Intensity function of lightsources

ρ(λ, p) The reflectance of anobject surface

Sk(λ) Sensor spectral sensitivityfunction

11 / 26

Appearance Variation Under Cast Shadow

Part or total light sources are blocked by foregroundobjectsAmbient illumination may be slightly changed (from BGA toBG ′

A)

12 / 26

Color Feature Vector

We encode the difference vector between background andshadow value as our color feature.

xs,t(p) = [αt(p), θt(p), φt(p)]T (in spherical coordinate

system)

Illumination attenuation

αt(p) =||vt(p)||

||BGt(p)||

Angle information

θt(p) = arctan(vG

t (p)

vRt (p)

)

φt(p) = arccos(vB

t (p)

||vt(p)||)

13 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

14 / 26

Overview

1 Perform background subtraction to obtain foregroundcandidates (i.e., including real foreground and castshadows)

2 Apply weak shadow detector as a pre-filter to obtainshadow candidates (e.g., filter out those pixels whoseillumination values are larger than the correspondingbackground values)

3 For these shadow candidates, learn the color featurevector xs,t(p) using GMM over time

4 Detecting cast shadow using the learned cast shadowmodel

15 / 26

Weak Shadow Detector

Criterion for shadow candidates

rl(p) =‖BGt(p)‖

‖xt(p)‖ cos(ψ(p))

ψ(p) = arccos( 〈xt(p), BGt(p)〉

‖xt(p)‖‖BGt(p)‖

)

rmin < rl(p) < rmax, ψ(p) < ψmax

16 / 26

Incorporating Spatial Information

Prior Knowledge of Cast Shadows

Cast shadows would not enhance the spatial gradient intensity

Introduce ωt(p) as a confidence value of cast shadows

ωt(p) =ε + |∇(Bt(p))|

ε+ max{|∇(It(p))|, |∇(Bt(p))|},

where ε is a smooth term.

To accelerate the learning speed of the pixel-basedshadow model, take ωt(p) as confidence value to updateshadow model at pixel p

Penalize samples having larger gradient intensity thanbackground by lessening the learning rate

17 / 26

Detecting Shadows at Light/Shadow Border

Shadows at light/shadow border show different behaviorfrom shadows inside the shaded region

Solution: Detecting cast shadows only with angleinformation

(a)αt(p)

(b)θt(p)

(c)φt(p)

18 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

19 / 26

Qualitative Evaluation

(a) (b) (c) (d)

Figure: (a) Original images, (b) Background posterior probability, (c)Shadow posterior probability, and (d) Forground posterior probability

20 / 26

Quantitative Evaluation

Performance Evaluation Metrics [Prati et al. PAMI 2003]

Shadow Detection Rate η

η =TPS

TPS + FNS

Shadow Discriminative Rate ξ

ξ =TPF

TPF + FNF

Sequence Highway I Highway II HallwayMethod η% ξ% η% ξ% η% ξ%

Proposed 72.34 84.98 72.70 79.89 71.69 88.25Kernel 70.50 84.40 68.40 71.20 72.40 86.70

LGf 72.10 79.70 - - - -GMSM 63.30 71.30 58.51 44.40 60.50 87.00

21 / 26

Effect of Shadows at Shadow/Light border

(a) (b) (c)

Figure: Effect of shadows at shadow/light border (a) Original frame ofsequence “Highway I". (b)(c) Foreground posterior without/withconsidering shadows at shadow/light border.

22 / 26

Outline

1 Introduction

2 Related Works

3 Physical Model for Cast Shadows

4 Learning and Detecting Cast Shadows

5 Experimental Results

6 Conclusion and Future Work

23 / 26

Conclusion

Provide a better description for background surface valuevariation under cast shadow

Incorporate spatial information to accelerate the learning ofpixel-based shadow model

Take shadows at light/shadow border into consideration

24 / 26

Future Work

Derive physics-based features for building a global shadowmodel in a scene

Jia-Bin Huang and Chu-Song Chen, “Moving Cast ShadowDetection using Physics-based Features", CVPR 2009

Extend the physical model to handle more general cases(e.g., surface with specular reflection, spatial varingambient illumination, etc.)

25 / 26

The End

Thank you

26 / 26