Spatiotemporal Saliency Detection and Its Applications in Static and Dynamic Scenes

IEEE TCSVT 2011Wonjun Kim

Chanho Jung

Changick Kim

Outline

IntroductionProposed MethodExperiment ResultApplicationConclusion

Introduction

Problem occurs when background is highly textured

Proposed Method

Feature RepresentationEdge orientation histogram (EOH)Color orientation histogram (COH)Temporal Feature

Self-ordinal MeasureSaliency MapScale-invariant Saliency Map

Edge Orientation Histogram (EOH)

1. Compute the edge orientation of every pixel in the local region center at the pixel

2. Quantized into K angle in the range of [,]

3. Compute the histogram of edge orientation

m(x,y,n):edge magnitude(x,y,n):quantized orientation

𝑖

local region

Color Orientation Histogram (COH)

1. Quantize the angle in HSV color space in the range of [,] into H angles

2. Compute the histogram of color orientation

s(x,y,n):saturation value(x,y,n):quantized hue value

Temporal Feature

Compute the intensity differences between frames

Feature at the pixel of frame

P :total number of pixels in local regionj :index of those pixels in P :user-defined latency

Self-ordinal Measure

Define a 1(K+1) rank matrix by ordering the elements of EOH(COH) ex:

Self-ordinal Measure

Saliency Map of Edge and Color

Compute the distance from the rank matrix of center region to surrounding regions

Saliency Map of Edge Saliency Map of Color

N :total number of local regions in a center-surround window

, :maximum distance between two rank matrices

Spatial Saliency Map

Combine the edge and color saliency

Combining with Temporal Saliency

Compute the SAD of temporal gradients between center and the surrounding regions

Combine the spatial and temporal saliency

Scale-invariant Saliency Map

Combine 3 different scales of saliency Map(3232, 6464, 128128)

3232 1281286464

Algorithm

Experiment Result

Static ImagesVideo Sequences

Experiment Result

Static ImageLocal region = 55center-surround window = 77K = 8, H= 6 = 40, = 24

Video Sequence = 49Speed: 23ms per frame (43 fps)

Static Images

Static Images

Video Sequences

Video Sequences

Application

Image RetargetingMoving Object Extraction

Image Retargeting

Image Retargeting

Moving Object Detection

G:the set of salient pixels in the ground truth imageP:salient pixels in the binarized object mapCard(A):the size of the set A

Moving Object Detection

Conclusion

Ordinal signature can tolerate more local feature distribution than sample values.

The proposed scheme performs in real-time and can be extended in both static and dynamic scenes.