On the Use of Computable On the Use of Computable Features for Film Features for Film

ClassificationClassificationZeeshan Rasheed,Yaser SheikhZeeshan Rasheed,Yaser Sheikh

Mubarak ShahMubarak Shah

IEEE TRANSCATION ON CIRCUITS AND SYSTEMS IEEE TRANSCATION ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, JAN 2005FOR VIDEO TECHNOLOGY, JAN 2005

OutlineOutline

• Introduction• Computable video features

– Average Shot Length– Color Variance– Motion Content– Lighting Key

• Mean Shift Classification• Results• Conclusion

IntroductionIntroduction

• Films are a means of expression– Explicitly, with the delivery of lines by actors– Implicitly, with the background music, lighting,

camera movements and so on

• Study domain is the movie preview, which often emphasizes the theme of a film

IntroductionIntroduction

• Maybe a need to extract the “genre” of scenes

• With scene-level classification, it would allow a more flexible system of scene ratings, ex filter and recommendation of movies

Related WorkRelated Work

• Work by Fischer et al. and Truong et al. distinguished between newscasts, cartoon, commercials, sports through decision tree with examples

• Kobla et.al used DCT coefficients, and motion vector information of MPEG video for indexing and retrieval

Computable video featuresComputable video features

• Identify four major genres– Action, Comedy, Horror, and Drama– because most movies can be classified and lo

w-level discriminant analysis is most likely to succeed

• Employ for features– Average Shot Length, Shot Motion Content, Li

ghting Key, Color Variance

Shot Detection and Average Shot LengthShot Detection and Average Shot Length

• First proposed by Vasconcelos

• Can direct audience’s attention with controling the tempo of the scene

• Ex. Dramas have larger average length, whereas action movies shorter shot length

Shot Detection and Average Shot LengthShot Detection and Average Shot Length

• Detection of shot boundaries using color histogram intersection in the HSV space– H: hue (color), 8 bins– S: saturation, 4 bins– V: value (brightness), 4 bins

• S(i) represent the intersection of histograms

and of frames i and i-1iH 1iH

Shot Detection and Average Shot LengthShot Detection and Average Shot Length

min

bin1 bin2 bin1 bin2

Frame i Frame i-1

Shot Detection and Average Shot LengthShot Detection and Average Shot Length

min

When S(i) is less than a fixed threshold shot boundaries !!bin1 bin2 bin1 bin2

Frame i Frame i-1

Shot Detection and Average Shot LengthShot Detection and Average Shot Length

17 shots identified by a human observer

Number of shots detected: 40; Correct: 15; False positive: 25; False negative: 2

Shot Detection and Average Shot LengthShot Detection and Average Shot Length

To improve the accuracy, an iterative smoothing of the 1-D function is performed firstNumber of shots detected: 18; Correct: 16; False positive: 2; False negative: 1

Color VarianceColor Variance

• variance of color has a strong correlational structure with genres intuitively– For instance, comedies with a large variety of

bright colors– whereas horror films with only darker hues

• Employ variance of CIE Luv– L: luminancy (發光度 )– u,v: chrominancy (色差 )

Color VarianceColor Variance

Generalized variance is obtain

ps. All key frames presented in a preview are used to find this feature

Motion ContentMotion Content

• The visual disturbance of a scene can be represented as the motion content present

• Action films with higher value for such a measure, and dramatic or romantic movies with less visual disturbance

Motion ContentMotion Content

Horizontal slice: I(x,t)

Motion ContentMotion Content

Hx, Ht are the partial derivatives of I(x,t)

Lighting KeyLighting Key

• There are numerous ways to illuminate a scene, one of the common used is Three Point Lighting

Keylight:

The main source of light on the subject and it is the source of greatest illumination

Backlight:

Help emphasize the contour of the object, and it also separates it from adark background

Fill-light:

Secondary illumination source which helps to soften some of the shadows thrown by the keylight and backlight

Lighting KeyLighting Key

• High-key lighting:– An abundance of bright light– More action, less dramatic– Ex. Comedy & action movies

• Low-key lighting:– Ex. Film noir or horror films

Lighting KeyLighting Key

• Many algorithms exist that compute the position of a light source in a given image

• Unfortunately, assumptions typically made in existing algorithms are violated, for example, single light source

• Compute the key of the lighting with brightness value of pixels

Lighting KeyLighting Key

Lighting KeyLighting Key

• Key frame i with m*n pixels, find the mean and standard deviation of the value component of the HSV space

• Lighting quantity – Horror movies with small value– Comedy movies with large value

iii ),(

Mean Shift ClassificationMean Shift Classification

• Mean shift procedure has been shown to have excellent properties for clustering and mode-detection with real data

Xi: video features

hi: their bandwidth parameters

Action + drama

drama Comedy+

drama

comedy Action+

comedy horror

ResultsResults

• Conduct 101 film previews obtained from the Apple website

• The total number of outliers in the final classification was 17 and 83% genre classification accurate

ConclusionsConclusions

• Propose a method to perform genre classification of previews using low-level computable features

• Classification is performed using mean shift clustering in the 4-D feature space of average shot length, color variance, motion content, and the lighting key