+ All Categories
Home > Education > Computer Vision For Computer Music

Computer Vision For Computer Music

Date post: 11-Sep-2014
Category:
View: 3,791 times
Download: 5 times
Share this document with a friend
Description:
Presentation given in the Computer Music PhD program at the School of Arts (http://artes.ucp.pt) Jorge Cardoso - http://jorgecardoso.eu
Popular Tags:
of 32 /32
Computer Vision Techniques for Musical Interaction Using Processing Computer Music Doctoral Program http://www.artes.ucp.pt/si/doutoramento/index.html 16-05-2009 Jorge Cardoso 1
Transcript
Page 1: Computer Vision For Computer Music

Computer Vision Techniques for Musical Interaction

Using Processing

Computer Music Doctoral Program

http://www.artes.ucp.pt/si/doutoramento/index.html

16-05-2009 Jorge Cardoso 1

Page 2: Computer Vision For Computer Music

Computer Music Doctoral Program

Topics

• Introduction to session

– What (is this session about)

– How (the session will go through)

– Who (am I)

• Techniques – Examples and live demos

– Brightness/Color detection

– Background subtraction

– Movement detection

– Movement estimation

– Face Detection

• Techniques – Code Overview

– (Depending on time left)

– Mixed Initiative (tell me what you want to see in more detail)

16-05-2009 Jorge Cardoso

2

Page 3: Computer Vision For Computer Music

Computer Music Doctoral Program

What

• Overview of some computer vision techniques

– How they work and what are the main limitations

– How to use them in Processing

• Some –very simple(istic)– examples of how they canbe used for musical interaction

“Computer vision is the science and technology of machines that see. As a scientific discipline, computer vision is concerned with the theory for building artificial systems that obtain information from images” – Computer vision. (2009, April 24). In Wikipedia, The Free Encyclopedia. Retrieved 11:53,

May 9, 2009, from http://en.wikipedia.org/w/index.php?title=Computer_vision&oldid=285848177

16-05-2009 Jorge Cardoso

3

Page 4: Computer Vision For Computer Music

Computer Music Doctoral Program

What

• Techniques

– Color detection (where in the image, does a given colorappear – can be used for tracking objects)

– Brightness detection (what is the position of thebrightest pixels)

– Background subtraction for object vectorization

– Movement detection (where –coordinates– in theimage has movement occurred)

– Movement estimation (what direction are the objectsmoving)

– Face detection (is there a face in the image? where?)

16-05-2009 Jorge Cardoso

4

Page 5: Computer Vision For Computer Music

Computer Music Doctoral Program

How

• General overview of how those CV techniques work

– Will try not to dwelve much into code in this phase

• Examples of what can be accomplished with thetechniques

– Vídeos

– Live demos

• Depending on time left

– Tell me if you want to see something in more detail

– More detailed description of code (libraries needed, etc)

16-05-2009 Jorge Cardoso

5

Page 6: Computer Vision For Computer Music

Computer Music Doctoral Program

Who

• Jorge Cardoso (http://jorgecardoso.eu)

• Teacher of Interactive Video Art course(School of Arts)

– Mainly computer vision techniques for (full) bodyinteraction with screen projections

– Students use Processing and CV techniques for interactive art projects

16-05-2009 Jorge Cardoso

6

Page 7: Computer Vision For Computer Music

Computer Music Doctoral Program

Color detection

• Isolate color regions in an image

• Procedure:

– Go through the image, pixel by pixel

– Calculate the distance between the color of the currentpixel and the reference color

– If distance is smaller than a given threshold, keep the pixel

16-05-2009 Jorge Cardoso

7

Page 8: Computer Vision For Computer Music

Computer Music Doctoral Program

Color Detection

• Q: How do we calculate the “distance” between twopixels?

• A: Simple version: euclidean distance.

– Take a color as a point in 3D space (RGB -> XYZ)

– Calculate the distance between the two points

• Subtract each component (dR = R1-R2, dG = G1-G2, dB = B1-B2)

• Dist = Sqrt(dR*dR + dG*dG + dB*dB)

• A1: Slightly (perceptively) better version: weight each component differently:

• Dist = Sqrt(2*dR*dR + 4*dG*dG + 3*dB*dB)

16-05-2009 Jorge Cardoso

8

Page 9: Computer Vision For Computer Music

Computer Music Doctoral Program

Color Detection

• Q: What’s it good for?

• A:

– Catch of the day [video]

– Play-doh as Piano Keyboard [video]

– ColorDifferenceInstrument [live demo]

– ColorDifferenceDistanceInstrument [live demo]

– ColorDifferenceAngleInstrument [live demo]

16-05-2009 Jorge Cardoso

9

Page 10: Computer Vision For Computer Music

Computer Music Doctoral Program

Brighteness detection

• Slight variation on color detection when we’re only interestedin tracking bright pixels

– For example, to track a light

• Procedure:

– Same as color detection but extract the brightness of each pixel andkeep only the highest

16-05-2009 Jorge Cardoso

10

Page 11: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

11

Brighteness detection

• Q: How do we calculated the “brightness” of a color?

• A:

– In RGB:

• (perceptive) Brightness = (0.2126*R) + (0.7152*G) + (0.0722*B)

• (physical/energy) Brightness = (R+G+B)/3

• …

– In HSB:

• Brightness = B ☺

– Just use the brightness() function in whatever language…

Page 12: Computer Vision For Computer Music

Computer Music Doctoral Program

Brighteness detection

• Q: Why not just color detection?

• A:

– Brightness detection is more robust (i.e., lessinfluenced by lighting conditions)

– You don’t care about color, just the brightness(for example to detect a lantern)

16-05-2009 Jorge Cardoso

12

Page 13: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

13

Brighteness detection

• Q: What’s it good for?

• A:

– Drawing With Light [video]

– Burning the Sound [video]

Page 14: Computer Vision For Computer Music

Computer Music Doctoral Program

Background subtraction

16-05-2009 Jorge Cardoso

14

• If we have a fixed camera, and a background imagefor reference, we can subtract the background to each frame to isolate new objects

This - This = This

Page 15: Computer Vision For Computer Music

Computer Music Doctoral Program

Background subtraction

• Q: How do we “subtract” two images?

• A: Pixel by pixel

• Q: How do we “subtract” two pixels?

• A: Channel by channel (color component bycolor component, usually R, G and B)

16-05-2009 Jorge Cardoso

15

Page 16: Computer Vision For Computer Music

Computer Music Doctoral Program

Background subtraction

• Pixel subtraction alone doesn’t work very welldue to noise or shadows in the image

• The resulting subtraction must be thresholdedto eliminate noise

16-05-2009 Jorge Cardoso

16

Page 17: Computer Vision For Computer Music

Computer Music Doctoral Program

Background subtraction

• After subtracting and thresholding, we canbinarize the image to get a mask

16-05-2009 Jorge Cardoso

17

Page 18: Computer Vision For Computer Music

Computer Music Doctoral Program

Background Subtraction

• Q: What’s it good for?

• A: Isolating objects in a scene. The mask canbe vectorized, giving us access to it’s contour(polygon).

– Bg Subtraction [Live demo]

– Position Through Background [Video]

– BouncingBalls [Live demo]

– SoundingPolygon [Live demo]

16-05-2009 Jorge Cardoso

18

Page 19: Computer Vision For Computer Music

Computer Music Doctoral Program

Movement Detection

• If we subtract two consecutive video frames, threshold andbinarize it, we get a rough view of the movement in the video

• Since we don’t need a reference frame, this technique is more rubost in face of lighting conditions

16-05-2009 Jorge Cardoso

19

Page 20: Computer Vision For Computer Music

Computer Music Doctoral Program

Movement Detection

• Q: What’s it good for?

• A:

– WebCam Piano [video]

– Air Piano [live demo]

16-05-2009 Jorge Cardoso

20

Page 21: Computer Vision For Computer Music

Computer Music Doctoral Program

Motion Estimation (aka Optical Flow)

16-05-2009 Jorge Cardoso

21

• Estimate the movementvectors of blocks in a sequence of two images

• Procedure

– Divide both images intoblocks

– For each block in imageone, find the closest (more similar) block in image two Image: http://grauonline.de/wordpress/wp-content/uploads/ovscreenshot1.png

Page 22: Computer Vision For Computer Music

Computer Music Doctoral Program

Motion Estimation

16-05-2009 Jorge Cardoso

22

• Q: What’s it good for?

• A:

– BryanChung Example [video]

– OpticalFlowMidi [live demo]

Page 23: Computer Vision For Computer Music

Computer Music Doctoral Program

Face detection

• Not recognition!

• Needs a configuration file that determins what to detect (front faces, side faces, body, etc)

• Returns the position and sizeof detected faces

16-05-2009 Jorge Cardoso

23

Page 24: Computer Vision For Computer Music

Computer Music Doctoral Program

Face Detection

• Q: What’s it good for?

• A:

– Face Detection With A Smiley [video]

– Funny Eyes [video] [live demo]

16-05-2009 Jorge Cardoso

24

Page 25: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

25

Summary

• Color detection

– Needs good lighting

– Can track (several) objects by color

• Brighteness detection

– Robust, needs only slightly controled lightingconditions (low light)

– High accuracy of position and movement

– Can track several objects at the same time (withsome restrictions)

Page 26: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

26

Summary

• Background subtraction

– Needs initial setup on startup (reference frame)

– Needs highly controled lighting conditions.

– Isolates objects without any marker (color or light)

– Can be used to determine distance to camera (withoutgreat accuracy)

– Can be used to detect movement

Page 27: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

27

Summary

• Movement Detection/Estimation

– No startup configuration

– Doesn’t need very controlled lighting conditions

– Determins regions where movement occured

– Hard to determine number of objects moving

Page 28: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

28

Summary

• Face detection

– Needs relatively controlled lighting conditions

– Needs relatively controlled positioning of people(near and looking directly at the camera)

– Detects position and size (approximate) of faces

Page 29: Computer Vision For Computer Music

Computer Music Doctoral Program

References

• Videos– Catch of the day: http://www.youtube.com/watch?v=G8-Nvyx0xtk

– Play-Doh as Piano Keyboard!: http://vimeo.com/465726?pg=embed&sec=

– Drawing with Light: http://www.youtube.com/watch?v=VDP3e20uYMI

– Burning the Sound: http://www.vimeo.com/3096584

– Position through background subtraction:

http://www.youtube.com/watch?v=yyc8aionno4

– Webcam piano: http://www.vimeo.com/1219327?pg=embed&sec=1219327

– Bryan Chung optical flow: http://www.youtube.com/watch?v=WCdQ8KQ_Wyo&

– Face detection with a smiley: http://www.youtube.com/watch?v=r4XArSFPwPc

– Funny Eyes: http://www.youtube.com/watch?v=9WXLs4qQ5nk

16-05-2009 Jorge Cardoso

29

Page 30: Computer Vision For Computer Music

Computer Music Doctoral Program

References

16-05-2009 Jorge Cardoso

30

• Webpages

– Computer vision techniques for artists (by Golan Levin): http://www.flong.com/texts/essays/essay_cvad/

– Processing Website: http://processing.org/

– Processing Books: http://processing.org/learning/books/

– Portuguese Processing Forum: http://processing.jorgecardoso.eu

– Color metrics: http://www.compuphase.com/cmetric.htm

– Bryan Chung (Optical Flow videos):

http://www.bryanchung.net/?p=262

– OpenCV Processing library: http://ubaa.net/shared/processing/opencv/

Page 31: Computer Vision For Computer Music

Computer Music Doctoral Program16-05-2009 Jorge Cardoso

31Stuff you need to make the examplesrun

• Processing

• OpenCV processing library

• Webcam

Page 32: Computer Vision For Computer Music

Computer Music Doctoral Program

The End

16-05-2009 Jorge Cardoso

32

• Code Overview – anything in particular?

• Questions?

• …


Recommended