Hands On: Multimedia Methods for Large Scale Video Analysis(Project Meeting)
Dr. Gerald Friedland, [email protected]
1
Wednesday, August 29, 12
Today
Wednesday, August 29, 12
Today
• Project Requirements
Wednesday, August 29, 12
Today
• Project Requirements• Data available
Wednesday, August 29, 12
Today
• Project Requirements• Data available• Compute Architecture at ICSI
Wednesday, August 29, 12
Today
• Project Requirements• Data available• Compute Architecture at ICSI• Some Project Ideas
Wednesday, August 29, 12
Project Requirements
3
Wednesday, August 29, 12
Project Requirements
• Form a team of 2 to 3 people.
3
Wednesday, August 29, 12
Project Requirements
• Form a team of 2 to 3 people.• Each team work on one project idea
3
Wednesday, August 29, 12
Project Requirements
• Form a team of 2 to 3 people.• Each team work on one project idea• Project must be on big multimedia
data, e.g. at least ten-thousands of videos, hundred-thousands of sounds pieces/images.
3
Wednesday, August 29, 12
Project Requirements
• Form a team of 2 to 3 people.• Each team work on one project idea• Project must be on big multimedia
data, e.g. at least ten-thousands of videos, hundred-thousands of sounds pieces/images.
• Team delivers written project report at the end of semester, reports on progress during the semester
3
Wednesday, August 29, 12
Each Project Report...
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
• ...needs to report on
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
• ...needs to report on – accuracy
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
• ...needs to report on – accuracy– efficiency
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
• ...needs to report on – accuracy– efficiency– scalability
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
• ...needs to report on – accuracy– efficiency– scalability– limits of the approach
4
Wednesday, August 29, 12
Each Project Report...
• ...needs to explain the idea of the project and show evidence that the project has been performed.
• ...needs to report on – accuracy– efficiency– scalability– limits of the approach
• ... and reports on problems occured.4
Wednesday, August 29, 12
Project Resources
5
Wednesday, August 29, 12
Project Resources• Data:
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
– Use your own (after discussion with instructor)
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
– Use your own (after discussion with instructor)
• Compute:
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
– Use your own (after discussion with instructor)
• Compute:– Use your own laptop initially then
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
– Use your own (after discussion with instructor)
• Compute:– Use your own laptop initially then– Use ICSI’s compute pool then
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
– Use your own (after discussion with instructor)
• Compute:– Use your own laptop initially then– Use ICSI’s compute pool then– if needed, use Amazon EC2
5
Wednesday, August 29, 12
Project Resources• Data:
– Use the data provided in the class (TrecVID, 1M Songs, MediaEval)
– Use your own (after discussion with instructor)
• Compute:– Use your own laptop initially then– Use ICSI’s compute pool then– if needed, use Amazon EC2– Use any other compute resource that you
have access to. 5
Wednesday, August 29, 12
Project Resources II
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
• Time:
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
• Time:– Start work on your project as early as
possible!
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
• Time:– Start work on your project as early as
possible!
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
• Time:– Start work on your project as early as
possible!
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
• Time:– Start work on your project as early as
possible!
6
Wednesday, August 29, 12
Project Resources II
• Hard disk:– Once in the ICSI phase use ICSI’s ttmp
structure– At Amazon: Need to buy space if needed.
• Time:– Start work on your project as early as
possible!
•
6
Wednesday, August 29, 12
Project Idea
7
Wednesday, August 29, 12
Project Idea
• Come up with your own project idea as a team, inspired by the class content, co-students, the data, and/or other input.
7
Wednesday, August 29, 12
Project Idea
• Come up with your own project idea as a team, inspired by the class content, co-students, the data, and/or other input.
• Discuss the project idea with the class and the instructor
7
Wednesday, August 29, 12
Available BIGDATA
8
Wednesday, August 29, 12
Available BIGDATA
• MediaEval 2012
8
Wednesday, August 29, 12
Available BIGDATA
• MediaEval 2012• TrecVid MED 2011
8
Wednesday, August 29, 12
Available BIGDATA
• MediaEval 2012• TrecVid MED 2011• 1M Songs
8
Wednesday, August 29, 12
Placing TaskAutomatically guess the location of a Flickr video:• i.e., assign geo-coordinates (latitude and
longitude)• Using one or more of:
– Visual/Audio content– Metadata (title, tags, description, etc)– Social information 9
Wednesday, August 29, 12
Data Description (2010)
• Training Data– 15k videos/metadata/visual
keyframes (+features)/geo-tags– 6M photos/metadata/visual
features• Test Data
– 5k video/metadata/visual keyframes (+features)
– no geotags• Test/Training split: by UserID
10
Wednesday, August 29, 12
Example
11
Wednesday, August 29, 12
A ‘Good’ video
14
• Title: CIMG0254• Keywords: southamerica, june, 2008, video, pearce, vacation,
iguazufalls, iguassufalls, iguaçufalls, waterfall, argentina• Description: (None) Wednesday, August 29, 12
How about this one?
15
• Title: MVI_6423_rau• Keywords: usa08, puertorico• Description: (None)
Wednesday, August 29, 12
Another ‘bad’ video
16
• Title: Stillness #1• Keywords: henrywcoestatepark, henrycoe, california, 2009,
hiking, landscape, nature, chinaholetrail, video• Description: When I hike, I like to stop at random spots and just
stand still for a few minutes and listen and look at my surroundings. On this hike, I decided to record a few seconds during those moments.
Wednesday, August 29, 12
Metadata
• 98.8% of videos were annotated with at least one title, tags, or description
17
Wednesday, August 29, 12
TrecVID MED 2011 detailed
16
Wednesday, August 29, 12
TrecVID MED 2011 detailed
16
• Videos all “consumer produced” , typically 1-5 minutes long
Wednesday, August 29, 12
TrecVID MED 2011 detailed
16
• Videos all “consumer produced” , typically 1-5 minutes long
• Given 15 concepts, 5 for training, 10 for eval
Wednesday, August 29, 12
TrecVID MED 2011 detailed
16
• Videos all “consumer produced” , typically 1-5 minutes long
• Given 15 concepts, 5 for training, 10 for eval
• About 100 sample videos per concept
Wednesday, August 29, 12
TrecVID MED 2011 detailed
16
• Videos all “consumer produced” , typically 1-5 minutes long
• Given 15 concepts, 5 for training, 10 for eval
• About 100 sample videos per concept• Testset 2011: 50k videos, open set
Wednesday, August 29, 12
TrecVID Dataset
17
Consumer-Produced, Unfiltered Videos...Wednesday, August 29, 12
What is Video Concept Detection?
18
Wednesday, August 29, 12
What is Video Concept Detection?
18
• A concept (as of TrecVID MED 11):
Wednesday, August 29, 12
What is Video Concept Detection?
18
• A concept (as of TrecVID MED 11):– is a complex activity occurring at a specific place and
time;
Wednesday, August 29, 12
What is Video Concept Detection?
18
• A concept (as of TrecVID MED 11):– is a complex activity occurring at a specific place and
time;– involves people interacting with other people and/or
objects;
Wednesday, August 29, 12
What is Video Concept Detection?
18
• A concept (as of TrecVID MED 11):– is a complex activity occurring at a specific place and
time;– involves people interacting with other people and/or
objects;– consists of a number of human actions, processes,
and activities that are loosely or tightly organized and that have significant temporal and semantic relationships to the overarching activity;
Wednesday, August 29, 12
What is Video Concept Detection?
18
• A concept (as of TrecVID MED 11):– is a complex activity occurring at a specific place and
time;– involves people interacting with other people and/or
objects;– consists of a number of human actions, processes,
and activities that are loosely or tightly organized and that have significant temporal and semantic relationships to the overarching activity;
– is directly observable.
Wednesday, August 29, 12
Event Kit?
19
Wednesday, August 29, 12
Event Kit?
19
• An Event Kit consists of:
Wednesday, August 29, 12
Event Kit?
19
• An Event Kit consists of:– A textual (natural language) event definition
Wednesday, August 29, 12
Event Kit?
19
• An Event Kit consists of:– A textual (natural language) event definition – A textual (NL) event explication, which is a
glossart for the definition
Wednesday, August 29, 12
Event Kit?
19
• An Event Kit consists of:– A textual (natural language) event definition – A textual (NL) event explication, which is a
glossart for the definition– Evidential description: A textual listing of
attributes that are indicative of an event instance.
Wednesday, August 29, 12
Event Kit?
19
• An Event Kit consists of:– A textual (natural language) event definition – A textual (NL) event explication, which is a
glossart for the definition– Evidential description: A textual listing of
attributes that are indicative of an event instance.
– A set of illustrative video examples each containing an instance of the event.
Wednesday, August 29, 12
Event Kit: Example
20
Event name: Attempting a board trick
Definition: One or more people attempt to do a trick on a skateboard,snowboard, surfboard, or other boardsport board.
Explication: Boardsports are sports where a person stands, sits, or lays on a board and moves and controls the board. Tricks consist of intentional motions made with the board that are not simply slowing down/stopping the board or steering the board as it moves. Steering around obstacles or steering a board off of a jump and landing on the ground are not considered tricks in and of themselves.
Common tricks involve actions like sliding the board along the top of anobject (e.g. a swimming pool rim or railing), jumping from the ground orthe surface of water into the air, and spinning or flipping in the air.
Evidential description:
scene: outside, often in a skatepark
objects/people: skateboard, snowboard, surfboard, ramps, rails, safety gear, crowds
activities: standing, sitting or laying on the board; jumping with the board; flipping the board and landing on it; spinning the board; sliding the board across various objects.
audio: sounds of board hitting surface during trick; crowd cheeringWednesday, August 29, 12
2011 Concepts
21
Event Category Train DevTest E001 Board Tricks 160 111E002 Feeding Animal 160 111E003 Landing a Fish 122 86E004 Wedding 128 88E005 Woodworking 142 100E006 Birthday Party 173 0E007 Changing Tire 110 0E008 Flash Mob 173 0E009 Vehicle Unstuck 131 0E010 Grooming animal 136 0E011 Make a Sandwich 124 0E012 Parade 134 0E013 Parkour 108 0E014 Repairing Appliance 123 0E015 Sewing 116 0Other Random other N/A 3755
Wednesday, August 29, 12
Sample Video 1: “Board Tricks”
22
Wednesday, August 29, 12
Sample Video 2: “Board Tricks”
23
Wednesday, August 29, 12
Test Video: “Board Tricks”
24
Wednesday, August 29, 12
Test Video: “Board Tricks”
24
NOT A POSITIVE!
Wednesday, August 29, 12
25
Wednesday, August 29, 12
25
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.
Wednesday, August 29, 12
25
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:
Wednesday, August 29, 12
25
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:• To encourage research on algorithms that scale to
commercial sizes
Wednesday, August 29, 12
25
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:• To encourage research on algorithms that scale to
commercial sizes• To provide a reference dataset for evaluating research
Wednesday, August 29, 12
25
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:• To encourage research on algorithms that scale to
commercial sizes• To provide a reference dataset for evaluating research• As a shortcut alternative to creating a large dataset with APIs
(e.g. The Echo Nest's)
Wednesday, August 29, 12
25
The Million Song Dataset is a freely-available collection of audio features and metadata for a million contemporary popular music tracks.Its purposes are:• To encourage research on algorithms that scale to
commercial sizes• To provide a reference dataset for evaluating research• As a shortcut alternative to creating a large dataset with APIs
(e.g. The Echo Nest's)• To help new researchers get started in the MIR field
Wednesday, August 29, 12
26
Wednesday, August 29, 12
26
• Dataset is a sqllite database of about 100 attributes for each song.
Wednesday, August 29, 12
26
• Dataset is a sqllite database of about 100 attributes for each song.
• Subset of 10k song attributes available
Wednesday, August 29, 12
26
• Dataset is a sqllite database of about 100 attributes for each song.
• Subset of 10k song attributes available• The song IS NOT part of the database.
Need to access using a different database using metadata (e.g. Grooveshark).
Wednesday, August 29, 12
27
Wednesday, August 29, 12
Compute Architecture at ICSI
Wednesday, August 29, 12
Compute Architecture at ICSI
• Each enrolled student gets account at ICSI
Wednesday, August 29, 12
Compute Architecture at ICSI
• Each enrolled student gets account at ICSI
• ICSI = International Computer Science Institute:http://www.icsi.berkeley.eduLocation: 1947 Center Street, 6th floor
Wednesday, August 29, 12
Compute Architecture at ICSI
Wednesday, August 29, 12
Compute Architecture at ICSI
• Accounts belong to “speech group” and allow access to Unix compute cluster of that group
Wednesday, August 29, 12
Compute Architecture at ICSI
• Accounts belong to “speech group” and allow access to Unix compute cluster of that group
• Compute Cluster is currently 160 CPUs
Wednesday, August 29, 12
Compute Architecture at ICSI
• Accounts belong to “speech group” and allow access to Unix compute cluster of that group
• Compute Cluster is currently 160 CPUs
• Main compute: “Squids” 8x8 CPUs, 8x2 GPUs
Wednesday, August 29, 12
Compute Architecture at ICSI
Wednesday, August 29, 12
Compute Architecture at ICSI
• Most important page for technical Info:https://speechwiki.icsi.berkeley.edu/speechwiki/index.php/Main_Page
Wednesday, August 29, 12
Compute Architecture at ICSI
• Most important page for technical Info:https://speechwiki.icsi.berkeley.edu/speechwiki/index.php/Main_Page
• Includes “Ganglia Monitoring”
Wednesday, August 29, 12
Compute Architecture at ICSI
• Most important page for technical Info:https://speechwiki.icsi.berkeley.edu/speechwiki/index.php/Main_Page
• Includes “Ganglia Monitoring”
Wednesday, August 29, 12
Compute Architecture at ICSI
• Most important page for technical Info:https://speechwiki.icsi.berkeley.edu/speechwiki/index.php/Main_Page
• Includes “Ganglia Monitoring”
Wednesday, August 29, 12
Compute Pool
Wednesday, August 29, 12
Compute Pool• Main usage policy: “Be nice to
each other”
Wednesday, August 29, 12
Compute Pool• Main usage policy: “Be nice to
each other”• If you want to start a job using
more than 16 CPUs, please send email to: [email protected].
Wednesday, August 29, 12
Compute Pool• Main usage policy: “Be nice to
each other”• If you want to start a job using
more than 16 CPUs, please send email to: [email protected].
• Never login into a compute machine to run jobs!
Wednesday, August 29, 12
Compute Pool• Main usage policy: “Be nice to
each other”• If you want to start a job using
more than 16 CPUs, please send email to: [email protected].
• Never login into a compute machine to run jobs!
• If in doubt or trouble: Send me email.
Wednesday, August 29, 12
ICSI Compute Pool: Basic Usage
Easy compared to Amazon!
Wednesday, August 29, 12
ICSI Compute Pool: Basic Usage
• showjobs -- Shows job currently running
Easy compared to Amazon!
Wednesday, August 29, 12
ICSI Compute Pool: Basic Usage
• showjobs -- Shows job currently running
• run-command -- Starts a job
Easy compared to Amazon!
Wednesday, August 29, 12
ICSI Compute Pool: Basic Usage
• showjobs -- Shows job currently running
• run-command -- Starts a job• ssh machine /bin/kill -1 -- -pid -- Kill a job
Easy compared to Amazon!
Wednesday, August 29, 12
ICSI Compute Pool: Data Storage
Wednesday, August 29, 12
ICSI Compute Pool: Data Storage
• Home/Project directory (backed up, strictly quota’d)
Wednesday, August 29, 12
ICSI Compute Pool: Data Storage
• Home/Project directory (backed up, strictly quota’d)
• Local scratch space (fast, local machine, not backed up)
Wednesday, August 29, 12
ICSI Compute Pool: Data Storage
• Home/Project directory (backed up, strictly quota’d)
• Local scratch space (fast, local machine, not backed up)
• Networked scratch space (not backed up)
Wednesday, August 29, 12
ICSI Compute Pool: Data Storage
• Home/Project directory (backed up, strictly quota’d)
• Local scratch space (fast, local machine, not backed up)
• Networked scratch space (not backed up)
• Temporary space (so-called ttmp, networked, deleted automatically)
Wednesday, August 29, 12
ICSI Compute Pool: Availabilty
There are always other jobs...Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
• Build the “any tag” detector
Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
• Build the “any tag” detector• City-scale location estimation
Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector
Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector• Correlate Users among videos
Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector• Correlate Users among videos• Find videos from a certain country
Wednesday, August 29, 12
Some Project Ideas (MediaEval Dataset)
• Build the “any tag” detector• City-scale location estimation• Rural/Non-Rural Detector• Correlate Users among videos• Find videos from a certain country• Correlate video quality with tag
quality
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
• Build a simple TrecVid MED system!
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
• Build a simple TrecVid MED system!
• Build a keyword-spotter
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
• Build a simple TrecVid MED system!
• Build a keyword-spotter• Build a face detector
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
• Build a simple TrecVid MED system!
• Build a keyword-spotter• Build a face detector• Sort videos by visual similarity
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
• Build a simple TrecVid MED system!
• Build a keyword-spotter• Build a face detector• Sort videos by visual similarity
Wednesday, August 29, 12
Some Project Ideas (TrecVid Dataset)
• Build a simple TrecVid MED system!
• Build a keyword-spotter• Build a face detector• Sort videos by visual similarity
Wednesday, August 29, 12
Some Project Ideas (1M Song Dataset)
Wednesday, August 29, 12
Some Project Ideas (1M Song Dataset)
• Build a dubbed-song recognition system for the TrecVid MED set
Wednesday, August 29, 12
Some Project Ideas (1M Song Dataset)
• Build a dubbed-song recognition system for the TrecVid MED set
• Correlate song similarity with user ratings
Wednesday, August 29, 12
Some Project Ideas (1M Song Dataset)
• Build a dubbed-song recognition system for the TrecVid MED set
• Correlate song similarity with user ratings
• Cluster and sort songs by tags and correlate with acoustic clustering
Wednesday, August 29, 12
Some Project Ideas (1M Song Dataset)
• Build a dubbed-song recognition system for the TrecVid MED set
• Correlate song similarity with user ratings
• Cluster and sort songs by tags and correlate with acoustic clustering
• Try to align lyrics with songs
Wednesday, August 29, 12
Some Project Ideas (1M Song Dataset)
• Build a dubbed-song recognition system for the TrecVid MED set
• Correlate song similarity with user ratings
• Cluster and sort songs by tags and correlate with acoustic clustering
• Try to align lyrics with songs
Wednesday, August 29, 12
This Week (Lecture)
38
•Architectural Considerations for Large Scale Conten Analysis
Wednesday, August 29, 12
Next Week (Project Meeting)
39
• Amazon EC2 and how to use it
Wednesday, August 29, 12