Jay LeBoeufImagine Research
jayatimagine-researchcom
July 2008
Intelligent Audio Systems A review of the foundations and applications of
semantic audio analysis and music information retrieval
WIKI REFERENCEShellip
These lecture notes contains hyperlinks to the CCRMA Wiki
On these pages you can find additional supplement the lecture material found in the class - providing extra tutorials support references for further reading or demonstration code snippets for those interested in a given topic
Click on the symbol on the lower-left corner of a slide to access additional resources
bull ccrmastanfordeduworkshops2008
bull Introductionsndash A little about yourselfndash Backgroundndash List of your region of interest and any specific items of
interest that yoursquod like to see covered
Administration
Final projects
Motivation by your personal interests or audio collections
Motivated by examining large audio collectionsExamples
httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type
ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc
Example Seedhellip
Queries and Evaluation
bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer
bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet
bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or
inferred)
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
WIKI REFERENCEShellip
These lecture notes contains hyperlinks to the CCRMA Wiki
On these pages you can find additional supplement the lecture material found in the class - providing extra tutorials support references for further reading or demonstration code snippets for those interested in a given topic
Click on the symbol on the lower-left corner of a slide to access additional resources
bull ccrmastanfordeduworkshops2008
bull Introductionsndash A little about yourselfndash Backgroundndash List of your region of interest and any specific items of
interest that yoursquod like to see covered
Administration
Final projects
Motivation by your personal interests or audio collections
Motivated by examining large audio collectionsExamples
httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type
ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc
Example Seedhellip
Queries and Evaluation
bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer
bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet
bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or
inferred)
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull ccrmastanfordeduworkshops2008
bull Introductionsndash A little about yourselfndash Backgroundndash List of your region of interest and any specific items of
interest that yoursquod like to see covered
Administration
Final projects
Motivation by your personal interests or audio collections
Motivated by examining large audio collectionsExamples
httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type
ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc
Example Seedhellip
Queries and Evaluation
bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer
bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet
bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or
inferred)
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Final projects
Motivation by your personal interests or audio collections
Motivated by examining large audio collectionsExamples
httpenwikipediaorgwikiList_of_Music_Genome_Project_attributes_by_type
ndash Instrument IDndash Chord recognitionndash SFX Loop searchesndash Transcription by classifierndash Speaker ID and characteristicsndash Narrator segmentationndash etc
Example Seedhellip
Queries and Evaluation
bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer
bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet
bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or
inferred)
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Example Seedhellip
Queries and Evaluation
bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer
bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet
bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or
inferred)
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Queries and Evaluation
bull Query by Hummingndash Lots of academic workndash Demo with Midomi or Fraunhofer
bull Query by audio ID ndash Gracenote ID Shazam Audible Magicndash Noisy audio snippet
bull Query by examplendash Find more like this (where ldquothisrdquo has to be specified or
inferred)
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Opportunities
bull Genre ID (labels exist but even humans disagree)bull Artist classification
ndash Tricks use voice only to improve accuracy to 70 (out of 100 artists)
bull Artist similarity ndash Ground truth from wwwmusicseercomndash Really what is the similarity
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
FUTURE SOFTWARE CAPABILITIES
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
TempoKey 000000
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
TempoKey Analyzing tempohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey Calculating Bars Beatshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey Estimating Keyhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Instrument IDhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Instrument Groupinghellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Guitar Chord Analysishellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Transcribing Drumshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Transcribing Pianohellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Identifying Phraseshellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Segmenting Songhellip
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Tempo125 BPMKey A major Done
Track 1
Track 2
Track 3
Track 4
Track 5
Track 6
Track 7
Track 8
Track 9
Track 10
Kick
Snare
Drum OH L
Drum OH R
Track 1Dirty Guitar
Elec Bass
Male Vocals
Male Vocals
Clean Guitar
Piano
Kick
Snare
Drum OH L
Drum OH R
Male Vocals
Male Vocals
Dirty Guitar
Elec Bass
Clean Guitar
Piano
A C D G A C D G A C D G Em D Em D A C D G A
A A G A A G A A G A
Intro Verse 1Verse 1 Chorus Chorus 2 Bridge Verse Chorus
Drums
Snare
Drum OH L
Drum OH R
Kick
Snare
Drum OH L
Drum OH R
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Why MIR
bull Find specific itembull Find something vaguebull Find something interesting or newbull Google for audio
ndash Quaero ndash French ldquoGoogle killer with image and audio example queries
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull Retrieval based on similarity (IR and creative applications)
bull Live analysis of audiobull Music Discovery Recommendationbull Query for musicbull Assisted Music Transcriptionbull Audio fingerprintbull Creative applications
Commercial Applications
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Motivations Demos
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtmlbull httpwwwpragprogcomarticlesa-pragmatic-
project-live-in-concertthe-methodology
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull Transcriptionist vs Descriptionist approach
ndash Music Transcription (restoration) ndash piano company from MIDI
bull httpzenphcomlistenhtml
- More transcription
Motivations Demos
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
BASIC SYSTEM OVERVIEW
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Basic system overview
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Segmentation
(Frames Onsets Beats Bars Chord
Changes etc)
Feature Extraction
(Time-based spectral energy
MFCC etc)
Analysis Decision Making
(Classification Clustering etc)
Basic system overview
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
TIMING AND SEGMENTATION
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
Timing and Segmentation
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
1 second1 second
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull Slicing up by fixed time sliceshellipndash 1 second 80 ms 100 ms 20-40ms etc
bull ldquoFramesrdquondash Different problems call for different frame lengths
bull Onset detectionbull Beat detection
ndash Beatndash Measure Bar Harmonic changes
bull Segments ndash Musically relevant boundariesndash Separate by some perceptual cue
Timing and Segmentation
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull What is an Onsetbull How to detect
ndash Envelope is not enoughndash Need to examine frequency bands
Onset detection
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Filterbank
Detect change in band 1
Detect change in band 2
hellip
Detect change in band N
Combine results
audio
Detection
output
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
1
2
3
4
5
6
7
8
octa
ve
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
threshold
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Adaptive
threshold
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull Beat detectionndash Tempo (eg 125 bpm)
bull Detecting periodicities from the onset detection curvendash Beat
bull AKA ldquoTactusrdquo ndash the ldquofoot tapping raterdquobull Time-frequency analysis -gt Resonators -gt Probablistic
modelndash Measure
bull Musical change rate bull Harmonic change rate
Beat and Tempo Detection
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Many many many approaches
Some simple ones 1 Autocorrelation function of the onset detection curve2 Spectral decomposition of the onset detection curve3 Combine both strategies the autocorrelation function is
translated into the frequency domain in order to be compared to the spectrum curve - two curves are then multiplied
Peak picking is applied to the autocorrelation function or to the spectrum representation
Beat and Tempo Detection
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
beats
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
FEATURE EXTRACTION
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Frame 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
FRAME 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Zero crossing rate = 9FRAME 1
ZERO CROSSING RATE
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
Frame 2
Zero crossing rate = 423
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
ANALYSIS AND DECISION MAKING
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
bull Example ldquoCowbellrdquo on just the snare drum of a drum loop ldquoSimplerdquo instrument recognition
bull Use basic thresholds or simple decision tree to form rudimentary transcription of kicks and snares
bull Time for more sophistication
Heuristic Analysis
gt End of Lecture 1
gt End of Lecture 1
