Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
Pre-Viva Talk:
Parsing Jazz:Harmonic Analysis of Music UsingCombinatory Categorial Grammar
Mark Granroth-Wilding
Supervisors:Mark Steedman Sharon Goldwater
School of InformaticsUniversity of Edinburgh
15th March 2013
Handout
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
1/24
Introduction
• Structures underly music
• Hierarchical structures
• Metrical structure
• Harmonic structure
1
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
1/24
Introduction
• Structures underly music
• Hierarchical structures
• Metrical structure
• Harmonic structure
1
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
2/24
Approaches to Musical Analysis
• Varying goals:
1. model/aid compositional process2. model listener’s cognition3. suggest interpretations
• Often not clearly defined
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
3/24
Thesis
• Tonal harmony has a syntax like that of language
• Statistical parsing can be used to infer harmonic structure
Contributions:
• Formal grammar for syntax of harmony
• Harmonic analysis by parsing
• Practical statistical parsing of chord sequences
• Extension to analysis of performance data
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
4/24
A Few Applications
• Automatic generation:• melodic variations• accompaniments
• Song identification
• Language modelling for transcription
(speculative)
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
5/24
Consonance and Harmony
• Simultaneous notes createdissonance / consonance
• Used by composers:tension / relaxation
• Harmony :formation of chord phrases
�������
��
� ��
� �����
�����
���
�����
���
����
����
• Relationships between chords
• Expectation / fulfilment
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
6/24
Approaches to Harmonic Analysis
Rameau (1722): Traite de l’harmonie
Roman numeral analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
D: I IV I IV V7 I IV6 V I
Key of D
��VI��� VIIV� ����II���
I
���IV��� �III��� �
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
6/24
Approaches to Harmonic AnalysisRameau (1722): Traite de l’harmonie
Roman numeral analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
D: I IV I IV V7 I IV6 V I
Riemann (1893): Vereinfachte Harmonielehre
Functional analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
T S T S D T S D T
Key of D
� ���Tp
��D
��D 7
���Sp
���T
���S
��� �Dp
���
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
6/24
Approaches to Harmonic AnalysisRameau (1722): Traite de l’harmonie
Roman numeral analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
D: I IV I IV V7 I IV6 V I
Riemann (1893): Vereinfachte Harmonielehre
Functional analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
T S T S D T S D T
Lerdahl & Jackendoff (1983):
A Generative Theory of Tonal Music
�������
����
� ��
����
�����
����
����
���
��������
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
6/24
Approaches to Harmonic AnalysisRameau (1722): Traite de l’harmonie
Roman numeral analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
D: I IV I IV V7 I IV6 V I
Riemann (1893): Vereinfachte Harmonielehre
Functional analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
T S T S D T S D T
Lerdahl & Jackendoff (1983):
A Generative Theory of Tonal Music
�������
����
� ��
����
�����
����
����
���
��������
Winograd (1968), Keiler (1978),
Steedman (1984), Rohrmeier (2011)
Structured functional analysis
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
6/24
Approaches to Harmonic AnalysisRameau (1722): Traite de l’harmonie
Roman numeral analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
D: I IV I IV V7 I IV6 V I
Riemann (1893): Vereinfachte Harmonielehre
Functional analysis
�������
��
� ��
� �����
�����
���
�����
���
����
����
T S T S D T S D T
Lerdahl & Jackendoff (1983):
A Generative Theory of Tonal Music
�������
����
� ��
����
�����
����
����
���
��������
Winograd (1968), Keiler (1978),
Steedman (1984), Rohrmeier (2011)
Structured functional analysis
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
7/24
Tonal Space
• Longuet-Higgins’ formalization ofharmonic tonal theory
• Tonal relations between notes
• Ambiguous in performance
E[[
G[
B[
D
F]
B[[
D[
F
A
C]
F[
A[
C
E
G]
C[
E[
G
B
D]
G[
B[
D
F]
A]
D[
F
A
C]
E]
yMajor 3rd
4:5
xPerfect 5th
2:3
zOctave
1:2
• Harmonic analysis disambiguates tonal relations
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
8/24
Harmony in the Tonal Space
E[[[
G[[
B[[
D[
F
A
C]
B[[[
D[[
F[
A[
C
E
G]
F[[
A[[
C[
E[
G
B
D]
C[[
E[[
G[
B[
D
F]
A]
G[[
B[[
D[
F
A
C]
E]
D[[
F[
A[
C
E
G]
B]
A[[
C[
E[
G
B
D]
F]]
E[[
G[
B[
D
F]
A]
C]]
B[[
D[
F
A
C]
E]
G]]
F[
A[
C
E
G]
B]
D]]
C[
E[
G
B
D]
F]]
A]]
Dominant
Tonic
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
9/24
Functional Harmony
E[[[
G[[
B[[
D[
F
A
C]
B[[[
D[[
F[
A[
C
E
G]
F[[
A[[
C[
E[
G
B
D]
C[[
E[[
G[
B[
D
F]
A]
G[[
B[[
D[
F
A
C]
E]
D[[
F[
A[
C
E
G]
B]
A[[
C[
E[
G
B
D]
F]]
E[[
G[
B[
D
F]
A]
C]]
B[[
D[
F
A
C]
E]
G]]
F[
A[
C
E
G]
B]
D]]
C[
E[
G
B
D]
F]]
A]]
DominantTonic
Subdominant
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
10/24
Harmonic Analysis
• Functional harmonic structure
• Segmentation into chords
• Identification of keys
• Functional relationships between chords
C E7 A7 Dm7 G7 Dm7 D[7 C
dom dom dom
dom
dom dom
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
11/24
Harmonic Analysis
• Chords function as: dominant, subdominant or tonic
• Dominant-tonic resolution
• Subdominant-tonic resolution
• Recursion
• Substitution
• Delayed resolution: coordination
D[
F
A
A[
C
E
E[
G
B
G7 C
dom
F C
subdom
D7 G7 C
dom dom
D7 D[7 C
dom dom
D7 G7 D7 D[7 C
dom dom dom
dom
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
11/24
Harmonic Analysis
• Chords function as: dominant, subdominant or tonic
• Dominant-tonic resolution
• Subdominant-tonic resolution
• Recursion
• Substitution
• Delayed resolution: coordination
D[
F
A
A[
C
E
E[
G
B
G7 C
dom
F C
subdom
D7 G7 C
dom dom
D7 D[7 C
dom dom
D7 G7 D7 D[7 C
dom dom dom
dom
Mark bought and Greg read the book
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
11/24
Harmonic Analysis
• Chords function as: dominant, subdominant or tonic
• Dominant-tonic resolution
• Subdominant-tonic resolution
• Recursion
• Substitution
• Delayed resolution: coordination
D[
F
A
A[
C
E
E[
G
B
G7 C
dom
F C
subdom
D7 G7 C
dom dom
D7 D[7 C
dom dom
D7 G7 D7 D[7 C
dom dom dom
dom
C E7 A7 Dm7 G7 Dm7 D[7 C
dom dom dom
dom
dom dom
B[[
D[
F
A
F[
A[
C
E
C[
E[
G
B
G[
B[
D
F]
D[
F
A
C]
A[
C
E
G]
E[
G
B
D]
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
12/24
Harmonic Combinatory Categorial Grammar
G7 C
dom
C
D[
F
A
A[
C
E
E[
G
B
B[
D
F]
G7 ⇒ G/C : λx . dom(x)
C ⇒ C : C
G7 C
G/C C>
G –C
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
12/24
Harmonic Combinatory Categorial Grammar
G7 C
dom
C
D[
F
A
A[
C
E
E[
G
B
B[
D
F]
G7 ⇒ G/C : λx . dom(x)
C ⇒ C : C
G7 C
G/C : λx . dom(x) C : C>
G –C : dom(C)
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
13/24
Harmonic CCG: Recursion
D7 G7 C
dom dom
C
D7 G7 C
D/G : λx . dom(x) G/C : λx . dom(x) C : C>
G –C : dom(C)>
D–C : dom(dom(C))
Dom: G7 ⇒ G/C : λx . dom(x) Ton: C ⇒ C : C
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
13/24
Harmonic CCG: Recursion
D7 G7 C
dom dom
C
D7 G7 C
D/G : λx . dom(x) G/C : λx . dom(x) C : C>B
D/C : λx . dom(dom(x))>
D–C : dom(dom(C))
Dom: G7 ⇒ G/C : λx . dom(x) Ton: C ⇒ C : C
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
14/24
Harmonic CCG: Substitution
D7 D[7 C
dom dom
D7 D[7 C
D/G G/C C>B
D/C>
D–C
Dom: G7 ⇒ G/C Ton: C ⇒ C
Dom tritone: D[7 ⇒ G/C
D7 G7 C
D/G G/C C>B
D/C>
D–C
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
15/24
Harmonic CCG: Coordination
D7 G7 D7 D[7 C
dom dom dom
dom C
D7 G7 D7 D[7 C
D/G G/C D/G G/C C>B >B
D/C D/C&
D/C>
D–C
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
16/24
Derivation Model
• Supervised statistical parsing model
• Parsing model: Hockenmaier & Steedman (2002)1
• Model of CCG derivations: PCCG
• Supervised training, smoothing
• CKY parser with beam
D7 G7 D7 D[7 C
D/G G/C D/G G/C C>B >B
D/C D/C&
D/C>
D–C
1Generative models for statistical parsing with Combinatory Categorial Grammar. ACL
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
17/24
Supertagging
D7 G7 D7 D[7 C
D/G G/C D/G G/C C
>B >BD/C D/C
&D/C
>D–C
D/A G/D A[/D[ D[/G [ E
>B–E
B[/F A/D C/F B/E C/F
· · ·
• Experimented with n-gram models
• Small corpus: trigrams don’t help
• For parsing experiments:• bigram (HMM)• Katz backoff• Witten-Bell discounting• adaptive supertagging (Clark & Curran, 2007)2
2Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
17/24
Supertagging
D7 G7 D7 D[7 C
D/G G/C D/G G/C C
>B >BD/C D/C
&D/C
>D–C
D/A G/D A[/D[ D[/G [ E
>B–E
B[/F A/D C/F B/E C/F
· · ·
• Experimented with n-gram models
• Small corpus: trigrams don’t help
• For parsing experiments:• bigram (HMM)• Katz backoff• Witten-Bell discounting• adaptive supertagging (Clark & Curran, 2007)2
2Wide-Coverage Efficient Statistical Parsing with CCG and Log-Linear Models
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
18/24
Baseline: HmmPath
• Construct path with HMM:
(0,0) (-1,0) (-1,0)
D7 G7 CM7
D[
F
A
A[
C
E
E[
G
B
B[
D
F]
F
A
C]
C E7 A7 Dm7 G7 Dm7 D[7 C
dom dom dom
dom
dom dom
B[[
D[
F
A
F[
A[
C
E
C[
E[
G
B
G[
B[
D
F]
D[
F
A
C]
A[
C
E
G]
E[
G
B
D]
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
18/24
Baseline: HmmPath
• Construct path with HMM:
(0,0) (-1,0) (-1,0)
D7 G7 CM7
D[
F
A
A[
C
E
E[
G
B
B[
D
F]
F
A
C]
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
19/24
Jazz Corpus
• Annotated training corpus
• Jazz chord sequences
• Full grammatical derivationannotated
• → full harmonic analysis
• 74 sequences: ∼3k chords
• Cross-validation
CM7 F♯φ7 B7♭9 Em7 A7 Dm7 Gm7 Dm7 G7 CM7
CM7 F♯φ7 B7♭9 Gm7 Dm7 Am7 Dm7 G7 CM7
F6 G♯φ7 G7 E♭7 Am7 Em7 Am7 Dm7 F F7
GM7 Bφ7 Em7 A7 Dm7 Am7 Fm7 GM7
CM7 F♯φ7 B7♭9 Em7 A7 Dm7 Gm7 Dm7 G7 CM7
C E7 A7 Dm7 G7 Dm7 D[7 C
dom dom dom
dom
dom dom
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
19/24
Jazz Corpus
• Annotated training corpus
• Jazz chord sequences
• Full grammatical derivationannotated
• → full harmonic analysis
• 74 sequences: ∼3k chords
• Cross-validation
CM7 F♯φ7 B7♭9 Em7 A7 Dm7 Gm7 Dm7 G7 CM7
CM7 F♯φ7 B7♭9 Gm7 Dm7 Am7 Dm7 G7 CM7
F6 G♯φ7 G7 E♭7 Am7 Em7 Am7 Dm7 F F7
GM7 Bφ7 Em7 A7 Dm7 Am7 Fm7 GM7
CM7 F♯φ7 B7♭9 Em7 A7 Dm7 Gm7 Dm7 G7 CM7
C E7 A7 Dm7 G7 Dm7 D[7 C
dom dom dom
dom
dom dom
Afternoon in ParisAlice in WonderlandAnthropologyBeauty and the BeastBlack OrpheusBlackberry WinterBlue in GreenBoplicityBud PowellByrd LikeCall Me IrresponsibleCan’t Help Lovin’ Dat ManChelsea BridgeA Child is BornChippieChitlins Con CarneComo En VietnamConfirmationCrescentDear Old Stockholm
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
20/24
Evaluation Metric
Tonal space edit distance
D[
F
A
A[
C
E
E[
G
B
B[
D
F]
F
A
C]
DDT,T
E[[
G[
B[
D
B[[
D[
F
A
F[
A[
C
E
C[
E[
G
BT,T
DT
Precision, recall, f-score
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
21/24
Evaluation: Dependency Recovery
• Dependency recovery of harmonic analysis
C D7 D[7 C
C
dom dom
C
C D7 D[7 C
E
dom
D[ C
Gold standard:
Parse result:
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
22/24
Results
• Tonal space metric:
P (%) R (%) F (%) Cov. (%)
HmmPath 77.44 84.87 80.98 100PCCG 92.29 88.78 90.50 97.37St+PCCG 90.18 92.79 91.46 100
• Dependency recovery:
P (%) R (%) F (%)
PCCG 90.25 86.83 88.51St+PCCG 88.22 90.78 89.48
(0,0) (-1,0) (-1,0)
D7 G7 CM7
D7 G7 D7 D[7 C
D/G G/C D/G G/C C>B >B
D/C D/C&
D/C>
D–C
D7 G7 D7 D[7 C
D/G G/C D/G G/C C
>B >BD/C D/C
&D/C
>D–C
D/A G/D A[/D[ D[/G [ EB[/F A/D C/F B/E C/F
· · ·
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
23/24
Conclusion
• Harmonic analysis in the tonal space
• Hierarchical structure in harmony
• Harmonic adaptation of CCG
• Statistical parsing, adapted from NLP
• Chord sequence treebank
D[
F
A
A[
C
E
E[
G
B
B[
D
F]
F
A
C]
D7 G7 D7 D[7 C
dom dom dom
dom
D7 G7 D7 D[7 C
D/G G/C D/G G/C C>B >B
D/C D/C&
D/C>
D–C
D7 G7 D7 D[7 C
D/G G/C D/G G/C C
>B >BD/C D/C
&D/C
>D–C
D/A G/D A[/D[ D[/G [ EB[/F A/D C/F B/E C/F
· · ·CM7 F♯φ7 B7♭9 Em7 A7 Dm7 Gm7 Dm7 G7 CM7
CM7 F♯φ7 B7♭9 Gm7 Dm7 Am7 Dm7 G7 CM7
F6 G♯φ7 G7 E♭7 Am7 Em7 Am7 Dm7 F F7
GM7 Bφ7 Em7 A7 Dm7 Am7 Fm7 GM7
CM7 F♯φ7 B7♭9 Em7 A7 Dm7 Gm7 Dm7 G7 CM7
Introduction Harmonic Analysis Harmonic CCG Statistical Parsing Conclusion
24/24
Conclusion
Thesis:
• Tonal harmony has a syntax like that oflanguage
• Statistical parsing can be used to infer harmonicstructure
• CCG grammar of harmonic structure
• Statistical parser for harmonic analysis
• Parser outperforms HMM baseline
• Extension to analysis of performances
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Bibliography I
Clark, S., & Curran, J. R. (2007).Wide-coverage efficient statistical parsing with CCG andlog-linear models.Computational Linguistics, 33 , 493–552.
Hockenmaier, J., & Steedman, M. (2002).Generative models for statistical parsing with CombinatoryCategorial Grammar.In Proceedings of the 40th Meeting of the Association forComputational Linguistics, (pp. 335–342). Philadelphia, PA:Association for Computational Linguistics.
Keiler, A. (1981).Two views of musical semiotics.In W. Steiner (Ed.) The Sign in Music and Literature, (pp.138–168). Austin TX: University of Texas Press.
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Bibliography II
Lerdahl, F., & Jackendoff, R. (1983).A Generative Theory of Tonal Music .Cambridge, MA: MIT Press.
Rameau, J. P. (1722).Traite de l’harmonie.Jean-Baptiste-Christophe Ballard.
Riemann, H. (1893).Vereinfachte Harmonielehre oder die Lehre von den tonalenFunktionen der Akkorde.Augener & Co.Trans. H. Bemerunge, as Harmony Simplified, or the Theory ofthe Tonal Functions of Chords.
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Bibliography III
Rohrmeier, M. (2011).Towards a generative syntax of tonal harmony.Journal of Mathematics and Music , 5 , 35–53.
Steedman, M. (1984).A generative grammar for jazz chord sequences.Music Perception, 2 , 52–77.
Winograd, T. (1968).Linguistics and the computer analysis of tonal harmony.Journal of Music Theory , 12 , 2–49.
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Atomic Categories
G7 C
G/C C>
G –C
C G7 C
C G/C C>
G –Cdev
C –C = C
BM7 G7 C
B G/C C>
G –Cdev
B–C
C –C
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Parsing Performances
• Can we parse a musical performance?
G C G C G D7 G
• Proof-of-concept extension
• HMM chord recognizer
• Parse chord labels as before
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Parsing Performances
• Parser beats baseline
• Much lower results than chord task
• Chord recognizer over-commits, parse a lattice
• Need rhythmic/metrical models
• Harder modelling task:voice-leading, polyphony, chord inversion, octave separation, . . .
• Evaluation harder: model does segmentation
• Unlabelled data
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
MIDI Parsing
64 (C)
68 (E)
71 (G)
73 (A)72 (G♯)
75 (B)
66 (D)
63 (B)
MIDI input
Chord recognizer: lattice
Lattice-based supertagger/parser
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Tonal Space
E[[[
G[[
B[[
D[
F
A
C]
B[[[
D[[
F[
A[
C
E
G]
F[[
A[[
C[
E[
G
B
D]
C[[
E[[
G[
B[
D
F]
A]
G[[
B[[
D[
F
A
C]
E]
D[[
F[
A[
C
E
G]
B]
A[[
C[
E[
G
B
D]
F]]
E[[
G[
B[
D
F]
A]
C]]
B[[
D[
F
A
C]
E]
G]]
F[
A[
C
E
G]
B]
D]]
C[
E[
G
B
D]
F]]
A]]
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Roman Numeral Tonal Space
[[[III−
[[V−
[[VII−
[II−
IV−
VI−
]I−
[[[VII−
[[II−
[IV−
[VI−
I−
III−
]V−
[[IV−
[[VI−
[I−
[III−
V−
VII−
]II−
[[I−
[[III−
[V−
[VII−
II−
]IV−
]VI−
[[V
[[VII
[II
IV
VI
]I
]III
[[II
[IV
[VI
I
III
]V
]VII
[[VI
[I
[III
V
VII
]II
]]IV
[[III
[V
[VII
II
]IV
]VI
]]I
[[VII+
[II+
IV+
VI+
]I+
]III+
]]V+
[IV+
[VI+
I+
III+
]V+
]VII+
]]II+
[I+
[III+
V+
VII+
]II+
]]IV+
]]VI+
Bibliography Atomic Categories MIDI Parsing Tonal Space Parsing Time Index
Extra slide
Parsing Time
Average parse time per chord sequence:
Model Mean (std. dev.)
HmmPath 0:03 (0:01)PCCG 34:17 (75:23)St+PCCG 9:22 (33:32)