Computer-Aided Autocompletion of Cadential Harmony Gabriel Lesnick May 11, 2016 Abstract The most popular Computer-Aided Algorithmic Composition systems are primarily algorith- mic, having little human input to aid. We investigate the merits of more human-involved CAAC systems through analysis of our implementation of an autocomplete-type Hidden Markov Model that determines algorithmically the cadential harmony of a human-composed chord progression. We find that this system demonstrates potential to help humans compose music in creatively and stylistically consistently. 1 Introduction Musicians have been quick to attempt to inte- grate computer processing techniques and elec- trical sound manipulations into composition and performance ever since the advent and subse- quent development of computer and electrical engineering in the last century. Composers es- pecially have taken a keen interest in technol- ogy as a vehicle for the Avant-Garde [6, 8, 9], although there is a strong following in musical analysis as well [7]. Composing with computers usually involves setting the composer’s idea of music-making in the form of an algorithm in or- der to have the computer produce music as the composer would. Such purely algorithmic sys- tems, whether based in automata or neural nets, whether deterministic or stochastic, are invari- ably limited, however, in that their composing is unsupervised. In other words, the computer is given a series of inputs and then outputs a com- pleted work, but without any interaction with the human composer in between. Without hu- man input, it is unclear to what degree compo- sition algorithms, given the current progress in artificial intelligence, can actually compose [5]. However, whether it is the human programmer or the algorithm itself creating the music, the important point is that a purely contained algo- rithmic system that takes some input and then produces a musical output is strictly less power- ful than a system that interacts with a human before outputting, simply because the external interaction gives the system a broader creative state-space in which to compose. It is of inter- est, then, to experiment with exactly how to set up a composition system with human interaction so as to make use of this expanded state-space and produce maximally creative music. This idea of artist as union of human and computer can be reached equally well from the notion of musical complexity. With trends to- ward increasing complexity [11] and chaos [10] in contemporary music, there is more and more 1
Transcript
Computer-Aided Autocompletion of Cadential Harmony
Gabriel Lesnick
May 11, 2016
Abstract
The most popular Computer-Aided Algorithmic Composition systems are primarily algorith-mic, having little human input to aid. We investigate the merits of more human-involved CAACsystems through analysis of our implementation of an autocomplete-type Hidden Markov Modelthat determines algorithmically the cadential harmony of a human-composed chord progression.We find that this system demonstrates potential to help humans compose music in creatively andstylistically consistently.
1 Introduction
Musicians have been quick to attempt to inte-grate computer processing techniques and elec-trical sound manipulations into composition andperformance ever since the advent and subse-quent development of computer and electricalengineering in the last century. Composers es-pecially have taken a keen interest in technol-ogy as a vehicle for the Avant-Garde [6, 8, 9],although there is a strong following in musicalanalysis as well [7]. Composing with computersusually involves setting the composer’s idea ofmusic-making in the form of an algorithm in or-der to have the computer produce music as thecomposer would. Such purely algorithmic sys-tems, whether based in automata or neural nets,whether deterministic or stochastic, are invari-ably limited, however, in that their composing isunsupervised. In other words, the computer isgiven a series of inputs and then outputs a com-pleted work, but without any interaction with
the human composer in between. Without hu-man input, it is unclear to what degree compo-sition algorithms, given the current progress inartificial intelligence, can actually compose [5].However, whether it is the human programmeror the algorithm itself creating the music, theimportant point is that a purely contained algo-rithmic system that takes some input and thenproduces a musical output is strictly less power-ful than a system that interacts with a humanbefore outputting, simply because the externalinteraction gives the system a broader creativestate-space in which to compose. It is of inter-est, then, to experiment with exactly how to setup a composition system with human interactionso as to make use of this expanded state-spaceand produce maximally creative music.
This idea of artist as union of human andcomputer can be reached equally well from thenotion of musical complexity. With trends to-ward increasing complexity [11] and chaos [10]in contemporary music, there is more and more
1
potential for computers to assist humans withthe great tedium and increasingly unfathomablescope of immense, complicated scores. Suchcomputer assistance could allow a human com-poser to focus more on form, development, andother important large-scale structural elementsof a composition by automating lower-level ne-cessities. A computer that could simplify andexpedite the process of transcribing to the no-tated score the thoughts and ideas of the com-poser would increase the fidelity, and theoreti-cally also the creative potential, of such ideas.
Between these two distinct but related pur-suits of purely algorithmic composition and com-plex human composition lies a vast gulf of po-tential in Computer-Aided Algorithmic Compo-sition (CAAC) that remains largely underdevel-oped. Upcoming CAAC programs will allow hu-man composers to achieve high degrees of com-plexity without unreasonable effort while alsogiving algorithmic composition systems the hu-man supervision they need in order to achievecreative results [5]. Still, some extant CAACsoftware like OpenMusic [1] developed at IR-CAM already show a lot of promise, with off-shoot products like Composer’s Assistant en-abling Finale users to generate chords and fillin missing melodies [4].
Along this vein, we have implemented a CAACsystem to perform autocompletion for chord pro-gressions. We employ a Hidden Markov Model(HMM) to algorithmically finish (i.e. provide acadential chord for) human-composed chord pro-gressions. The hypothesis behind this is thatin general, a good chord progression has only afew possible cadences so that a sufficiently welltrained machine learning model should be ableto find and suggest one of these few possibilities.This system is obviously only the beginning ofwhat would be necessary to unite the composi-
tional abilities of human and machine. However,the system we have created, rudimentary andlow-level as it may be, gives evidence towards theimmense creative potential available from such aunion.
2 Implementation
Our CAAC program is written in Python (seeAppendix) using music21 [2] to process theMIDI-formatted music data and the hmmlearn[3] branch of scikit-learn to power the HMM. Theprogram consists of three overarching stages,namely those of data processing, HMM initial-ization, and finally cadence prediction.
In the data processing stage of the program,a selected corpus of music data is collated andmade python-compatible, from which the finalcadence of each piece in the corpus is extracted.In the next stage, system initialization and train-ing, the number of states for the HMM is opti-mized and the system is trained on the cadencedata. In the final stage, the trained HMM is usedto suggest a cadential chord, either by using theHMM’s score function to assess randomly gen-erated cadences or by using the transition prob-abilities of the HMM to directly emit a proba-bilistically sampled cadence.
Setting up the training data well is one ofthe most important things to do to get good re-sults from an HMM. To this end, our programinterfaces with two large datasets from which wecan extract thematic subsets to train the pro-gram. The first dataset is the corpus that comesinstalled with music21 and from which subsetscan be generated by a simple keyword search.We used this corpus, for example, to generate adata subset of 471 cadences from various Bach
2
Generatecadences
Initializesystem
SetupdataSave/loadcadencedata
Save/loadHMMmodels
Setcadenceparameters
Setdataparameters
Checkwithtestcases
Settrainingparameters
Figure 1: System block diagram.
pieces using the keyword “bwv”. The seconddataset is from the website kunstderfuge.com,which hosts a wide range of sequenced MIDIfiles for pieces ranging from Baroque to Modern.This MIDI data can be easily manipulated withmusic21 and lets us create more contemporary-leaning training data than the music21 corpus.
After collecting a large and complicated datasubset from one of these sources, the next stepof the program is to extract cadential chord pro-gressions from each piece in the dataset. For sim-plicity, and because we have access to so manypieces, we decided to extract just one cadencefrom each piece, namely the final cadence sinceit is the easiest to isolate. The pieces from whichwe extract the chords, however, are not neces-sarily very regular in that there may be passingchords, a variable number of notes per chord,and a different progression length in each piece.In order to make these progressions compatiblewith the HMM, then, we regularize them by en-suring that each progression has the same num-ber of chords and that every chord has the samenumber of notes. We use the global parame-ters CHORDS PER CADENCE and NOTES PER CHORD
to specify these numbers, and in addition we
also use the BEAT STRENGTH MINIMUM parameterto pick chords only on strong beats to avoid pass-ing chords. One final thing that must be doneto ensure consistency is to normalize each chordprogression to the same key. We use the keyof the cadential chord, calculate by music21, forthis purpose, subtracting the key and then find-ing the remainder modulo 12 to normalize eachnote to a value between 0 and 11. These meth-ods for regularization were chosen for their abil-ity to make the data compatible with the HMM,but it is certainly possible that there are bettermethods. This would be a good task for futurework.
One final consideration for the training datais that in order for the HMM to “know” that thedata it sees is cadential, the data must have somestructure in addition to the note values of thechords. The additional structure we impose is anextra field designated to mark progress towardsthe cadence. In other words, if NOTES PER CHORD
is set to 4, for example, then each data point fedto the HMM will actually consist of five values:the four note values and an extra value designat-ing whether or not this chord is a cadence. Forthe sake of control, this cadence-marking abilitycan be configured, so that the CADENCE MARKER
value determines the weight of the cadence mark.If so desire, the cadenceMark function could bereset as well to implement any other algorithmfor cadence marking.
Initialization of the system can take placeonce the training data has been processed. Weinstantiate a GaussianHMM object and then callits fit method on each of the six-chord caden-tial progressions to train it. Before we can train,however, we have to ensure the HMM is prop-erly configured with an optimal number of states.
3
The program allows the user to either specifya number of states, or if a good number is notknown, the program can be made to determinethe optimal number of states.
The method used for automatically optimizingthe number of states employs the training datato self-assess how accurately the HMM fits thisdata. Starting with a small number of states, wetrain a model HMM on half the training data.We then use this model system to score the otherhalf of the training data. We repeat this train-ing and scoring process, incrementing the num-ber of states, until the score stops increasing.We want to find the number of states that bestfits the data without overfitting - the number ofstates reached when the score stops increasing isa very good (and efficient) estimate of this opti-mal number.
Generating the output cadence becomespossible once the system is properly initialized.Given a complete cadential chord progressionwithout a cadence, the HMM can algorithmi-cally suggest the cadence. Our program actuallyoffers two methods for suggesting this cadence,each tailored to a slightly different functionality.The first method is to search the space of all ca-dences and locally maximize the chord progres-sion’s score with stochastic gradient ascent whilethe second method is to emit a cadential chordusing the transition probabilities of the HMM it-self and the highest-likelihood sequence of statesassociated with the inputted chord progression.
The gradient ascent method makes the mostsense when the cadence-marking option is en-abled. Specifically, we can force this method toprovide us with a cadential chord by constrainingthe state-space from which the potential chordsare picked to the set of chords that are marked as
cadential. In other words, since we are (stochas-tically) choosing chords to feed into the gradi-ent ascent, we can generate these chords so thatthey are definitely cadences. This is as opposedto the second method, where we have no abilityto constrain the chord that is generated. In thesecond method, however, we are more likely toget high-fidelity data since instead of just ran-domly searching the state-space of possible ca-dences for a good match, we use the preexistingstructure of the HMM itself to decide what themost likely chord is. So while the first methodis only guaranteed to find a local maximum, thismethod will theoretically find the global maxi-mum for the best-fit cadential chord, assumingthat such a global maximum exists.
3 Results
We used a number of test cases to see how ourprogram fares in comparison to real human com-posers. Since this program is concerned primar-ily with new composition, we used mostly mod-ern or contemporary test cases so as to test ca-pacity for contemporary ideas. For complete-ness, though, we also include a sample fromBeethoven.
All of these test cases are generated from anHMM trained on a corpus of Bach music. Thisdataset obviously gives the system a bias towardsconsonance and tonality, but for contemporarypieces that are quasi-tonal enough to be consid-ered to have chord progressions, this is of inter-est. In Figures 2-6, the original cadential pro-gression is followed by two generated cadences,the first generated by gradient ascent and thesecond from state emission.
Note that we did not yet get good data forthese test cases from any training sets besides
4
Figure 2: Beethoven - Symphony No. 9, cadence to Turkish March.
Figure 4: Lutos lawski - Concerto for Orchestra, mm. 271-286.
5
Figure 5: Messiaen - L’ascension, First and Third movements
Figure 6: Ligeti - Hamburg Concerto, Hymnus mm. 13-15.
Figure 7: Chord sequence from random seed.
the Bach corpus. The kunstderfuge.com data isinteresting to test the system with but is unfor-tunately not at a sufficient stage of developmentto enable meaningful results.
In any case, there is still plenty to be gleanedfrom just this Bach data, especially because
the Bach data is easy to understand - thewell-defined tonal nature of Bach’s progressionsmakes it possible to distinguish between whatthe HMM is getting from the training data andwhat it understands about the style of each testcase itself. It is clear, for example, that as the
6
test cases go from triadic harmony (Beethoven,Figure 2) to pure atonality (Ligeti, Figure 6),the generated chords remain essentially triadic.This is of course a result of training the HMM ontriadic data - it only produces chords of a kindthat it has seen before. This is a good sign! Itmeans the HMM is actually picking up relevantinformation from the training data. It wouldbe good to see how this would work with a fullcompendium of contemporary data to make surethis training works universally - it is possible, al-though seemingly unlikely, that the current suc-cess of the training with Bach data is due to thespecific constraints set up in the parsing of thedata and that other data would not be as suc-cessful. Further testing would be necessary tocompletely understand this.
Continuation beyond the cadence is anobvious extension for a chord progression au-tocompleter. Even though the program is de-signed, in its current form, to produce only sin-gle chords, it is still intriguing to see what resultsfrom such a misuse. To do this, we began witha random seed chord produced by sampling thetrained HMM (trained still with Bach) and thenadded “cadential” chords one at a time using thegradient ascent method. One such realization ofthis process is seen in Figure 7. Although thisresult is interesting, many other runs quickly sta-bilized around a single chord, usually a triad.
4 Conclusion
It is evident from the test cases that our programis able to emulate a lot of the ideas that humancomposers have about chord progressions. El-ements like voice leading, triadic harmony, andthe dominant-tonic cadential relationship clearly
come through from these examples. However,it is hard to tell how much the system actually“understands.” A lot of these factors are sim-ple to approximate but very difficult to emulateexactly. Voice leading, for example, is easy toapproximate heuristically with the simple rulethat each voice should move by at most two semi-tones between each chord. However, leaps largerthan a whole tone are obviously important instandard voice leading and this rule would leavethem out. So it is necessary to assess just howmuch the HMM is able to get from the train-ing data and on the contrary how much of thesuccess of the program is due to the rule-basedmusical constraints we have added to it.
Ability to match human creativity is oneway to assess the HMM, and comparing the gen-erated cadences in the test cases to the originalcadences is a great way to do this. A brief lookat the test cases shows that at least voice leadingis well emulated, and it is apparent that also thestyle of each test case is at least partially pre-served in the generated cadences. One obviousthing that is missing in the generated cadences,however, is that they are never exactly the sameas the original cadence, meaning that the HMMis not as powerful as the humans it is trying toemulate.
The success of the voice leading, although im-pressive, is also a bit deceptive. In order to makethe HMM indifferent to key, the input chord pro-gression is normalized to be rooted on C0. Thenotes of the generated cadence are then octave-transposed until they are as close as possible tothose of the penultimate chord. So the octavepositions of each chord are not determined by theHMM. Still, the fact that the generated chordsare so often within a semitone or two of the pre-
7
ceding chord is encouraging, since this variationwithin the octave is indeed decided by the HMM.
Even more encouraging is the fact that thegenerated chords seem to have approximatelythe same style as the original cadences, indeedthe same style as the rest of the chords in the testcases as well. In the Beethoven test case (Figure2), for example, all the chords are triadic, so thegenerated cadences are triadic as well. Similarlyin the Schoenberg test case (Figure 3), the chordsare not as consonant but are still mostly triadic,so the generated cadences are again triadic. TheLigeti (Figure 6), being manifestly non-triadic, iscompleted with a much more condensed seventhchord, which despite being made from stackedthirds, is more dissonant than the pure harmoniccadences of the Beethoven or Schoenberg. Thefact that all the generated cadences consist ofstacked thirds might be disconcerting until youremember that we used Bach as training data.So with more contemporary (less triadic) train-ing data, we would perhaps see a greater degreeof variation in consonance or dissonance of thegenerated cadences depending on how consonantor dissonant the test case is.
Now the lack of exact replication of the orig-inal cadences by the HMM is troubling, sincethe HMM probabilistically considers most pos-sible cadences and therefore rejects the originalcadence as inferior to whatever cadence it comesup with. So further fine-tuning would be nec-essary before this program could more preciselyemulate the original human composers of thesetest cases. However, we are looking for evidenceof creativity in this system, not ability to copyhumans exactly, so this is beside the point. A hu-man composer given these test cases and askedto complete them with their own cadences wouldsimilarly have trouble recreating the exact origi-nals but might still be able to come up with some
creative solutions of their own.
Potential for new creativity is anothergood way to assess the HMM. Ignoring the orig-inal cadences in the test cases, what can be saidabout the ability of the HMM to be creative?
The most readily apparent shortcoming of thisHMM is its inability to recognize global patternslike tonal center or position within the progres-sion. In the Lutos lawski test case (Figure 4),the generated chords are of the correct (triadic)sonority, but are in the wrong key. Now in someways this is irrelevant; the HMM simply has apredilection for progressive tonality. But with-out an ability to generate chords based not juston immediate neighbors but also greater struc-ture like tonal center or form, this HMM will bedoomed.
In a smaller scale, though, without regardto larger patterns like this, the HMM performsspectacularly. In the Messiaen test case (Figure5), the HMM generates novel, creative cadencesthat sound good despite being in the “wrong”key. Unless the listener already knew the pieceor knew Messiaen’s style very well, they wouldmost likely hear the generated cadences as cor-rect.
The HMM is clearly able to come up with in-teresting chords with good notions of voice lead-ing and consonance and dissonance. Of course,there are still problems to be worked out. TheHMM is not always this good; it often takes afew runs before a good chord is generated. Bet-ter training data would have to be collected totest its capability for more contemporary styles.The constraints and parameters of the programwould have to be modified to fully optimize it.So it may not be perfect, but it definitely passesthe mark for creativity.
8
5 Future Work
Although we have made significant progress inchord progression autocompletion, this is just asmall feature with respect to the full potentialof realizable CAAC systems. Even just withinthe subdomain of chord autocompletion, thereis a lot more room for innovation. We added alot of constraints, for example, to make the pro-gram easier to build, like all chords having thesame number of notes, all training progressionshaving the same number of chords, and allowingonly one cadential chord to complete each pro-gression. It would be worth spending some timein the future unconstraining this program, or re-searching to see if there are perhaps alternativeconstraints that would be more conducive to cre-ating cadences and would provide better results.It is doubtless that, given the inclination, a vastnumber of potential avenues for improvement ofthis program could be pursued; we will presentjust a few of these now.
Creating a good GUI for this programwould make it utilizable for its intended pur-pose: helping humans compose. It is still use-ful as proof of concept in its current state, butit would only be useful in during compositionif it were fleshed out with an interface. Ideally,the composer would use notation software likeSibelius or Finale to write music and would beable to invoke this progression auto-completionfeature as a plugin with one click of a button.The plugin would suggest a cadential chord tothe user and if it were to the user’s liking it wouldthen be added to the score. This functionalityas described is a long way away. But it is goodthat we now have a proof of concept for this kindof plugin - we have taken the first step.
Training with more accurate data wouldhelp to make the HMM more accurate. However,this is much easier said than done. The act of ac-curately isolating and homogenizing data from avariety of different musical styles and eras is closeto intractable. Even just within the Bach data,for example, it is exceedingly difficult to algo-rithmically decide which chords are part of thefunctional harmony, which notes are not passingtones, and where a chord progression starts andends. It almost seems appropriate to create anentire HMM to process the music data.
Having a meta-system like this to process datato train the real system is impractical, but theend result of having good training data is neces-sary to get the full system working with high fi-delity. Without high quality training data, thereis no chance that the HMM will be able to pickup on the full complexity and nuance of musiccomposition. The question now is how to gethigh quality data.
Generating more than one chord at a timewould allow for more novel capabilities. Chordprogression autocompletion would itself be ben-efitted by the possibility of multi-chord cadencessince then chord progressions could be success-fully completed in ways potentially more cleverand creative than just appending a single chord.This idea could be extended further, however, tosuch novelties as modality bridging, that is, au-tomatically generating a chord or set of chordsto pivot between modalities, or even melody har-monization. Such higher level modes of chordgeneration would necessitate better and morefluid ways of representing chord data and wouldrequire reformulation, or removal, of some of theother constraints set in the current version of theprogram beyond just the number of chords gen-
9
erated at a time. For example, pivoting froma modality expressed in three-part harmony toone expressed in four-part harmony would onlybe possible if the constraint that all chords havethe same number of notes were lifted.
Other machine learning models might bebetter suited to chord autocompletion than Hid-den Markov Models. HMMs make intuitive sensesince music often feels as if it proceeds fromstate to state, with recurring musical elementsconnected by transitional material in often well-defined ways. But it is entirely possible thatsuch a heuristic idea misses some other sort ofmachine learning model that might behave bet-ter. Just because music might seem Markov-likedoes not mean that the best way to create musicis with an HMM - at least, most human com-posers would balk at the idea of writing musicin such a way. This is not to say that HMMsare badly suited for the task at hand of chordgeneration, but only that it would definitely beworth assessing the capabilities of other modelsas well.
A few small bugs still remain that should befixed. In particular, the algorithm that trains theHMM conflicts with the cadence marking whenthis functionality is activated, often producingan error from degeneracy. We are currently get-ting around this by keeping the CADENCE MARKER
value small (a value of around 0.3 works) so asto minimize the degeneracy in the data and letthe HMM training proceed. It would be nice,however, to be rid of this degeneracy altogether,but unfortunately this is work for the future.
References
[1] http://repmus.ircam.fr/openmusic/home.
[2] http://mit.edu/music21/.
[3] http://github.com/hmmlearn/hmmlearn.
[4] Products of interest. Computer Music Jour-nal, 26(2):113–125, 2002.
[5] C. Ariza. The interrogator as critic: Theturing test and the evaluation of generativemusic systems. Computer Music Journal,33(2):48–70, 2009.
[6] A. R. Burton. Generation of musical se-quences with genetic techniques. ComputerMusic Journal, 23(4):59–73, 1999.
[7] M. S. Cuthbert and C. Ariza. music21: Atoolkit for computer-aided musicology andsymbolic music data. In International So-ciety for Music Information Retrieval, 11thConference, 2010.
[8] S. R. Holtzman. Using generative gram-mars for music composition. Computer Mu-sic Journal, 5(1):51–64, 1981.
[9] K. McAlpine, E. Miranda, and S. Hoggar.Making music with algorithms: A case-study system. Computer Music Journal,23(2):19–30, 1999.
[10] R. Steinitz. Music, maths and chaos. TheMusical Times, 137(1837):14–20, 1996.
[11] B. Truax. The inner and outer complex-ity of music. Perspectives of New Music,32(1):176–193, 1994.
10
6 Appendix: Python Code
1 import numpy as np
2 from hmmlearn import hmm
3 import pickle
4 from music21 import *
56 # constants used in isolating chord progressions from corpus data
7 NOTES_PER_CHORD = 4
8 CHORDS_PER_CADENCE = 6
9 BEAT_STRENGTH_MINIMUM = 0 # set to nonzero to select for strong beats
10 CADENCE_MARKER = .3 # set to nonzero to use cadence marker
11 NUMBER_STATES = 13 # set to zero to use state optimization algorithm
