Post on 27-Oct-2021
transcript
DOI: 10.4018/IJACDT.2019070102
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
Copyright©2019,IGIGlobal.CopyingordistributinginprintorelectronicformswithoutwrittenpermissionofIGIGlobalisprohibited.
20
MixMash:An Assistive Tool for Music Mashup Creation from Large Music CollectionsCatarina Maçãs, CISUC - Department of Informatics Engineering University of Coimbra, Coimbra, Portugal
Ana Rodrigues, CISUC - Department of Informatics Engineering University of Coimbra, Coimbra, Portugal
Gilberto Bernardes, INESC TEC & University of Porto, Faculty of Engineering, Porto, Portugal
https://orcid.org/0000-0003-3884-2687
Penousal Machado, CISUC - Department of Informatics Engineering, University of Coimbra, Coimbra, Portugal
ABSTRACT
ThisarticlepresentsMixMash,aninteractivetoolwhichstreamlinestheprocessofmusicmashupcreationbyassistingusersintheprocessoffindingcompatiblemusicfromalargecollectionofaudiotracks.ItextendstheharmonicmixingmethodbyBernardes,DaviesandGuedeswithnoveldegreesofharmonic,rhythmic,spectral,andtimbralsimilaritymetrics.Furthermore,itrevisesandimprovessomeinterfacedesignlimitationsidentifiedintheformermodelsoftwareimplementation.Anewuserinterfacedesignbasedoncross-modalassociationsbetweenmusicalcontentanalysisandinformationvisualisationispresented.Inthisgraphicmodel,alltracksarerepresentedasnodeswheredistancesandedgeconnectionsdisplay theirharmonic compatibility as a result of a force-directedgraph.Besides,avisuallanguageisdefinedtoenhancethetool’susabilityandfostercreativeendeavourinthesearchofmeaningfulmusicmashups.
KeywoRDSCross-Modal Associations, Dissonance, Force-Directed Layout, Harmonic Mixing, Information Visualisation, Music Mashup, Perceptual Relatedness, Rhythmic Density, Spectral Region, Timbral Similarity
INTRoDUCTIoN
Mashupcreationisamusiccompositionpracticestronglylinkedtovarioussub-genresofElectronicDanceMusic(EDM)andtheroleoftheDJ(Shiga,2007).Itentailstherecombinationofexistingpre-recordedmusicalaudioasameansofcreativeendeavour(Navas,2014).Thispracticehasbeennurturedbytheexistingandgrowingmediapreservationmechanismsthatallowuserstoaccesslargecollectionsofmusicalaudioindigitalformatfortheirmixes(Vesna,2007).However,thescalabilityof thesegrowingaudiocollectionsalsoraises the issueof retrievingmusicalaudio thatmatchesparticularcriteria(Schedl,Gómez,&Urbano,2014).Inthiscontext,bothindustryandacademiahavebeendevotingefforttodeveloptoolsforcomputationalmashupcreation,whichstreamlinethetime-consumingandcomplexsearchforcompatiblemusicalaudio.
Early research on computational mashup creation, focused on rhythmic-only attributes,particularlythoserelevanttothetemporalalignmentoftwoormoremusicalaudiotracks(Griffin,Kim&Turnbull,2010).Recentresearch(Davies,Hamel,Yoshii,&Goto,2014;Gebhardt,Davies,
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
21
&Seebe,2016;Bernardes,Davies,&Guedes,2018)hasexpandedtherangeofmusicalattributesunderconsiderationtowardsharmonic-andspectral-drivenattributes.Theformeraimstoidentifythedegreeofharmoniccompatibilityinmusicalaudio,commonlyreferredtoasharmonicmixing.Thelatteraimstoidentifythespectralregionoccupiedbyaparticularmusicalaudiotrackacrossthefrequencyrange(e.g.,theconcentrationofenergyinlow,middle,andhighfrequencybands),whichcanthenguidethespectraldistributionofthemix.
Theinterfacedesignofearlysoftwareimplementationmodelsadoptsaone-to-manymappingstrategybetweenauser-definedtrackandarankedlistofcompatibletrackstoshowtheresultstotheuser(MixedinKey,n.d.;NativeInstruments,n.d.;Daviesetal.,2014).Recently,Bernardesetal.(2018)proposedaninterfacedesignwhichadoptsamany-to-manymappingstrategy,whichoffersaglobalviewofthecompatibilitybetweenalltracksinamusiccollectionandpromotesserendipitousnavigation(Figure1).Itrepresentseachaudiotrackinacollectionasagraphicalelementinanavigable2-dimensionalinterface.Distancesamongtheseelementsindicateharmoniccompatibilityandtheadditionalgraphicvariablesoftheseelements,suchascolourandshape,indicaterhythmicandspectralinformationrelevanttomashupcreation.Byexposinguserstothecompatibilitybetweenalltracksinacollection,thisinterfacedesignaimstopromoteanoverviewoftherelationsbetweentracks.
Inshort,advancesincomputationalmashupcreationmodels,emphasizeagradualincreaseinthenumberofextracteddata-drivenattributesfrommusicalaudioandaglobalviewoftheaudiocollectionsthroughinformationvisualization.Thistendencyacknowledgesthesubjectivenatureofthetaskandenhancesthedegreeofpersonalizationinthesearchforcompatibleaudioinmashupcreation.However,scalabilityoftheseaudiocollectionsnowraisesconcernsattheusabilitylevelinmany-to-manyinterfacedesign(Bernardes,Davies,&Guedes,2018;Maçãsetal.,2018).Figure1b)highlightsthethreemainlimitations:(i)theclutterresultingfromthesuperpositionofgraphicelements;(ii)thestaticrepresentationofthetracks,whichdoesnotpromoteafinerexplorationofparticulardenseareas;and(iii)thereducednumberofexistinggraphicattributesinthevisualtracksrepresentation todepictmusical audiocontent-driven information.From these limitations itwaspossibletodefinethethreemaingoalsforthepresentwork:(i)thepreventionofoverlappinggraphicelementsandsubsequentsimplificationof thevisualisation;(ii) thecreationofasystemcapableofdynamicallyadapt to theuser interactions;and(iii) thecompleterepresentationof the tracks’musicalcharacteristicsandtheirharmoniccompatibility.Toaddresstheidentifiedinterfacedesignlimitations,itwasadoptedinMaçãsetal.(2018)amethodologybasedontheThreeCycleViewofDesignScienceResearch(Hevner,2007).Thismethodologypromotedtheiterativeimplementation
Figure 1. Screenshot of the visualisation of the original MixMash interface representing (a) 50 musical audio tracks and (b) 200 musical audio tracks. Refer to Bernardes et al. (2018) for a detailed interpretation.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
22
ofthevisualisationmodelandthedefinitionofthevisualrepresentationsoftheattributesofmusicalaudiotracks.
The current paper extends Maçãs et al. (2018) in four aspects. First, it provides a detailedoverviewofcomputationalmashupcreation,fromtheearlyrhythmic-onlyalignmentstothecurrentmultidimensional attribute spaces, which to the best of our knowledge has not been addressedelsewhere.Second, itexpands therangeofmusicalaudioattributesunderconsideration,notablyincluding timbreasa relevantdimension, following recentevidenceon its significant impactonlisteningpreferenceaslistenersareabletoreliablyevokechangesintimbretowardstheirpreferences(Dobrowohl,Milne,&Dean,2019).Third, itdetailsthesignalprocessingunderlyingallmetricsadoptedtoextractcontent-driveninformationfrommusicalaudio.Fourth,theforce-directedalgorithm,visualmappingsandinteractiveinterfacearethoroughlydescribed.
Theremainderofthepaperisorganisedasfollows.TheBackgroundsummarisestherelatedwork.MixMash:CompatibilityMethoddescribestheaudioanalysismethodsthatsupportanovelmusic visualisation system for assisting music mashup creation. The Methodology presents thedevelopmentstrategiesforthevisualisationmodel.MixMash:VisualisationModelpresentsanewapproachforthevisualisationofcompatiblemusicalaudiotracks,adescriptionoftheestablishedassociationsbetweengraphicelementsandmusicalaudioattributes,andtheinterfacedesign.TheConclusionstatestheconclusionsofthisworkanditsfuturedirections.
BACKGRoUND
Thepresentsectionisdividedintothreeparts,inwhichtheauthorspresentrelatedworkconcerningthefollowingtopics:harmonicmixingmethods,thevisualisationmethodsappliedtorepresentmusic,andthecharacterisationofforce-directedgraphsandtheirapplicationinthevisualisationofmusiccollections.
Harmonic MixingTherearefourmajorharmonicmixingmethodsintheliterature:keyaffinity,chromavectorssimilarity,sensorydissonanceminimisation, andhybrid (hierarchical)models.The initial three approachesfocusonsingleharmonicattributesonly,andthelatterapproachprovidesahierarchicalviewoverharmoniccompatibility.
Keyaffinityisoneoftheearliestcomputationalapproachestoharmonicmixing.Ithasbeenproposedbyindustry(MixedinKey,n.d.,NativeInstruments,n.d.)andiscomputedasdistancesintheCamelotWheelorcircleoffifthsrepresentation(MixedinKey,n.d.).Thisapproachenforcessomedegreeoftonalstabilityandlarge-scaleharmoniccoherenceofthemashupbyprivilegingtheuseofthesamediatonickeypitchset.
Chromavectorssimilarityinspectsthecosinedistancebetweenchromavectorrepresentationsofpitch-shiftedversionsoftwogivenaudiotracksasameasureoftheircompatibility(Daviesetal.,2014;Lee,Lin,Yao,Li,&Wu,2015).Thesimilarityistypicallycomputedatthetimescaleofbeatdurations,thusprivilegingsmall-scalealignmentsoverlarge-scaleharmonicstructurebetweenaudiosliceswithhighlysimilarpitchclasscontent.
Sensorydissonancemodelsfollowthesamelocalandsmall-scalealignmentstrategyaschromavectorssimilaritybetweenpitch-shiftedversionsofoverlappingmusicalaudio,yetadoptamorerefinedmetricofcompatibility,whichaimstominimizetheircombinedlevelofsensorydissonance(Gebhardt,Davies&Seebe,2016);amotivationwell-rootedintheWesternmusical traditionbyfavouringalessdissonantharmoniclexicon.
Recently,ahybridhierarchicalmodelforharmonicmixinghasbeenproposedbyBernardesetal.(2018).Itcombinespreviousapproachesfor(small-andlarge-scale)harmoniccompatibilityinasingleframework.Furthermore,itproposesanovelinterfacedesignapproach,whichoffersaglobal
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
23
viewover theharmoniccompatibilityofanentiremusiccollection (many-to-many),beyond theexistingone-to-manyrelationshipsbetweenauser-definedtrackandanaudiocollection(Bernardesetal.,2018).
Music VisualisationHistorically,numerousartistshavecreatedaudiovisualassociations,laterreferredtoasgraphicnotationandvisualmusic.PioneeringworksbyKandinsky,Pfenninger,Cage,Fischinger,andWhitneyexplorecombinationsofvisualprinciples—mainlycolourandshape—toemphasisetheaudiovisualexperience(McDonnell,2007).Inthepresentwork,thevisualtranslationofacollectionofmusicalaudiotracksisapproachedfromtwodifferentpointsofview:musicvisualisationandthevisualrepresentationofalargecollectionofmusicalaudiotracks.Regardingmusicvisualisation,someauthorshavetriedtosolvethisproblembyfocusingonthegeometryofmusicalstructure(Bergstrom,Karahalios,&Hart,2007),whileothershavefocusedonasolutionbasedonmappingsbetweenaspecificsetofmusicalcharacteristicsandsomevisualcharacteristics(Snydal&Hearst,2005;Wattenberg,2002;Sapp,2001).Forexample,Wattenberg(2002)usesarcdiagramstoconnectsequencescontainingthesamepitchcontent,revealingthestructureofmusicalcompositions.Ontheotherhand,Sapp(2001)presentedamulti-timescalevisualisationoftheharmonicstructureandthekeyrelationsinamusicalcomposition.AnotherexperimentalapproachhasbeenpresentedbyRodrigues,Cardoso,andMachado(2016),whereavisualisationmodeliscreatedtoprovideaperceptuallyrelevantexperiencefortheuser.Overall,theaforementionedvisualisationsdonotallowgreatmodularityofdata,oftenbindingthevisualclarityofeachelementandlimitingthecomparisonofothermusicalcharacteristics.
Although visualisations of large collections of musical audio collections have already beenaddressedbysomeauthors(Grill&Flexer,2012;Hamasaki&Goto,2013;Rauber,Pampalk,Merkl,2003;SchwarzandN.Schnell,2009;Gulik,Vignoli,&VandeWetering,2004),itstillisanareainneedofgreatdevelopment.GrillandFlexer(2012),similarlytotheworkofBernardesetal.(2018),developedavisualisationstrategycapableofrepresentingperceptualqualitiesfromalargecollectionofsounds.Althoughtheirmusicalfocuswasinsoundtextures,theyaimedatfindinganintuitiveandmeaningfulinterface.Forthispurpose,theybuiltanaudiovisuallanguagebasedonthecross-modalmechanismsofhumanperception.Inthisproject,subjectswereabletosuccessfullyassociatesoundswiththecorrespondinggraphicrepresentations.HamasakiandGoto(2013),Guliketal.(2004)andSchwartzandSchnell(2009)alsoproposedinteractivevisualisationsofseverallayersofinformation;however,theydonotrepresenttheperceptualrelevanceofthemusicalcharacteristicsatavisuallevel.
Force-Directed GraphsThe visualisation of graphs handles the representation of relational structures in data, aiding intheanalysis,modelling,andinterpretationofcomplexnetworksystems(Meirelles,2013).Graphvisualisation is characterised by the existence of two main elements: (i) nodes, representing anentity(e.g.,person,cell,machine);and(ii)edges,representingtherelationshipbetweennodes.Suchmodelsareoftenappliedtolargeandcomplexdatasets,whichentailsasetofproblemsrelatedtoperformanceandclutter.Asthegraphsgrowinsize,therequiredvisualspaceandcomputationalresourcesalsoincrease(Herman,Melançon,&Marshall,2000).Tosolvethis,clusteringtechniquesareappliedbygroupingsimilarnodesby their semanticsand/orpositionon thegraph.Throughclustering,itispossibletoreducevisualclutterandcomplexity,enhanceclarityandperformance,andcreateasimplifiedoverviewofthenetwork’sstructure(Hermanetal.,2000;Kimelman,Leban,Roth,&Zernik,1994).
Thepositioningofnodesinspacecanbedefinedthroughadifferentsetofgraphlayouts,suchasarcdiagrams,treemaps,circular,andforce-directedlayouts.Inthispaper,thelatterisadopted.Force-directedgraphsarebasedonaphysicalsystemthatorganisesthenetworkthroughforcesofrepulsion and attraction applied continuously to each node. This technique facilitates the visual
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
24
interpretationoftheinherentstructureofthedata,improvingtheanalysisandcomprehensionoftherelationswithincomplexnetworks(Jacomy,Venturini,Heymann,&Bastian,2014).
Force-directed graphs are used in a variety of areas, such as biology, medicine, literature,sociology(EnrightandC.A.Ouzounis,2001;Gohetal.,2007,Chen,2006;HeerandBoyd,2005;Gilbert,Simonetto,Zaidi,Jourdan,&Bourqui,2011;Fuetal.,2007),butalsointhevisualisationofmusiccollections(Vavrille,n.d.;Gibney,n.d.).IntheworkofMuelder,Provan,andMa(2010),aforce-directedlayoutwasappliedtovisualisemusiclibraries.Eachpieceofmusicrepresentsanode,positioneddependingonitssimilaritieswithothermusicalaudiotracks(e.g.,sameartist).Songrium(Hamasaki&Goto,2013),isamusicbrowsingservicethatvisualisesrelationsamongoriginalsongsandtheirderivativeworks.Eachnoderepresentsavideo-clipandeachedgeconnectsanoriginalworktoitsderivatives.Inaddition,songswithsimilarmoodsgetstrongerforces,thusarepositionedclosertogether.
Thecurrentworkexpandsthestate-of-the-artbyapplyingaforce-directedgraphtothemashupcreationprocess.Toenabletheusertoanalysethegraphandrelatemusicalaudiotracksfromlargecollections,theharmoniccompatibilitymetricsareappliedtotheforcesofthegraphnodes.Thisimprovesthevisualseparationofdistinctmusicalaudiotracks,whichcanhaveapositiveimpactonusercreativity(Henry,Fekete,&McGuffin,2007).
MIXMASH: CoMPATIBILITy MeTHoD
MixMashisasoftwareapplicationwhichaimstoassistusers infindingmusicalmashupsin thecontextofmashupcreation.ItbuildsonmetricsandmethodspresentedinBernardesetal.(2018)and expands the state-of-the-art of harmonic mixing by providing a greater amount of relevantinformationtotheprocessofmusicmashup.Itsmainnoveltyliesinahierarchicalharmonicmixingmethod,whichincludesmetricsforbothsmall-andlarge-scalestructurallevels,i.e.,local(e.g.,beatsorphrases)andglobal(e.g.,largesectionsoroverallmusicalmashup)harmonicalignmentsbetweenmusicalaudiotracks,respectively.Moreover,thismethodconsidersthreeadditionaldimensionsthatcanhelpusersdefiningthecompatibilityofmusicalaudiotracksandremainingcompositionalgoalsintermsofrhythmic(onsetdensity),spectral(region)andtimbralqualities.
Topromotean intuitivesearch forcompatible tracks inamusiccollection,amany-to-manymappingstrategywaspreviouslyintroducedintheinterfacedesign.Thisdesignopposestherankedtrack list toauser-definedtrack,adopted inprevioussystemsharmonicmixingsoftware(MixedinKey,n.d.;Native Instruments,n.d.),which is (i) inefficientcomputationally,as it recomputesintensiveaudiosignalanalysiseverytimeadifferentaudiotrackisselectedastarget,and(ii)limitedinpromotingcreativeendeavorandserendipity(Bernardesetal.,2018).Aflexiblemany-to-manymappingstrategywasallowedbytheadoptionofnovelsignalprocessingmethodsforsmall-andlarge-scaleharmoniccompatibilitymetricsinaconfinedspatialconfiguration.ThismethodisatthebasisoftheMixMashvisually-driveninterfacestrategy.Thesignalprocessingmethodsusedtocomputetheseharmoniccompatibilitymetrics,followedbytheadditionalrhythmic,spectral,andtimbralattributesaredescribednext.
Small- and Large-Scale Harmonic CompatibilityMixMashadoptstheperceptually-motivatedTonalIntervalSpace(Bernardes,Cocharro,Caetano,Guedes,&Davies,2016)forrepresentingtheharmoniccontentofmusicalaudiotracks.Eachtrackexistsasa12-dimensional(12-D)TonalIntervalVector(TIV),T k( ) ,whoselocationsrepresentuniqueharmonicconfigurations.AnaudiotrackTIV,T k( ) ,iscomputedastheweightedDiscreteFourierTransform(DFT)ofanL
1normalizedchromavector,c n( ) ,suchthat:
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
25
T k w k c n e kn
N j kn
N( ) = ( ) ( ) ∈=
− −
∑0
1 2π
, (1)
whereN = 12 isthedimensionofthechromavector,eachofwhichexpressestheenergyofthe12pitchclasses,andw k( ) = { }3 8 11 5 15 14 5 7 5, , . , , . , . areweightsderivedfromempiricalratingsofdyadsconsonanceusedtoadjustthecontributionofeachdimension k (Bernardesetal.,2018).k issetto1 6≤ ≤k forT k( ) sincetheremainingcoefficientsaresymmetric.T k( ) usesc n( )
whichisc n( ) normalizedbytheDCcomponentT c nn
N
00
1
( ) = ( )=
−
∑ toallowtherepresentationand
comparisonofmusicatdifferenthierarchicallevelsoftonalpitch(Bernardesetal.,2016).Torepresentvariable-lengthaudiotracks,thechromavectors,c n( ) ,resultingfrom16384samplewindowsanalysisata44.1kHzsamplingrate(≈372ms)with50%overlapareaccumulatedacrossthetrackduration.
FromtheaudiotracksTIVs,twometricsthatcapturetheharmoniccompatibilitybetweenTIVstobemixedarecomputed.Ofnoteisthesplitbetweensmall-andlarge-scaleharmoniccompatibility,whichroughlycorrespondtothesoundobjectandmesoormacrotimescalesofmusic,respectively.Inotherwords,thesmallscaledenotesthebasicunitsofmusicalstructure,fromnotestobeats,andthelargescaleinspectsthestructurallevelsbetweenthephraseandtheoverallmusicalpiecearchitecture(Roads,2001).Inthecontextofthecurrentwork,thefirstaimsmostlyatfindinggoodharmonicmatchesbetweenthetracksinacollection,andthesecondinguaranteeingcontrolovertheoverallharmonicstructureofamix,i.e.,thetonalchangesatthekeylevelacrossitstemporaldimension.
Equation2computesthesmall-scaleharmoniccompatibility, Si p,
,betweentwogivenaudiotracks,i andp ,representedbytheirTIVs,T k
i ( ) andT kp ( ) ,asthecombinationoftwoindicators:
perceptual relatedness, Ri p,
, (Equation 3) and dissonance, Di p,
, (Equation 4). The smaller theperceptualrelatedness,R
i p,,thegreatertheaffinitybetweentwogiventracks,asshownbyBernardes
etal.(2016).Thesmallerthedegreeofdissonance,Di p,
,thegreatertheircompatibility,asshownbytheempiricaldatainGebhardtetal.(2016).
S R Di p i p i p, , ,= ⋅ (2)
where:
R T k T ki p
ki p,
= ( )− ( )=∑1
6 2 (3)
DaT k a T k
a a w ki p
i i p p
i p
,= −
( )+ ( )+ ( )
1 (4)
aianda
paretheamplitudesofT k
i ( ) andT kp ( ) ,respectively.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
26
Large-scaleharmoniccompatibility, Li p,
, isaderivationof theperceptual relatedness, Ri p,
,indicator,asitexpressestherelatednessofagiventrackTIVfromthem = 24 majorandminorkeyTIVs,andcanbeinterpretedasthedegreeofassociationofagiventracktoamusicalkey(Bernardes,Davies, & Guedes, 2017). As such, the large-scale harmonic compatibility can be computed byinterpretingT k
i ( ) andT kp ( ) inEquation3asatrackTIVandakeyTIV,respectively.Them = 24�
majorandminorkeysTIVarecomputedbyadoptingthe12shiftsoftheCmajorandCminorkeyschromavectors,c n( ) ,showninFigure2,inEquation1.
Rhythmic, Spectral and Timbral AttributesThreeadditionaldescriptionsofrhythmic,spectral,andtimbralaudiotrackattributesarecomputed.Theyaresubsidiaryoftheprimarysmall-andlarge-scaleharmoniccompatibilitymetricsandaimtorefinethesearchamongcompatibleaudiotracks.Next,adescriptionofthemetricandtheirmusicalinterpretationinthecontextofmusicmashupcreation,mostnotablyinMixMash,isprovided.ReaderscanrefertoBrent(2010)foracomprehensivedescriptionoftheircomputation.
Eachaudiotrack’srhythmiccontentisdescribedbyitsnoteonsetdensity,Oi,ofamusicaltrack
i ,andiscomputedbyathreefoldstage.First,aspectralfluxfunctionforeachtrackiscomputed,
Figure 2. Sha’ath’s (2011) key profiles for the C major and C minor keys
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
27
usingthetimbreIDlibrary(Brent,2010)withinPureData.Thisfunctiondescribestheamountofnoveltyfromawindowedpowerspectrumrepresentationoftheaudiosignal(windowsize≈46mswith 50% overlap). Second, the peaks from the function above a user-defined threshold, t , areextractedandinterpretedasnoteonsetlocations.Priortothepeakdetectionstage,abi-directionallow-passIIRfilter,withacut-offfrequencyof5Hz,wasappliedtoavoidspuriousdetections.Thenoteonsetdensity,O
i,isthencomputedastheratiobetweenthenumberofonsetsandtheentire
durationofthetrack(inseconds).Anindicatorofthespectralregion,B
i,ofamusicaltrack i ,isgivenbythecentroidofthe
accumulatedBarkspectrum,b, acrossthedurationofanaudiotrack(Equation5).TheBarkspectrum,b ,iscomputedbythetimbreIDlibrary(Brent,2010),whichbalancesthefrequencyresolutionacrossthehumanhearingrange,bywarpingapowerspectrumrepresentationtothe24criticalbands,h ,ofthehumanauditorysystem(i.e.,Barkbands).
Bb h
bi
h h
h h
=⋅
=
=
∑∑�
�1
24
1
24 (5)
wherebh
istheenergyoftheBarkbandh .TheBiindicatorcanrangefrom1to24Barkbands.
FollowingPachetandAucouturier(2004),thetimbralsimilarity,Ci p,
,betweentwotracks, i andp ,isgivenbythecosinesimilaritybetweentheirmel-frequencyspectrumcoefficients(MFCC),MiandM
p(Equation6).MFCCvectors,M ,include38componentsresultingfromapplyinga
100mel-scaledfilterbankspacinginthetimbreIDlibrary(Brent,2010).
CM M
M Mi p
i p
i p,=
⋅ (6)
Thetimbralsimilaritymetric,Ci p,
,rangesbetween1and-1,whichcorrespondstotrackswithequaltimbreandthemostdissimilartimbre,respectively.
MeTHoDoLoGy
ThemethodologyusedforthedevelopmentofthepresentworkisbasedonA Three Cycle View of Design Science Research(Hevner,2007).Thismethodologyfacilitatesthedevelopmentofinteractiveapplicationsandpromotesquickeriterationsbetweentheseveralphasesoftheapplicationdevelopment,suchasthevisualisationimplementation,thevisualcomponent’svalidation,andtheguidelinesforitsrefinement(Figure3).
Thepresentmethodologyisdividedintothreeparts:TheRelevanceCycle,theDesignCycle,andtheRigorCycle.Firstly,intheRelevanceCycle,theresearchcontextisdefined.Therequirementsandproblemsfromtheprevioussystemwerehighlighted,leadingtothedefinitionofthekeyobjectivesforthenewsystem(seeIntroduction).
Secondly,intheDesignCycle,thenewsystemwasdevelopedthroughaloopofresearchbetweenthe implementation of thevisualisation components and their assessment. This iterative processpromotestheanalysisandimprovementofpreviousstepsandtheexperimentationandrefinementofdifferentapproaches.Inparticular,thiscyclewasdividedintofourmainphases:(i)thedataanalysis,whichisrelatedtothecomputationoftheharmoniccompatibilitymatrix(seeMixMash:Compatibility
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
28
Method);(ii)thedesignofthevisualisationmodel,whichrelatestotheiterativeprocessbetweendefinitionof thegraphicalvariablestorepresent themusicalcharacteristicsandthevisualisationmodelanditsvalidationwithexpertsinInformationVisualisation;(iii)theimplementationofthemodel,wheretheForceAtlas2algorithmwasstudiedandimplemented;and(iv)theevaluationofthesystem.ThedefinitionofsuchphasesisalignedwiththemethodologiesproposedbyWilkins(2003)andFry(2004)fromtheInformationVisualisationfield.Thevalidationofthesystemwasconductedthroughan initial informalevaluation (Lametal.,2011), inwhich thevisualisationcomponentswerediscussedbetweentheauthorsandexternalexpertsfromthevisualisationfieldtoassessthevisualisationintuitivenessandusability.
Finally,theRigorCycleconnectswiththecentralDesignCyclethroughaniterativeexchangeofknowledge,bothfromscientificfoundationsasfromthevisualisationvalidation.Thisfinalcycleischaracterisedbyfine-tuningthevisualisationmodelthroughknowledgeacquiredfromtheevaluationsandrelatedwork.
MIXMASH: VISUALISATIoN MoDeL
Thevisualrepresentationoftherelationshipsbetweenaudiotrackswereguidedbythreeobjectives(seeIntroduction):(i)theimplementationofanadaptivevisualisationmodel(seesubsectionForce-directedSystem);(ii)thedistinctionbetweenthedifferentsoundcharacteristicsoftheseveralmusicalaudiotracks(seesubsectionGraphicVariablesandAudiovisualMappings);and(iii)theconceptionandimplementationofasimpleandintuitiveinterface(seesubsectionInterfaceDesignandInteraction).
Toimprovethescalability,interaction,andvisualisationoftheinterfacepreviouslypresentedinBernardesetal.(2018),aforce-directedgraphlayoutbasedontheForceAtlas2algorithm(Jacomy,Venturini,Heymann,&Bastian,2014)wasimplemented.Thisvisualisationtechniqueenabledthecreationofanemergent,organic,andappealingenvironmentfortheuser.Furthermore,itimprovesthereadabilityofthepreviousinterfaceofBernardesetal.,bypreventingoverlaps,arrangingthetracksinspacebytheirharmoniccompatibility,andbyenablingtheusertoexploreandinteractwithtracksofinteresteasily.Additionally,tocharacterise,distinguishandimprovethereadabilityofthetracksandtoaugmentthegraphicattributesthatrepresentanddistinguishthetracks,agraphicalrepresentationforthemusicalaudiotrackswasalsostudiedandapplied.
Toenabletheusertofilterthetrackcollectionaccordingtoharmonic,rhythmic,spectral,andtimbral attributes, a graphic interface was also implemented (Figure 4). With the force-directedalgorithm,theusercandetectthemostharmonicallycompatibletracksthroughtheirvisualproximityinthecanvas.Thisiscausedbytheforcesappliedtoeachtrack,whichdependdirectlyontheharmoniccompatibility.However,theusercanmanipulatetheimpactoftheforcesofattractionandrepulsion
Figure 3. The Three Cycle View of design science research
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
29
betweentracks,easingthecomprehensionofmoreclutteredzones.Additionally,highlycompatibletracksareclusteredtoreduceundesiredclutter.Theseclustersarevisuallydistinguishedfromthetracksandcanbeexpandedorwithdrawnthroughinteraction.Theforce-directedalgorithmandeachcomponentoftheinterfacearedescribedinmoredetailinthefollowingsubsections.
Force-Directed SystemTheForceAtlas2algorithm(Jacomy,Venturini,Heymann,&Bastian,2014)canbecharacterisedbyitsabilitytoplacethenodeswithinagraphaccordingtotheirconnectionsweight.Thealgorithmsimulatesaphysicalsystemthatspatiallyarrangesthenetwork’snodesinanautomaticform.Thenodeshaveforcesofrepulsiontopreventthemfromoverlapping,andtheedgesbetweennodesapplyforcesofattractiontobringthenodescloser.Theseedgeshavedifferentforcevaluesaccordingtothesimilaritybetweennodes(e.g.,theirharmoniccompatibility).Byapplyingcontinuously,thedifferentforces,thegraphconvergestoabalancedstatethataidsthesemanticinterpretationofthenetwork.TheForceAtlas2algorithmwasfullyimplemented,thusforadetaileddescriptionofthealgorithm,refertoJacomyetal.(2014).
Inthisproject,twotypesofnodesaredefined:theonesrepresentingeachmusicalaudiotrackinthecollection,t,andtheonesrepresentingeachoneofthem=24majorandminormusicalkeys.ThevisualdistinctionbetweenthenodesisdiscussedatlengthinsubsectionGraphic Variables and Audiovisual Mappings.Byrepresentingthemusicalkeys,m,throughnodes,andconsequentlytheirrelationtothemusicalaudiotracks,themostcompatiblekeyofeachmusicalaudioisindicated.Thisenablestheusertovisuallydetectthetracksandsetsoftracksmoreharmonicallycompatiblewiththedifferentkeysandrelatetracksaccordingtothiscompatibility.
Theweightoftheedgebetweentwonodesismappedaccordingtothecompatibilityvalueinthet + msquarematrix.Thissquarematrixiscomputedthroughagivenauser-definedcollectionoftaudiotracksandm=24majorandminorkeysandexpressesthemetricsforsmall-scale(Si,p)and large-scale (Li,p) scaleharmoniccompatibility (seeMixMash:CompatibilityMethod).As inForceAtlas2, theweightdefinestheforceofattractionbetweennodes.Forthisproject, themore
Figure 4. Screenshot of the visualisation interface. On the left, the interaction panel; at the centre, the graph.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
30
compatiblethenodes,thehighertheforceofattraction,andconsequentlythecloserthenodes.Aforceofrepulsionisappliedtoallnodestoavoidthemtooverlap.Theseforcesareappliedtothenodesindependentlyoftheirtype,facilitatingtheinterpretationofthegraphandinteractionwhensearchingforharmonicallycompatibletracks,t.
Twomechanismswereimplementedtoenabletheusertorefinethegraphlayout:(i)aconnectivitythreshold;and(ii)musicalkeyrestriction.Bothmechanismscanbeexploredbytheuserthroughtheleftpaneloftheinterface(Figure4).Throughtheconnectivitythreshold,theusercandefineathresholdvaluetodeterminewhethertwonodesareconnected.Foreachnode,theconnectionstoothernodesonlyoccurwhentheirharmoniccompatibilityvalueislowerthanthepredefinedvalue.Thesecondmechanismlimitstheconnectionsbetweennodesandkeys.Throughthismechanism,theusercandefinewhetheratrackisconnectedtoallcompatiblemusicalkeysoronlytoitsmostprobablekey.Thismechanismisintendedtoreduceclutterandenhancetheassociationbetweenkeysandmusicalaudiotracks.
Asthenumberoftracksinacollectioncanvarygreatly,anagglomerativeclusteringalgorithm(Rokach&Maimon,2005)isimplementedtopreventclutteredgraphs.Thisalgorithmaggregatesthenodesbytheircompatibilityvalues.Aminimumnumberofthreenodesperclusterisrequiredtopreventsmallclusters,whichwouldn’tenhancetheclarityofthevisualisation.Eachclusterisalsoconnectedtothecompatiblenodesandclusters,thusexposingtheirharmonicrelationtotheneighbourhoodelements.Thesecompatiblenodesareretrievedfromthelistofcompatiblenodesofeachnodewithinthecluster.Ifanouternodeisconnectedtoaninnernodeofacluster,aconnectionbetweenthenodeandtheclusterisestablished.Theattractionforcebetweenaclusterandacompatiblenodeisequaltotheaverageforceofallforcesbetweeninnernodesandtheconnectedouternode(asdepictedinFigure5).
AsinForceAtlas2,allnodesgravitatearoundthecentreofthecanvas.Thiseffectresultsfromtheattractionforceappliedtoallnodestowardsthecanvascentrepoint.Thegravitationalforceissignificantlyweakerthantheothersand,asitisappliedequallytoallnodes,itdoesnotinterferewiththedistancebetweennodes,andonlypreventsthenodesfromdispersinginthecanvas.
Bydefault, themusicalkeynodesarepositionedby theforce-directed layout,dependingontheirrelationswiththeothertracks.Thiscausesthekeynodestohavedifferentpositionsateveryrun,whichcancreatesomeuserfatiguewhilesearchingforaspecificmusicalkeyanditsconnectedtracks.Tofacilitatethissearch,amechanismthatpositionsthekeynodesaccordingtotheCamelotWheelorcircleoffifthsrepresentationwasimplemented(Figure6).Withthismechanism,allmusicalkeyshaveafixedpositioninthecanvas,andonlythetracknodesareinfluencedbytheattractionandrepulsionforcesofthealgorithm.
Figure 5. Scheme of the forces representing outer and inner nodes of a cluster. The forces between clusters and nodes (b) are computed through the average force between the inner and outer nodes (a). The lines depict compatible nodes.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
31
Graphic Variables and Audiovisual MappingsThevisualrepresentationofdataelementsstronglyimpactsthevisualisationoflargeamountsofdata.Inthissection,theadoptedvisualrepresentationofmusicalconceptsisdiscussedinlightofacarefullydesignedinteractivevisualisation.
The development of the visualisation model complies with the following guidelines andsubsequent challenges: (i) to each track there is a corresponding visual representation based onitsmusicalattributes,creatingaconsistentvisualfeedbackbetweensimilartracks;(ii)perceptualfoundationsareusedtoguidetheaestheticalchoicesconcerningvisualrepresentationoftracks;(iii)anaturalandintuitiveinteractionwiththetoolispromotedallowingtheusertoeasilynavigateamongtrackstocreatehis/hermashup.
Foreachtrack,thespectralregion,Bi,onsetdensity,Oi,andtimbralsimilarity,Ci,p,aremappedtoacorrespondingvisualvariable.Additionally,bothspectralregion,Bi,andonsetdensity,Oi,aresubdividedintothreelevelsofmagnitude,allowingamoreaccurateinterpretationandanalysisofmusicaldata.
Thespectralregionattribute,Bi,issplitintohigh,medium,andlowregions,anditscorrespondingvisualrepresentationisprimarilycharacterisedbyshape.Althoughthereisatendencytoassociatecolour with pitch levels in similar audiovisual mapping problems (McDonnell, 2007; Mudge,1920;Ward,Huckstep,&Tsakanikos,2006), ithasalsobeenproposed that theuseof shapeormonochromecolourismoreefficientthanhuetodistinguishseverallayersofinformationforthehumaneye(Arnheim,1974;Chatterjee,2013).Basedonsuchstudies(Arnheim,1974;Chatterjee,2013;Ramachandran2003;Spence,2011),high-frequencyregionsofthespectrumareassociatedwithsharpcontours(triangle),lowfrequencieswereassociatedwithroundedcontours(circle),andmediumfrequenciesareassociatedwithneutralcontours,achievedthroughtheuseofstraightlinesoftherectangle(Figure7a).Colourhue,isusedtoreinforcetheseassociations.Higherfrequenciesarerelatedtoacoldcolour(blueandlowfrequenciestoawarmcolour(orange;seeFigure7c).
Figure 6. Fixing the keys in the circle of fifths. By clicking on the Fix Tones button, all nodes that represent a key are fixed on the canvas according to their positioning in the circle of fifths.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
32
Onsetdensity,Oi,issplitintohigh,mediumandlow-densitylevels,andthenconveyedinthevisualdomainasaparameterofshape-filling,asitrelatestothenumberofnotesinatrack.Lowdensitycorrespondstoanemptyshape(onlycontoursvisible),mediumdensitytoahalf-filledshape,andhighdensitytoacompletelyfilledshape(Figure7b).
Timbreisoftenreferredtointheliteratureasthecolourofinstruments(Mudge,1920;Wardetal.,2006).Inspiredbythisdefinition,timbralsimilarity,Ci,p,isconveyedinthevisualdomainthroughcolouredcirclesinthetopsideofthenodes,whichareonlyvisiblewhenacertainnodeisselected.Thechoiceofcoloursdoesnotrelyonanytypeofassociation,andforthisreason,arerandomlyselectedfromapredefinedsetofcolours.Theysimplyaimtoprovideacleardistinctionbetweendifferenttimbreswhenmultipletimbresareselected.Byselectingonetracknode,acolouredcircleandalineconnectingtheformertothenodearedrawn(Figure8a).Then,alltrackswithsimilartimbreswillalsogainacirclecolouredwiththesamecolourcodeastheclickedone(Figure8b).Ifatracknodeissimilartomultipletimbres,multiplecolouredcircleswillbedrawnoverthenode(Figure8c).
Thevisualdistinctionbetweennodesrepresentingmusicalaudiotracksandkeysishighlightedbythecolouredoutlineofthekeynodes.Thelatterhasaredoutlineandincludethetypographicrepresentationofthekey’stonicpitch,whichallowsthedirectreadingofthekey(Figure4).
Thevisualrepresentationofaclusterisdefinedbytherespectiveinnernodestoavoidtheuseofadditional(andpotentially,morecomplex)visualelements.Morespecifically,allnodesthatbelongtoaclusterarerepresentedandpositionedwithinacircularshapeoutlinedwithblackdots.Thesizeofthecircularshapedependsonthenumberofelementswithinthecluster(Figure9).Assuch,theusercandifferentiatetheclustersfromthenodes,and,simultaneously,getanoverviewoftheirinnernodes.
Figure 7. Audiovisual mappings: (a) Shape—Spectrum Frequency from low frequencies (circle) to higher ones (triangle); (b) Object Fill—Onset Density. Empty shapes correspond to low density, half-filled shapes to normal/mid density, and full shape corresponds to high density; (c) Colour—Spectrum Frequency from low frequencies (orange) to higher ones (blue).
Figure 8. Timbre representations. (a) One track is selected and there are no other tracks with similar timbres. (b) One track is connected timbre wise to the selected track. (c) The track can have more than one timbre similarities (d) The track is not related to any of the selected tracks.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
33
Interface Design and InteractionMixMashenablestheusertoexplorethegraphbyallowinghim/herto(i)listenandselectindividualnodetracksofinterest(Figures10a,10b,10c);(ii)highlightnodeswithinclusters(Figure10d)oraccordingtotheirsoundcharacteristics(Figure11);(iii)modifyitsorganisation(e.g.,fixingthekeysaccordingtothecircleoffifths)(Figures10e,10f);(iv)changehowthetracknodesconnecttothekeynodes(Figures10g,10h);(v)changetheconnectionthresholdbetweennodes;and(vi)adjusttheforcesofattractionandrepulsion.Thisisallaccessiblethroughaninteractivepanelontheleftsideoftheinterface(Figure4)andthroughmouseinteractions.Avideoregardingtheseinteractionsandpossibilitiescanbeaccessedathttps://vimeo.com/270076175.
Oncethesystemhasbeeninitialised,theuserwillseethenodesandclustersestablishingtheirpositioninthecentreofthecanvas.Inadditiontothefunctionalitiespresentintheleftpanel,theusercaninteractwiththevisualisationthroughmouseinteractions.Theusercanzoomandpanthegraphtoviewmoredetails.Then,theusercaninteractwiththetracksindividually.Tolistentothetracks,theuserhastomovethemouseovereachnode.Toselectanode,theuserneedstoleft-click.Withthisaction,he/shewilllistenagaintothetracksound.Toviewthecompatibletimbresofacertaintrack,theuserhastoright-click.Bydoingso,theclosesttracks(intermsoftimbre)arecomplementedwithacolouredcircleasexplainedinsubsectionGraphic Variables and Audiovisual Mappings.Interactionswiththeclusterswerealsoimplemented.Toexpandacluster,theuserhastoleft-clickoverit.AllnodesinsidetheclusterwillbeaffectedbytheforcesaccordingtotheForceAtlas2algorithm,andadoughnutshapefigurewillbemadevisible.Thedoughnutshape,positionedinthecentreofallcorrespondingnodes,behaveslikeabuttonand,whenclicked,itclosesthecluster.Iftheuserdoesmouseoveronthislattershape,allthenodesbelongingtotheclusterwillbehighlightedthroughamagentastroke(Figure10d).Finally,allthetracksthathavenosimilaritytootherswillbeplacedatthebottomrightsideofthewindow.
Figure 9. Representation of two clusters with different sizes
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
34
Additionalkeyboardinteractionswereimplemented.Whenlisteningtoanaudiotrack,theusercanclicktheSkeyonthekeyboard,andthemusicwillstopplaying.Bycontinuouslyselectingnodes,theuserissavingthetracks,whichcanbeheardatthesametimebyusingthespacekey.
Overall,thissetofinteractiontechniquesareimportanttoachieveanintuitiveandmeaningfulinteractivevisualisationtoolinthecontextofmusicalmashupcreation.Withthis,theauthorsaimtoenhancetheunderstandingofthetrack’sharmoniccompatibilityandfosterusercreativity,byallowingtheusertoefficientlyexplorealargemusicalaudiocollectiontowardsspecificcompositiongoals.
Figure 10. Visual representation of the interactions with the model: (a) no selection, (b) mouse over, (c) mouse click, (d) clusters’ nodes highlight, (e) force-directed layout, (f) circle of fifths layout, (g) connections to all compatible keys, (h) connection to the most compatible key (1st tone option)
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
35
CoNCLUSIoN
Theauthorsproposedanovelvisualisationsystemwhichreliesonforcesofattractionandrepulsiontopositionthetracksdependingontheirharmoniccompatibility.ThevisualisationdevelopmentwasguidedbyamethodologyproposedbyHevner(2007),whichconsistsofthreecyclesandleadtoclearandconsistentinteractionsbetweenthemusicmashupandtheinformationvisualisationmodel.
Regardingtheforce-directedalgorithm,controlledlevelsofattractionandrepulsionallowthereductionofclutterinthevisualizationoflargemusiccollections(ofroughly50musicalaudiotracksormore).Clutterwasalsominimizedbytheadoptionofclusteringtechniques,whichenhancethevisualizationofcombinedhierarchicallevelsofharmoniccompatibilityinthesamerepresentationandtheuser-controloverclusteringquality(i.e.,distancethreshold).
Afluidre-organizationofthevisualizationwasachievedbydraggingconnectedelementsintheinterface,therebyenhancinghighlydenseareasofparticularinteresttotheuser.Ontheotherhand,theFixTonesstrategyexploresastaticvisualizationofthemusicalaudiocollectionbyfixingthelocationofkeysintheTonalIntervalSpace.TheresultingrepresentationisoneofthemostfamiliarmapsoftonalregionsinWesternmusic.
Theauthorswereabletoexpandthenumberofcontent-basedmusicalaudioattributesunderconsiderationtocoverbothrhythmic,harmonic,spectral,andtimbralattributes.Thedevelopmentof a specificgraphic representation supported themusicvisualisationbyprovidingaperceptualassociation,andtherefore,theintuitiveassociationbetweenvisualandmusicalattributes.Ingeneral,thepresentedvisualsolutionwasabletopromoteamorefluidvisualisation.However,itstillhassomelimitationsduetothehighnumberofsamplesthatarebeingdisplayedinrealtimetotheuser.
Asfuturework,theauthorsintendtoimprovetheclusteringalgorithmbygivingtheuserthepossibility toclusternodesaccording todifferent audiocontent-basedattributes, e.g.,key,onsetdensity,spectralregionortimbralsimilarity.Toimprovereadability,differentsolutionstothenodes’sizedependingontheircompatibilitywillbestudied,e.g.,nodeswithhighercompatibility,growinsize,emphasisinghighlycompatibletracks.Asthenumberoftrackscanincreasedependingontheuser,afish-eyezoomtechniquewillbeimplementedsotheusercanhavedetailincertainareas
Figure 11. Frequency highlight by color
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
36
withoutlosingcontextofthesurroundingtracks.Finally,atimelinewillbedesigned,sothatuserscanarrangeselectedtracksintime,thusenablingthecompositionofmusicalmashupswithcomplextemporalstructures.
ACKNowLeDGMeNT
The work is supported by Portuguese National Funds through the FCT-Foundation for Scienceand Technology, I.P., under the grants SFRH/BD/129481/2017 and SFRH/BD/139775/2018.Experimentationinmusic inPortugueseculture:History,contexts,andpractices in the20thand21st centuries—Project co-funded by the European Union, through the Operational ProgramCompetitivenessandInternationalization,initsERDFcomponent,andbynationalfunds,throughthePortugueseFoundationforScienceandTechnology.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
37
ReFeReNCeS
Arnheim,R.(1974).Art and visual perception.UniversityofCaliforniaPress.
Bergstrom,T.,Karahalios,K.,&Hart,J.(2007).Isochords:Visualizingstructureinmusic.InProceedingsofGraphicsInterface(pp.297–304).ACM.
Bernardes,G.,Cocharro,D.,Caetano,M.,Guedes,C.,&Davies,M.(2016).Amulti-leveltonalintervalspaceformodellingpitchrelatednessandmusicalconsonance.Journal of New Music Research,45(4),281–294.doi:10.1080/09298215.2016.1182192
Bernardes,G.,Davies,M.,&Guedes,C.(2017).Automaticmusicalkeyestimationwithadaptivemodebias.InProceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing(pp.316–320).IEEEPress.doi:10.1109/ICASSP.2017.7952169
Bernardes,G.,Davies,M.,&Guedes,C.(2018).Ahierarchicalharmonicmixingmethod.InM.Aramakietal.(Eds.),InternationalSymposiumonComputerMusicMultidisciplinaryResearch(pp.151–170).SpringerNatureSwitzerland.
Brent,W.(2010).AtimbreanalysisandclassificationtoolkitforPureData.InProceedings of the International Computer Music Conference.AcademicPress.
Chatterjee,A.(2013).The aesthetic brain: How we evolved to desire beauty and enjoy art.OxfordUniversityPress.doi:10.1093/acprof:oso/9780199811809.001.0001
Chen,C.(2006).CiteSpaceII:Detectingandvisualizingemergingtrendsandtransientpatternsinscientificliterature.Journal of the Association for Information Science and Technology,57(3),359–377.
Davies,M.,Hamel,P.,Yoshii,K.,&Goto,M.(2014).Automashupper:Automaticcreationofmulti-songmusicmashups.IEEE Transactions on Audio, Speech, and Language Processing,22(12),1726–1737.doi:10.1109/TASLP.2014.2347135
Davies,M.,Stark,A.,Gouyon,F.,&Goto,M.(2014).Improvasher:Areal-timemashupsystemforlivemusicalinput.Proceedings of the International Conference on New Interfaces for Musical Expression,London,UK(pp.541–544).AcademicPress.
Dobrowohl,F.A.,Milne,A.J.,&Dean,R.T.(2019).Timbrepreferencesinthecontextofmixingmusic.Applied Sciences,9(8),1695.doi:10.3390/app9081695
Enright,A.,&Ouzounis,C.(2001).Biolayout—Anautomaticgraphlayoutalgorithmforsimilarityvisualization.Bioinformatics (Oxford, England),17(9),853–854.doi:10.1093/bioinformatics/17.9.853
Fry,B.J.(2004).Computational information design[PhDthesis].MassachusettsInstituteofTechnology.
Fu,X.,Hong,S.-H.,Nikolov,N.,Shen,X.,Wu,Y.,&Xuk,K.(2007).Visualizationandanalysisofemailnetworks.Proceedings of the International Asia-Pacific Symposium on Visualization(pp.1–8).IEEE.
Gebhardt,R.,Davies,M.,&Seeber,B.(2016).Psychoacousticapproachesforharmonicmusicmixing,Applied Sciences, 6(5).
Gibney,M.(n.d.)Music-map,thetouristmapofmusic.Retrievedfromhttps://www.music-map.com
Gilbert,F.,Simonetto,P.,Zaidi,F.,Jourdan,F.,&Bourqui,R.(2011).Communitiesandhierarchicalstructuresindynamic socialnetworks:Analysis andvisualization.Social Network Analysis and Mining,1(2),83–95.doi:10.1007/s13278-010-0002-8
Goh,K.-I.,Cusick,M.,Valle,D.,Childs,B.,Vidal,M.,&Barabási,A.-L.(2007).Thehumandiseasenetwork.Proceedings of the National Academy of Sciences of the United States of America, 104(21), 8685–8690.doi:10.1073/pnas.0701361104
Griffin,G.,Kim,Y.,&Turnbull,D.(2010).Beat-sync-mash-coder:Awebapplicationforreal-timecreationofbeat-synchronousmusicmashups.Proceedings of the International Conference on Acoustics, Speech, and Signal Processing(pp.437–440).AcademicPress.doi:10.1109/ICASSP.2010.5495743
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
38
Grill, T., & Flexer, A. (2012). Visualization of perceptual qualities in textural sounds. Proceedings of the International Computer Music Conference.AcademicPress.
Hamasaki,M.,&Goto,M.(2013).Songrium:Amusicbrowsingassistanceservicebasedonvisualizationofmassiveopencollaborationwithinmusiccontentcreationcommunity.Proceedings of the International Symposium on Open Collaboration.ACM.doi:10.1145/2491055.2491059
Heer,J.,&Boyd,D.(2005).Vizster:Visualizingonlinesocialnetworks.Proceedings of the IEEE Symposium on Information Visualization(pp.32–39).IEEE.
Henry,N.,Fekete,J.-D.,&McGuffin,M.(2007).Nodetrix:Ahybridvisualizationofsocialnetworks.IEEE Transactions on Visualization and Computer Graphics,13(6),1302–1309.doi:10.1109/TVCG.2007.70582
Herman,I.,Melançon,G.,&Marshall,M.(2000).Graphvisualizationandnavigationininformationvisualization:Asurvey.IEEE Transactions on Visualization and Computer Graphics,6(1),24–43.doi:10.1109/2945.841119
Hevner,A.R. (2007).Athreecycleviewofdesignscienceresearch.Scandinavian Journal of Information Systems,19(2),4.
Jacomy,M.,Venturini,T.,Heymann,S.,&Bastian,M.(2014).ForceAtlas2,acontinuousgraphlayoutalgorithmforhandynetworkvisualizationdesignedfortheGephisoftware.PLoS One,9(6),e98679.doi:10.1371/journal.pone.0098679
Kimelman,D.,Leban,B.,Roth,T.,&Zernik,D.(1994).Reductionofvisualcomplexityindynamicgraphs.Proceedings of theInternational Symposium on Graph Drawing(pp.218–225).Springer.
Lam,H.,Bertini,E.,Isenberg,P.,Plaisant,C.,&Carpendale,S.(2011).Seven guiding scenarios for information visualization evaluation.UniversityofCalgary.
Lee,C.,Lin,Y.,Yao,Z.,Li,F.,&Wu,J.(2015).Automaticmashupcreationbyconsideringbothverticalandhorizontalmashabilities.Proceedings of the International Society for Music Information Retrieval Conference,Malaga,Spain(pp.399–405).AcademicPress.
Maçãs,C.,Rodrigues,A.,Bernardes,G.,&Machado,P.(2018).MixMash:Avisualisationsystemformusicalmashupcreation.Proceedings of the 2018 22nd International Conference Information Visualisation (IV)(pp.471–477).IEEE.doi:10.1109/iV.2018.00088
McDonnell,M.(2007).Visualmusic.InVisual Music Marathon.BostonCyberartsFestivalProgramme.
Meirelles,I.(2013).Design for information.Rockportpublishers.
MixedinKey.(n.d.).Mashup2[software].Retrievedfromhttp://mashup.mixedinkey.com
Mudge,E.(1920).Thecommonsynaesthesiaofmusic.The Journal of Applied Psychology,4(4),342–345.doi:10.1037/h0072596
Muelder,C.,Provan,T.,&Ma,K.-L.(2010).Contentbasedgraphvisualizationofaudiodataformusiclibrarynavigation. Proceedings of the IEEE International Symposium on Multimedia (pp. 129–136). IEEE Press.doi:10.1109/ISM.2010.27
Native Instruments. (n.d.)TraktorPro2 [software].Retrieved fromhttps://www.native-instruments.com/en/products/traktor/dj-software/traktor-pro-2/
Navas,E.(2014).Remix theory: The aesthetics of sampling.Birkhäuser.
Pachet,F.,&Aucouturier,J.-J.(2004).Improvingtimbresimilarity:Howhighisthesky.Journal of Negative Results in Speech and Audio Sciences,1(1),1–13.
Ramachandran,V.S.,&Hubbard,E.M.(2003).Hearingcolors,tastingshapes.Scientific American,288(5),52–59.doi:10.1038/scientificamerican0503-52
Rauber,A.,Pampalk,E.,&Merkl,D.(2003).TheSOM-enhancedjukebox:Organizationandvisualizationofmusiccollectionsbasedonperceptualmodels.Journal of New Music Research,32(2),193–210.doi:10.1076/jnmr.32.2.193.16745
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
39
Rodrigues,A.,Cardoso,A.,&Machado,P.(2016).Harmonicconstellation:Anaudiovisualenvironmentoflivingorganisms.Proceedings of theFourth International Workshop on Musical Metacreation.AcademicPress.
Rokach,L.,&Maimon,O.(2005).ClusteringMethods.InO.Maimon&L.Rokach(Eds.),Data mining and knowledge discovery handbook.Boston:Springer.doi:10.1007/0-387-25465-X_15
Sapp,C.(2001).Harmonicvisualizationsoftonalmusic.Proceedings of the International Computer Music Conference(pp.419–422).AcademicPress.
Schedl,M.,Gómez,E.,&Urbano,J.(2014).Musicinformationretrieval:Recentdevelopmentsandapplications.Foundations and Trends in Information Retrieval,8(2–3),127–261.doi:10.1561/1500000042
Schwarz,D.,&Schnell,N.(2009).Soundsearchbycontent-basednavigationinlargedatabases.Proceedings of the Sound and Music Computing Conference.AcademicPress.
Shaath,I.(2011).Estimation of key in digital music recordings[Master’sthesis].BirkbeckCollege,UniversityofLondon.
Shiga,J.(2007).Copy-and-persist:Thelogicofmash-upculture.Critical Studies in Media Communication,24(2),93–114.doi:10.1080/07393180701262685
Snydal,J.,&Hearst,M.(2005).Improviz:Visualexplorationsof jazzimprovisations.InCHI’05ExtendedAbstractsonHumanFactorsinComputingSystems(pp.1805–1808).ACM.
Spence,C.(2011).Crossmodalcorrespondences:Atutorial review.Attention, Perception & Psychophysics,73(4),971–995.doi:10.3758/s13414-010-0073-7
VanGulik,R.,Vignoli,F.,&VandeWetering,H.(2004).Mappingmusicinthepalmofyourhand,exploreanddiscoveryourcollection.InProceedings of the International Conference on Music Information Retrieval.Retrievedfromhttp://www.ee.columbia.edu/~dpwe/ismir2004/CRFILES/paper153.pdf
Vavrille,F.(n.d.).Liveplasma.Retrievedfromhttp://www.liveplasma.com/artist-Hard-Fi.html
Vesna,V.(2007).Database aesthetics: Art in the age of information overflow.UniversityofMinnesotaPress.doi:10.5749/j.cttts7q7
Ward,J.,Huckstep,B.,&Tsakanikos,E.(2006).Sound-coloursynaesthesia:Towhatextentdoesitusecrossmodalmechanismscommontousall?Cortex,42(2),264–280.doi:10.1016/S0010-9452(08)70352-6
Wattenberg,M.(2002).Arcdiagrams:Visualizingstructureinstrings.Proceedings of the IEEE Symposium on Information Visualization(pp.110–116).IEEE.
Wilkins, B. (2003). MELD: A pattern supported methodology for visualisation design [Doctoral thesis].UniversityofBirmingham.
eNDNoTe
1 Intheaudiodomain,12-elementchromavectorsreporttheenergyofthetwelvepitchclasses,i.e.,allchromaticnotesoftheequal-temperedscale,bywrappingthespectralenergycontentofanaudiosignalintoasingleoctave.
International Journal of Art, Culture and Design TechnologiesVolume 8 • Issue 2 • July-December 2019
40
Catarina Maçãs has an Undergraduate and Master degree in Design and Multimedia from University of Coimbra. Currently is a computational designer and researcher at the Computational Design and Visualization Lab, is enrolled in the Doctoral Program of Information Science and Technology of the University of Coimbra, and is also a teaching assistant at the Department of Informatics Engineering of the University of Coimbra. In the past two years, started to work on Information Visualization and had already published in conferences such as, IJCAI and IVAPP.
Ana Rodrigues is a Computational Designer and Ph.D. Student. Aside from being currently enrolled in the Program of Information Science and Technology at the University of Coimbra, working on her thesis entitled “Cross-Modal Associations for Design Languages”, she is also doing research at the Computational Design and Visualization Lab. At the same institution she has previously completed a B.Sc. degree and a M.Sc. degree in Design and Multimedia. Currently, she also performs the role of invited assistant lecturer. She is particularly interested in exploring bridges that may emerge from crossing domains of visual communication, music, neurocognitive science, and computational creativity.
Gilberto Bernardes has a multifaceted activity as a saxophonist and researcher in sound and music computing. He holds a PhD in digital media from the University of Porto under the auspices of the University of Texas at Austin; a Master of Music, cum laude, from the Amsterdamse Hogeschool voor de Kunsten and a Degree in music performance from the Superior School of Music and Performing Arts of the Polytechnic Institute of Porto and from the ENM d’Issy-les-Moulineaux (France). Bearnardes pursues a research agenda focused on sampling-based synthesis techniques and tonal pitch spaces, whose findings have been shared to the scientific and artistic communities in over 30 scientific publications. His artistic activity counts with regular concerts in venues and festivals with recognized merit, such as Asia Culture Center (Korea); New York University (USA); Concertgebouw and Bimhuis (Holland); Teatro Monumental de Madrid (Spain) and Casa da Música (Portugal). He is a member of the Portuguese Symphonic Band and Oficina Musical. Bernardes is currently an Assistant Professor at the University of Porto, Faculty of Engineering and an fellow researcher at INESC TEC.
Penousal Machado is Associate Professor at the Department of Informatics of the University of Coimbra in Portugal. He is a deputy director of the Centre for Informatics and Systems of the University of Coimbra (CISUC), the coordinator of the Cognitive and Media Systems group and the scientific director of the Computational Design and Visualization Lab. of CISUC. His research interests include Evolutionary Computation, Computational Creativity, Artificial Intelligence and Information Visualization. He is the author of more than 200 refereed journal and conference papers in these areas, and his peer-reviewed publications have been nominated and awarded multiple times as best paper. As of February 28, 2018, his publications gathered over 1828 citations, an h-index of 21, and an i10-index of 40. He is also the chair of several scientific events, including, amongst the most recent, PPSN XV, EvoStar 2016, and EuroGP 2015; member of the Programme Committee and Editorial Board of some of the main conferences and journals in these fields; member of the Steering Committee of EuroGP, EvoMUSART and Evostar; and executive board member of SPECIES. He is the recipient of several scientific awards, including the prestigious EvoStar Award for outstanding Contribution to Evolutionary Computation in Europe, and the award for Excellence and Merit in Artificial Intelligence granted by the Portuguese Association for Artificial Intelligence.Penousal Machado has been invited to perform keynote speeches in a wide set of domains, from evolutionary computation to visualization and art. His work was featured in the Leonardo journal, Wired magazine and presented in venues such as the National Museum of Contemporary Art (Portugal) and the “Talk to me” exhibition of the Museum of Modern Art, NY (MoMA). He has been the Principal Investigator and Coordinator of several national and international projects, both at an academic and industry level.