Taptop:UsingtheLaptopChassisasaMusicalControllerOliverThompson1andChristopherHarte2
1DepartmentofElectronics,UniversityofYork,[email protected]
2MelodientLimited,Leeds,[email protected]
Abstract:ThispaperpresentsTaptop,anewgesturecontrolapplicationdesignedforcontrollingdrummachines,samplersandothersoftwarevirtualinstrumentsrunningonlaptopcomputers.Withoutrequiringexternalelectronichardware,Taptopturnsthechassisofthelaptopintoavelocitysensitivecontrolsurfacefortriggeringsamplesbycombiningdatacollectedfromthebuilt-inwebcamandmicrophonetoproduceMIDImessages.Theusertapsontheunusedspaceeithersideoftheirlaptopmousetrackpadwhichisdividedintoanumberofvirtual‘drumpads’.EachvirtualpadcausesadifferentMIDInotetobeoutputwhentapped,withthevelocityofthatnotedeterminedbytheforceofthetap.ThiscontrolsolutionaimstoprovideamoreaffordableandportablealternativetothehardwareMIDIcontroller,andincreasethepotentialofthecomputerasastandaloneperformanceinstrument.
Keywords:GestureControl,DrumPads,VelocitySensitive,LivePerformance,MusicalInterface
IntroductionSoftwaredrummachinesandsamplersareanimportantpartoftheworkflowofmanyelectronicmusicproducers.Yet,despitetheprominentroleoftheseinstrumentsinthemodernmusicstudio,theoptionsforcontrollingthemremainrestrictedtoalimitednumberofmethods.Thisisanissuenotonlyassociatedwithsoftwaredrummachinesandsamplers,butothervirtualinstrumentstoo.Whilesignificantadvancescontinuetobemadeindevelopingmorepowerfulandflexiblevirtualinstruments,thewaysinwhichmusiciansinteractwiththoseinstrumentshasremainedlargelythesamesincetheinceptionoftheDigitalAudioWorkstation(DAW)Theberge(1997).Whencomposingwithavirtualinstrument,amusicianwilloftenarrangesequencesofnotesviatheirDAWpianoroll.Thismethodofcontrolobscuresthemusician’sinteractionwiththeirinstrumentbehindsoftware,incontrasttoanacousticinstrumentwherethephysicalgesturesthatproducethesoundareimmediatelyapparenttotheaudience(Dobrian2001).Wellthoughtoutmappingofphysicalgesturestomusicalparameterscouldthereforeprovideamoreexpressivemethodofcontrolforsoftwareinstruments,andhelptoreintroducesomeoftheperformanceelementthatisoftenlostwhenmakingmusicwithacomputer(Bown,Bell,andParkinson2014).
Forthepurposesofmusicalinteraction,itisimportantthatagesturecontrolsystemprovidesthemusicianwiththeopportunitytodeveloparepertoireandadegreeofmusicalityovertimethroughpractice,aswouldbethecasewithamoretraditionalinstrument.Whenaskedwhatinstrumenttheyplay,fewcomputermusicianswouldrespondwith‘Iplaythecomputer’.Thisislargelyduetothefactthatthemajorityofhuman-computerinteractioninterfacesleavelittleroomforprogressionorvirtuosity(WesselandWright2002).Thefamiliarmethodofcontrollingacomputerthroughakeyboardandmouseortrackpadmaybeintuitive,butitisalsoincrediblyrestrictiveasamusicalinterface(Pavlovic,Huang,andSharma1997).Suchdevicesareunabletocapturevelocityinformationandhencelimitmusicalexpression.However,thekeyboardandmousearenottheonlysensorsavailabletousonamodernlaptop,indeedFiebrinketalhaveinvestigatedtheuseofvariousbuilt-insensorsincludingmicrophones,accelerometers,andwebcamsamongothersforcontrollingmusic(Fiebrink,Wang,andCook2007).HongandYeousedacousticfeedbackbetweenthecomputerspeakeranditsownmicrophonetodriveaudioDSPforexperimentalaudio-visualperformance(HongandYeo2013).
Figure1.Taptopinuse.Anangledmirrorallowsthesystemtodetecttheuser’shandmotioninthedrumpadareausingthebuiltinwebcam.Theuserhasachoiceofdividingthelaptopchassisinto2,4or8drumpads,selectedviaaseriesofbuttonsontheGUI.Thedrumpadsincreaseinwidthasfewerpadsareselected,sothattheentiretyofthelaptopchassiswidthisalwaysbeingutilised.Theangleofthemirrorattachmentisadjustablesothatitcanbeused
withavarietyoflaptopmodels.
ExternalMIDIcontrollersgosomewayinenhancingthemusicalperformancecapabilitiesofthecomputer,buttheseareoftenexpensivedevicesandconstituteanotherpieceofequipmentforamusiciantotransportoraccommodateintheirstudio(GillianandNicholls2012).AsystemthatprovidesthesamefunctionalityasaMIDIcontrollerwithouttherequirementforexternalelectronichardwarewouldthereforepossesssignificantadvantages.Tothisend,wehavedevelopedTaptop,agesturecontrolsoftwareapplicationthatusesthelaptop’sownmicrophoneandcameratodetecttheuser’sactions.Usingthesebuilt-insensors,weallowunusedspaceonthelaptopchassisitselftobeusedasawellestablishedinterfaceforcontrollingvirtualdrummachinesandsamplers,namelyabankofdrumpads.Drumpadsareanintuitiveinterfacethatanyonecanusetotriggersamples,buttheyalsoallowaperformertodeveloptheirskillssufficientlyenoughovertimetobeconsideredanexpert(Zamborlinetal.2014);theelectronicmusicianJeremyEllisbeingagoodexample(Ellis2011).
Theconceptofusingmicrophonestoturnagenericsurfaceintoaninterfaceforcontrollingdigitalaudioisnotnew.Anumberofsystemshavealreadybeensuccessfullyimplementedusingthisidea,themostwidelyknownprobablybeing‘Mogees’(Zamborlin2016).Mogeesaresmalldevicescontainingacontactmicrophonewhich,whenconnectedtoasmartphoneorcomputer,allowtheusertodriveasynthesisengineinaccompanyingsoftwaresimplybyhittingorrubbingasurface.Asinglemicrophoneallowsthesystemtocapturetimbreandtransientinformationbutsomeotherapproacheshaveusedmultipletransducerstoallowpositioninformationtobecapturedaswell.ForexampleBisbyetal.(2014)developedaninstrumentusinganarrayofpiezostoachieveaccuratelocalisationofsoundsourcesonasurface,whileNovelloandRaijekoff(2015)utilisedapairofcontactmicrophonestolocatethegesturesoftheuseronasurface,mappingthepositionofmovementtothepitchoftheinstrument.
SystemOverviewTaptopwasimplementedinthePureDatagraphicalprogramminglanguage,makingextensiveuseoftheimageandvideoprocessingobjectsthatcomprisetheGEMlibrary.Insteadofphysicaldrumpads,theusertapsintheunusedspaceeithersideoftheirlaptopmousetrackpadwhichisdividedintoanumberofhorizontallyarrangedvirtualpads,representedbycolouredboxesinfigure1.Thelaptopwebcamfieldofviewispositionedoverthisareabymeansofamirrorattachment;theonlyexternalaccessoryrequiredtouseTaptop.Toassisttheuserinpositioningthemirrorattachment,thewebcamstreamisdisplayedinawindowoverlaidwithablackrectanglethatindicateswherethemousetrackpadshouldbelocated.
TaptopcontrolsvirtualinstrumentsviaMIDI.InordertotriggeraMIDInote,twocriteriamustbemet.Thisprocessisdescribedintheflowchartinfigure2.Firstly,theaudibletapoftheuser’sfingeronthechassisofthelaptopmustbedetectedthroughanalysisofthelaptopmicrophonesignal.ThelevelofthistapcorrespondstothevelocityofanyMIDInotesubsequentlytriggered.Secondly,movementmustbefoundtobeoccurringinatleastoneofthevirtualdrumpadsbyamotiondetectionalgorithmappliedtothewebcamsignal.EachofthesedrumpadscorrespondstoadifferentMIDInotenumber.Whenatapisdetectedinthemicrophonesignal,MIDInoteonmessagesaresenttothevirtualinstrumentbeingcontrolledwithnotenumbersdeterminedbythemovementoccurringinthedrumpads.
Figure2.Dataflowinthesystem.TaptoptranslatesdatafromthelaptopwebcamandmicrophonetoproduceMIDImessageswhicharethenusedtocontrolavirtualinstrument.
DetectingMovement
Taptopcropsthelaptopwebcamsignalintoanumberofregions,eachrepresentingavirtualdrumpad.Itisthemovementoftheuser’sfingersintheseregionsthatdeterminewhichnotenumbersareencodedintotheMIDInoteonmessagesthataresent.Movementisdetectedbycomparingthecurrentandpreviousframesofeachregiontodeterminewhichpixelshavesignificantlychanged.ThevideostreamisconvertedfromanRGBtoanRGBAcolourspacesothatpixelsthatdemonstrateachangeinthesesubsequentframescanbestoredintheemptyalphachannel.Ifthenumberofpixelsstoredinthealphachannelisaboveasetthreshold,thesystemassumesthatmovementisoccurringinthatdrumpad.
DetectingTransients
Todetecttheaudibletapoftheuser’sfingeronthelaptopchassis,weusedamethodbasedonthe‘Bonk’PureDataobject(Puckette1997).Bonkisatransientdetectorthathastheabilitytodeterminewhichinstrumentwasresponsibleforaparticularattackbycomparingspectralchangestoasetofstoredtemplates.Tapsonthelaptopchassiswererecordedatavarietyofvelocities,thenstoredastemplatessothatBonkcouldbetaughttoignoreotheraudiobeingpickedupbythemicrophone.Thishelpedaddressthepossibleissueoffeedbackinthesystem,whereplayingbackdrumsamplesthroughloudspeakerscouldcausetheapplicationtotriggeritself.Whilesomedrumhitsmayhaveasimilarenvelopetothetemplates,thespectraldifferencesbetweenthetwoshouldcausethemtobeignoredbyBonkinmostcases.
Transient detection
Motion detection
Motion detection
Motion detection
Partition and crop
Pad 1
Pad 2
Pad n
Note x
Note y
Note z
Video
Audio Events with velocity
MIDI output
Virtual instrument
MIDI Note On
Pad→note mapping
Pad→note mapping
Pad→note mapping
Mirror
User taps chassis
Web cam
Microphone
UserTestingAseriesofusertestswerecarriedoutinordertodeterminethesuccessofTaptopasamusicalinterface.Tenparticipantswereselectedwhoeachhadbackgroundsinelectronicmusicproduction.Thiswastoensurethatthosetakingpartinthetestswouldhavetherelevantknowledgetoprovideinformedopinionsontheperformanceandfeaturesoftheapplication.Twolinesofinvestigationwereemployed;afocusgrouptoestablishiftheconceptwaspopularamongmusicproducers,followedbymoredetailedbetatestingtoascertainhowproducersutilisedTaptopoveranextendedperiodoftime.Thefocusgroupandbetatestinvolvedfiveparticipantseach.
Methodology
Inthefocusgroup,participantswerefirstgivenanintroductiontotheconceptbehindTaptopandthecurrentissuesinvirtualinstrumentcontrolthatitwasdesignedtoaddress.Theyweretheninstructedonhowtousetheapplicationbeforebeinggiventenminutesalonetotryitoutforthemselvescontrollingavirtualinstrument.Thetestswerecarriedoutundercontrolledconditionsinaroomwheretheleveloflightingcouldbeadjusted.ThismeantthattheperformanceofTaptopcouldbeevaluatedinbright,dark,andoptimallightingtoreflectthedifferentenvironmentsthatitmaybeusedin.Afterthetestswerecompleted,theparticipantseachansweredaquestionnaireabouttheirexperienceswiththeapplication.
Forbetatesting,theparticipantswereeachprovidedwithacopyofTaptoptotryintheirownstudiosetups.Overtheperiodofaweek,theycouldusetheapplicationinwhateverwaytheychose.Afterthistimehadelapsed,theparticipantssubmittedfeedbackonhowithadperformed.ThetwokeythingstoestablishfromthiswerethevariouswaysinwhichtheparticipantsutilisedTaptop,andiftheywouldthenconsideradoptingtheapplicationovertheirusualmethodofcontrolattheendofthetrialperiod.
Results
Inthefocusgroupquestionnaireresults,thetwomostfrequentanswersgivenbytheparticipantsforthefeaturesofTaptoptheylikedthemostwereitsself-contained,‘inthebox’format,anditsabilitytocontroltheplaybackvelocityofsamplesthroughaphysicalaction.TheconceptofamusicalinterfacebeingcompletelyintegratedintothesamedeviceasthesoundsourceprovedattractivetothoseparticipantswhomakeuseofhardwareMIDIcontrollers.Althoughmanufacturersputeffortintomakingsuchdevicesmoreportable,nothavingtoaccommodateoneatallwouldbedesirabletoamusicianwhohastotravelorwhohaslimitedstudiospace.ForthoseparticipantswhouseaDAWpianorolltocontroltheirvirtualinstruments,theinputofnotevelocitythroughgesturewasseenasamoreexpressivealternativetoprogrammingvelocityautomationwithinaDAW.
ThemostcommonfeatureofTaptopthattheparticipantsdislikedwasitsreliability.Iflightingwasnotatanoptimumlevelthentheapplicationhaddifficultyindetectingtheusersmovement,meaningthatsamplesweresometimesnottriggeredwhentheyweresupposedtobe.Sincetheenvironmentsthatelectronicmusicisperformedinaregenerallydimlylitwithsporadicburstsofverybrightlight,thisisasignificantproblemthatneedstobeaddressed.Anotherconcernraisedbytheparticipantswasthattappingalaptoptoohardinanareawheretheharddriveislocatedmayresultinthedrivebeingdamaged.Thisisespeciallyrelevantformagneticdiskdriveswhicharemoresusceptibletoimpactdamagethansolid-statestorage(Pinheiro,Weber,andBarroso2007).Ithasyettobeestablishedwhetherornotlongtermuseoftheapplicationhasanaffectontheharddriveofthelaptopthatitisrunon.
TheresultsofthebetatestingaredisplayedinTable1.Feedbackwasgenerallypositivewith3outofthe5participantsstatingthattheywouldconsiderusingTaptopinplaceoftheirusualmethodofcontrol.Participant2encounteredissueswithPureDatabeingunabletostreamvideofromthelaptopwebcam,thoughttobecausedbyacompatibilityproblembetweenthe‘pix_video’PureDataobjectandcertainversionsoftheMacOSXoperatingsystem.Forthisreason,theywereunabletogettheapplicationworkingintheirstudiosetupandthereforestatedthattheypreferredtheirusualmethodofcontroltoTaptop.
Table1.Resultsofbetatesting.FeedbackwasgenerallypositivewiththreecandidateswillingtoconsiderusingTaptoptoreplacetheircurrentcontrolmethod.*Participant2wasunabletorunthesystemduetoacompatibility
problemwithPureDataontheirlaptop.
Participant5alsostatedthattheypreferredtheirusualmethodofcontrol,aMIDIkeyboard,toTaptop.Whiletheythoughtthattheapplicationwasusefulforgeneratingideasonthemove,theyenjoyedthetactilefeedbackofkeysbeingdepressedwhenperformingonahardwaredevice.ItisinterestingtonotethatParticipant4usedTaptopinconjunctionwithsoftwaresynthesisersratherthanwithsoftwaredrummachinesandsamplers,aswasitsintendeduse.Theycommentedthattheapplicationprovidedamoreimmediatewayofinterfacingwithsynthesiserswhenauditioningpatchesthantheirusualmethodofcontrol,aDAWpianoroll.Participants1and3usedTaptopforitsoriginalpurpose,controllingdrummachineandsamplervirtualinstruments.TheybothstatedthattheypreferredTaptopovertheirusualhardwaresetupsduetotheeaseofportabilitythatitaffordedthem.
ConclusionWehavedemonstratedareal-timemethodofvirtualinstrumentcontrolthatusesonlythebuilt-inwebcamandmicrophonethatcomeasstandardwithmostlaptopcomputers.Thedevelopmentofthisapplicationsoughttoaddressanumberoftheissuesassociatedwithcurrentmethodsofvirtualinstrumentcontrol,particularlyconcerningsoftwaredrummachinesandsamplers.OneobjectivewastoreleasemusiciansfromtheburdenoftransportingandaccommodatinghardwareMIDIcontrollersthroughagesturecontrolsystemthatpossessedthesamefunctionality.Thiswasachievedbyexclusivelyutilisingthebuilt-infeaturesofalaptopcomputertocontrolthevirtualinstrument;theonlyexternalcomponentrequiredbeingasmallmirrorattachment.BybasingTaptop’scontrolmetaphoronabankofdrumpads,thefunctionalityoftheapplicationislargelyidenticaltoanalreadywellestablishedmusiccontrolinterface.
Anotherobjectivewastoexplorethepotentialofthecomputerasaperformanceinstrumentthroughthedevelopmentofanexpressivemethodofcontrolforvirtualinstruments.Normally,withouttheuseofanexternaldevice,musiciansarerestrictedtocomposingonacomputerviatheirDAWpianoroll.Sincethismethodofcontrolisincapableoffacilitatingreal-timeperformance,itisunderstandablewhymanymusicianshesitatetoclassthecomputerasamusicalinstrument(WesselandWright2002).However,withsuchawealthofmusicnowbeingmadeexclusivelyusingcomputers,itseemsstrangethatthisshouldbethecase.Taptopexpandsthepotentialofthecomputerasamusicalinstrumentbyenablingalaptoptobeusedforstandaloneperformance,wheretheaudiencecandirectlyrelatethephysicalgesturesoftheperformertothemusictheyarehearing(CicilianiandMojsysz2014).
FurtherworkneedstobedoneinimprovingTaptop’sreliability,particularlywhenbeingusedinadverselightingconditions.Ifthelightingistoobrightortoodark,themotiondetectionalgorithmislesslikelytobeabletodetectusermovement.Makingthemotiondetectionalgorithmadaptivetodifferentlightingconditionscouldbetheanswertothis,sothatindifficultenvironmentsthesensitivityisautomaticallyloweredincreasingthechancesofusermovementbeingdetected.
ReferencesBisby,Helena,AdamCooper,StephenJames,AlecRobertson,andKiaNg.2014.“ATactileAudioVisualInstrumentUsingSoundSourceLocalisation.”InProceedingsofthe2014ElectronicVisualisationandtheArtsConference,200–206.
Bown,Oliver,RenickBell,andAdamParkinson.2014.“ExaminingthePerceptionofLivenessandActivityinLaptopMusic:ListenerInferenceAboutWhatthePerformerIsDoingfromtheAudioAlone.”InProceedingsofthe2014InternationalConferenceonNewInterfacesforMusicalExpression,13–18.
Ciciliani,Marko,andZenonMojsysz.2014.“EvaluatingaMethodfortheAnalysisofPerformancePracticesinElectronicMusic.”InProceedingsofthe2014InternationalConferenceonLiveInterfaces,45–61.
Dobrian,Christopher.2001.“AestheticConsiderationsintheUseofVirtualMusicInstruments.”JournaloftheSocietyforElectro-AcousticMusicintheUnitedStates16(2):23–33.
Ellis,Jeremy.2011.“JeremyEllisPerformingonMaschine-UnlikeAnyOther.”https://www.youtube.com/watch?v=FcJCxe1VSLA[Accessed:21Feb.2016].
Fiebrink,Rebecca,GeWang,andPerryCook.2007.“Don’tForgettheLaptop:UsingNativeInputCapabilitiesforExpressiveMusicalControl.”InProceedingsofthe2007InternationalConferenceonNewInterfacesforMusicalExpression,164–67.
Gillian,Nicholas,andSarahNicholls.2012.“AGesturallyControlledImprovisationSystemforPiano.”InProceedingsofthe2012InternationalConferenceonLiveInterfaces.
Hong,DaeRyong,andWoonSeungYeo.2013.“Laptap:LaptopComputerasaMusicalInstrumentUsingAudioFeedback.”InProceedingsofthe2013InternationalConferenceonNewInterfacesforMusicalExpression,233–36.
Novello,Alberto,andAntonyRaijekoff.2015.“APrototypeforPitchedGesturalSonificationofSurfacesUsingTwoContactMicrophones.”InProceedingsofthe2015InternationalConferenceonNewInterfacesforMusicalExpression,170–73.
Pavlovic,Vladimir,ThomasHuang,andRajeevSharma.1997.“VisualInterpretationofHandGesturesforHuman-ComputerInteraction:AReview.”IEEETransactionsonPatternAnalysisandMachineIntelligence19(7):677–95.
Pinheiro,Eduardo,WolfWeber,andLuizBarroso.2007.“FailureTrendsinaLargeDiskDrivePopulation.”InProceedingsofthe5thUSENIXConferenceonFileandStorageTechnologies,17–18.
Puckette,Miller.1997.“PureData:RecentProgress.”InProceedingsoftheThirdIntercollegeComputerMusicFestival,1–4.
Theberge,Paul.1997.AnySoundYouCanImagine:MakingMusic,ConsumingTechnology.Hanover,NewHampshire:UniversityPressofNewEngland.
Wessel,David,andMatthewWright.2002.“ProblemsandProspectsforIntimateMusicalControlofComputers.”ComputerMusicJournal26(3):11–22.
Zamborlin,Bruno.2016.“Mogees.”http://mogees.co.uk[Accessed:28Apr.2016].
Zamborlin,Bruno,FredericBelivacqua,MarcoGillies,andMarkd’Iverno.2014.“FluidGestureInteractionDesign:ApplicationsofContinuousRecognitionfortheDesignofModernGesturalInterfaces.”ACMTransactionsonInteractiveIntelligentSystems3(4):30–45.