Audio Engineering Society

Student DesignCompetitionPresented at the 141st Convention

2016 September 29 – October 2, Los Angeles, CA, USA

ADEPT: A Framework for Adaptive Digital Audio EffectsOwen Campbell1, Curtis Roads1, Andrés Cabrera1, Matthew Wright3, and Yon Visell2

1Media Arts and Technology , University of California Santa Barbara2Media Arts and Technology and Electrical and Computer Engineering , University of California Santa Barbara3Center for Computer Research in Music and Acoustics , Stanford University

Correspondence should be addressed to Owen Campbell (owencampbell@mat.ucsb.edu)

ABSTRACT

The field of music information retrieval (MIR) has enabled research in ‘intelligent’ audio processing. Emergingapplications of MIR techniques in music production might provide mixing engineers, musicians and composerscontrol over extant effects processing plugins based on the audio content of the recorded music. In adaptivedigital effects (ADAFx) processing, mapping functions are used to modulate algorithm parameters via featuresthat are extracted from the input audio signal or signals. We present an audio software plugin that is designedto facilitate feature-parameter mappings within digital audio workstations. We refer to it as the Adaptive DigitalEffects Processing Tool (ADEPT).

1 Introduction

The field of music information retrieval (MIR) has ledto the development of numerous signal processing algo-rithms for extracting semantic information from audiosignals. By repurposing some of these audio featureextraction algorithms, emerging applications of MIRtechniques in music production might provide mixingengineers, musicians and composers control over ex-tant effects processing and synthesis plugins driven bythe content of musical performances.

In adaptive effects processing, mapping functions areused to modulate algorithm parameters via features thatare extracted from the input audio signal or signals suchthat the effects operate in a signal-dependent way [1].There are many examples of adaptive effects, such asdynamic range processors, gates, and pitch-correctorsthat have long been mainstays of music production.

However, in principle any effect could become an adap-tive effect if one or more of its processing parametersettings could be dynamically driven by signal content.

For example, a vocalist could create a control mappingthat increases the amount of reverb as he or she singshigher and higher pitches, or the dissonance of elec-tric guitar music could be mapped to the intensity ortimbre of a distortion effect. These examples are just asmall subset of the possible mappings and novel adap-tive effects made possible by a flexible adaptive effectsframework.This document describes a proposed sys-tem for flexible ADAFx routing using existing musicproduction software tools.

2 Methods

As in [2] and [3], this work proposes an approach toadaptive effects processing in which existing audiosoftware tools can be augmented to facilitate arbitrary

Campbell et al. A Framework for Adaptive Digital Audio Effects

control mappings between the content of input audiosignals and the behavior of audio processing and syn-thesis algorithms. Modern digital audio workstations(DAWs) possess all the capabilities necessary to enablethe control mappings required for a generalized frame-work for adaptive effects processing. An abundance ofdiverse, high-quality effects processing and synthesisalgorithms in the form of audio plugins and support forinter-application communication for remote DAW con-trol are two critical components of the system. DAWslack only the feature extraction pipeline and mappingcapabilities required to complete the system.

The ADEPT adaptive audio processing software im-plements feature extraction and a mapping algorithmin order to facilitate ADAFx processing (Fig. 1). Asa result it can control any processing parameters thatare made available as automatable controls within theDAW. Developed using JUCE, a C++ framework whichenables cross-platform deployment of audio plugins, itcommunicates with Ableton Live using Open SoundControl [4] and the LiveOSC MIDI Remote Script .ADEPT works with the DAW and existing plugins, pro-viding immediate benefits due to the large variety andquality of available effects processing and synthesisalgorithms, and integration with production and perfor-mance practices.

Each instance of the ADEPT plugin computes a streamof audio features using Essentia [5], a C++ audio fea-ture extraction library. Audio features from a givensource may be mapped to any plugin parameter inthe DAW session, including those belonging to plu-gins residing on other tracks. At present, the feature-parameter mappings are one-to-many; a single featurestream may feed multiple mappings controlling differ-ent parameters. Initial experimentation with the systemsuggests that there is great potential in this capability,and there are far more possible mappings to investigatethan can be explicitly described in this document. Forany given effect or synthesis algorithm there are a num-ber of parameters which, when modulated in real time,produce output that sounds markedly different fromthe same algorithm with static settings. Some of thesepotential mappings will produce useful or interestingresults and others will not. Much of this work has beendevoted to discovering the former.

3 Results

To illustrate the capabilities of the ADEPT software, wedescribe two case examples of adaptive effects using

the software. Examples were tested using segments ofhigh-quality multi-track recordings from MedleyDB[6].

Case example 1: One of the simplest but most com-pelling examples is the mapping from vocal pitch tothe decay time of a reverb unit (Fig. 2, Fig. 3a). Asthe detected pitch [7] increases, so does the reverb’sdecay time, resulting in a very dry application of theeffect for the lowest notes of the singer’s range and avery wet, sustained reverb for the highest notes. Thisrelationship, though applied in a rather extreme mannerin this example, is useful for automatically avoidingmuddy low notes with too much reverb while applyinga desirable amount of reverb to higher notes.

ADEPTPlugin

ParameterAutomation

API

Digital Audio

FX Plugin

Audio In Audio OutUnaltered

Audio

Audio Feature DrivenControl

Messages

FX ParameterUpdates

Fig. 1: Each instance of the plugin computes audiofeatures a stream of audio features which canfeed mapping functions for controlling audioplugin behavior.

AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2Page 2 of 5

Campbell et al. A Framework for Adaptive Digital Audio EffectsO

utpu

t Aud

ioD

ecay

Tim

e P

aram

eter

Va

lues

Extra

cted

Pitc

h V

alue

s (H

z)In

put A

udio

(Analysis Frames)

(Analysis Frames)

(Samples)

(Samples)

(Seconds)

(Seconds)

Fig. 2: Example time series of audio feature and effect parameter data with mapping from vocal pitch to reverbdecay time.

AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2Page 3 of 5

Campbell et al. A Framework for Adaptive Digital Audio Effects

(a) View of the ADEPT plugin GUI with mapping from vocal pitch to reverb decay time.

(b) View of the ADEPT plugin GUI with mapping from guitar dissonance to distortion drive.

Fig. 3: Users can activate mappings between selected audio features and processing parameters (background) andaccess simple controls and visualization of mapping function behavior (foreground) via the plugin’s GUI.

AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2Page 4 of 5

Campbell et al. A Framework for Adaptive Digital Audio Effects

Case example 2: In another example, the dynamicapplication of a guitar distortion effect is based onthree different audio feature mappings. The primaryadaptive processing relationship maps higher detectedlevels of dissonance [8] to lower settings for the distor-tion unit’s ‘drive’ parameter (Fig. 3b) , thus applyingheavy distortion to power chords and single notes andmoderate distortion to more complex chords. In thenext mapping, the ‘tone’ parameter tracks the detectedbrightness value (as approximated by spectral centroid[9]). Finally, the detected loudness [10] modulates the‘dry/wet’ parameter to automatically make the distor-tion wetter at low levels and dryer at high levels tomanage the dynamic range of the effect’s output.

4 Discussion

The adaptive reverb effect in case example 1 abovedemonstrates that the introduction of even a single adap-tive control mapping to a familiar effect can have dra-matic results. By combining multiple such mappings,it is easy to turn static effects into highly dynamic ones,as in the adaptive distortion described in case example2 above. There are many other potential applications,among them several that have been implemented usingthe ADEPT system, including a cross-adaptive auto-mated panning scheme, an onset-ducking delay effect,and a peak-tracking resonant filter.

With more development time the ADEPT plugin willsupport logical comparison or arithmetic combinationof multiple feature streams for implementing many-to-many mappings, enabling the full taxonomy of adaptiveeffects outlined in [1]. Future work will also explorethe use of additional audio features, more complex map-ping paradigms, and extensions to the user interface.

5 Summary

We developed a conceptual framework for flexible adap-tive effects routing in the DAW environment and in-vestigated potential use cases for such a system. Asoftware implementation of this framework was cre-ated in the form of an audio plugin, where we exploredthe utility of a number of audio features and an interfacefor defining control mappings.

References

[1] Verfaille, V., Zolzer, U., and Arfib, D., “AdaptiveDigital Audio Effects (a-DAFx): A New Class ofSound Transformations,” IEEE Transactions onAudio, Speech and Language Processing, 14(5),pp. 1817–1831, 2006, ISSN 1558-7916, doi:10.1109/TSA.2005.858531.

[2] Stabile, M. E., Roads, C., Pope, S. T., Turk, M.,and Kuchera-Morin, J., “Adapt: A NetworkablePlug-in Host for Dynamic Creation of Real-TimeAdaptive Digital Audio Effects,” 2010.

[3] Brandtsegg, O., “A Toolkit for Experimentationwith Signal Interaction,” in Proceedings of the18th International Conference on Digital AudioEffects (DAFx-15), pp. 42–48, 2015.

[4] Wright, M., Freed, A., and Momeni, A., “Open-sound Control: State of the Art 2003,” in Proceed-ings of the 2003 Conference on New Interfacesfor Musical Expression, pp. 153–160, NationalUniversity of Singapore, 2003.

[5] Bogdanov, D., Wack, N., Gómez, E., Gulati, S.,Herrera, P., Mayor, O., Roma, G., Salamon, J.,Zapata, J. R., and Serra, X., “Essentia: An AudioAnalysis Library for Music Information Retrieval.”in ISMIR, pp. 493–498, Citeseer, 2013.

[6] Bittner, R. M., Salamon, J., Tierney, M., Mauch,M., Cannam, C., and Bello, J. P., “MedleyDB: AMultitrack Dataset for Annotation-Intensive MIRResearch.” in ISMIR, pp. 155–160, 2014.

[7] Brossier, P. M., Automatic Annotation of MusicalAudio for Interactive Applications, Ph.D. thesis,Queen Mary, University of London, 2006.

[8] Plomp, R. and Levelt, W. J. M., “Tonal conso-nance and critical bandwidth,” The journal of theAcoustical Society of America, 38(4), pp. 548–560, 1965.

[9] Peeters, G., “{A large set of audio features forsound description (similarity and classification)in the CUIDADO project},” 2004.

[10] Vickers, E., “Automatic Long-Term Loudness andDynamics Matching,” in Audio Engineering Soci-ety Convention 111, Audio Engineering Society,2001.

AES 141st Convention, Los Angeles, CA, USA, 2016 September 29 – October 2Page 5 of 5