Hard real-time constraints in the Digital MusicIndustry
Leonardo Gabrielli
Universita Politecnica delle Marche, Ancona, Italy
June, 26th, 2014 - RTS-LIKE - Pisa
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 1
3MediaLabs - what
3MediaLabs: Multimedia Information Processing
Made of two research sub-groups: A3LAB and SEMEDIA.LEADER: Prof. Francesco Piazza
A3LAB: DSP Algorithms and Adaptivesystems for Audio applications
Data Processing Approach
People: 2 Assistant Professors, 5PostDocs, 4 Phd Students
SEMEDIA: Semantic Web and Multime-dia
MetaData Processing Approach
People: 3 PostDocs
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 2
3MediaLabs - A3LAB
Research Topics
Computational Intelligence: Neural Networks, Extreme LearningMachine, Blind Signal Processing.
Speech and Music Processing : Speech Enhancement, Speech/SpeakerRecognition, Acoustic Echo and Feedback Cancellation,Speech-interfaced Systems for HMI, Music Signal Analysis-Synthesis.
Immersive Audio Rendering : Digital Equalization, Wave-field Synthesisand Analysis, Loudspeakers array, Digital effects, 3D Audio.
In-car Audio: Advanced Car Information System, Professional Audio.
Projects
Privately and publicly (EU, National, Regional) funded
More than 1.5 Million Euro in the last 5 years
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 3
Collaborations
Companies: Texas Instruments, Faital Spa, Leaff Engineering,VideoWorks, CRN Yacht, FBT, Radvision, and so on so forth.
Universities: Technical University of Munich, Aalto University, Universityof Illinois at Chicago, South China University of Technology, ImperialCollege (UK), Chinese Academy of Sciences and others.
Member of the Texas Instrument European University Program
Related Companies
Leaff Engineering : former Spin-off operating in the Digital Audio field.
Sensible Logic: former Spin-off operating in the Semantic Web andDigital Libraries field.
DowSee: brand-new Spin-off operating in the Energy Management andSmart Grid fields.
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 4
The New Lab
Big acoustic semi-anechoic chamber (11m x 9m x 6m, ISO3744, ISO3745, Ft100Hz)
Media-lab, Texas Instrument DSP lab, and more
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 5
The New Lab
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 6
Speaker’s Bio
LEONARDO GABRIELLI
PhD candidate at UNIVPM:Physics modeling of ElectroacousticMusical InstrumentsMusic Technology: wirelessconnection and embedded platforms
Co-founder and CTO of DowSee srl:
Spinoff company of UNIVPMWireless Sensor Networks for theWater GridComputational Intelligence forDomestic Gas, Water andElectricity
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 7
Physically Informed Sound Synthesis
Analysis and Synthesis of Stringed Instruments
Virtual Acoustic Feedback: digital emulation of feedbacks betweensound sources and capture devices
e.g. guitar - loudspeaker feedbackcan be used to augment any kind of pitched instrument
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 8
WeMUST
A tool for live music at home / studioslow-latency, uncompressed audiocan be integrated into existingtechnologies (e.g. PCs, tablets)built on existing technologieseasy configuration and connectionaudio + MIDI + OSC...
Emphasis is put on embedded devices
Applications
Live: rehearsal studios
Laptop orchestras
Art installations and exhibitions
Currently we are improving reliability and studying techniques foroutdoor/stage use
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 9
Some history
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 9
40 years of Digital Music Industry
Prophet 5 (1978)
Arch: MCU controlling a full Analog signal path
5-voices polyphony
MCU: (Zilog Z80)
VCO, VCF: SSM (Rev.1,2), Curtis CEM (Rev.3)
6000 units sold
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 10
40 years of Digital Music Industry
Fairlight CMI (1979)
Sampling system
dual Motorola 6800
Memory: 16 kB per voice, System: 64 kB, Video: 16 kB (512x256pixels)
street price: 28,000£!!
Compare: KORG M-1 (1988): cost 1500£, 250,000 units sold
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 11
40 years of Digital Music Industry
MIDI (Musical Instrument Digital Interface) (1981)
Current-loop UART at 31250 baud
3-byte control messages
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 12
40 years of Digital Music Industry
Yamaha DX7 (1983)
Digital FM synthesis, 6 operators
Patches, Storage disk
early MIDI adopter
200,000 units sold
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 13
40 years of Digital Music Industry
Computer Music SW
Max Mathews’ MUSIC (1957), computer language
Miller Puckette’s Max (mid-1980s): visual programming language
...
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 14
40 years of Digital Music Industry
Virtual Analog
DSP to emulate analog waveshapes: Virtual Analog
Motorola 56002 DSP (80MHz, 40MIPS, 24bit fixed point)
Motorola 68331 MCU
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 15
RECAP: HW
Transition era: late 70s to the 80s:MCUs control a full Analog/Discrete Digital signal path
1990s: DSP era (TI, AD, Motorola)Sample-wise, Assembly, Fixed-pointEarly integration with MCU cores (e.g. TMS320C5x + ARM7)
2000s: PCs get to grips with real musicIntel SSE (1999) enhances FP calculus in x86 CPUsMany contemporary music software houses take offNote: Atari/Apple music SW since the late 80s
2000s: Peak point in discrete DSPs usage:Floating+Fixed point, 100MHz-1GHz, C coding, frame-wiseApplication processors: DSP + MCU
Note: ARM attempts successfully at Sig.Proc in 2005 with NEON
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 16
Current HW /2
2008: begin of the smartphone eraARM cores everywhere!
Consequences:ARM dominating in handheldsx86 dominating in PCsLast years: end of the all-SW era: musicians want tangible interfaceshybridization between interface (control) and engine (synthesis):
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 17
40 years of Digital Music Industry
Tux is playin’ (and payin’)
Viscount UNICO Organs (Italy, 2004)
KORG Oasys (2005): Intel Pentium 4 and a ”customized Linux”
KORG Triton (2011): Intel Atom and a ”custom OS over Linux”
Viscount Physis piano (Italy, 2011)
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 18
40 years of Digital Music Industry
Physis Piano (2011)
6 TI OMAP L137 (C67x + ARM9) + ARM9 master MCU
Master core runs Debian
Kernel drivers contributions
And latest news: Japanese manufacturer to release a low-end workstationbased on all-ARM processing?
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 19
all-ARM musical instruments
converging to standardized HW platforms
no SW convergence
side note: Android audio API are lame!
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 20
Constraints
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 20
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 21
Technical Constraints and Issues
Response below perceptual latency (ON-LINE vs REAL-TIME!)
No dropouts
FLOPS!
Robust against EM noise and surge
Gig-proof Build quality
User Requirements
Aesthetics
Sound Quality
Flexibility
User-friendliness
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 22
Typical Team
2 HW Eng
N SW Eng (2-3x Algo, 1x Control, 1x DSP-Sys Integration, 1x Linuxspecialist)
1-3 ”Musicians”
1 Tech Officer
Manufacturing staff
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 23
Tech stuff
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 23
ARM-based systems and GNU/Linux
ARM cores are not designed for DSP, but:leveraging SDKs reduces time to marketSW libraries include optimized DSP routinesReference designs facilitate HW designEVM boards allow parallel HW/SW developmentDSP programming headaches avoided (parallelism, mem sharing, ...)
Viable solutions for ARM development:
eCos
freeRTOS
QNX
Windows CE
Linux
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 24
Current SW stack for audio processing
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 25
Audio servers
JACK is *the* solution
Pulseaudio resamples everything (no synch, adds overhead)
ALSA can only give exclusive control
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 26
Latency: HW and SW
With JACK, latency is HW-dependent only
synchronous processing
Time unit = 1 period p of 2i samples
Total input-output latency:
Lio = (Ni +No)× pFs
+AL
with Ni, No number of buffers (min:2, ping-pong)
AL is the algorithmic latency (0 if no components with memory)
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 27
Scheduling Jack
Jack currently runs as a SCHED FIFO thread plus some SCHED OTHER
threads.
Jack clients run as a DSP SCHED FIFO thread plus SCHED OTHER threads
every Jack client gets a RT priority lower than that of Jack
so Jack can preempt clients breaking the deadline
when a client does not finish within the next IRQ: XRUN!
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 28
Scheduling Jack
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 29
Scheduling Jack
SCHED DEADLINE is aware of periodicity and CPU bandwidth
Can SCHED DEADLINE yield better performance than SCHED FIFO?
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 30
Previous Works
Cucinotta et al. ”Low-latency Audio on Linux by Means of Real-TimeScheduling”, LAC2011
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 31
Scheduling ALSA
Remove the need for:
synchrony
interconnection
extra features of JACK
Take exclusive control of ALSA and have a single audio process running on thesystem.Periodicity can be gained with SCHED DEADLINE
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 32
References
Cucinotta, Faggioli, and Bagnoli. ”Low-latency audio on linux by meansof real-time scheduling.” In Proceedings of the Linux Audio Conference(LAC 2011)
Gabrielli, Leonardo, Stefano Squartini, Emanuele Principi, and FrancescoPiazza. ”Networked beagleboards for wireless music applications.” InEducation and Research Conference (EDERC), 2012 5th European DSP,pp. 291-295. IEEE, 2012.
Gabrielli, Leonardo, Stefano Squartini, and Francesco Piazza.”Advancements and Performance analysis on the Wireless Music Studio(WeMUST) framework.” In Audio Engineering Society Convention 134.Audio Engineering Society, 2013.
June, 26th, 2014 - RTS-LIKE - Pisa, Gabrielli - UNIVPM 33