Web & SoundPlaying with Sound APIs recently introduced to the Web

Overview + Demo

New Sound APIs

◇ Web Audio (http://webaudio.github.io/web-audio-api/)

￭ Real-time audio synthesis￮ cf. SuperCollider, Pure Data, Max/MSP

￭ SFX (Pan, Delay, Filter, etc)◇ Web MIDI (http://webaudio.github.io/web-midi-api/)

￭ Real-time communication￮ Web ⇄ Musical Instruments

● Synths, DJ and lighting controllers◇ WebRTC (http://w3c.github.io/webrtc-pc/)

￭ Camera & Microphone

Web Audio Demo

◇ Web Audio Playground (http://webaudioplayground.appspot.com/)

￭ Straight-forward GUI wrapper◇ WebPd (https://github.com/sebpiq/WebPd)

￭ Pure Data clone◇ CoffeeCollider (https://github.com/mohayonao/CoffeeCollider)

￭ Coffee Script + SuperCollider◇ T’SS JS edition (https://github.com/toyoshim/tss)

￭ Chiptune sound emulation (demo, tsdsp)

MIDI is old, and new◇ 30th anniversary◇ MIDI is not only SMF but a protocol

￭ SMF: Standard MIDI File￮ Protocol dump file with timestamps

￭ Real-time communication - lighting and more￮ 冨田勲×初音ミク「イーハトーヴ」@ YouTube

◇ De facto for music production tools￭ Software Synth in DAW: Audio Unit, VSTi, DXi￭ Hardware: DIN, USB, FireWire, RTP, Bluetooth(*)

￮ (*): supported by Yosemite, Windows 10, Android M

Web MIDI Demo

◇ Physical controller for online sound systems￭ Web Audio Synth v2 (http://aikelab.net/websynthv2/)

￭ Web FM Synthesizer (http://www.taktech.org/takm/WebFMSynth/)

￭ Online DAW: audiotool (http://www.audiotool.com/)

￭ Online VJ: MajVj (GGT 2014 demo)

◇ Demo and Productions by YAMAHA￭ Pocket Miku (Product page & Web App)

￭ Web Music Platform (toyoshim’s fork)

￭ Reface + Soundmondo (Product page)￮ Social to share your sound and music via Web MIDI

WebRTC

◇ microphone￭ getUserMedia({audio:true}, …)

￮ Can connect to Web Audio via createMediaStreamSource()

◇ camera￭ getUserMedia({video: true}... )

￮ Can use it as a texture for WebGL

Web Apps be Secure!

◇ Hardware is hard to use securely￭ What happens on WebGL?

￮ Need GPU process and GPU command validator￭ Recent APIs to use hardware

￮ Permission prompt (infobar/bubble UI)￮ Secure origin, e.g. https or localhost, is mandatory

￭ Real P0 issue happened on Web MIDI￮ Microsoft GS Wavetable SW Synth crashes

● on receiving a special valid sequence● outside Chrome sandbox, inside the driver

How to learn

◇ Web Audio￭ Getting Started with Web Audio API html5rocks

￭ Web Audio API Ryoya Kawai

￭ Web Audio API 解説 g200kg

◇ Web MIDI￭ Web MIDI API (x-webmidi編) Ryoya Kawai

￭ ブラウザで電子楽器を作ってみよう HTML5 Experts

￭ 最高に乱暴なWeb MIDI API利用方法 tadfmac

Lesson: Web Audio 1

◇ Simple sine wave

var audio = new AudioContext();

var osc = audio.createOscillator();

osc.frequency.value = 440; // in Hz

osc.connect(audio.destination);

osc.start(0);

Lesson: Web Audio 2◇ Other nodes

￭ Audio source￮ AudioBufferSourceNode (from Audio resources, e.g., mp3, m4a)￮ MediaElementAudioSourceNode (from DOM media object)￮ MediaStreamSource (from microphone via getUserMedia)￮ ScriptProcessorNode (deprecated) => AudioWorkerNodeProcessor (from JS)￮ OscillatorNode (from various kinds of oscillators)

￭ Effect￮ GainNode￮ DelayNode￮ (Stereo)PannerNode / AudioListener￮ ConvolverNode (e.g., Vocoder, Reverb, and so on)￮ DynamicsCompressorNode / WaveShaperNode￮ BuquadFilterNode (e.g., LPF, HPF, BPF, and so on)

￭ Misc￮ AnalyserNode (FFT)￮ ChannelSplitterNode / ChannelMergerNode (Mixer)

Lesson: Web MIDI 1

◇ Device Query

var promise = navigator.requestMIDIAccess();

promise.then(function(midi) {

for (var input of midi.inputs.values())

console.log(input); // MIDIInput

for (var output of midi.outputs.values())

console.log(output); // MIDIOutput

});

Lesson: Web MIDI 2

◇ MIDIInput

for (var input of midi.inputs.values())

input.onmidimessage = handler;

function handler (e) {

// Single MIDI message in Array.

console.log(e.data);

}

Lesson: Web MIDI 3

◇ MIDIOutput

for (var output of midi.outputs.values()) {

// Send single MIDI message or multiple

// MIDI messages.

output.send([0x90, 0x40, 0x7f]); // Note ON

}

Lesson: Promise◇ New replacement for callbacks

foo(successCallback, errorCallback);↓

foo().then(successCallback, errorCallback);

foo().then(bar)

.then(baz)

.catch(qux);

◇ Chaining

Promise.all([foo(),

bar()])

.then(baz, qux);

◇ Wait multiple async functions

Lesson: Maplike◇ readonly maplike<DOMString, MIDIInput> inputs◇ readonly maplike<DOMString, MIDIOutput> outputs

◇ keys() … MIDIPort.id is used as an unique key◇ values() … MIDIInput / MIDIOutput

for (var input of inputs.values()) { // New syntax for ES6 iterable}

Lesson: Web MIDI 4

◇ Hotplug

midi.onstatechange = function (e) {

var newport = e.port;

if (newport.state == “connected”)

console.log(newport.name + “ is attached”;

else if (newport.state == “disconnected”)

console.log(newport.name + “ is detached”;

};

Fine ||

References

◇ Slide template￭ from Slides Carnival

￮ http://www.slidescarnival.com/imogen-free-presentation-template/399