WebRTCA front-end perspective
Opera SoftwareFronteers2014
@shwetank
About me
Shwetank DixitOpera Software
Fronteers2014
@shwetankDeveloper Relations, Opera Software
http://www.thehindu.com/news/national/other-states/using-pcos-in-mumbai-show-id-proof/article4241422.ece
Source: https://www.flickr.com/photos/rambow/858719954
Source: https://www.flickr.com/photos/rambow/858719954Source: https://www.flickr.com/photos/rambow/858717314
What is WebRTC?
Real Time Peer-to-Peer
Communication
but …So far I’ve seen very
few web people talking about it
Lack of Awareness
Not supported by MS
Not supported by Apple
Other
0% 5.5% 11% 16.5% 22%
Whats the biggest barrier to WebRTC adoption?*
* WebRTC World Outlook Survey 2014
WebRTC is for Web Developersbut …
Mediastream
Mediastream
RTCPeerConnection
Mediastream
RTCPeerConnection
RTCDataChannel
` `
MicrophoneDevice Camera
mediaStream
+
` `
MicrophoneDevice Camera
getUserMedia
+
Sqwiggle
gUM + CSS(Filters, Masks, Blend Modes)
Create interesting visual effects from camera output in real-time using CSS
gUM + Web Audio
Sound powered actions and navigation
World’s first webcam
http://en.wikipedia.org/wiki/Trojan_Room_coffee_pot
- Motion Detection (Baby Monitors, Security Cam) - Hotspots - Gesture Recognition - Emotion Recognition - Sound Detection and Visualisation - Voice / Sound Activated Controls - Capture video (Kind of) (whammy.js, gifshot)
What can you achieve with gUM?
gUM: Current way to do it
navigator.getUserMedia*
*with prefixes
gUM: Broken Error Handling
Exceptions are not caught by the current API. Promises can fix this.
Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Sep/0171.html
gUM: Current Way
var failure = function(reason){ log("Failed to show camera: " + reason.message); } !navigator.getUserMedia(constraints, function(stream) { try { $video.src = window.URL.createObjectURL(stream); $video.play(); } catch(e) { failure(e); } //failure must tolerate non-MediaStreamError }, failure);
Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Sep/0171.html
gUM: In the future ?
navigator.mediaDevices.getUserMedia
gUM: In the future ? Promises Proposal
navigator.mediaDevices.getUserMedia(constraints) .then( function(stream) { videoElement.src = window.URL.createObjectURL(stream); videoElement.play(); }).catch(function (reason){ log(“Failed to show camera: ” +reason.message); });
Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Sep/0171.html
Debate
This is still up for debate. Some are eager to just ship v1.0 and add support for promises in gUM at a later stage. !
Based on: http://lists.w3.org/Archives/Public/public-media-capture/2014Oct/0003.html
gUM: Other concerns
Impossible for a gUM app to use the device flash for low-light
!
Use (auto) zoom Use (auto) focus
`
gUM: Other concerns
“If you look further into this API then it doesn’t offer any advanced features
that you expect in a good camera app: focus, flash, zoom etc. It is clear that …(getUserMedia).. has been designed
for use in P2P applications and not as a dedicated camera experience.”
- Paul Kinlan
Mediastream Image-Capture
!
Will provide a deeper level of control for taking photos
!
w3.org/TR/image-capture/
`
Mediastream Image-Capture
White Balance ISO
Red-Eye Reduction Exposure
Brightness Contrast
Saturation Sharpness
Zoom takePhoto()
Focus? Flash? ¯\_(ツ)_/¯
Mediastream Image-Capture
Getting permission is not easy
Some users don’t seem to notice the permissions dialog
in certain browsers
Getting permission is not easy
Some users don’t seem to notice the permissions dialog
in certain browsers
Some users don’t seem to notice the permissions dialog
in certain browsers
Some users don’t seem to notice the permissions dialog
in certain browsers
Some users don’t seem to notice the permissions dialog
in certain browsers
`
Getting permission is not easy
Permissions dialogs are different on different platforms and browsers …
especially mobile
HTTPS helps …
If the app is on HTTPS, then the browser will remember the last
granted permission.
RTCPeerConnection
RTCPeerConnection
* May or may not wax a chump like a candle.
ICE - Interactive Connectivity Establishment
STUN
Whats my external IP Address?
Your external IP Address is x.x.x.x
TURN around…
Total eclipse of the (p2p) call …
TURN
Basically relays stuff from one party to the other.
Big Fat TURN Server
TURN
Basically relays stuff from one party to the other.
In this case, connection won’t be p2p, but at least the
connection will be made!
TURN
Basically relays stuff from one party to the other.
In this case, connection won’t be p2p, but at least the
connection will be made!
Generally speaking, a TURN server will include a STUN server too.
Free ICE Project
github.com/DamonOehlman/freeice(List of free STUN/TURN servers to use in your projects)
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Session Description (SDP)
v=0 o=- 680121471469462884 2 IN IP4 127.0.0.1 s=- t=0 0 a=group:BUNDLE audio video a=msid-semantic: WMS GUKF430Khp9jEQiPrdYe0LbTAALiNAKAIfl2 Audio Lines m=audio 54278 RTP/SAVPF 111 103 104 0 8 106 105 13 126 c=IN IP4 180.6.6.6 a=rtcp:54278 IN IP4 180.6.6.6 !ICE Candidates a=candidate:4022866446 1 udp 2113937151 192.168.0.197 36768 typ host generation 0 a=candidate:4022866446 2 udp 2113937151 192.168.0.197 36768 typ host generation 0 a=candidate:2706108158 1 tcp 1509957375 192.168.0.197 0 typ host generation 0 a=candidate:2706108158 2 tcp 1509957375 192.168.0.197 0 typ host generation 0 a=candidate:1853887674 1 udp 1845501695 46.2.2.2 36768 typ srflx raddr 192.168.0.197 rport 36768 generation 0 a=candidate:1853887674 2 udp 1845501695 46.2.2.2 36768 typ srflx raddr 192.168.0.197 rport 36768 generation 0 a=candidate:2157334355 1 udp 33562367 180.6.6.6 54278 typ relay raddr 46.2.2.2 rport 38135 generation 0 a=candidate:2157334355 2 udp 33562367 180.6.6.6 54278 typ relay raddr 46.2.2.2 rport 38135 generation 0
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Creates Offer
local description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Receives Offer
local description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Receives Offer
remote description
local description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Creates and sends Answer
local description
remote description
local description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Receives Answer
local description
remote description
local description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Receives Answer
local description
remote description
local description
remote description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
Connectivity Checks
local description
remote description
local description
remote description
The WebRTC Signalling Process
WebSocket || XHR || SIP || Carrier Pigeon
P2P Connection Established
Libraries
!
720p : 1-2 Mbps // VP8 at 30fps 360p : 0.5 - 1 Mbps // VP8 at 30fps 180p : 0.1 - 0.5 Mbps // VP8 at 30fps
Audio : .06 - 0.51 Mbps // Opus
Bandwidth Numbers
Debugging
opera://webrtc-internals chrome://webrtc-internals
about:webrtc !
The getStats() method !
callstats.io !
Data Channels
Once RTCPeerConnection is established, we can use Data Channels to transfer data P2P.
Data can be text or binary data
Whats the advantage?
Data Channels provide a * High Performance * P2P * Low Latency Connection to others
Working with Data
datachannel .onopen // Data Channel opened .onclose // Data Channel closed .onerror // Error occurred
.onmessage // New message arrived
.send() // Send a message
DataChannels can work exactly like WebSockets if you want, but they don’t
have to …
SCTP // Transport Protocol DTLS // Encryption
UDP
notable options
ordered send data packets in ordered fashion or not
!
maxRetransmitTime Max time to try to send a failed message
!
maxRetransmits Max number of tries to deliver failed message before it
gives up
In the wild
Peerflix PeerCDN WebTorrent
In the wild
Peerflix !
PeerCDN WebTorrent
@mafintosh
@feross
Two ModesIsWebRTCReadyYet.com
ORTC
- WebRTC 1.1 (ORTC) - SDP exchange not required - Advanced stuff like simulcast - More low level than WebRTC - Shim for WebRTC to work on top of ORTC - Important goal is compatibility. Existing
apps should not be affected
Communication