Date post: | 12-Nov-2014 |
Category: |
Technology |
Upload: | vline |
View: | 6,440 times |
Download: | 2 times |
A Practical Guide to Building WebRTC Apps
Ben [email protected]://vline.com@vlineinc
vLine
10/09/2013
8/2011
My Background with WebRTC
Founded vLine in early 2011 to build tools and infrastructure for WebRTC
EDGE LOCATIONS ON 5 CONTINENTS
Global Infrastructure
Developer SDKs
Built what was probably the first WebRTC App
8/2013
vLine
vLine 3
Topics
1. Making Mobile Work Well
2. Choosing a Strategy for Multi-Party Conferencing
3. UI Considerations
vLine 4
Mobile: What is Possible?
[DEMO]
vLine 5
Mobile: The Basics
Android
• Works in Chrome 29+ and Firefox 24+
• Google Maintains Java PeerConnection APIs
• Works pretty well
iOS
• Not supported in any browser (thanks to Apple’s browser engine policies)
• Google Maintains Obj-C APIs
• Voice works well
• Video can be made to work well (this will get easier)
vLine 6
Android: Browser vs. Native
Browser Pros Native App Pros
• Re-use app built for desktop browsers.
• Supports the most powerful part of WebRTC: Links
• Push notifications!!!
• Other native APIs like address book and background services
• Easier to tune for particular devices
• Slightly better performance
vLine 7
Android: Browser vs Native
Our Answer: Both
• Make your web app work well on mobile, so that users without native app have low-friction experience
• Run the same app in PhoneGap with push notifications, address book integration, etc.
vLine 8
Mobile: Video Quality
• Mobile devices are less powerful than laptops and desktops(duh)
• Wide range of capabilities among mobile devices(smaller gap between high-end devices and low-end laptops than between low-end and high-end mobile devices)
• WebRTC methods for adapting to device capabilities don’t work well (yet)
vLine 9
Mobile: How WebRTC Adapts
Two Methods:
• Video Adapter reduces encode resolution if CPU is overloaded
• Remote Bitrate Estimator tells other side to reduce encode bitrate if decoder isn’t keeping up
Problems
• Only kick in after quality is visibly degraded
• Encoder and decoder compete for CPU, with unpredictable interactions
• Result is often bitrates of 50kbps on devices and networks that could support much better.
vLine 10
Mobile: Device Adaptation
The unfortunate reality
You’ll get much better results by imposing limits on resolution, framerate, and bitrate than by relying on the built-in adaptation methods.
vLine 11
Device Adaptation: The Knobs
Framerate and resolution are controlled by MediaConstraints passed to getUserMedia().function getStream(width, height, framerate) { var constraints = { video: { mandatory:{ minWidth:width, maxWidth:width, maxFramerate:framerate } } }; navigator.webkitGetUserMedia(constraints, onStream1, onFailure);}
Note: On Chrome, if another LocalMediaStream is active, the constraints will not take effect. Stop all other streams first!
vLine 12
Device Adaptation: The Knobs
Max bitrate and quantization set by adding VP8 fmtp parameters to SDP (Chrome only)
...a=rtpmap:100 VP8/9000a=fmtp:100 x-google-max-bitrate=700; x-google-max-quantization=20...
vLine 13
Mobile: Choosing Parameters
Two Strategies:
1. If at least one party is on mobile, set very low limits on resolution, framerate, and bitrate (e.g., 320x240 at 15fps and 400kbps)
2.Detect device capabilities, signal them to peers, and adapt encoding parameters appropriately
• Set encoding parameters to minimum of what the local device can encode and the remote device can decode
vLine 14
Mobile: Choosing Parameters
Detecting Device Capabilities
1. For native apps, Android and iOS APIs will give you fine-grained capabilities
2.For browser-based apps, run a javascript benchmark
• Run it once in a single web worker.
• Run it in parallel on 4 web workers.
• Infer CPU clock rate and number of cores.
3.Never encode at a higher resolution than the remote device’s screen!
15
Mass-Market TelepresenceWith HDMI outputs, 1080p-capable hardware codecs, and support for bluetooth conference mics, Android devices make the perfect embedded telepresence systems, starting at $100 per room.
A Mobile Bonus
HP Slate 21 - $400Tegra 4 processor and 21” screen. Perfect on a desktop or wall-mounted in a small conference room.
Galaxy NX - $1600Exnyos-5 Octa processor plus APS-C sensor. Mount it on a wall and plug in TV for a room system with broadcast-quality video.
HDMI Dongle - $50Rk3188 processor. Plug in a USB camera and a TV for a 1080p capable room system.
Note 10 - $500Exnyos-5 Octa processor. Put it on a conference table and plug in a wall-mounted TV and USB camera to power a larger room.
vLine 16
Multi-Party: Two Strategies
Mesh Topology Star Topology
vLine 17
Multi-Party: What is Possible?
[DEMO]
vLine 18
Multi-Party: Mesh Topology
Mesh Pros
• No server (simpler and cheaper infrastructure)
Mesh Cons
• More complex session management
• Only scales to about 4 participants
• Poor results on mobile devices (which are often hard-pressed to support a single peer)
vLine 19
Multi-Party: Star Topology
Star Pros
• Less processing and network load on clients
• Higher quality on low-end devices
• Potentially unlimited number of participants.
• Good place to record, do speaker detection, etc.
Star Cons
• Have to run lots of servers (complex and expensive)
• For fast machines on fast network connections, may be worse quality (mostly remedied with geo-distributed servers)
vLine 20
Multi-Party: Star Topology
How the Star Topology Works
• All clients connect to central server (usually called MCU or Media Router)
• Clients encode and send one stream. MCU fans it out.
vLine 21
Multi-Party: Star Topology
MCU/Router Strategies
• Mix/composite once and re-encode a single stream for each client (best quality. lowest load on client. very little flexibility on client. high load on server)
• Re-encode each stream for each client (high quality. lots of flexibility on client. more load on client. very high load on server)
• Restrict bandwidth, resolution, framerate, to lowest common denominator and just forward (low load, potentially poor quality)
• Advanced strategies: temporal scaling and simulcast
vLine 22
Multi-Party: Star Topology
Mixed approach is probably best
• Send two streams to server from each client (low and high res)
• For small differences, adapt to lowest-common denominator and forward.
• For large differences, re-encode.
• Use temporal scaling to fine-tune (if you control the encoder)
vLine 23
UI Considerations
For Video, Bigger is Better
• Use full window
• Encourage full-screen (especially on mobile)
• HD is best of all, but make sure devices on both ends can handle the CPU load, or you’ll get a worse experience.
vLine 24
UI Considerations
No one notices the GetUserMedia permissions UI!
• Show an arrow to point at it and don’t let them do anything else before accepting
• Be aware:• In Chrome, button positions vary by platform. • Firefox is different• On Android Phone devices, buttons are at bottom.
vLine 25
UI Considerations
Feedback for states and errors
• If you have a mute button, make it obvious when you're muted• Give the user feedback that you are trying to connect (or reconnect)
• If a session ends, distinguish between the other party hanging up and losing connection
• Use stats api to detect poor network conditions and provide feedback. User can then do something about it (e.g., move closer to wifi access point)
• If video bitrate stays at 50kbps, encourage (or force) user to switch to voice only