Date post: | 13-Jul-2015 |
Category: |
Technology |
Upload: | alive-kuo |
View: | 222 times |
Download: | 2 times |
Web Developer or OS Developer?
Experience from FirefoxOS
Alive COSCUP 2013
me @alivedise http://alivedise.github.io http://github.com/alivedise [email protected] [email protected] ~2.5year Frontend Exp.
聽說COSCUP一定要放貓
Operating System
?
http://www.flickr.com/photos/klaikong/6041270516/
Operating System
FirefoxOS….OS?
Storage Management
fileSystem(yet)
DeviceStorageWatcher
Memory Management
Page Visibility
mozAppType
User Interface
Multi Window Management
Statusbar
System Overlay
App Overlay
Web Activity
Keyboard
Hardware Resource
Management
Wi-Fi
Radio
GPS
Bluetooth
Power
CPU WakeLock
Audio/Media
Multi-Tasking
mozBrowser remote=true
Card view
• Multi Channel • Audio Competing • Background Play
Audio Management
The beginning of everything..
http://bugzil.la/805333
805333
Why do we need multi- channel on OS?
Sounds within FxOS - <audio/>
Sounds within FxOS - radio
Sounds within FxOS - alarm
Sounds within FxOS - voice
The world of single channel..
Peace
But..
Too peaceful.
This means:
When you mute the device, your alarm also is muted. So..
Overslept ♪♪♪~
Zzzzz..
How does OS avoid that?
Different settings for audios on different purposes.
On the other hand..
Competing
On Desktop browser, we don’t need competing.
How about mobile OS?
Cases • one <audio/> in one app. • two <audio/> in one app. • two <audio/> in two apps. • one <audio/> and one not-audio-sound in two apps.
It’s all about UX.
User wants to listen music in background.. User has difficulty to mute the background app.. User needs the alarm to sound anyway..
Android solution: audio focus • http://developer.android.com/training/managing-audio/audio-focus.html
• App is responsible to request the audio focus on demand and release the focus actively.
Problems • How to request audio focus in web? • Volume control of different type of audio source. • Telephony is not using audio tag. • FM is not using audio tag. • …
Proposal • New attribute for media tags. • Background play and competing needs permission check.
• Expose current occupied media info to system.
Page Visibility • HTML5 Page Visibility • The page visibility state is simple on desktop browsers by toggling tabs.
• A little complicated on FirefoxOS • Window(App) has opening/closing transitions. • Screen off means page visibility is hidden, too. • Background page has higher possibility to be killed. (Memory Management!)
mozAudioChannelType
telephony alarm
notification content
normal(default)
Multi channel volume settings
• ringer, desktop-notification
audio.volume.notification
• music, FMRadio
audio.volume.content
• alarm
audio.volume.alarm
• telephony
audio.volume.voice
• Bluetooth SCO
audio.volume.bt_sco
Usage
OR
manifest.webapp
app.html
app.js
Interrupted Event
Current Competing Policies • Channel with higher priority interrupts the
lower channel. • Foreground page is always playable. But
interruption occurs or not depends on the channel of the media element.
• When screen is off, the current app or page is not brought to background now if it’s playing normal channel audio now. (Memory and Power concern!)
• Audio elements being interrupted could know by ‘mozinterruptbegin’ and ‘mozinterruptend’.
content� normal�
Gecko sidemozAudioChannelType
audio
App
nsIDOMHTMLAudioElement
nsIDOMHTMLMediaElement
nsHTMLMediaElement
nsIAudioChannelAgentCallback
nsIAudioChannelAgent
nsBuiltinDecoder
nsMediaOmxDecoder
nsMediaDecoder
DataSource
MediaStreamSource
MediaResource
- SetReadMode()
create
IOMX
OMX
MediaExtractorOMXCodec
MediaSource - Tell()- Seek()- Read()- GetLength()
hw codec
sw codec
MediaBuffer
- CanPlayChanged()
- "mozinterruptbegin" event- "mozinterruptend" event
OmxDecoder
nsHTMLAudioElement
nsIChannel
create
nsMediaOmxReader
nsBuiltinDecoderReader
create
create
MediaOmxStateMachine
nsBuiltinDecoderStateMachine
nsDecoderStateMachine
create
nsIThreadnsIThread
control
create
nsDocument
- NotifyOwnerDocumentActivityChanged()
FileMediaResource
ChannelMediaResource
nsAudioStream
nsNativeAudioStream
libsydneyaudio
AudioTrack
AudioFlinger
- Write()- Pause()- Resume()- IsPaused()- Drain()- GetMinWriteSize()- GetPosition()- SetVolume()
platform: Firefox OS(gonk)branch: Firefox OS 1.01 (b2g18)
will be replaced to libcubeb in v1.2
Audio Streams Diagram
https://github.com/sotaroikeda/firefox-diagrams
Since API is implemented..
World peace?
No…
Don’t forget the dependency tree.
Incoming Issues • Silent mode under multi channels • Background play ability of existing web pages • Background play and competing for video • Short playing sound is hard to adjust. • DTMF tones • Power consumptions. • Bluetooth earphone has its own profile. • …
Policy for Silent mode • Easy for single channel • Multi channels
• The default channel for volume rockers is ‘notification’. (Same as Android)
• Silent mode interesting channels • notification • content/normal
• Alarm is not affected by silent mode.
Policy for video element • <video> doesn’t need to play at background.
• Currently, use ‘visibilitychange’ event. • <video> by default shall be able to compete the background playing content channel. • Separate background play and audio competing for video element.
Background play issues • Existing web pages: “Hey, I’m a music site!”
• grooveshark • youtube • …
• How to play when screen is off without adding mozAudioChannelType to each web site? • Depart screen off state from invisible page visibility state.
CE Certification • EN 50332-1 • Output level through earphone
should be under 85 dBA. • Prompt a warning when earphone is
plugged and the volume is greater than 85 dBA.
• The cool down time of CE warning popup is continuous 20 hours music listening.
OS level policy for CE • Only apply to content/normal channel. • Output power is hardware-dependent. Exactly, we don’t know the db value for specific devices. • The CE maximum value is customizable for different OEM via default settings.
After all..
UX logic in backend and frontend���
backend
frontend
Find out solutions for existing web sites without new API.���
• As a platform, it’s difficult to make decisions. • New API takes time to be mature.
The scope is undefined before you look into it. ���
To be honest, I wrote few code in these issues. But lots of online/offline discussions.
The position of Frontend
https://speakerdeck.com/josephj/f2e-the-keystone
• It doesn’t matter you are making a web site, a web app or an operating system.
• Frontend is the one connecting UX and Backend, and struggle for ideal design and real requirement.
Thanks! Not the end, to be continued..