Confidential
Media Playback Flow
Javid Hsu
Confidential
Outline
MediaPlayer Subsystem
Related Files
MediaPlayer Frame of Playing Flow
─ StageFright and AwesomePlayer Relatin
─ AwesomePlayer Frame and Playing Flow
Simple Playing Implement
Confidential
Binder
MediaPlayer Subsystem
Java MediaPlayer
(frameworks/base/media/java/android/media)
libmedia_jni.so
JNI
libmedia.so
(frameworks/av/media/libmedia)
libmediaplayerservice.so
(frameworks/av/media/
libmediaplayerservice)
libstagefright.so
libstagefright_omx.so
libstagefright_soft_XXX.so
(codec)
(frameworks/av/media/
libstagefright/*)
(frameworks/base/media/jni)
libaudioflinger.so
(frameworks/av/services/
audioflinger)
libstagefrighthw.so
libOMXVideoEncoderXX.so
libOMXVideoDecoderXX.so
(vendor codec (hw codec))
Confidential
Related Files
libmedia_jni.so (frameworks/base/media/jni) libmedia.so (frameworks/av/media/libmedia)
Confidential
Related Files
libmediaplayerservice.so
(frameworks/av/media/libmediaplayerservice)
libstagefright.so
(frameworks/av/media/libstagefright)
Confidential
Related Files
libaudioflinger.so
(frameworks/av/services/audioflinger)
Confidential
MediaPlayer Frame of Playing Flow
Stagefright and AwesomePlayer Relation
Confidential
MediaPlayer Frame of Playing Flow
Ref: http://blog.csdn.net/tx3344/article/details/8084912
SurfaceFlinger
AudioFlinger
AwesomePlayer Frame and Playing Flow
Ref: http://blog.csdn.net/tx3344/article/details/8109185
Confidential
Simple Playing Implement
//For example:
//Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE); mp.prepare();
mp.start();
Ref: http://developer.android.com/reference/android/media/MediaPlayer.html
MediaPlayer State Diagram
Confidential
Simple Playing Implement
//For example: //Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();
Confidential
New MediaPlayer
Applications
Applications Framework
• Initiation and Setup
Media Player
Application
android_media_MediaPlayer MediaPlayer.java
android_media_MediaPlayer_native_init()
android_media_MediaPlayer_native_setup()
Static{ System.loadLibrary("media_jni"); native_init(); }
static void android_media_MediaPlayer_native_init (JNIEnv *env){ clazz = env->FindClass("android/media/MediaPlayer"); : fields.post_event = env->GetStaticMethodID(clazz, "postEventFromNative", "(Ljava/lang/Object;IIILjava/lang/Object;)V"); : fields.context = env->GetFieldID(clazz, "mNativeContext", "I"); : }
static void android_media_MediaPlayer_native_setup (JNIEnv *env, jobject thiz, jobject weak_this) { : sp<MediaPlayer> mp = new MediaPlayer(); // create new listener and give it to MediaPlayer sp<JNIMediaPlayerListener> listener = new JNIMediaPlayerListener(env, thiz, weak_this); mp->setListener(listener); // Stow our new C++ MediaPlayer in an opaque field in the Java object. setMediaPlayer(env, thiz, mp); }
static sp<MediaPlayer> setMediaPlayer (JNIEnv* env, jobject thiz, …) { Mutex::Autolock l(sLock); : env->SetIntField(thiz, fields.context, (int)player.get()); : }
Confidential
Simple Playing Implement
//For example: //Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();
Confidential
Libraries
(libmedia.so)
MediaPlayer player->setDataSource
IMediaPlayer
Media Player
Application
android_media_MediaPlayer
MediaPlayer.java
Applications
Applications Framework
MediaPlayerService::
Client
Media Set Data Source Flow (setDataSource)
SetDataSource
Using file type to choose
setDataSource_pre(player_type
playerType)…..
Using real player to set data source
setDataSource()
android_media_MediaPlayer_setDataSourceFD()
setDataSource()
status_t MediaPlayerService::Client::setDataSource(int fd, int64_t offset, int64_t length) { : player_type playerType = MediaPlayerFactory::getPlayerType(this, fd, offset, length); sp<MediaPlayerBase> p = setDataSource_pre(playerType); if (p == NULL) { return NO_INIT; } : // now set data source setDataSource_post(p, p->setDataSource(fd, offset, length)); return mStatus; }
Confidential
Media Set Data Source Flow (setDataSource)
Get Player Type
Create Player Instance
Player Set Data Source
Using file type to get player type
get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Confidential
Media Set Data Source Flow (setDataSource)
Using file type to get player type
get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Get Player Type
Create Player Instance
Player Set Data Source
Confidential
Libraries
Media Set Data Source Flow (setDataSource)
Using file type to get player type
MediaPlayerFactory.cpp
-factory: IFactory*
-sFactoryMap
NuPlayerFactory
+createPlayer(): return new NuPlayerDriver
SonivoxPlayerFactory
+createPlayer(): return new MidiFile
TestPlayerFactory
+createPlayer(): return new TestPlayerStub
StagefrightPlayerFactory
+createPlayer(): return new StagefrightPlayer
(libmedia.so)
MediaPlayer
player->setDataSource
MediaPlayerFactory
IMediaPlayer
Media Player
Application
android_media_MediaPlayer MediaPlayer.java
Applications
Applications Framework
MediaPlayerService::
Client
a. playerType:getPlayerType()
MediaPlayerFactory:
StagefrightPlayerFactory
createPlayer() return new StagefrightPlayer();
setDataSource()
android_media_MediaPlayer_setDataSourceFD()
setDataSource()
b. sp<MediaPlayerBase>: createPlayer(playerType)
a. getPlayerType:
1. sFactoryMap add all player factory while MediaPlayerService
started.
2. scoreFactory function using compare file
to get corresponding player type:
a. NuPlayerFactory
if(!strncasecmp("http://", url, 7)
|| !strncasecmp("https://", url, 8)
|| !strncasecmp("file://", url, 7))
b. SonivoxPlayerFactory
if (!strncasecmp(url + start, FILE_EXTS[i], len))
static const char* const FILE_EXTS[] = { ".mid",
".midi",
".smf",
".xmf",
".mxmf",
".imy",
".rtttl",
".rtx",
".ota" };
c. StagefrightPlayerFactory
1. compare raw data header if is equal to 0x5367674f
2. Default Player Type
Confidential
Media Set Data Source Flow (setDataSource)
Get Player Type Using file type to get player type
get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Get Player Type
Create Player Instance
Player Set Data Source
Confidential
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
Libraries
• Load software and hardware codec
StagefrightPlayer MediaPlayerFactory
new StagefrightPlayer()
connect()
OMXClient
OMX
OMXMaster
OMX()
mMaster(new OMXMaster)
Software Codec Hardware Codec
SoftOMXPlugin
AwesomePlayer
new AwesomePlayer()
libstagefrighthw.so
Confidential
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
Libraries StagefrightPlayer MediaPlayerFactory
new StagefrightPlayer()
RegisterDefaultSniffers()
AwesomePlayer
new AwesomePlayer()
RegisterSniffer_l(SnifferFunc func)
MPEG4Extractor.cpp
MP3Extractor.cpp
bool SniffMP3(...)
:
:
1. SniffMPEG4 <--> MPEG4Extractor
2. SniffMatroska <--> MatroskaExtractor
3. SniffOgg <--> OggExtractor
4. SniffWAV <--> WAVExtractor
5. SniffFLAC <--> FLACExtractor
6. SniffAMR <--> AMRExtractor
7. SniffMPEG2TS <--> MPEG2TSExtractor
8. SniffMP3 <--> MP3Extractor
9. SniffAAC <--> AACExtractor
10. SniffMPEG2PS <--> MPEG2PSExtractor
11. SniffWVM <--> WVMExtractor
12. SniffAVI <--> AVIExtractor
All Sniff register in gSniffers
(List<SnifferFunc> gSniffers)
bool SniffMPEG4(...)
typedef bool (*SnifferFunc)(
const sp<DataSource> &source, String8 *mimeType,
float *confidence, sp<AMessage> *meta);
DataSource
• Register default sniffers
FileSource
DataSource
- static List<SnifferFunc> gSniffers
+static sp<DataSource> CreateFromUR()
+static void RegisterDefaultSniffers()
+static void RegisterSniffer_l()
+typedef bool (*SnifferFunc)()
+bool sniff()
+bool getUInt16()
+bool getUInt64()
+virtual ssize_t readAt()
+virtual String8 getMIMEType() const()
HTTPBase NuCachedSource2
:
Confidential
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
Libraries StagefrightPlayer MediaPlayerFactory
new StagefrightPlayer() new AwesomePlayer()
AwesomePlayer
mVideoEvent = new AwesomeEvent(this, &AwesomePlayer::onVideoEvent); mStreamDoneEvent = new AwesomeEvent(this, &AwesomePlayer::onStreamDone); mBufferingEvent = new AwesomeEvent(this, &AwesomePlayer::onBufferingUpdate); mVideoLagEvent = new AwesomeEvent(this, &AwesomePlayer::onVideoLagUpdate); mThumbnailEvent = new AwesomeEvent(this, &AwesomePlayer::onThumbnailEvent);
• Create AwesomeEvents of onVideoEvent …
Ref: http://blog.csdn.net/tx3344/article/details/8053106
Confidential
Applications
Applications Framework
Libraries
• set Notify Callback (before new MediaPlayer()) : AP -> HAL
APP : setOn##Listener(m##Listener);
MediaPlayer.java : setOn##Listener(listener){mOn##Listener=listener}
Media Player
Application
android_media_MediaPlayer_native_init(){ : fields.post_event = env->GetStaticMethodID(clazz, "postEventFromNative", "(Ljava/lang/Object;IIILjava/lang/Object;)V"); : }
android_media_MediaPlayer MediaPlayer.java
Static{ System.loadLibrary("media_jni"); native_init(); }
Media Set Data Source Flow (setDataSource)
Confidential
android_media_MediaPlayer MediaPlayer.java
(libmedia.so)
MediaPlayer
android_media_MediaPlayer_native_setup()
• set Notify Callback (new MediaPlayer()) : HAL -> C++
Applications
Applications Framework
Libraries
Media Player
Application
new MediaPlayer()
setListener(const sp<MediaPlayerListener>& listener) {mListener = listener;}
listener = new JNIMediaPlayerListener(env, thiz, weak_this);
Media Set Data Source Flow (setDataSource)
Confidential
Media Player
Application
android_media_MediaPlayer MediaPlayer.java
MediaPlayerService (libmedia.so)
MediaPlayer
player->setDataSource
MediaPlayerService::
Client
IMediaPlayer
• set Notify Callback (setDataSource(…)) : client -> service
Applications
Applications Framework
sp<Client> c = new Client( const sp<MediaPlayerService>& service, pid_t pid, int32_t connId, const p<IMediaPlayerClient>& client, int audioSessionId, uid_t uid) { : mClient = client; : }
service->create(this, mAudioSessionId)
Libraries
setDataSource()
android_media_MediaPlayer_setDataSourceFD()
setDataSource()
Media Set Data Source Flow (setDataSource)
Confidential
Libraries
MediaPlayerFactory a. player_type:getPlayerType()
b. sp<MediaPlayerBase>:createPlayer(playerType, this, notify)
MediaPlayerService::
Client
• set Notify Callback (setDataSource(…)) : service -> player
StagefrightPlayer:public MediaPlayerInterface:
public MediaPlayerBase
void setNotifyCallback( void* cookie, notify_callback_f notifyFunc) { Mutex::Autolock autoLock(mNotifyLock); mCookie = cookie; mNotify = notifyFunc; } void sendEvent(int msg, int ext1=0, int ext2=0, const Parcel *obj=NULL) { Mutex::Autolock autoLock(mNotifyLock); if (mNotify) mNotify(mCookie, msg, ext1, ext2, obj); }
Media Set Data Source Flow (setDataSource)
Media Player
Application
Applications
android_media_MediaPlayer
Applications Framework
MediaPlayer.java
StagefrightPlayer AwesomePlayer
mPlayer(new AwesomePlayer)
mPlayer->setListener(this)
Confidential
Libraries
(libmedia.so)
MediaPlayer
IMediaPlayer
Media Player
Application
android_media_MediaPlayer::
JNIMediaPlayerListener MediaPlayer.java
Applications
Applications Framework
MediaPlayerService::
Client
• set Notify Callback (setDataSource(…)) : send message
AwesomePlayer.cpp
notifyListener_l(int msg, int ext1, int ext2) { : listener->sendEvent(msg, ext1, ext2); : }
StagefrightPlayer:
public MediaPlayerInterface:
public MediaPlayerBase
Notify(int msg, …)
sendEvent(int msg, …)
Notify(int msg, …)
Notify((int msg, …) postEventFromNative(int what, …)
• Media Event Type:
Media Set Data Source Flow (setDataSource)
Confidential
Media Set Data Source Flow (setDataSource)
Using file type to get player type
get real player instance
• Load software and hardware codec
• Register default sniffers
• Create AwesomeEvents of onVideoEvent, onStreamDone, onBufferingUpdate,
onVideoLagUpdate, onThumbnailEvent, onCheckAudioStatus and
onAudioTearDownEvent.
• setNotifyCallback
Using real player to set data source
• new FileSource(fd, …) insert to MediaExtractor::Create()
─ Sniff DataSource and get mine type
─ Get extractor with mime type
• Separate video (sp<MediaSource> mVideoTrack) and
audio (sp<MediaSource> mVideoSource) track
Get Player Type Get Player Type
Create Player Instance
Player Set Data Source
Confidential
Media Set Data Source Flow (setDataSource)
AwesomePlayer.cpp
setDataSource_l(dataSource)
MediaExtractor.cpp
Create(dataSource)
if (!strcasecmp(mime, MEDIA_MIMETYPE_CONTAINER_MPEG4) || !strcasecmp(mime, "audio/mp4")) { ret = new MPEG4Extractor(source); : : return ret;
DataSource.cpp
sniff(mimeType,…)
setDataSource_l(extractor)
extractor
Using Real Player Set Data Source • MediaExtractor::Create(const sp<DataSource> &source,...)
─ Sniff DataSource and get mine type
Read at filesource
Compabile ?
Write mine type
─ Get extractor with mime type
Confidential
Media Set Data Source Flow (setDataSource)
─ Sniff DataSource and get mine type
Read at filesource
Compabile ?
Write mine type
─ Get extractor with mime type Compabile ?
Read at filesource: Ex: Golden Flower mp4
Write mine type
Confidential
Media Set Data Source Flow (setDataSource)
Using Real Player Set Data Source • Separate video and audio track
Video track
Audio track
Confidential
Simple Playing Implement
//For example: //Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();
Confidential
AwesomePlayer.cpp
status_t prepareAsync(){
return prepareAsync_l(); }
status_t prepareAsync_l(){
new AsyncPrepareEvent;
postEvent();
return OK;
}
onPrepareAsyncEvent()
initVideoDecoder
initAudioDecoder
notifyVideoSize_l
notifyListener_l
AwesomePlayer.cpp
status_t prepare(){
prepareAsync_l(…)
condition.wait(Mutex)
}
status_t prepareAsync_l(){
:
:
}
onPrepareAsyncEvent()
initVideoDecoder initAudioDecoder
condition.broadcast();
• prepare • prepareAsync
prepare V.S. prepareAsync (AwesomePlayer)
Media Set Data Source Flow (setDataSource)
Confidential
Libraries MediaPlayerService::
Client
(libmedia.so)
MediaPlayer
IMediaPlayer
Media Player
Application
android_media_MediaPlayer MediaPlayer.java
Applications
Applications Framework
Media Set Data Source Flow (setDataSource)
prepareAsync
prepareAsync()
android_media_MediaPlayer_prepareAsync()
prepareAsync()
mPlayer->prepareAsync()
StagefrightPlayer
prepareAsync() prepareAsync()
AwesomePlayer
AwesomePlayer::
initVideoDecoder() AwesomePlayer::
initAudioDecoder()
TimedEventQueue::
start()
Confidential
Media Set Data Source Flow (setDataSource)
initVideoDecoder • How to choose codec
AwesomePlayer
mVideoSource
initVideoDecoder()
findMatchingCodecs(mime, componentName, …)
MediaCodecList : cat /etc/media_codecs.xml
OMXCodec
create(…MetaData, MediaSource, …)
makeComponentInstance(name, callbacks…)
Load the corresponding module and register callback functions. Callback Function: OMX_CALLBACKTYPE OMXNodeInstance::kCallbacks = { &OnEvent, &OnEmptyBufferDone, &OnFillBufferDone };
libstagefrighthw.so
Confidential
Media Set Data Source Flow (setDataSource)
Video Playback Flow
Ref: http://www.cnblogs.com/shakin/p/4729534.html
Confidential
Simple Playing Implement
//For example: //Simple Playing Implement:
MediaPlayer mp = new MediaPlayer();
mp.setDataSource(PATH_TO_FILE);
mp.prepareAsync();
mp.start();
Confidential
Libraries MediaPlayerService::
Client
(libmedia.so)
MediaPlayer
IMediaPlayer
Media Player
Application
android_media_MediaPlayer MediaPlayer.java
Applications
Applications Framework
MediaPlay Start
Start()
start() android_media_MediaPlayer_start()
start()
mPlayer->start ()
StagefrightPlayer
start()
play()
AwesomePlayer AudioPlayer start()
postVideoEvent_l()
AwesomePlayer::
onVideoEvent()
Confidential
Video Audio Synchronization
Playing point
Audio Samples
Video Samples
Start Point
The Time in AudioPlayer Real Playing
The Time in Audi Real Playing out
Playback Time Line:
Playing point
latenessUs (VideoPlayer Lateness Time)
Confidential
The End