Android Audio & OpenSL

www.dsp-ip.comFast Forward Your Development

Android Audio with OpenSL

Yossi CohenDSP-IP

Fast Forward Your Development

Overview•Android Audio APIs•OpenSL Overview•OpenSL Android implementation (code)

2


Playing Audio on Android•MediaPlayer•SoundPool•AudioTrack / AudioRecord (OpenMAX)•OpenSL

3


MediaPlayer•Java Based, Rich API•Good for playing files and streams•Not Good for gaming sounds or audio

conferencing duo to very large delay

4


OpenMAX•OpenMAX, a Khronos multimedia

integration API exposes AudioTrack() and AudioRecord()

•Those functions enables the creation of “channels” which play or capture audio

•Those API enables audio effects like reverb or distortion

•This API also has a significant delay

5


OPENSL | ESOverview


OpenSL•OpenSL is a Khronos Open Sound Library•OpenSL ES is OpenSL for embedded

systems•Android 2.3 provides PARTIAL

implementation of the OpenSL ES More on the limitations of And

•More on the OpenSL ES functions supported by Android in this Article

7


OpenSL•Advantages

▫Enables Low level audio tweaking and control

▫Device independent between android phone (Android 2.3+)

•Disadvantages▫Doesn’t improve the long audio delay of

android phones

8


OpenSL Features•OpenSL supports the following

features•Android OpenSL

Implementation does not support most of those features

9

Doppler

MIDI messages

Preset Reverb

Equalizer PitchVolume

Buffer QueuesRate Metadata extraction

Environmental Reverb

Virtualization Stereo widening LED & Vibra

3D positioning


Android and OpenSL

10

Open SL

OpenSL ES

Android OpenSL ES


OpenSL, OpenMAX and Android•Base on Android logging we noticed that

OpenSL calls ALSA and OpenMAX API functions

11

Applications

Media silicon(audio HW, CPUs, DSPs)

OpenMax (AudioTrack / AudioRecord)ALSA


Profiles•OpenSL has three profiles:

▫Phone▫Music ▫Game

Game Music

PhoneBasic mobile phonesRing tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games

Game-centric mobile devicesAdvanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc.

Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support


OpenSL API and Architecture•OpenSL is a C API which implements a

COM like API •Each object must implements the

SLObjectItf Interface (like Iunknown in COM)

•OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL

•OpenSL is not a channel like architecture like OpenMAX AL

13


OpenSL Objects•After creation, object should be realized in

order to use them.

14


OpenSL Architecture•OpenSL uses a few objects

▫Engine – a singleton object which must be implemented so we could use OpenSL

▫AudioPlayer – enables audio playing▫OutputMixer – enable mixing and audio

effects (such as reverb) ▫AudioRecorder – enable audio capturing

from microphone. Since Android does NOT support OpenSL microphone enumeration & control functions so AudioRecorder is the only way to capture audio in OpenSL

15


CODING OPENSL IN ANDROID


Android OpenSL Sample•To use the sample, you should install the

Android NDK. •Download Android NDK from

http://developer.android.com/sdk/ndk/index.html no need to install just unzip

•The samples is under:▫Android-ndk-r5b/Samples/native-audio

17


Editing the project•Select “Create new

project” in eclipse select “android project” and follow the settings according to the screen capture:

18


Native Audio Overview•Native Audio is a very simple (too simple)

OpenSL sample.•It enables Half duplex utilization of the

OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer.

•What is missing?•Most voice applications will require Full

Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers.

19


Loopback sample•For RT loopback behavior we need to add:

▫Small buffers instead of 5 seconds buffers▫A two threaded application one for capture

and one for record•Since this presentation is not about how

to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI

•Import it from samples as we did for Native-Audio

20


Building an OpenSL application•Add the following include files:#include <SLES/OpenSLES.h>#include <SLES/OpenSLES_Android.h>Add to the makefile

LOCAL_LDLIBS += -lOpenSLES

21


Add Audio Premissions•Add the following user permissions for

Audio capture and play in the Manifest permission tab:

•Internet is required for sending / receiving the audio packets

22


Application thread•We will implement the recorder in the

application thread.•Recorder is create from the objects in the

diagram, each object is created() and Realized()

23


Code Example – Create Engine// create engine__android_log_print(ANDROID_LOG_INFO, "NativeAudio", "initEngine");

SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE,

(SLuint32) SL_BOOLEAN_TRUE};result = slCreateEngine( &engineObject, 1, EngineOption, 0, NULL, NULL);__android_log_print(ANDROID_LOG_INFO, "NativeAudio", "create engine result %d", result);assert(SL_RESULT_SUCCESS == result);// realize the engineresult = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);assert(SL_RESULT_SUCCESS == result);// get the engine interface, which is needed in order to create other objectsresult = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);assert(SL_RESULT_SUCCESS == result);__android_log_print(ANDROID_LOG_INFO, "NativeAudio", "get engine result %d", result);

24


Play Thread creation

result = pthread_create( &audioThread, NULL, AudioInOutThread, NULL);

25

•After creating the recorder like in the native-Audio sample we will create the player thread

void *AudioInOutThread( void *ptr ){SLresult result;initPlayer();// set the player's state to playingresult = (*bqPlayerPlay)->SetPlayState(bqPlayerPlay,

SL_PLAYSTATE_PLAYING);assert(SL_RESULT_SUCCESS == result);}


Player CallBack Function

void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context){

int playershift = playFrame*RECORDER_FRAME; SLresult result; // enqueue another buffer result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue,

&recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short));}

26

•We will enqueue the next sample in the callback (need to verify there is a next sample)

•Recorder Enqueue is the same but also preform the first enqueue for the player to start it


Player Thread•The player implements the following

objects

27


Resources•WiseAndroid Blog•Mobile Pearls Blog•MindTheRobot Blog

28


DSP-IP Contact informationDownload slides at: Download slides at: www.dsp-ip.comwww.dsp-ip.com

For course & Development consulting For course & Development consulting contactcontact

www.dsp-ip.com www.dsp-ip.com Mail : [email protected] Mail : [email protected] Phone: Phone: +972-545-313092+972-545-313092Fax : +972-50- 8962910Fax : +972-50- 8962910

Yossi CohenYossi [email protected] [email protected] +972-545-313092+972-545-313092

Date post:	15-Jan-2015
Category:	Technology
Upload:	yossi-cohen
View:	23,112 times
Download:	9 times

Android Audio & OpenSL

Technology