Skip to content

Lock-free audio IO with OpenSL ES on Android

October 29, 2012

As we saw in our previous discussion of OpenSL audio IO, this API operates, like a number of others, by employing a callback mechanism. Here, we have a very simple interface, where the callback is only used to notify the application that a new buffer can be enqueued (for playback or for recording). In other APIs, the callback also handles pointers to the audio buffers that are to be filled or consumed, but in OpenSL you have a choice just to use the callbacks to operate a signalling mechanism of some sort and keep all the processing in your audio processing thread. This would include enqueueing the required buffers, after the relevant signals are received.

The implementation we showed in the previous example uses thread locks to do this signalling, leading to what could be described as a blocking operation, which is kept synchronous with the notifications issued by the callbacks. This is one of the two ways to go about implementing realtime audio IO. Once upon a time, blocking interfaces were all there was. In the grand UNIX tradition, realtime audio APIs such as the SGI Irix Audio Library provided an open-read-write mechanism, similar to file streams, for the programmer. This was simple and worked relatively well. However, newer low-latency systems (e.g. CoreAudio, Jack, etc.) generally require a more responsive operation, and these now provide exclusively a non-blocking interface, based on callbacks running on specialised audio IO threads. To use them, we need to pay attention to the fact that data is shared between the callback and main process and make sure wires do not get crossed, and the operations occur in the correct order.

While it is possible to employ some types of locks to regulate the communication quite successfully (as noted in our previous implementation of audio IO with  OpenSL), one of the basic techniques of inter-thread data sharing is to use a lock-free circular buffer (called a ‘ringbuffer’ in other quarters). It would be interesting then to see how we could re-implement the audio IO part of our OpenSL example with such methods.

A circular buffer interface

The first thing to do then is to prepare a circular buffer interface that our code can use. We need four functions: to create a circular buffer, to write to it, to read from it, and to destroy it. The circular buffer itself will be a black box to our application:

circular_buffer* create_circular_buffer(int bytes);
int read_circular_buffer_bytes(circular_buffer *p, char *out, int bytes);
int write_circular_buffer_bytes(circular_buffer *p, const char *in, int bytes);
void free_circular_buffer (circular_buffer *p);

The way this is supposed to work is that the read operation will only read the number of requested bytes up to what has been written in the buffer already. The write function will only write the bytes for which there is space in the buffer. They will return a count of read/written bytes, which can be anything from zero to the requested number.

The consumer thread (the audio IO callback in the case of playback, or the audio processing thread in the case of recording) will read from the circular buffer and then do something with the audio. At the same time, asynchronously, the supplier thread will be filling the circular buffer, only stopping if it gets filled up. With an appropriate circular buffer size, the two threads will cooperate seamlessly.

Lock-free audio IO

With the above interface, we can write our audio IO functions and OpenSL callbacks to use it. Let’s take a look at the input side:

// this callback handler is called every time a buffer finishes recording
void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->inBufSamples*sizeof(short);
 write_circular_buffer_bytes(p->inrb, (char *) p->recBuffer,bytes);
 (*p->recorderBufferQueue)->Enqueue(p->recorderBufferQueue,p->recBuffer,bytes);
}
// gets a buffer of size samples from the device
int android_AudioIn(OPENSL_STREAM *p,float *buffer,int size){
 short *inBuffer;
 int i, bytes = size*sizeof(short);
 if(p == NULL || p->inBufSamples == 0) return 0;
 bytes = read_circular_buffer_bytes(p->inrb, (char *)p->inputBuffer,bytes);
 size = bytes/sizeof(short);
 for(i=0; i < size; i++){
 buffer[i] = (float) p->inputBuffer[i]*CONVMYFLT;
 }
 if(p->outchannels == 0) p->time += (double) size/(p->sr*p->inchannels);
 return size;
}

In the callback function (lines 2-8), which will be called every time a new full buffer (recBuffer) is ready, we write all of it to the circular buffer. Then the recBuffer is ready to be enqueued again (line 7). The audio processing function, lines 10 to 21, tries to read the requested number of bytes (line 14) into inputBuffer and then copies that number of samples to the output (converting it into float samples). The function reports the number of copied samples.

For output, we will do the reverse:

// puts a buffer of size samples to the device</pre>
int android_AudioOut(OPENSL_STREAM *p, float *buffer,int size){

short *outBuffer, *inBuffer;
int i, bytes = size*sizeof(short);
if(p == NULL || p->outBufSamples == 0) return 0;
for(i=0; i < size; i++){
p->outputBuffer[i] = (short) (buffer[i]*CONV16BIT);
}
bytes = write_circular_buffer_bytes(p->outrb, (char *) p->outputBuffer,bytes);
p->time += (double) size/(p->sr*p->outchannels);
return bytes/sizeof(short);
}

// this callback handler is called every time a buffer finishes playing
void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->outBufSamples*sizeof(short);
 read_circular_buffer_bytes(p->outrb, (char *) p->playBuffer,bytes);
 (*p->bqPlayerBufferQueue)->Enqueue(p->bqPlayerBufferQueue,p->playBuffer,bytes);
}

The audio processing function (lines 2-13) takes in a certain number of float samples, converts to shorts and then writes the full outputBuffer into the circular buffer, reporting the number of samples written. The OpenSL callback (lines 16-22) reads all of the samples and enqueues them.

For this to work properly, the number of samples read from the input needs to be passed along with the buffer to the output. Here’s the processing loop code that loops the input back into the output:


while(on)

samps = android_AudioIn(p,inbuffer,VECSAMPS_MONO);

for(i = 0, j=0; i < samps; i++, j+=2)
 outbuffer[j] = outbuffer[j+1] = inbuffer[i];
 android_AudioOut(p,outbuffer,samps*2);
 }

In that snippet, lines 5-6 loop over the read samples and copy them to the output channels. It is a stereo-out, mono-in setup, and for this reason we copy the input samples into two consecutive output buffer locations.

Also, now that the enqueueing is happening in the OpenSL threads, in order to start the callback mechanism, we need to enqueue a buffer for recording and another for playback after we start audio on the device. This will ensure the callback is issued when buffers need to be replaced.

Full code

Here is the full code for a lock-free OpenSL IO module; below you will see a simple C example utilising it, which is in fact very similar to our previous blogpost example. You can substitute the three files in the previous project for these, and the application should run as before.

/*
opensl_io.c:
Android OpenSL input/output module header
Copyright (c) 2012, Victor Lazzarini
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
 * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 * Neither the name of the <organization> nor the
 names of its contributors may be used to endorse or promote products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#ifndef OPENSL_IO
#define OPENSL_IO

#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>
#include <stdlib.h>
#include <unistd.h>
#ifdef __cplusplus
extern "C" {
#endif
typedef struct _circular_buffer {
 char *buffer;
 int wp;
 int rp;
 int size;
} circular_buffer;

typedef struct opensl_stream {

 // engine interfaces
 SLObjectItf engineObject;
 SLEngineItf engineEngine;

// output mix interfaces
 SLObjectItf outputMixObject;

// buffer queue player interfaces
 SLObjectItf bqPlayerObject;
 SLPlayItf bqPlayerPlay;
 SLAndroidSimpleBufferQueueItf bqPlayerBufferQueue;
 SLEffectSendItf bqPlayerEffectSend;

// recorder interfaces
 SLObjectItf recorderObject;
 SLRecordItf recorderRecord;
 SLAndroidSimpleBufferQueueItf recorderBufferQueue;

 // buffers
 short *outputBuffer;
 short *inputBuffer;
 short *recBuffer;
 short *playBuffer;

circular_buffer *outrb;
 circular_buffer *inrb;

// size of buffers
 int outBufSamples;
 int inBufSamples;

double time;
 int inchannels;
 int outchannels;
 int sr;

} OPENSL_STREAM;

/*
 Open the audio device with a given sampling rate (sr), input and output channels and IO buffer size
 in frames. Returns a handle to the OpenSL stream
 */
 OPENSL_STREAM* android_OpenAudioDevice(int sr, int inchannels, int outchannels, int bufferframes);
 /*
 Close the audio device
 */
 void android_CloseAudioDevice(OPENSL_STREAM *p);
 /*
 Read a buffer from the OpenSL stream *p, of size samples. Returns the number of samples read.
 */
 int android_AudioIn(OPENSL_STREAM *p, float *buffer,int size);
 /*
 Write a buffer to the OpenSL stream *p, of size samples. Returns the number of samples written.
 */
 int android_AudioOut(OPENSL_STREAM *p, float *buffer,int size);
 /*
 Get the current IO block time in seconds
 */
 double android_GetTimestamp(OPENSL_STREAM *p);

#ifdef __cplusplus
};
#endif

#endif // #ifndef OPENSL_IO

This is the opensl_io.c module:

</pre>
/*
 opensl_io.c:
 Android OpenSL input/output module
 Copyright (c) 2012, Victor Lazzarini
 All rights reserved.

Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 * Neither the name of the <organization> nor the
 names of its contributors may be used to endorse or promote products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
 DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#include "opensl_io2.h"
#define CONV16BIT 32768
#define CONVMYFLT (1./32768.)
#define GRAIN 4

static void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context);
static void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context);
circular_buffer* create_circular_buffer(int bytes);
int checkspace_circular_buffer(circular_buffer *p, int writeCheck);
int read_circular_buffer_bytes(circular_buffer *p, char *out, int bytes);
int write_circular_buffer_bytes(circular_buffer *p, const char *in, int bytes);
void free_circular_buffer (circular_buffer *p);

// creates the OpenSL ES audio engine
static SLresult openSLCreateEngine(OPENSL_STREAM *p)
{
 SLresult result;
 // create engine
 result = slCreateEngine(&(p->engineObject), 0, NULL, 0, NULL, NULL);
 if(result != SL_RESULT_SUCCESS) goto engine_end;

// realize the engine
 result = (*p->engineObject)->Realize(p->engineObject, SL_BOOLEAN_FALSE);
 if(result != SL_RESULT_SUCCESS) goto engine_end;

// get the engine interface, which is needed in order to create other objects
 result = (*p->engineObject)->GetInterface(p->engineObject, SL_IID_ENGINE, &(p->engineEngine));
 if(result != SL_RESULT_SUCCESS) goto engine_end;

engine_end:
 return result;
}

// opens the OpenSL ES device for output
static SLresult openSLPlayOpen(OPENSL_STREAM *p)
{
 SLresult result;
 SLuint32 sr = p->sr;
 SLuint32 channels = p->outchannels;

if(channels){
 // configure audio source
 SLDataLocator_AndroidSimpleBufferQueue loc_bufq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};

switch(sr){

case 8000:
 sr = SL_SAMPLINGRATE_8;
 break;
 case 11025:
 sr = SL_SAMPLINGRATE_11_025;
 break;
 case 16000:
 sr = SL_SAMPLINGRATE_16;
 break;
 case 22050:
 sr = SL_SAMPLINGRATE_22_05;
 break;
 case 24000:
 sr = SL_SAMPLINGRATE_24;
 break;
 case 32000:
 sr = SL_SAMPLINGRATE_32;
 break;
 case 44100:
 sr = SL_SAMPLINGRATE_44_1;
 break;
 case 48000:
 sr = SL_SAMPLINGRATE_48;
 break;
 case 64000:
 sr = SL_SAMPLINGRATE_64;
 break;
 case 88200:
 sr = SL_SAMPLINGRATE_88_2;
 break;
 case 96000:
 sr = SL_SAMPLINGRATE_96;
 break;
 case 192000:
 sr = SL_SAMPLINGRATE_192;
 break;
 default:
 return -1;
 }

 const SLInterfaceID ids[] = {SL_IID_VOLUME};
 const SLboolean req[] = {SL_BOOLEAN_FALSE};
 result = (*p->engineEngine)->CreateOutputMix(p->engineEngine, &(p->outputMixObject), 1, ids, req);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// realize the output mix
 result = (*p->outputMixObject)->Realize(p->outputMixObject, SL_BOOLEAN_FALSE);

 int speakers;
 if(channels > 1)
 speakers = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT;
 else speakers = SL_SPEAKER_FRONT_CENTER;
 SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM,channels, sr,
 SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16,
 speakers, SL_BYTEORDER_LITTLEENDIAN};

SLDataSource audioSrc = {&loc_bufq, &format_pcm};

// configure audio sink
 SLDataLocator_OutputMix loc_outmix = {SL_DATALOCATOR_OUTPUTMIX, p->outputMixObject};
 SLDataSink audioSnk = {&loc_outmix, NULL};

// create audio player
 const SLInterfaceID ids1[] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
 const SLboolean req1[] = {SL_BOOLEAN_TRUE};
 result = (*p->engineEngine)->CreateAudioPlayer(p->engineEngine, &(p->bqPlayerObject), &audioSrc, &audioSnk,
 1, ids1, req1);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// realize the player
 result = (*p->bqPlayerObject)->Realize(p->bqPlayerObject, SL_BOOLEAN_FALSE);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// get the play interface
 result = (*p->bqPlayerObject)->GetInterface(p->bqPlayerObject, SL_IID_PLAY, &(p->bqPlayerPlay));
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// get the buffer queue interface
 result = (*p->bqPlayerObject)->GetInterface(p->bqPlayerObject, SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
 &(p->bqPlayerBufferQueue));
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// register callback on the buffer queue
 result = (*p->bqPlayerBufferQueue)->RegisterCallback(p->bqPlayerBufferQueue, bqPlayerCallback, p);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// set the player's state to playing
 result = (*p->bqPlayerPlay)->SetPlayState(p->bqPlayerPlay, SL_PLAYSTATE_PLAYING);

if((p->playBuffer = (short *) calloc(p->outBufSamples, sizeof(short))) == NULL) {
 return -1;
 }

(*p->bqPlayerBufferQueue)->Enqueue(p->bqPlayerBufferQueue,
 p->playBuffer,p->outBufSamples*sizeof(short));

 end_openaudio:
 return result;
 }
 return SL_RESULT_SUCCESS;
}

// Open the OpenSL ES device for input
static SLresult openSLRecOpen(OPENSL_STREAM *p){

SLresult result;
 SLuint32 sr = p->sr;
 SLuint32 channels = p->inchannels;

if(channels){

switch(sr){

case 8000:
 sr = SL_SAMPLINGRATE_8;
 break;
 case 11025:
 sr = SL_SAMPLINGRATE_11_025;
 break;
 case 16000:
 sr = SL_SAMPLINGRATE_16;
 break;
 case 22050:
 sr = SL_SAMPLINGRATE_22_05;
 break;
 case 24000:
 sr = SL_SAMPLINGRATE_24;
 break;
 case 32000:
 sr = SL_SAMPLINGRATE_32;
 break;
 case 44100:
 sr = SL_SAMPLINGRATE_44_1;
 break;
 case 48000:
 sr = SL_SAMPLINGRATE_48;
 break;
 case 64000:
 sr = SL_SAMPLINGRATE_64;
 break;
 case 88200:
 sr = SL_SAMPLINGRATE_88_2;
 break;
 case 96000:
 sr = SL_SAMPLINGRATE_96;
 break;
 case 192000:
 sr = SL_SAMPLINGRATE_192;
 break;
 default:
 return -1;
 }

 // configure audio source
 SLDataLocator_IODevice loc_dev = {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT,
 SL_DEFAULTDEVICEID_AUDIOINPUT, NULL};
 SLDataSource audioSrc = {&loc_dev, NULL};

// configure audio sink
 int speakers;
 if(channels > 1)
 speakers = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT;
 else speakers = SL_SPEAKER_FRONT_CENTER;
 SLDataLocator_AndroidSimpleBufferQueue loc_bq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};
 SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM, channels, sr,
 SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16,
 speakers, SL_BYTEORDER_LITTLEENDIAN};
 SLDataSink audioSnk = {&loc_bq, &format_pcm};

// create audio recorder
 // (requires the RECORD_AUDIO permission)
 const SLInterfaceID id[1] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
 const SLboolean req[1] = {SL_BOOLEAN_TRUE};
 result = (*p->engineEngine)->CreateAudioRecorder(p->engineEngine, &(p->recorderObject), &audioSrc,
 &audioSnk, 1, id, req);
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

// realize the audio recorder
 result = (*p->recorderObject)->Realize(p->recorderObject, SL_BOOLEAN_FALSE);
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

// get the record interface
 result = (*p->recorderObject)->GetInterface(p->recorderObject, SL_IID_RECORD, &(p->recorderRecord));
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

 // get the buffer queue interface
 result = (*p->recorderObject)->GetInterface(p->recorderObject, SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
 &(p->recorderBufferQueue));
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

// register callback on the buffer queue
 result = (*p->recorderBufferQueue)->RegisterCallback(p->recorderBufferQueue, bqRecorderCallback,
 p);
 if (SL_RESULT_SUCCESS != result) goto end_recopen;
 result = (*p->recorderRecord)->SetRecordState(p->recorderRecord, SL_RECORDSTATE_RECORDING);

if((p->recBuffer = (short *) calloc(p->inBufSamples, sizeof(short))) == NULL) {
 return -1;
 }

(*p->recorderBufferQueue)->Enqueue(p->recorderBufferQueue,
 p->recBuffer, p->inBufSamples*sizeof(short));

 end_recopen:
 return result;
 }
 else return SL_RESULT_SUCCESS;
}

// close the OpenSL IO and destroy the audio engine
static void openSLDestroyEngine(OPENSL_STREAM *p){

// destroy buffer queue audio player object, and invalidate all associated interfaces
 if (p->bqPlayerObject != NULL) {
 SLuint32 state = SL_PLAYSTATE_PLAYING;
 (*p->bqPlayerPlay)->SetPlayState(p->bqPlayerPlay, SL_PLAYSTATE_STOPPED);
 while(state != SL_PLAYSTATE_STOPPED)
 (*p->bqPlayerPlay)->GetPlayState(p->bqPlayerPlay, &state);
 (*p->bqPlayerObject)->Destroy(p->bqPlayerObject);
 p->bqPlayerObject = NULL;
 p->bqPlayerPlay = NULL;
 p->bqPlayerBufferQueue = NULL;
 p->bqPlayerEffectSend = NULL;
 }

// destroy audio recorder object, and invalidate all associated interfaces
 if (p->recorderObject != NULL) {
 SLuint32 state = SL_PLAYSTATE_PLAYING;
 (*p->recorderRecord)->SetRecordState(p->recorderRecord, SL_RECORDSTATE_STOPPED);
 while(state != SL_RECORDSTATE_STOPPED)
 (*p->recorderRecord)->GetRecordState(p->recorderRecord, &state);
 (*p->recorderObject)->Destroy(p->recorderObject);
 p->recorderObject = NULL;
 p->recorderRecord = NULL;
 p->recorderBufferQueue = NULL;
 }

// destroy output mix object, and invalidate all associated interfaces
 if (p->outputMixObject != NULL) {
 (*p->outputMixObject)->Destroy(p->outputMixObject);
 p->outputMixObject = NULL;
 }

// destroy engine object, and invalidate all associated interfaces
 if (p->engineObject != NULL) {
 (*p->engineObject)->Destroy(p->engineObject);
 p->engineObject = NULL;
 p->engineEngine = NULL;
 }

}
// open the android audio device for input and/or output
OPENSL_STREAM *android_OpenAudioDevice(int sr, int inchannels, int outchannels, int bufferframes){

 OPENSL_STREAM *p;
 p = (OPENSL_STREAM *) malloc(sizeof(OPENSL_STREAM));
 memset(p, 0, sizeof(OPENSL_STREAM));
 p->inchannels = inchannels;
 p->outchannels = outchannels;
 p->sr = sr;

 if((p->outBufSamples = bufferframes*outchannels) != 0) {
 if((p->outputBuffer = (short *) calloc(p->outBufSamples, sizeof(short))) == NULL) {
 android_CloseAudioDevice(p);
 return NULL;
 }
 }

if((p->inBufSamples = bufferframes*inchannels) != 0){
 if((p->inputBuffer = (short *) calloc(p->inBufSamples, sizeof(short))) == NULL){
 android_CloseAudioDevice(p);
 return NULL;
 }
 }

if((p->outrb = create_circular_buffer(p->outBufSamples*sizeof(short)*4)) == NULL) {
 android_CloseAudioDevice(p);
 return NULL;
 }
 if((p->inrb = create_circular_buffer(p->outBufSamples*sizeof(short)*4)) == NULL) {
 android_CloseAudioDevice(p);
 return NULL;
 }

if(openSLCreateEngine(p) != SL_RESULT_SUCCESS) {
 android_CloseAudioDevice(p);
 return NULL;
 }

if(openSLRecOpen(p) != SL_RESULT_SUCCESS) {
 android_CloseAudioDevice(p);
 return NULL;
 }

if(openSLPlayOpen(p) != SL_RESULT_SUCCESS) {
 android_CloseAudioDevice(p);
 return NULL;
 }

p->time = 0.;
 return p;
}

// close the android audio device
void android_CloseAudioDevice(OPENSL_STREAM *p){

if (p == NULL)
 return;

openSLDestroyEngine(p);

if (p->outputBuffer != NULL) {
 free(p->outputBuffer);
 p->outputBuffer= NULL;
 }

if (p->inputBuffer != NULL) {
 free(p->inputBuffer);
 p->inputBuffer = NULL;
 }

if (p->playBuffer != NULL) {
 free(p->playBuffer);
 p->playBuffer = NULL;
 }

if (p->recBuffer != NULL) {
 free(p->recBuffer);
 p->recBuffer = NULL;
 }

free_circular_buffer(p->inrb);
 free_circular_buffer(p->outrb);

free(p);
}

// returns timestamp of the processed stream
double android_GetTimestamp(OPENSL_STREAM *p){
 return p->time;
}
// this callback handler is called every time a buffer finishes recording
void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->inBufSamples*sizeof(short);
 write_circular_buffer_bytes(p->inrb, (char *) p->recBuffer,bytes);
 (*p->recorderBufferQueue)->Enqueue(p->recorderBufferQueue,p->recBuffer,bytes);
}

// gets a buffer of size samples from the device
int android_AudioIn(OPENSL_STREAM *p,float *buffer,int size){
 short *inBuffer;
 int i, bytes = size*sizeof(short);
 if(p == NULL || p->inBufSamples == 0) return 0;
 bytes = read_circular_buffer_bytes(p->inrb,(char *)p->inputBuffer,bytes);
 size = bytes/sizeof(short);
 for(i=0; i < size; i++){
 buffer[i] = (float) p->inputBuffer[i]*CONVMYFLT;
 }
 if(p->outchannels == 0) p->time += (double) size/(p->sr*p->inchannels);
 return size;
}

// this callback handler is called every time a buffer finishes playing
void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->outBufSamples*sizeof(short);
 read_circular_buffer_bytes(p->outrb, (char *) p->playBuffer,bytes);
 (*p->bqPlayerBufferQueue)->Enqueue(p->bqPlayerBufferQueue,p->playBuffer,bytes);
}

// puts a buffer of size samples to the device
int android_AudioOut(OPENSL_STREAM *p, float *buffer,int size){

short *outBuffer, *inBuffer;
 int i, bytes = size*sizeof(short);
 if(p == NULL || p->outBufSamples == 0) return 0;
 for(i=0; i < size; i++){
 p->outputBuffer[i] = (short) (buffer[i]*CONV16BIT);
 }
 bytes = write_circular_buffer_bytes(p->outrb, (char *) p->outputBuffer,bytes);
 p->time += (double) size/(p->sr*p->outchannels);
 return bytes/sizeof(short);
}

circular_buffer* create_circular_buffer(int bytes){
 circular_buffer *p;
 if ((p = calloc(1, sizeof(circular_buffer))) == NULL) {
 return NULL;
 }
 p->size = bytes;
 p->wp = p->rp = 0;

 if ((p->buffer = calloc(bytes, sizeof(char))) == NULL) {
 free (p);
 return NULL;
 }
 return p;
}

int checkspace_circular_buffer(circular_buffer *p, int writeCheck){
 int wp = p->wp, rp = p->rp, size = p->size;
 if(writeCheck){
 if (wp > rp) return rp - wp + size - 1;
 else if (wp < rp) return rp - wp - 1;
 else return size - 1;
 }
 else {
 if (wp > rp) return wp - rp;
 else if (wp < rp) return wp - rp + size;
 else return 0;
 }
}

int read_circular_buffer_bytes(circular_buffer *p, char *out, int bytes){
 int remaining;
 int bytesread, size = p->size;
 int i=0, rp = p->rp;
 char *buffer = p->buffer;
 if ((remaining = checkspace_circular_buffer(p, 0)) == 0) {
 return 0;
 }
 bytesread = bytes > remaining ? remaining : bytes;
 for(i=0; i < bytesread; i++){
 out[i] = buffer[rp++];
 if(rp == size) rp = 0;
 }
 p->rp = rp;
 return bytesread;
}

int write_circular_buffer_bytes(circular_buffer *p, const char *in, int bytes){
 int remaining;
 int byteswrite, size = p->size;
 int i=0, wp = p->wp;
 char *buffer = p->buffer;
 if ((remaining = checkspace_circular_buffer(p, 1)) == 0) {
 return 0;
 }
 byteswrite = bytes > remaining ? remaining : bytes;
 for(i=0; i < byteswrite; i++){
 buffer[wp++] = in[i];
 if(wp == size) wp = 0;
 }
 p->wp = wp;
 return byteswrite;
}

void
free_circular_buffer (circular_buffer *p){
 if(p == NULL) return;
 free(p->buffer);
 free(p);
}
<pre>

This is a simple loopback example using the opensl_io.c module:


/*
opensl_example.c:
OpenSL example module
Copyright (c) 2012, Victor Lazzarini
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
 * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 * Neither the name of the <organization> nor the
 names of its contributors may be used to endorse or promote products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#include <android/log.h>
#include "opensl_io.h"

#define BUFFERFRAMES 1024
#define VECSAMPS_MONO 64
#define VECSAMPS_STEREO 128
#define SR 44100

static int on;
void start_process() {
 OPENSL_STREAM *p;
 int samps, i, j;
 float inbuffer[VECSAMPS_MONO], outbuffer[VECSAMPS_STEREO];
 p = android_OpenAudioDevice(SR,1,2,BUFFERFRAMES);
 if(p == NULL) return;
 on = 1;
 while(on) {
 samps = android_AudioIn(p,inbuffer,VECSAMPS_MONO);
 for(i = 0, j=0; i < samps; i++, j+=2)
 outbuffer[j] = outbuffer[j+1] = inbuffer[i];
 android_AudioOut(p,outbuffer,samps*2);
 }
 android_CloseAudioDevice(p);
}

void stop_process(){
 on = 0;
}

Advertisements

From → Android

22 Comments
  1. ksam permalink

    Hi Victor,

    Thanks to your valuable insights on the usage of the openSLES in android devices. As with your previous code, I was getting the loss in captured data. And using the circular buffers there is much reduction in the loss of the data. I modified your code into the class version, and while creating the instances for the capture and playback I am facing error as “slCreateEngine while another engine is active”. And while deleting the objects also it is giving error as “Destroy for engine ignored.” Any inputs on this is greatly helpful.

    Thank you in advance.

  2. Regarding loss in capture data, all I can say, without looking at the code, is to check buffer sizes. I’ve never experienced it.
    Regarding your modification, just check whether you are not somehow running the same code twice. That would explain the messages.

  3. scott permalink

    Hi Victor,

    I am wondering about this way of keeping track of time:

    p->time += (double) size/(p->sr*p->outchannels);

    (Have seen something similar in the Supercollider android audio driver)

    What happens if I’m doing something intensive directly in my audio callback, and it takes longer than it should (and drops out / glitches) – I’m presuming either a callback gets missed or they get pushed forward in time, and the time counter becomes inaccurate – not good if I’m using it to get events out of a queue fed from Java, as synthesis starts to lag increasingly behind the sequencing.

    Does the separate pthread to keep the callback topped up prevent this? In that, the callback will always happen on time – time will always move forward uniformly – and if anything glitches it will be the pthread doing all the synthesis work?

    Any pointers/thoughts appreciated

    • It really depends, I think, in what the time reference you want to take. If it is stream time, then the code will be accounting for it. If it is any other form, you are better off reading the system clock.

  4. scott micheli smith permalink

    i ended up using stream time and checking the cpu clock every 16 callbacks to see if it had fallen behind time, and if so, bumping the time forward. works alright.
    Thanks anyway, decent blog U have here.

  5. Chris permalink

    Hi Victor, When I run your android-audiotest application with these files instead of the locking io files, I get white noise out. I’ve tried just putting a sine wave out of the output and it still happens. I’m using API level 17. Do you know if there’s anything new here that could break your code?
    Thanks.

  6. Sam permalink

    Hi Victor, many thanks for your awesome code.. It works perfectly with the loopback test example you supplied on my Motorola Xoom. However when I try to just feed android_AudioOut with a sine wave buffer or other audio sample I just get corrupted audio out.. My hunch is there may be a bug in the circular buffer code somewhere because if I dramatically increase the circular buffer size the audio plays fine for a second or two. I plan on taking a closer look at the internals tonight to see if I can work out what wrong. Have you tried just pumping android_AudioOut with data other than microphone input? Cheers!

    • There could be possible a bug, I have to have a look. I have added a similar code to Csound and it works OK there with both full-duplex or single-direction audio.

  7. amaury permalink

    Hi Victor. Many thanks for sharing your openSL implementations. I have been comparing both thread-lock and circular buffer implementations in terms of CPU usage on my new Samsung Galaxy Ace 2 (800Mhz Dual core Armv7). In the test app I have an empty activity that launches audio processing. The input is copied to the output. I obtain the CPU usage figure by launching ‘adb shell top’ after the app is running. thread-lock implementation: 1.4% CPU, circular buffer implementation: 46% CPU. I am still trying to understand the figures (top may also fail to report correct measurements). Do you have an idea why there would be such differences in CPU usage?
    Best, Amaury

    • Well, that looks pretty bad, does it not? I would not think the code per se would cause more CPU use, because there is not really any more processing. I will see if I can take a look.

    • I think I know why this is happening. The processing thread spins as it is idle and uses up CPU. I’ll put it to sleep to save CPU cycles. It was an oversight of mine.

      • Just to close this discussion, to solve this issue, the calling code needs to be modified:

        #include
        #include “opensl_io.h”

        #define BUFFERFRAMES 2048
        #define VECSAMPS_MONO 64
        #define VECSAMPS_STEREO 128
        #define SR 44100
        #define MICROS 1000000

        static int on;
        void start_process() {
        OPENSL_STREAM *p;
        int samps, i, j;
        float inbuffer[VECSAMPS_MONO], outbuffer[VECSAMPS_STEREO];
        p = android_OpenAudioDevice(SR,1,2,BUFFERFRAMES);
        if(p == NULL) return;
        on = 1;
        while(on) {
        while((samps = android_AudioIn(p,inbuffer,VECSAMPS_MONO)) == 0)
        usleep(MICROS*VECSAMPS_MONO/SR);
        for(i = 0, j=0; i < samps; i++, j+=2)
        outbuffer[j] = outbuffer[j+1] = inbuffer[i];
        android_AudioOut(p,outbuffer,samps*2);
        }
        android_CloseAudioDevice(p);
        }

        void stop_process(){
        on = 0;
        }

  8. amaury permalink

    Thanks for the update! I am quite new to OpenSL and my attempts to fix it have not been successful so far.

  9. risteaandrei permalink

    I’m using OpenSL to play sounds on Android. I keep getting this Error : Encountered unknown AudioTrack event 5 for CAudioPlayer. According to android_AudioPlayer.cpp, this shouldn’t happen, since the value 5 (EVENT_BUFFER_END) is not treated as an error. I use Android 4.1.1.
    Does anyone know what the problem is?

    • This could be related to enqueueing a buffer before a notification is given by the callback. Is this stopping the audio altogether?

  10. risteaandrei permalink

    Thanks for the reply!
    No, there is no audio stopping, nor glitch. The sound is OK…
    I was wondering what I could do in order for the warning to disappear.

  11. I see that you are creating a new engine every time you create a queue that you can add sound data to. Do you have any other examples along the same general lines as this which could support possibly many different sounds being played at once? Thanks

  12. Note that the circular buffer implementation writes a variable from one thread, and reads it from another. This is not safe. If run under the clang thread sanitizer (TSAN), it would show as a data race, and has been pointed out extensively in the last year, no data race is benign/safe in C or C++, unless you use C++11’s atomic primitives.

    And no, volatile does not solve the issue. 🙂

    • You’re quite right there. I was aware that the circular buffer index gets updated from the two threads and that is open to thread-safety issues. I think I can solve it with gcc’s atomic intrinsics, and keep the code in C, for simplicity. Interesting enough, I know that a similar circular buffer code is used elsewhere in a heavily used audio pipeline code (the Jack Connection Kit), and it does not employ atomic operations, AFAIK.

      • The issue with TSAN bugs is that they’re compiler (and instance of compilation) dependent – the language definition allows the compiler to do things that will bite you if a) the compiler generates a particular spill/etc, and b) a thread scheduling event of the wrong sort happens to occur. So most “benign”-but-actually-TSAN-data-races will in practice be always safe or virtually so – but you can’t be sure the next compile (or build with a different compiler, etc) will be safe as well, unless you inspect the generated code.

        Atomics are the safe way to address this without explicit locks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: