Skip to content

An update on the Android audio latency issue

With the advent of newer versions of Android (4.1, 4.2) and newer devices, the topic of latency has come a bit more into the mainstream of system development. Newer APIs in the Android SDK are providing means to enquire devices about low-latency parameters (buffer sizes and sampling rates).  But how does that translate in practical terms to latency reduction?

My tests with Csound

Well, I got hold of a Google Nexus 7 tab this week and went about measuring it. Using the latest Csound, whose OpenSL IO code is (I expect) optimal, I first measured touch-to-sound, and then round-trip latency. The first measurement was made by recording an audible mechanical ‘tap’ as I touched the screen and the sound coming out of the speakers. Granted, it is not as fine-grained as a video test, but it is not too far off the mark and would be particularly close to user experience as one tries to manipulate the UI. The second was done by using a microphone attached to a pair of headphones, which would pick up a click from its source and then from the tablet output.

Results were as follows:

  • best touch-to-sound latency: ~120ms
  • best round-trip latency: ~105ms

The parameters for Csound were -B1024 and -b128, which means, in this particular case, circular buffer and audio buffer sizes of 1024 and 128  frames, respectively. What I observed was that the size of the circular buffer was critical: anything less than about 1024 would bring either intermittent or constant dropouts, regardless of the audio buffer size. Also, smaller sizes would not translate to lower latencies (e.g.  low-latency but with dropouts), the lower limit never changed. This means that although the buffer sizes can be made very small, there will be considerable jitter in the OpenSL buffering and that the lower layers of buffering, which we do not have access to, are the ones ultimately responsible for the latency (we knew that, but this confirms it).

Although the round-trip figure is far away from low-latency, it is actually becoming more usable. When testing little processing apps (echos, reverb, flangers), actually it does not feel too bad. I would say it can be used in live-electronics settings that are not too strict in terms of live-processed synchronisation, and I plan to use it in a piano piece I have been asked to write, next year.

What is a bit disappointing is the touch latency, which appears to contribute more to the overall latency than the single-direction audio delay. This is an issue for performance systems using the tablet/phone touch interface, although it might be solved via external controllers  (but do we want to depend on these?).

Other reports

I have reported these measurements to the Android audio discussion list and was interested to hear other results posted there, which confirm my numbers in the case of the same hardware and Android version.

Amongst these reports, in particular, there was a meticulous measurement made with a video camera, graphics and audio. After a touch,  graphics and audio would be generated, so touch latency could be evaluated separately. This is another way to split the measurements, if you are not looking to measure round-trip delay. The hardware used was a low-cost tablet and Android 4.0. Results were as follows: touch-graphics delay was ~90ms; touch to audio, ~250ms. These measurements also line up quite well with my own experiences with the Galaxy Tab 10.1 and 4.0.

Conclusions

I can drawn some interesting conclusions from this, and as the movie goes, some are good, some are bad, and some are definitely ugly:

1. The good: things are improving and we have a situation now that for some applications, audio processing can be usable. They seem to be listening to us.

2. The bad: development is far too slow. Older devices with fine computing capabilities (like my Galaxy Tab 10.1) are being left behind.

3. The ugly: a comment in the Android audio list says it all, “I wished that those shiny tablet magazines did simple tests like this, and then published the results instead of irrelevant values like their thickness and number of processing cores”. In my opinion, with the plethora of magazines and online review sites that exist today, someone has to provide us with better information in this regard. As it is now, they are only patting hardware and software vendors on the back, helping to preserve the status quo.

Post-script

As an aside, I have noticed that in Android platform-14, the OpenSL ES headers are providing an alternative streaming audio interface, identified by  SL_IID_ANDROIDBUFFERQUEUESOURCE. Well, I thought, let’s give it a go, maybe this is a door for devices to support lower latency streams. It turns out that although it is a published interface, it has not actually found its way into the library binary, which is odd. Furthermore, in the release 8b of the NDK, there is a SL_IID_BUFFERQUEUE in the library (but that is not in the headers), which has now disappeared in the latest, 8c, release.

Anyway, the documents still seem to say that the Android Simple Buffer Queue interface is to be used, although this does not mean much, as the documentation can sometimes lag behind the released code. I have filed a ticket with Google regarding the absence of the buffer queue source interface, so it will be interesting to see how they respond to this.

On the wrapping of Android NDK code with SWIG

In this post, I would like to detail a bit more the use of SWIG to wrap the android NDK C or C++ code in Java so that apps can use it.  This simplifies the integration of native code, since the Java Native Interface is a bit more longwinded and trickier to use.

All we need to have at hand is:

  1. The header file(s) containing the functions, data structures (if any) and/or classes (if in C++) that we want to wrap and expose to Java.
  2. An interface file for SWIG to find the relevant headers and wrapped them.

With this SWIG will generate the following

  1. A C or C++ wrapper code that will act as the interface between your C or C++code and Java. This, together with your original C or C++ code, will have to be compiled as a dynamic loadable module, the JNI library.
  2. Java classes representing your C code, which you will use in your app.

So let’s look at an example from this blog, a basic echo effect. In it, we have decided to have a minimal exposure to Java, so here’s the header (echo.c):


#ifdef __cplusplus
extern "C" {
#endif
 void main_process(float delay, float fdb);
 void stop_process();
#ifdef __cplusplus
};
#endif

Two functions, one to start the process and another one to close it. Now we have to define our interface file (echo_interface.i):


%module echo
%{
#include "echo.h"
%}

// Enable the JNI class to load the required native library.
%pragma(java) jniclasscode=%{
 static {
 try {
 java.lang.System.loadLibrary("echo");
 } catch (UnsatisfiedLinkError e) {
 java.lang.System.err.println("native code library failed to load.\n" + e);
 java.lang.System.exit(1);
 }
 }
%}

%include "echo.h"

The lines starting with % are SWIG commands. The first defines the module name, ‘echo’.  Then we tell SWIG to add a C #include statement at the top of the C wrapper code, which is obviously required if you want the JNI library to access your functions.

In the specific case of Java, we need to add some boilerplate code to one of the generated Java classes so that it can load the JNI library we will create. This is done next in lines 7-16. Finally, we tell swig to wrap the code in echo.h using the %include command.

Now it is time to invoke the swig command to do its magic. This is OSX, so the options reflect the location of the JDK headers, which you will need to adjust to match your system.


$ swig -java -package echo -includeall -verbose -outdir src/echo         -I/System/Library/Frameworks/JavaVM.framework/Headers -I./jni     -o jni/java_interface_wrap.c echo_interface.i

The option -package defines the Java package name in the generated files, and -outdir places these files in src/echo. The -o option defines the output C file from the SWIG compilation. If we were using C++, then we would require C++ processing with the option -c++. With a C source file, we have the option of just generating C, or if we want, C++ by adding that option. In any case, the generated source code is pretty much unreadable, so it’s not useful to show it here.

The generated code is then compiled, together will our other C sources into a JNI library. For Android, this means creating an Android.mk file like this in our ./jni directory :


LOCAL_PATH := $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE := echo
LOCAL_C_INCLUDES := $(LOCAL_PATH)
LOCAL_CFLAGS := -O3
LOCAL_CPPFLAGS :=$(LOCAL_CFLAGS)
###

LOCAL_SRC_FILES := echo.c \
opensl_io.c \
java_interface_wrap.c

LOCAL_LDLIBS := -llog -lOpenSLES

include $(BUILD_SHARED_LIBRARY)

The NDK build also asks for a minimal Application.mk in the same directory as above:


APP_STL := gnustl_static
APP_CPPFLAGS += -fexceptions -frtti
APP_ABI := armeabi-v7a

Then we can build the JNI library with the ndk-build command (we assume you have the NDK tools directory in your path):


$ ndk-build  TARGET_PLATFORM=android-9 V=1

Ok, now in your app you can use the wrapped C code in java, like this (see the functions being used in lines 16 and 24):


package com.echo;
import echo.echo;
import android.app.Activity;
import android.os.Bundle;

public class EchoActivity extends Activity {
Thread t;

@Override
public void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.main);
t = new Thread() {
public void run() {
setPriority(Thread.MAX_PRIORITY);
echo.main_process(0.5f, 0.7f);
}
};
t.start();
}

public void onDestroy(){
super.onDestroy();
echo.stop_process();
try {
t.join();
} catch (InterruptedException e) {
e.printStackTrace();
 }
}
t = null;
}
}

That’s all folks, you’ve been wrapped. If you need to add another function to the interface, say to control a parameter etc., all you need is to add it to your C header file (and of course implement it somewhere). Then run the swig command and re-build. Your new function will be then available to your application.

Lock-free audio IO with OpenSL ES on Android

As we saw in our previous discussion of OpenSL audio IO, this API operates, like a number of others, by employing a callback mechanism. Here, we have a very simple interface, where the callback is only used to notify the application that a new buffer can be enqueued (for playback or for recording). In other APIs, the callback also handles pointers to the audio buffers that are to be filled or consumed, but in OpenSL you have a choice just to use the callbacks to operate a signalling mechanism of some sort and keep all the processing in your audio processing thread. This would include enqueueing the required buffers, after the relevant signals are received.

The implementation we showed in the previous example uses thread locks to do this signalling, leading to what could be described as a blocking operation, which is kept synchronous with the notifications issued by the callbacks. This is one of the two ways to go about implementing realtime audio IO. Once upon a time, blocking interfaces were all there was. In the grand UNIX tradition, realtime audio APIs such as the SGI Irix Audio Library provided an open-read-write mechanism, similar to file streams, for the programmer. This was simple and worked relatively well. However, newer low-latency systems (e.g. CoreAudio, Jack, etc.) generally require a more responsive operation, and these now provide exclusively a non-blocking interface, based on callbacks running on specialised audio IO threads. To use them, we need to pay attention to the fact that data is shared between the callback and main process and make sure wires do not get crossed, and the operations occur in the correct order.

While it is possible to employ some types of locks to regulate the communication quite successfully (as noted in our previous implementation of audio IO with  OpenSL), one of the basic techniques of inter-thread data sharing is to use a lock-free circular buffer (called a ‘ringbuffer’ in other quarters). It would be interesting then to see how we could re-implement the audio IO part of our OpenSL example with such methods.

A circular buffer interface

The first thing to do then is to prepare a circular buffer interface that our code can use. We need four functions: to create a circular buffer, to write to it, to read from it, and to destroy it. The circular buffer itself will be a black box to our application:

circular_buffer* create_circular_buffer(int bytes);
int read_circular_buffer_bytes(circular_buffer *p, char *out, int bytes);
int write_circular_buffer_bytes(circular_buffer *p, const char *in, int bytes);
void free_circular_buffer (circular_buffer *p);

The way this is supposed to work is that the read operation will only read the number of requested bytes up to what has been written in the buffer already. The write function will only write the bytes for which there is space in the buffer. They will return a count of read/written bytes, which can be anything from zero to the requested number.

The consumer thread (the audio IO callback in the case of playback, or the audio processing thread in the case of recording) will read from the circular buffer and then do something with the audio. At the same time, asynchronously, the supplier thread will be filling the circular buffer, only stopping if it gets filled up. With an appropriate circular buffer size, the two threads will cooperate seamlessly.

Lock-free audio IO

With the above interface, we can write our audio IO functions and OpenSL callbacks to use it. Let’s take a look at the input side:

// this callback handler is called every time a buffer finishes recording
void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->inBufSamples*sizeof(short);
 write_circular_buffer_bytes(p->inrb, (char *) p->recBuffer,bytes);
 (*p->recorderBufferQueue)->Enqueue(p->recorderBufferQueue,p->recBuffer,bytes);
}
// gets a buffer of size samples from the device
int android_AudioIn(OPENSL_STREAM *p,float *buffer,int size){
 short *inBuffer;
 int i, bytes = size*sizeof(short);
 if(p == NULL || p->inBufSamples == 0) return 0;
 bytes = read_circular_buffer_bytes(p->inrb, (char *)p->inputBuffer,bytes);
 size = bytes/sizeof(short);
 for(i=0; i < size; i++){
 buffer[i] = (float) p->inputBuffer[i]*CONVMYFLT;
 }
 if(p->outchannels == 0) p->time += (double) size/(p->sr*p->inchannels);
 return size;
}

In the callback function (lines 2-8), which will be called every time a new full buffer (recBuffer) is ready, we write all of it to the circular buffer. Then the recBuffer is ready to be enqueued again (line 7). The audio processing function, lines 10 to 21, tries to read the requested number of bytes (line 14) into inputBuffer and then copies that number of samples to the output (converting it into float samples). The function reports the number of copied samples.

For output, we will do the reverse:

// puts a buffer of size samples to the device</pre>
int android_AudioOut(OPENSL_STREAM *p, float *buffer,int size){

short *outBuffer, *inBuffer;
int i, bytes = size*sizeof(short);
if(p == NULL || p->outBufSamples == 0) return 0;
for(i=0; i < size; i++){
p->outputBuffer[i] = (short) (buffer[i]*CONV16BIT);
}
bytes = write_circular_buffer_bytes(p->outrb, (char *) p->outputBuffer,bytes);
p->time += (double) size/(p->sr*p->outchannels);
return bytes/sizeof(short);
}

// this callback handler is called every time a buffer finishes playing
void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->outBufSamples*sizeof(short);
 read_circular_buffer_bytes(p->outrb, (char *) p->playBuffer,bytes);
 (*p->bqPlayerBufferQueue)->Enqueue(p->bqPlayerBufferQueue,p->playBuffer,bytes);
}

The audio processing function (lines 2-13) takes in a certain number of float samples, converts to shorts and then writes the full outputBuffer into the circular buffer, reporting the number of samples written. The OpenSL callback (lines 16-22) reads all of the samples and enqueues them.

For this to work properly, the number of samples read from the input needs to be passed along with the buffer to the output. Here’s the processing loop code that loops the input back into the output:


while(on)

samps = android_AudioIn(p,inbuffer,VECSAMPS_MONO);

for(i = 0, j=0; i < samps; i++, j+=2)
 outbuffer[j] = outbuffer[j+1] = inbuffer[i];
 android_AudioOut(p,outbuffer,samps*2);
 }

In that snippet, lines 5-6 loop over the read samples and copy them to the output channels. It is a stereo-out, mono-in setup, and for this reason we copy the input samples into two consecutive output buffer locations.

Also, now that the enqueueing is happening in the OpenSL threads, in order to start the callback mechanism, we need to enqueue a buffer for recording and another for playback after we start audio on the device. This will ensure the callback is issued when buffers need to be replaced.

Full code

Here is the full code for a lock-free OpenSL IO module; below you will see a simple C example utilising it, which is in fact very similar to our previous blogpost example. You can substitute the three files in the previous project for these, and the application should run as before.

/*
opensl_io.c:
Android OpenSL input/output module header
Copyright (c) 2012, Victor Lazzarini
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
 * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 * Neither the name of the <organization> nor the
 names of its contributors may be used to endorse or promote products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#ifndef OPENSL_IO
#define OPENSL_IO

#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>
#include <stdlib.h>
#include <unistd.h>
#ifdef __cplusplus
extern "C" {
#endif
typedef struct _circular_buffer {
 char *buffer;
 int wp;
 int rp;
 int size;
} circular_buffer;

typedef struct opensl_stream {

 // engine interfaces
 SLObjectItf engineObject;
 SLEngineItf engineEngine;

// output mix interfaces
 SLObjectItf outputMixObject;

// buffer queue player interfaces
 SLObjectItf bqPlayerObject;
 SLPlayItf bqPlayerPlay;
 SLAndroidSimpleBufferQueueItf bqPlayerBufferQueue;
 SLEffectSendItf bqPlayerEffectSend;

// recorder interfaces
 SLObjectItf recorderObject;
 SLRecordItf recorderRecord;
 SLAndroidSimpleBufferQueueItf recorderBufferQueue;

 // buffers
 short *outputBuffer;
 short *inputBuffer;
 short *recBuffer;
 short *playBuffer;

circular_buffer *outrb;
 circular_buffer *inrb;

// size of buffers
 int outBufSamples;
 int inBufSamples;

double time;
 int inchannels;
 int outchannels;
 int sr;

} OPENSL_STREAM;

/*
 Open the audio device with a given sampling rate (sr), input and output channels and IO buffer size
 in frames. Returns a handle to the OpenSL stream
 */
 OPENSL_STREAM* android_OpenAudioDevice(int sr, int inchannels, int outchannels, int bufferframes);
 /*
 Close the audio device
 */
 void android_CloseAudioDevice(OPENSL_STREAM *p);
 /*
 Read a buffer from the OpenSL stream *p, of size samples. Returns the number of samples read.
 */
 int android_AudioIn(OPENSL_STREAM *p, float *buffer,int size);
 /*
 Write a buffer to the OpenSL stream *p, of size samples. Returns the number of samples written.
 */
 int android_AudioOut(OPENSL_STREAM *p, float *buffer,int size);
 /*
 Get the current IO block time in seconds
 */
 double android_GetTimestamp(OPENSL_STREAM *p);

#ifdef __cplusplus
};
#endif

#endif // #ifndef OPENSL_IO

This is the opensl_io.c module:

</pre>
/*
 opensl_io.c:
 Android OpenSL input/output module
 Copyright (c) 2012, Victor Lazzarini
 All rights reserved.

Redistribution and use in source and binary forms, with or without
 modification, are permitted provided that the following conditions are met:
 * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 * Neither the name of the <organization> nor the
 names of its contributors may be used to endorse or promote products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
 WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
 DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
 (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
 ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
 (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#include "opensl_io2.h"
#define CONV16BIT 32768
#define CONVMYFLT (1./32768.)
#define GRAIN 4

static void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context);
static void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context);
circular_buffer* create_circular_buffer(int bytes);
int checkspace_circular_buffer(circular_buffer *p, int writeCheck);
int read_circular_buffer_bytes(circular_buffer *p, char *out, int bytes);
int write_circular_buffer_bytes(circular_buffer *p, const char *in, int bytes);
void free_circular_buffer (circular_buffer *p);

// creates the OpenSL ES audio engine
static SLresult openSLCreateEngine(OPENSL_STREAM *p)
{
 SLresult result;
 // create engine
 result = slCreateEngine(&(p->engineObject), 0, NULL, 0, NULL, NULL);
 if(result != SL_RESULT_SUCCESS) goto engine_end;

// realize the engine
 result = (*p->engineObject)->Realize(p->engineObject, SL_BOOLEAN_FALSE);
 if(result != SL_RESULT_SUCCESS) goto engine_end;

// get the engine interface, which is needed in order to create other objects
 result = (*p->engineObject)->GetInterface(p->engineObject, SL_IID_ENGINE, &(p->engineEngine));
 if(result != SL_RESULT_SUCCESS) goto engine_end;

engine_end:
 return result;
}

// opens the OpenSL ES device for output
static SLresult openSLPlayOpen(OPENSL_STREAM *p)
{
 SLresult result;
 SLuint32 sr = p->sr;
 SLuint32 channels = p->outchannels;

if(channels){
 // configure audio source
 SLDataLocator_AndroidSimpleBufferQueue loc_bufq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};

switch(sr){

case 8000:
 sr = SL_SAMPLINGRATE_8;
 break;
 case 11025:
 sr = SL_SAMPLINGRATE_11_025;
 break;
 case 16000:
 sr = SL_SAMPLINGRATE_16;
 break;
 case 22050:
 sr = SL_SAMPLINGRATE_22_05;
 break;
 case 24000:
 sr = SL_SAMPLINGRATE_24;
 break;
 case 32000:
 sr = SL_SAMPLINGRATE_32;
 break;
 case 44100:
 sr = SL_SAMPLINGRATE_44_1;
 break;
 case 48000:
 sr = SL_SAMPLINGRATE_48;
 break;
 case 64000:
 sr = SL_SAMPLINGRATE_64;
 break;
 case 88200:
 sr = SL_SAMPLINGRATE_88_2;
 break;
 case 96000:
 sr = SL_SAMPLINGRATE_96;
 break;
 case 192000:
 sr = SL_SAMPLINGRATE_192;
 break;
 default:
 return -1;
 }

 const SLInterfaceID ids[] = {SL_IID_VOLUME};
 const SLboolean req[] = {SL_BOOLEAN_FALSE};
 result = (*p->engineEngine)->CreateOutputMix(p->engineEngine, &(p->outputMixObject), 1, ids, req);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// realize the output mix
 result = (*p->outputMixObject)->Realize(p->outputMixObject, SL_BOOLEAN_FALSE);

 int speakers;
 if(channels > 1)
 speakers = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT;
 else speakers = SL_SPEAKER_FRONT_CENTER;
 SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM,channels, sr,
 SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16,
 speakers, SL_BYTEORDER_LITTLEENDIAN};

SLDataSource audioSrc = {&loc_bufq, &format_pcm};

// configure audio sink
 SLDataLocator_OutputMix loc_outmix = {SL_DATALOCATOR_OUTPUTMIX, p->outputMixObject};
 SLDataSink audioSnk = {&loc_outmix, NULL};

// create audio player
 const SLInterfaceID ids1[] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
 const SLboolean req1[] = {SL_BOOLEAN_TRUE};
 result = (*p->engineEngine)->CreateAudioPlayer(p->engineEngine, &(p->bqPlayerObject), &audioSrc, &audioSnk,
 1, ids1, req1);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// realize the player
 result = (*p->bqPlayerObject)->Realize(p->bqPlayerObject, SL_BOOLEAN_FALSE);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// get the play interface
 result = (*p->bqPlayerObject)->GetInterface(p->bqPlayerObject, SL_IID_PLAY, &(p->bqPlayerPlay));
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// get the buffer queue interface
 result = (*p->bqPlayerObject)->GetInterface(p->bqPlayerObject, SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
 &(p->bqPlayerBufferQueue));
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// register callback on the buffer queue
 result = (*p->bqPlayerBufferQueue)->RegisterCallback(p->bqPlayerBufferQueue, bqPlayerCallback, p);
 if(result != SL_RESULT_SUCCESS) goto end_openaudio;

// set the player's state to playing
 result = (*p->bqPlayerPlay)->SetPlayState(p->bqPlayerPlay, SL_PLAYSTATE_PLAYING);

if((p->playBuffer = (short *) calloc(p->outBufSamples, sizeof(short))) == NULL) {
 return -1;
 }

(*p->bqPlayerBufferQueue)->Enqueue(p->bqPlayerBufferQueue,
 p->playBuffer,p->outBufSamples*sizeof(short));

 end_openaudio:
 return result;
 }
 return SL_RESULT_SUCCESS;
}

// Open the OpenSL ES device for input
static SLresult openSLRecOpen(OPENSL_STREAM *p){

SLresult result;
 SLuint32 sr = p->sr;
 SLuint32 channels = p->inchannels;

if(channels){

switch(sr){

case 8000:
 sr = SL_SAMPLINGRATE_8;
 break;
 case 11025:
 sr = SL_SAMPLINGRATE_11_025;
 break;
 case 16000:
 sr = SL_SAMPLINGRATE_16;
 break;
 case 22050:
 sr = SL_SAMPLINGRATE_22_05;
 break;
 case 24000:
 sr = SL_SAMPLINGRATE_24;
 break;
 case 32000:
 sr = SL_SAMPLINGRATE_32;
 break;
 case 44100:
 sr = SL_SAMPLINGRATE_44_1;
 break;
 case 48000:
 sr = SL_SAMPLINGRATE_48;
 break;
 case 64000:
 sr = SL_SAMPLINGRATE_64;
 break;
 case 88200:
 sr = SL_SAMPLINGRATE_88_2;
 break;
 case 96000:
 sr = SL_SAMPLINGRATE_96;
 break;
 case 192000:
 sr = SL_SAMPLINGRATE_192;
 break;
 default:
 return -1;
 }

 // configure audio source
 SLDataLocator_IODevice loc_dev = {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT,
 SL_DEFAULTDEVICEID_AUDIOINPUT, NULL};
 SLDataSource audioSrc = {&loc_dev, NULL};

// configure audio sink
 int speakers;
 if(channels > 1)
 speakers = SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT;
 else speakers = SL_SPEAKER_FRONT_CENTER;
 SLDataLocator_AndroidSimpleBufferQueue loc_bq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};
 SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM, channels, sr,
 SL_PCMSAMPLEFORMAT_FIXED_16, SL_PCMSAMPLEFORMAT_FIXED_16,
 speakers, SL_BYTEORDER_LITTLEENDIAN};
 SLDataSink audioSnk = {&loc_bq, &format_pcm};

// create audio recorder
 // (requires the RECORD_AUDIO permission)
 const SLInterfaceID id[1] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
 const SLboolean req[1] = {SL_BOOLEAN_TRUE};
 result = (*p->engineEngine)->CreateAudioRecorder(p->engineEngine, &(p->recorderObject), &audioSrc,
 &audioSnk, 1, id, req);
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

// realize the audio recorder
 result = (*p->recorderObject)->Realize(p->recorderObject, SL_BOOLEAN_FALSE);
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

// get the record interface
 result = (*p->recorderObject)->GetInterface(p->recorderObject, SL_IID_RECORD, &(p->recorderRecord));
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

 // get the buffer queue interface
 result = (*p->recorderObject)->GetInterface(p->recorderObject, SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
 &(p->recorderBufferQueue));
 if (SL_RESULT_SUCCESS != result) goto end_recopen;

// register callback on the buffer queue
 result = (*p->recorderBufferQueue)->RegisterCallback(p->recorderBufferQueue, bqRecorderCallback,
 p);
 if (SL_RESULT_SUCCESS != result) goto end_recopen;
 result = (*p->recorderRecord)->SetRecordState(p->recorderRecord, SL_RECORDSTATE_RECORDING);

if((p->recBuffer = (short *) calloc(p->inBufSamples, sizeof(short))) == NULL) {
 return -1;
 }

(*p->recorderBufferQueue)->Enqueue(p->recorderBufferQueue,
 p->recBuffer, p->inBufSamples*sizeof(short));

 end_recopen:
 return result;
 }
 else return SL_RESULT_SUCCESS;
}

// close the OpenSL IO and destroy the audio engine
static void openSLDestroyEngine(OPENSL_STREAM *p){

// destroy buffer queue audio player object, and invalidate all associated interfaces
 if (p->bqPlayerObject != NULL) {
 SLuint32 state = SL_PLAYSTATE_PLAYING;
 (*p->bqPlayerPlay)->SetPlayState(p->bqPlayerPlay, SL_PLAYSTATE_STOPPED);
 while(state != SL_PLAYSTATE_STOPPED)
 (*p->bqPlayerPlay)->GetPlayState(p->bqPlayerPlay, &state);
 (*p->bqPlayerObject)->Destroy(p->bqPlayerObject);
 p->bqPlayerObject = NULL;
 p->bqPlayerPlay = NULL;
 p->bqPlayerBufferQueue = NULL;
 p->bqPlayerEffectSend = NULL;
 }

// destroy audio recorder object, and invalidate all associated interfaces
 if (p->recorderObject != NULL) {
 SLuint32 state = SL_PLAYSTATE_PLAYING;
 (*p->recorderRecord)->SetRecordState(p->recorderRecord, SL_RECORDSTATE_STOPPED);
 while(state != SL_RECORDSTATE_STOPPED)
 (*p->recorderRecord)->GetRecordState(p->recorderRecord, &state);
 (*p->recorderObject)->Destroy(p->recorderObject);
 p->recorderObject = NULL;
 p->recorderRecord = NULL;
 p->recorderBufferQueue = NULL;
 }

// destroy output mix object, and invalidate all associated interfaces
 if (p->outputMixObject != NULL) {
 (*p->outputMixObject)->Destroy(p->outputMixObject);
 p->outputMixObject = NULL;
 }

// destroy engine object, and invalidate all associated interfaces
 if (p->engineObject != NULL) {
 (*p->engineObject)->Destroy(p->engineObject);
 p->engineObject = NULL;
 p->engineEngine = NULL;
 }

}
// open the android audio device for input and/or output
OPENSL_STREAM *android_OpenAudioDevice(int sr, int inchannels, int outchannels, int bufferframes){

 OPENSL_STREAM *p;
 p = (OPENSL_STREAM *) malloc(sizeof(OPENSL_STREAM));
 memset(p, 0, sizeof(OPENSL_STREAM));
 p->inchannels = inchannels;
 p->outchannels = outchannels;
 p->sr = sr;

 if((p->outBufSamples = bufferframes*outchannels) != 0) {
 if((p->outputBuffer = (short *) calloc(p->outBufSamples, sizeof(short))) == NULL) {
 android_CloseAudioDevice(p);
 return NULL;
 }
 }

if((p->inBufSamples = bufferframes*inchannels) != 0){
 if((p->inputBuffer = (short *) calloc(p->inBufSamples, sizeof(short))) == NULL){
 android_CloseAudioDevice(p);
 return NULL;
 }
 }

if((p->outrb = create_circular_buffer(p->outBufSamples*sizeof(short)*4)) == NULL) {
 android_CloseAudioDevice(p);
 return NULL;
 }
 if((p->inrb = create_circular_buffer(p->outBufSamples*sizeof(short)*4)) == NULL) {
 android_CloseAudioDevice(p);
 return NULL;
 }

if(openSLCreateEngine(p) != SL_RESULT_SUCCESS) {
 android_CloseAudioDevice(p);
 return NULL;
 }

if(openSLRecOpen(p) != SL_RESULT_SUCCESS) {
 android_CloseAudioDevice(p);
 return NULL;
 }

if(openSLPlayOpen(p) != SL_RESULT_SUCCESS) {
 android_CloseAudioDevice(p);
 return NULL;
 }

p->time = 0.;
 return p;
}

// close the android audio device
void android_CloseAudioDevice(OPENSL_STREAM *p){

if (p == NULL)
 return;

openSLDestroyEngine(p);

if (p->outputBuffer != NULL) {
 free(p->outputBuffer);
 p->outputBuffer= NULL;
 }

if (p->inputBuffer != NULL) {
 free(p->inputBuffer);
 p->inputBuffer = NULL;
 }

if (p->playBuffer != NULL) {
 free(p->playBuffer);
 p->playBuffer = NULL;
 }

if (p->recBuffer != NULL) {
 free(p->recBuffer);
 p->recBuffer = NULL;
 }

free_circular_buffer(p->inrb);
 free_circular_buffer(p->outrb);

free(p);
}

// returns timestamp of the processed stream
double android_GetTimestamp(OPENSL_STREAM *p){
 return p->time;
}
// this callback handler is called every time a buffer finishes recording
void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->inBufSamples*sizeof(short);
 write_circular_buffer_bytes(p->inrb, (char *) p->recBuffer,bytes);
 (*p->recorderBufferQueue)->Enqueue(p->recorderBufferQueue,p->recBuffer,bytes);
}

// gets a buffer of size samples from the device
int android_AudioIn(OPENSL_STREAM *p,float *buffer,int size){
 short *inBuffer;
 int i, bytes = size*sizeof(short);
 if(p == NULL || p->inBufSamples == 0) return 0;
 bytes = read_circular_buffer_bytes(p->inrb,(char *)p->inputBuffer,bytes);
 size = bytes/sizeof(short);
 for(i=0; i < size; i++){
 buffer[i] = (float) p->inputBuffer[i]*CONVMYFLT;
 }
 if(p->outchannels == 0) p->time += (double) size/(p->sr*p->inchannels);
 return size;
}

// this callback handler is called every time a buffer finishes playing
void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context)
{
 OPENSL_STREAM *p = (OPENSL_STREAM *) context;
 int bytes = p->outBufSamples*sizeof(short);
 read_circular_buffer_bytes(p->outrb, (char *) p->playBuffer,bytes);
 (*p->bqPlayerBufferQueue)->Enqueue(p->bqPlayerBufferQueue,p->playBuffer,bytes);
}

// puts a buffer of size samples to the device
int android_AudioOut(OPENSL_STREAM *p, float *buffer,int size){

short *outBuffer, *inBuffer;
 int i, bytes = size*sizeof(short);
 if(p == NULL || p->outBufSamples == 0) return 0;
 for(i=0; i < size; i++){
 p->outputBuffer[i] = (short) (buffer[i]*CONV16BIT);
 }
 bytes = write_circular_buffer_bytes(p->outrb, (char *) p->outputBuffer,bytes);
 p->time += (double) size/(p->sr*p->outchannels);
 return bytes/sizeof(short);
}

circular_buffer* create_circular_buffer(int bytes){
 circular_buffer *p;
 if ((p = calloc(1, sizeof(circular_buffer))) == NULL) {
 return NULL;
 }
 p->size = bytes;
 p->wp = p->rp = 0;

 if ((p->buffer = calloc(bytes, sizeof(char))) == NULL) {
 free (p);
 return NULL;
 }
 return p;
}

int checkspace_circular_buffer(circular_buffer *p, int writeCheck){
 int wp = p->wp, rp = p->rp, size = p->size;
 if(writeCheck){
 if (wp > rp) return rp - wp + size - 1;
 else if (wp < rp) return rp - wp - 1;
 else return size - 1;
 }
 else {
 if (wp > rp) return wp - rp;
 else if (wp < rp) return wp - rp + size;
 else return 0;
 }
}

int read_circular_buffer_bytes(circular_buffer *p, char *out, int bytes){
 int remaining;
 int bytesread, size = p->size;
 int i=0, rp = p->rp;
 char *buffer = p->buffer;
 if ((remaining = checkspace_circular_buffer(p, 0)) == 0) {
 return 0;
 }
 bytesread = bytes > remaining ? remaining : bytes;
 for(i=0; i < bytesread; i++){
 out[i] = buffer[rp++];
 if(rp == size) rp = 0;
 }
 p->rp = rp;
 return bytesread;
}

int write_circular_buffer_bytes(circular_buffer *p, const char *in, int bytes){
 int remaining;
 int byteswrite, size = p->size;
 int i=0, wp = p->wp;
 char *buffer = p->buffer;
 if ((remaining = checkspace_circular_buffer(p, 1)) == 0) {
 return 0;
 }
 byteswrite = bytes > remaining ? remaining : bytes;
 for(i=0; i < byteswrite; i++){
 buffer[wp++] = in[i];
 if(wp == size) wp = 0;
 }
 p->wp = wp;
 return byteswrite;
}

void
free_circular_buffer (circular_buffer *p){
 if(p == NULL) return;
 free(p->buffer);
 free(p);
}
<pre>

This is a simple loopback example using the opensl_io.c module:


/*
opensl_example.c:
OpenSL example module
Copyright (c) 2012, Victor Lazzarini
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
 * Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.
 * Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.
 * Neither the name of the <organization> nor the
 names of its contributors may be used to endorse or promote products
 derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT HOLDER> BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

#include <android/log.h>
#include "opensl_io.h"

#define BUFFERFRAMES 1024
#define VECSAMPS_MONO 64
#define VECSAMPS_STEREO 128
#define SR 44100

static int on;
void start_process() {
 OPENSL_STREAM *p;
 int samps, i, j;
 float inbuffer[VECSAMPS_MONO], outbuffer[VECSAMPS_STEREO];
 p = android_OpenAudioDevice(SR,1,2,BUFFERFRAMES);
 if(p == NULL) return;
 on = 1;
 while(on) {
 samps = android_AudioIn(p,inbuffer,VECSAMPS_MONO);
 for(i = 0, j=0; i < samps; i++, j+=2)
 outbuffer[j] = outbuffer[j+1] = inbuffer[i];
 android_AudioOut(p,outbuffer,samps*2);
 }
 android_CloseAudioDevice(p);
}

void stop_process(){
 on = 0;
}

Update: Adding the NDK (JNI) build to an Eclipse project

In a previous post, I explored the use of the OpenSL ES library to implement audio IO natively on Android. This required the creation of a JNI library, which was added to the Eclipse project for the App. In that case, we would run a build script from the command line by changing to the project directory and running

$ sh build.sh

The contents of build.sh were discussed in that post.

To facilitate things, we can incorporate the build script by going into the Eclipse project properties (right click on the project and select ‘properties’) and:

  1. choose ‘Builders’
  2. click on ‘New’
  3. select ‘Program’
  4. give it a name: ‘JNI build’
  5. On the ‘Main’ tab, in ‘Location’, add the path to your bash command: e.g. /bin/sh
  6. Same tab, in Working Directory, click on Browse Workspace and select your project.
  7. Same tab, in Arguments, add your build script name: e.g. build.sh
  8. On the Environment tab, click on ‘New’:
  9. Create a PATH variable and add your command path to it (e.g. /usr/bin:/usr/local/bin:/usr/sbin:/bin:/sbin)
  10. Click on OK.
  11. On the Builder list window, click ‘up’ to move the JNI build to the top of the list.

This will integrate the JNI build in your Eclipse project. The example project in GIT has been updated with these changes:

$git clone https://bitbucket.org/victorlazzarini/android-audiotest


A simple synth in Android: step by step guide using the Java SDK

In this post, I will provide a step-by-step guide to building a simple audio app with Eclipse and the Android SDK. Although I would encourage readers to check out the NDK for audio development, this is meant as a Java-only introduction to the topic.

This guide expects you to have installed the Android SDK, Eclipse Classic and its ADT plugin. You can run the examples on the emulator or on an Android device. If the latter is chosen, make sure that the device is set for development (debug enabled in the Android settings), and of course connected to the development computer.

Step 1

Create a new blank App: File -> New -> Android App

Follow the instructions, add the App name, and to simplify things, uncheck “create custom launcher icon”. Then follow the next steps, create a BlankActivity, using all the default settings and finish.

Step 2

Contgratulations: you have created your (first?) App. At this point you can press on the ‘play’ button  to run it (either on the emulator or on a connected device). But does it do anything?

Let’s build the synth part of the app. We want to locate the MainActivity.java source file. Use the Package Explorer window (the application is called SoundTest in this example):

Now we can go and edit the code. In the MainActivity class, we want to first add some data members:

public class MainActivity extends Activity {
    Thread t;    
    int sr = 44100;
    boolean isRunning = true;

The first is a Thread object that will hold our audio processing thread, the second the sampling rate, and the third a means of switching the audio on and off.

Inside the OnCreate method, provided by the template code, we will instantiate the thread object. This is done by defining its run() function, which will hold the code to process audio.

 // start a new thread to synthesise audio        
    t = new Thread() {
         public void run() {
           // set process priority
           setPriority(Thread.MAX_PRIORITY);

We set the priority to max so that we can achieve good performance. Now we need to deal with creating an output audio object, which will be an AudioTrack one. First we set the buffersize, which will hold the size of the audio block to be output. Then we instantiate the audioTrack object:

int buffsize = AudioTrack.getMinBufferSize(sr, AudioFormat.CHANNEL_OUT_MONO, 
                                               AudioFormat.ENCODING_PCM_16BIT);
// create an audiotrack object
AudioTrack audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC, sr, 
                                       AudioFormat.CHANNEL_OUT_MONO, 
                                       AudioFormat.ENCODING_PCM_16BIT, 
                                       buffsize, 
                                       AudioTrack.MODE_STREAM);

Eclipse should take care of adding any needed library import lines for AudioTrack and AudioFormat, but if there is trouble, just follow the suggestions from the editor on how fix the errors by allowing it toadd the given ‘import …’ lines automatically.

Now we’re ready to take care of the synthesis side. We create a signal buffer and define some parameters:

short samples[] = new short[buffsize];
int amp = 10000;
double twopi = 8.*Math.atan(1.);
double fr = 440.f;
double ph = 0.0;

Start audioTrack running:

// start audio
audioTrack.play();

The next thing is to define the synthesis loop:

// synthesis loop
while(isRunning){

   for(int i=0; i < buffsize; i++){ 
     samples[i] = (short) (amp*Math.sin(ph));
     ph += twopi*fr/sr;
   }
   audioTrack.write(samples, 0, buffsize);
}

and then take care of the closing of the audio device etc. when the synthesis is stopped:

audioTrack.stop();
audioTrack.release();
}
};

This closes the definition of the run() function and the Thread object that holds it. The final thing we need to do in the OnCreate() method is to start the thread:

t.start();

Finally, we need to add a method to deal with switching off the audio when the App is closed. This is done by supplying an OnDestroy() method:

public void onDestroy(){
   super.onDestroy();    
   isRunning = false;
   try {
     t.join();
   } catch (InterruptedException e) {
     e.printStackTrace();
   }    
  t = null;
}

Step 3

Now you can run your (first) Audio app. Switch it on and you get a beep, works well as a tuning fork. But we want controls. This is what we do next. Open the activity_main.xml resource file:

Click on the “hello world” label and delete. Drag a slider and add it to the app:

Right click on the slider in the App layout and select ‘edit ID’. Rename the ID “frequency”. Save the file.

Now go back to the MainApplication source code. Add these two other data members to the class (after the boolean isRunning…  line):

    SeekBar fSlider;
    double sliderval;

Ok, the SeekBar object will handle the slider, and sliderval member will pick up the slider value so we can use it.

Now we need to connect the fSlider object to its ‘view’ , which is the actual slider widget.

// point the slider to the GUI widget
fSlider = (SeekBar) findViewById(R.id.frequency);

and create a ‘listener’ for it, so that changes in the slider position are detected:

// create a listener for the slider bar;
OnSeekBarChangeListener listener = new OnSeekBarChangeListener() {
        public void onStopTrackingTouch(SeekBar seekBar) { }
        public void onStartTrackingTouch(SeekBar seekBar) { }
        public void onProgressChanged(SeekBar seekBar,int progress, 

                                      boolean fromUser) {
            if(fromUser) sliderval = progress / (double)seekBar.getMax();
        }
};

The listener has to implement three methods, but we only need the data from onProgressChanged(). The sliderval will hold the slider value in the range 0 – 1.

We now tell the fSlider object to use this listener:

// set the listener on the slider

fSlider.setOnSeekBarChangeListener(listener);

The final changes are in the synthesis loop. We will need to update the sinewave frequency, by looking at the value of sliderval:

// synthesis loop
while(isRunning){

  fr =  440 + 440*sliderval;

  for(int i=0; i < buffsize; i++){
    samples[i] = (short) (amp*Math.sin(ph));
    ph += twopi*fr/sr;
   }
   audioTrack.write(samples, 0, buffsize);

}

And there you go. You can now control the pitch of the sound with a slider. This concludes this short step-by-step tutorial.

Full code

The full MainApplication.java code is shown below:

package com.example.soundtest;
import com.example.soundtest.R;
import android.media.AudioFormat;
import android.media.AudioManager;
import android.media.AudioTrack;
import android.os.Bundle;
import android.app.Activity;
import android.view.Menu;
import android.widget.SeekBar;
import android.widget.SeekBar.OnSeekBarChangeListener;

public class MainActivity extends Activity {
    Thread t;
    int sr = 44100;
    boolean isRunning = true;
    SeekBar fSlider;
    double sliderval;

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
       // point the slider to thwe GUI widget
        fSlider = (SeekBar) findViewById(R.id.frequency);

        // create a listener for the slider bar;
        OnSeekBarChangeListener listener = new OnSeekBarChangeListener() {
          public void onStopTrackingTouch(SeekBar seekBar) { }
          public void onStartTrackingTouch(SeekBar seekBar) { }
          public void onProgressChanged(SeekBar seekBar, 
                                          int progress,
                                           boolean fromUser) {
              if(fromUser) sliderval = progress / (double)seekBar.getMax();
           }
        };

        // set the listener on the slider
        fSlider.setOnSeekBarChangeListener(listener);

        // start a new thread to synthesise audio
        t = new Thread() {
         public void run() {
         // set process priority
         setPriority(Thread.MAX_PRIORITY);
         // set the buffer size
        int buffsize = AudioTrack.getMinBufferSize(sr,
                AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT);
        // create an audiotrack object
        AudioTrack audioTrack = new AudioTrack(AudioManager.STREAM_MUSIC,
                                          sr, AudioFormat.CHANNEL_OUT_MONO,
                                  AudioFormat.ENCODING_PCM_16BIT, buffsize,
                                  AudioTrack.MODE_STREAM);

        short samples[] = new short[buffsize];
        int amp = 10000;
        double twopi = 8.*Math.atan(1.);
        double fr = 440.f;
        double ph = 0.0;

        // start audio
       audioTrack.play();

       // synthesis loop
       while(isRunning){
        fr =  440 + 440*sliderval;
        for(int i=0; i < buffsize; i++){
          samples[i] = (short) (amp*Math.sin(ph));
          ph += twopi*fr/sr;
        }
       audioTrack.write(samples, 0, buffsize);
      }
      audioTrack.stop();
      audioTrack.release();
    }
   };
   t.start();        
}

    @Override
    public boolean onCreateOptionsMenu(Menu menu) {
        getMenuInflater().inflate(R.menu.activity_main, menu);
        return true;
    }

    public void onDestroy(){
          super.onDestroy();
          isRunning = false;
          try {
            t.join();
           } catch (InterruptedException e) {
             e.printStackTrace();
           }
           t = null;
     }
}

Csound iOS SDK released

Complementing the Csound for Android SDK, we have now released its Csound for iOS counterpart. Here is the official announcement by Steven Yi:

Victor and I are happy to announce the release of the Csound for iOS

SDK, version 5.17.3.  It is available at:

https://sourceforge.net/projects/csound/files/csound5/iOS/

There are currently two zips there:

* csound-iOS-5.17.3.zip – This includes a PDF manual for developing

applications with Csound for iOS, as well as includes the Csound for

iOS Examples project.  This project has a number of examples that

demonstrate differents aspects of the CsoundObj and Csound API,

including widget wrapping, MIDI handling, and using sensors. This

project includes precompiled binaries of libcsound.a and libsndfile.a

that are compiled for arm and intel, and work with both the simulator

and devices.  To create new projects based on Csound for iOS, we have

put everything that is the core of Csound for iOS in the “Csound iOS

Examples/csound-iOS” folder.  You can copy that entire folder into any

new project and will have everything necessary to build and run new

Csound-based apps.

* libsndfile-iOS.zip – This is a copy of the sourcecode and Xcode

project used to build libsndfile.a for iOS.  This zip is only

necessary if you are planning to build libsndfile.a yourself,

otherwise, you can just use the pre-compiled one that comes with

csound-iOS-5.17.3.zip.

Other notes:

* An “iOS” folder has been pushed to the csound5 GIT repository.  It

contains a build.sh script for rebuilding libcsound.a, an

updateExamplesLibAndHeaders.sh file for updating the csound-iOS folder

with the latest .a’s and headers, and a release.sh script for building

a release folder (same as what is used for the pre-compiled release

above).

* Building requires CMake and Xcode to be installed.  We are using

CMake to generate Xcode projects (using the same CMake files that are

used for building the desktop Csound), then using Xcode to build.

Also, the XCode commandline tools must be installed, as xcodebuild is

used to build the XCode projects from the commandline.

* iOS, like Android, will be maintained as first class platforms for

Csound.  New releases of Csound will now be built not only for Mac,

Windows, and Linux, but for iOS and Android as well.

Thanks and enjoy!

steve

The SDK’s release is accompanied by the launch of the first of a series of Csound-based apps, as discussed in the Create Digital Music site. These applications demonstrate some interesting uses of the SDK in music applications.

Some thoughts on Android development and the latency question

As some of us have discovered very quickly, audio latency on Android is high. There were some indications that the figure of 45ms would somehow be achievable (ie. a buffer of 2048 frames a 44.1kHz), however, I am not sure it is achievable anywhere. On my Galaxy Tab 10.1, which is brand new, I am getting around 200 ms, which is about four times that figure.

There have been some efforts showing that given the right software support, lower latencies are possible. In this blog, for instance, we see the use of pulseaudio on an Android phone, with a 20ms latency reported. The crucial point however is this: do we want to develop for a custom system? The answer, as far as I am concerned is no. In my view, we need to use the OS as-is, off-the-shelf. Customised systems, rooted devices, etc., might be good for playing around, but it is not a productive way to go. Eventually, we would like the user down the street to be able to run our apps on her/is device.

I see similarities with the way pro-audio developed on Linux. In the early days with OSS devices and non-preemtible kernels, many people took to customising their systems to get some sort of decent audio support. But then slowly things start to change. First, better support for devices came with ALSA, then Jack was released, followed by more readily-available low-latency kernel patches (and prebuilt kernels). Then stock kernels got preemption support, so that now 10-15ms latency (not ‘low’ but acceptable) is standard and millisecond-level latency is available in specialised kernels. People can develop for this, knowing that the Linux desktop user will be able to take advantage of it.

So, for the time being, we need to keep asking the Android team to give us lower latencies. Develop apps that will benefit from it. Show them how it can make a difference. The way I see this, there is enough in Android to make it a very good platform for what we do, we only need the latency issue addressed.  Hopefully, they are hearing this loud and clear.

Follow

Get every new post delivered to your Inbox.

Join 30 other followers