An update on the Android audio latency issue
With the advent of newer versions of Android (4.1, 4.2) and newer devices, the topic of latency has come a bit more into the mainstream of system development. Newer APIs in the Android SDK are providing means to enquire devices about low-latency parameters (buffer sizes and sampling rates). But how does that translate in practical terms to latency reduction?
My tests with Csound
Well, I got hold of a Google Nexus 7 tab this week and went about measuring it. Using the latest Csound, whose OpenSL IO code is (I expect) optimal, I first measured touch-to-sound, and then round-trip latency. The first measurement was made by recording an audible mechanical ‘tap’ as I touched the screen and the sound coming out of the speakers. Granted, it is not as fine-grained as a video test, but it is not too far off the mark and would be particularly close to user experience as one tries to manipulate the UI. The second was done by using a microphone attached to a pair of headphones, which would pick up a click from its source and then from the tablet output.
Results were as follows:
- best touch-to-sound latency: ~120ms
- best round-trip latency: ~105ms
The parameters for Csound were -B1024 and -b128, which means, in this particular case, circular buffer and audio buffer sizes of 1024 and 128 frames, respectively. What I observed was that the size of the circular buffer was critical: anything less than about 1024 would bring either intermittent or constant dropouts, regardless of the audio buffer size. Also, smaller sizes would not translate to lower latencies (e.g. low-latency but with dropouts), the lower limit never changed. This means that although the buffer sizes can be made very small, there will be considerable jitter in the OpenSL buffering and that the lower layers of buffering, which we do not have access to, are the ones ultimately responsible for the latency (we knew that, but this confirms it).
Although the round-trip figure is far away from low-latency, it is actually becoming more usable. When testing little processing apps (echos, reverb, flangers), actually it does not feel too bad. I would say it can be used in live-electronics settings that are not too strict in terms of live-processed synchronisation, and I plan to use it in a piano piece I have been asked to write, next year.
What is a bit disappointing is the touch latency, which appears to contribute more to the overall latency than the single-direction audio delay. This is an issue for performance systems using the tablet/phone touch interface, although it might be solved via external controllers (but do we want to depend on these?).
I have reported these measurements to the Android audio discussion list and was interested to hear other results posted there, which confirm my numbers in the case of the same hardware and Android version.
Amongst these reports, in particular, there was a meticulous measurement made with a video camera, graphics and audio. After a touch, graphics and audio would be generated, so touch latency could be evaluated separately. This is another way to split the measurements, if you are not looking to measure round-trip delay. The hardware used was a low-cost tablet and Android 4.0. Results were as follows: touch-graphics delay was ~90ms; touch to audio, ~250ms. These measurements also line up quite well with my own experiences with the Galaxy Tab 10.1 and 4.0.
I can drawn some interesting conclusions from this, and as the movie goes, some are good, some are bad, and some are definitely ugly:
1. The good: things are improving and we have a situation now that for some applications, audio processing can be usable. They seem to be listening to us.
2. The bad: development is far too slow. Older devices with fine computing capabilities (like my Galaxy Tab 10.1) are being left behind.
3. The ugly: a comment in the Android audio list says it all, “I wished that those shiny tablet magazines did simple tests like this, and then published the results instead of irrelevant values like their thickness and number of processing cores”. In my opinion, with the plethora of magazines and online review sites that exist today, someone has to provide us with better information in this regard. As it is now, they are only patting hardware and software vendors on the back, helping to preserve the status quo.
As an aside, I have noticed that in Android platform-14, the OpenSL ES headers are providing an alternative streaming audio interface, identified by SL_IID_ANDROIDBUFFERQUEUESOURCE. Well, I thought, let’s give it a go, maybe this is a door for devices to support lower latency streams. It turns out that although it is a published interface, it has not actually found its way into the library binary, which is odd. Furthermore, in the release 8b of the NDK, there is a SL_IID_BUFFERQUEUE in the library (but that is not in the headers), which has now disappeared in the latest, 8c, release.
Anyway, the documents still seem to say that the Android Simple Buffer Queue interface is to be used, although this does not mean much, as the documentation can sometimes lag behind the released code. I have filed a ticket with Google regarding the absence of the buffer queue source interface, so it will be interesting to see how they respond to this.