The Khronos Group, well known for its successful OpenGL standard API for graphic applications and gaming, recently announced the OpenSL ES Sound Language for mobile devices such as phones and personal media players. OpenSL ES is to provide a standardized, high-performance, low-latency method to access audio functionality for developers of native applications on embedded mobile multimedia devices, with the aim of straightforward cross-platform deployment of hardware and software audio capabilities, reducing implementation effort, and promoting the market for advanced audio.
In September 2007, Khronos issued OpenSL ES 1.0 as a provisional specification to provide the opportunity to incorporate feedback from the developer community before the specification is finalized. Khronos plans to finalize the specification towards the end of 2007, and in the mean time is seeking input from the industry.
Developer Hayden Porter conducted an interview about the OpenSL ES API with Nathan Charles, lead for the Khronos OpenSL ES working group and Software Architect at Creative Labs.
Hayden Porter: Can you provide a brief overview of the OpenSL ES API features? Are they a subset of the OpenAL (Open Audio Language) API? Are the features grouped together by different functionalities? For example is there a set of APIs for MIDI interaction, and a different set for audio?
Nathan Charles: OpenSL ES is an entirely new API developed from scratch. We looked at other open audio APIs, such as OpenAL, but these are mainly targeted for game use on PC and games consoles. Our demands were slightly different: OpenSL ES needs to support a wide range of applications beyond games and it must run on resource constrained devices. It was clear to us that a new API would be the best way of achieving our aims.
OpenSL ES supports all the features you'd expect from an advanced audio API targeted for mobile embedded devices. The feature list is fairly long, including basic digital audio and MIDI playback (with support for SP-MIDI, mDLS and mXMF) but also more advanced features such as 3D audio, virtualization, MIDI messages, and effects including reverb and EQ. Additionally OpenSL ES provides APIs for controlling the LED and vibra, and for accessing metadata included in audio files so that applications can display information such as the track name and author.
OpenSL ES uses an object-oriented based architecture. The most basic objects are "players" and "recorders", so for example, you can create MIDI players and digital audio recorders. The features are then encapsulated in interfaces, with each interface containing controls for similar functionality. For example, the volume interface exposes controls for volume, balance and pan. One of the nice things about this architecture is that it's possible to use the same interface on different objects. So, for example, the volume interface can be exposed on a digital audio player, a MIDI player, an audio recorder and the global output.
HP: Can you summarize the different device profiles and their capabilities?
NC: With such a large feature set it wouldn't be reasonable or even appropriate to expect all devices to support the full range of features. One solution is to make lots of the features optional. The problem with this is that developers cannot rely on particular features being present so they either have to produce many versions of their application or only use the features guaranteed to be present on all devices. This can lead to a bad developer experience and ultimately a lower adoption of the API.
Instead, we decided to segment the functionality into profiles. Profiles often have a bad name but we think we've got the profiles in OpenSL ES just right. OpenSL ES has three profiles: Phone, Music and Game. Each profile contains the functionality most suited to its application area, with some features in more than one profile.
The Phone profile is aimed at mobile phones and supports ring tone playback, device UI sounds and audio for simple 2D games. The Music profile is for music playback applications and includes the ability to support multiple audio codecs and supports features including stereo widening. The Game profile is aimed at advanced games, with support for 3D audio, reverb and advanced MIDI.
HP: Do the OpenSL ES API profiles build upon each other, for example, does the game profile include functionality of both the phone and music profiles, or are the features of each profile independent?
NC: Each profile has been designed independently so that it contains all the functionality appropriate for the market it supports. Consequently the profiles do overlap, so there are areas of functionality (e.g. basic playback of PCM) that are present in all three profiles. Importantly though, the profiles are not levels, so it's not true that any one profile contains all the functionality of another profile. The developer doesn't need to worry about this though, they just write an application using the profile that most suits their application and it guaranteed to work on devices supporting that profile.
HP: Is there a way to use the API itself to identify the type of profile implemented on a given device?
NC: Yes, there's a query mechanism to determine the profiles supported but as the profiles are market-oriented it should be clear what profile to expect on a given device. For example, a game-oriented phone is likely to expose both the Phone and Game profiles, whereas a portable music player may just expose the Music profile. A smart-phone may expose all three profiles.
HP: Is the OpenSL ES API based upon or very similar to any existing mobile sound APIs such as those from JSR135/234 or perhaps BREW imedia API? Or is it a new API that has the same functionality as these other APIs but a different language structure? How portable would this API be for applications based upon already existing APIs?
NC: Although OpenSL ES is not based on any pre-existing APIs most of the concepts will be familiar to those used to other audio APIs. We've purposely kept away from reinventing the wheel, exposing features in a similar manner to other audio APIs. So, for example, those coming from a JSR134/234 or a PC audio background shouldn't have much trouble understanding OpenSL ES. Of course it's always slightly tricky porting an application from one API to another but porting to OpenSL ES shouldn't require too much work. We've got a number of members of the JSR234 expert group in the OpenSL ES working group, so that's helped us keep things consistent.
