An Audio DSP Toolkit for Rapid Application Development in Flash

International Workshop on Multimedia Signal Processing (MMSP)
October 5-7, 2009 in Rio de Janeiro, Brazil
By: Travis M. Doll, Ray V. Migneco, Jeff J. Scott and Youngmoo E. Kim


Abstract


The Adobe Flash platform has become the de facto standard for developing and deploying media rich web applications and games. The relative ease-of-development and cross-platform architecture of Flash enables designers to rapidly prototype graphically rich interactive applications, but comprehensive support for audio and signal processing has been lacking. ActionScript, the primary development language used for Flash, is poorly suited for DSP algorithms. To address the inherent challenges in the integration of interactive audio processing into Flash-based applications, we have developed the DSP Audio Toolkit for Flash (DATF), which offers significant performance improvements over algorithms implemented in Java or ActionScript. By developing this toolkit, we hope to open up new possibilities for Flash applications and games, enabling them to utilize real-time audio processing as a means to drive gameplay and improve the experience of the end user.

We present an architecture for embedding DSP routines in a Flash application, available functionality, and performance results. Source code and example applications can be found at http://music.ece.drexel.edu/ALF.


Functionality


The following lists the core functions provided in the DSP Library which is accessible via the DATF (see Figure 1):

  • Fast Fourier Transform (FFT)
  • Linear Prediction: Estimates spectral shape
  • Sinusoidal Analysis: Estimates harmonic components
  • Intensity: Measures total energy in the audio spectrum
  • Centroid: Measures the “brightness” of the spectrum
  • Rolloff: Indicates the frequency below which 85% of the spectral energy lies
  • Flux: Measures changes in the spectra between frames
  • Bandwidth: Indicates range of frequencies in spectrum
  • Reverb: Simulates an acoustic room environment effect using a room impulse response
  • Additive Synthesis: Generates complex sounds based on time- and spectral-domain parameters


Architecture


Figure 1: Diagram showing components of the DATF architecture and integration into a Flash application

The DATF is realized in part by two, key additions to the Flash platform: dynamic, buffer-based audio control in Flash 10 and the Alchemy framework. Alchemy allows C code to be compiled into bytecode optimized for the ActionScript Virtual Machine, allowing our DSP routines to run at near-native speeds. Figure 1 shows how Alchemy can be used to create a self-contained audio centric application.

  • Audio processing Library for Flash (ALF): A top-level interface that manages user specified audio files, audio processing functions and returned audio features. Also handles audio playback.
  • DSP Audio Toolkit for Flash (DATF): Interacts with Alchemy DSP library to read/write audio buffers and invoke user-specified library functions from ALF.
  • Alchemy DSP Library: Contains Alchemy-compiled C/C++ DSP routines


DATF Performance


FFT Performance (ms)
FFT Size 16384 8192 4096 2048 1024 512 256
ActionScript (AS3mathlib) 96.377 45.157 20.818 9.276 4.460 2.041 0.925
Java (JTransforms) 29.517 20.703 9.393 4.345 1.956 0.901 0.385
C (DATF) 3.009 1.371 0.628 0.297 0.139 0.067 0.034

To assess the performance benefits of the DATF, we measured the average computation times of the Fast Fourier Transform (FFT) for the DATF implementation vs. pure ActionScript (AS) and Java-based implementations:

  • FFT performed on real-valued sequences with lengths at each power of 2 from 256 to 16384
  • Reported timings based on the average elapsed times for 10,000 FFT iterations on each platform

The DATF implementation yields performance ~30 times faster than AS and 10-15 times faster than Java-based FFTs.


Pulse 2


Pulse 2 is a side-scrolling platform game which is driven by songs in the player’s music library. The game is unique in that it utilizes the DATF’s spectral analysis functions to generate the graphical content based on the acoustic features of the chosen music. Elements of the game world, including obstacles and enemies, are mapped to specific DATF functions, thus immersing the player in a unique gaming experience for every track in their music collection.