Research projects

Musically-Aware Humanoid Robots

Human musicians make use of substantial auditory and visual information throughout a performance. Our humanoids research focuses on providing such capabilities (e.g., audio and visual beat detection, note onset and pitch detection, and basic control for musical keyboard performance), with the long-term goal of enabling a large humanoid to be an interactive participant in a live music ensemble.

Music Emotion Recognition

In developing automated systems to recognize the emotional content of music, we are faced with a problem spanning two disparate domains: the space of human emotions and the acoustic signal of music. To address this problem, we must develop models for both data collected from humans describing their perceptions of musical mood and quantitative features derived from the audio signal.

Magnetic Resonator Piano

The MRP is an augmentation of the acoustic grand piano. By using electromagnets to induce the strings to vibration, the MRP expands the piano's vocabulary to include infinite sustain, crescendos from silence, harmonics and new timbres. An optical keyboard sensor system lets the pianist continuously shape the sound of every note..

Structured Audio

Structured audio is a representation of sound content using symbolic or semantic information as a means of encoding (or representing) the data. It provides a framework to easily manipulate or transform content after it reaches the user. .

Analysis-Synthesis of Guitar Expression

During performance, a guitarist employs specific techniques in order to convey expressive intentions for a desired effect. These intentions often correspond to simultaneous variation of the source (guitarist-string interaction) and the filter (resonant string) parameters. We propose modeling expression at the signal level via joint estimation of the source and filter parameters of plucked guitar sounds in order to achieve realistic re-synthesis while incorporating expressive intentions.

Expressive Percussion Performance

Expressivity in percussion relates to the creative alteration of tempo, volume, timbre, and excitation techniques. We study performances through audio feature analysis, and develop classification techniques and models that will help to better understand the details of expression specific to percussion.

Modeling Musical Attributes of the Music Genome Project

In this work, we look to model attributes of instrumentation, rhythm, and sonority in music audio signals. We then explore whether musical genre can be modeled objectively with these musical attributes. This work is done in collaboration with Pandora and harnesses the power of their Music Genome Project, which is 500+ attribute and genre labels across more than 1.2Million songs.

Automated Multi-Track Mixing

This system uses machine learning techniques to automatically combine individual instrument tracks into a single mixed song. Acoustic features are extracted from the audio and used to train the system to predict fader values for each instrument to produce a final mix.

ALF: Audio processing Library for Flash

A research project dedicated to providing Flash developers with sophisticated signal processing routines for use in web-based MIR or game applications. We hope that this library encourages the growth of audio-centric Flash applications for research and entertainment.

LiveNote: Orchestral Performance Companion

We have developed a system that helps users by guiding them through the performance using a handheld application in real-time. Using audio features, we attempt to align the live performance audio with that of a previously annotated reference recording. The aligned position is transmitted to users’ handheld devices and pre-annotated inform-ation about the piece is displayed synchronously.

Multi-touch Music Interfaces

There has been a great deal of recent interest in the use of multi-touch displays (MTDs) as an interface for human-computer interaction. A multi-touch display (as opposed to a standard touch-screen) is able to detect and track multiple points of contact (e.g., fingers). This capability enables simultaneous control of multiple objects as well as multi-point gestures..

Archived Projects