The iNotes project was a winner of the 2011 Knight Arts Challenge Philadelphia.
Many people enjoy live orchestral performances, but those without musical training may find it hard to relate to the music. We have developed a system that helps users by guiding them through the performance using a handheld application in real-time. Using chroma features and dynamic time warping, we attempt to align the live performance audio with that of a previously annotated reference recording. The aligned position is transmitted to users’ handheld devices and pre-annotated information about the piece is displayed synchronously.
Automatic Score Following
Acoustic Features: Chroma
Frequencies are divided into 12 pitch classes corresponding to the 12 notes in the Western chromatic scale (A, A#, etc.). The chromagram reflects the intensity of pitch over time, regardless of octave.
Alignment Using Dynamic Time Warping
Dynamic Time Warping (DTW) is related to dynamic programming, and is designed to compare two signals with different timescales and align them.
The Tracking System
With an iPhone or iPod Touch, audience members use an application that helps guide them through the performance.
Annotation content is generated through collaborations with Philadelphia universities and organizations. In addition to user feedback, the musicolgists’ feedback helps to tailor the application to the information that they want to convey to their audience.