Music contributes substantially to every culture on Earth, and the enjoyment of music is nearly universal. The appeal of music is that it provides expression to our own emotions, and it follows logically that individuals would want some creative input to that expression. Historically the primary exposure to music has been through live performance, providing audiences an opportunity for interaction with the musicians and music, but today the vast majority of music is experienced through recordings. And though recent digital audio technologies have had a tremendous impact on the world of recorded music, its fundamental nature remains unchanged: once a recording is made, that single "performance" is forever fixed, preventing any true interaction with the listener. The proposed work integrates research in digital audio technology with educational activities under a common vision of transforming the act of listening to "recorded" music into an interactive experience in which the "performance" responds to the creative input of the listener. Central to this vision is a concept known as Structured Audio, a semantic object-based representation of sound.
The specific work activity consists of the following efforts: . Structured Sound Modeling (SSM) of musical instruments, which uses a novel signal processing and machine learning framework to facilitate a greater degree of musically expressive control than can be achieved with existing models for sound synthesis. . Development of the Structured Audio Platform (SAP), consisting of devices and software that employ these new instrument models to provide an expanded artistic palette for music producers and interfaces enabling nonmusicians to interact with their music by controlling creative aspects of the "performance." An outreach program that uses music technology to attract students to science, technology, engineering, and mathematics (STEM) by demonstrating the contributions of these disciplines to modern music production as well as the creativity inherent within STEM and related fields.
With respect to intellectual merit, there are many methods of modeling musical instrument sounds, but most are limited in their ability to capture the full expressive range of those instruments. The project will employ a novel framework for modeling time-varying sounds using jointly estimated parameters of an excitation-resonance model to derive a HiddenMarkovModel (HMM) of expressive sound. This framework employs an analysis/synthesis scheme: input sound is parameterized by the model and can be regenerated from the model parameters. This process has been used to model the singing voice and can be applied to other instruments, resulting in high sound quality while retaining a full range of musical expression. Knowledge gained through this project will also benefit other machine listening applications, such as audio identification and sound sensing. The research activity includes the development of a custom platform of devices and interfaces that offer artists new creative options and allow non-musicians to also provide creative input to their music. A key area of research will investigate appropriate mappings from the expressive controls of SSM instruments to a simple, low-dimensional interface for musical novices. Creativity is a difficult subject to study quantitatively, but this project presents a framework from which a small subset of creative expression can be assessed and analyzed.
The educational activities of this project are designed to broadly impact students at many levels. The proposed outreach program will introduce high school students, particularly from underrepresented groups, to the technology of digital music, revealing the influence of science and engineering on music performance, recording, and listening. By engaging students' affinity for popular music the program seeks to foster curiosity in science and technology. The research component of the proposal will also be directly integrated into this program via custom music devices that allow students to creatively interact with music in Structured Audio format. This confluence of engineering and art has the potential to incubate interest in both fields. Given the diminishing support for music education in the public school system, this multi-disciplinary approach may be a solution to sustaining creative arts in the curriculum. The proposed research integrates knowledge from a wide range of areas. Participating graduate students will receive interdisciplinary training in fields such as acoustics, music theory, signal processing, performance practice, and machine learning.
This CAREER project integrated computational media research investigating creative expression with educational activities to enhance STEM learning through music technology. The project team developed multiple systems for automatically assessing the emotional content of music, with a particular focus on how the acoustic properties change along with the emotions conveyed over the course of a song. These results could be used, for example, by digital music services to group songs with similar emotional content to improve music recommendations. This research included a new method for collecting music emotion labels using a novel online game, resulting in a highly detailed data set for hundreds of songs that was made publicly available. Several additional online activities were developed for collecting listening data, some of which also aimed to educate participants regarding specific mathematical and scientific concepts related to sound and acoustics. Over the course of this project, the research team published over 30 papers on these topics and organized multiple conferences and workshops specifically on the topic of emotion recognition in music. The researchers also developed a variety of new acoustic, electronic, and computing systems used to create interactive music performances. This work was also presented through novel music performances for large audiences as part of the Philadelphia Science Festival in 2012 and 2013. Another system for live music performance tracking developed through this project received considerable recognition in the popular media, including a front page article in The Philadelphia Inquirer in June, 2011. Additionally, this award enabled the development of the Summer Music Technology (SMT) program, a one-week camp for high school students that uses music technology to motivate interest in Science, Technology, Engineering, and Math (STEM). Since its inception in 2007, over 150 high school students have participated in SMT over 6 summers, and the program continues to attract a large number of applications each summer. Program activities cover topics including loudspeaker construction, musical instrument acoustics, sound synthesis, and digital sound effects, and the project team developed a suite of custom iPad apps to provide an exploratory and highly interactive component to each activity. Recent cohorts using the custom tools and curriculum had better learning outcomes than earlier groups using off-the-shelf tools and activities, and several of the activities have been used in local Philadelphia high school classrooms. The curriculum, lesson plans, worksheets, and custom software developed for the Summer Music Technology program are freely available online.
