The essence of music is structure, or as the composer Edgar Varese put it, music is "organized sound." For centuries, people have responded to their enjoyment of music by studying and describing that structure, yet there is still no satisfactory description of what music is, much less how or why it works. Furthermore, there is very little work that can truly be said to start from the minimum of assumptions and attempt to define music simply in terms of whatever statistical regularities actually occur in a large body of real, polyphonic music. The PI's goal in this project is to begin to construct such a definition. To this end he will develop algorithms that describe and explain musical structure using automatic signal analysis and machine learning. These algorithms will lay the foundation for software that mines a database of many thousands of examples of music audio for recurrent structures and patterns at successively higher levels. (The PI argues that in order to work with very large databases it is critical to start from audio, rather than notated forms, since this is the only 'canonical form' in which all music exists; by the same token, audio is the representation closest to the original music.) The software will use these detected structures to create rich but compact descriptions of the realizations of music, which complement and extend human musicological insights. Such software will enable the implementation of tools that revolutionize the way listeners find and organize music, by empowering them to describe their interests in terms of objective, yet relevant, properties of the music audio itself. For similar reasons, the new algorithms may lead to a fully automated system for music recommendation based on analysis of a listener's existing collection in terms of high-level attributes identified in the analysis, as well as to advances in other music-processing applications such as automatic accompaniment and computer-aided composition.
Broader Impacts: Because music is important in the lives of so many people, new technologies that can afford insight into the structure and 'function' of music have very broad potential impact. Project outcomes will likely generalize to enable the discovery of high-level, hierarchic structure from large databases in other, analogous domains such as multimedia content analysis and natural language processing, and they ultimately may also cast light on the techniques used to represent complex information within the brain. A significant aspect of the project is a plan for outreach to school-age students tied in to an existing summer school and public school program operated by Columbia University. The strong appeal of music to all people, but particularly to school-age students, presents an opportunity to broaden interest in science and technology by developing and deploying a set of classroom materials and other tools, which middle- and high-school students and teachers can use to analyze and modify the music of their own choice.
