The goal of this project is to engineer the Creative Artificially-Intuitive and Reasoning Agent CAIRA, an intelligent computer system designed for making decisions and producing creative work based on both reasoning and intuition ? the latter being the process of making a spontaneous decision when there is not sufficient time for reasoning. A network of humans and several CAIRA agents will be formed and studied to provide new understanding of learning and social interaction in dynamic environments where boundary conditions change over time and decisions have to be made within a given time interval. The project will use improvised avant-garde music to study the performance of the intelligent agent, but the outcome will be applicable to other time-based communication scenarios that require creative solutions and intelligent agents that can communicate naturally, as humans do. The machine listening abilities of CAIRA will be based on a music recognition system that simulates the human auditory periphery to perform an Auditory Scene Analysis (ASA), the process by which the auditory system extracts features such as pitch, timbre, and rhythm to organize sound into perceptually meaningful units. The simulation of cognitive processes will include a comprehensive cognitive calculus for reasoning and decision-making, and the system will also possess metareasoning, that is, an entity's ability to reflect upon its own reasoning. CAIRA's capacity for metareasoning will provide groundbreaking insight into the human willingness to systematically break rules and weigh the consequences in order to achieve a given goal, for example, to produce creative work. A machine correlate to human intuition will be another key component of CAIRA, and a major goal is to find out how different balances of reasoning and metareasoning on the one hand, and intuitive decisions, on the other hand, will lead to novel and captivating forms of creativity.
Broader Impact: Based on a unique interdisciplinary approach between scientists and artists, and also engaging students from engineering and other fields in project-related courses, the project will help to better understand the process of artistic creation as a product of balancing different sources of creativity (e.g., rule-based reasoning approaches vs. intuitive decision-making). The general architecture of CAIRA will be useful for various musical applications, including automated composition systems and internet-based music performances with co-located ensembles. CAIRA will also provide an enhanced interaction with creative machines that will allow non-specialists and ability-impaired users to engage in expressive musical creation. Two examples include CAIRA's ability to automatically add musical texture or an accompaniment to a simple melody or to serve as a duet partner. Beyond this, the model will further our understanding of the human mind's ability to balance creative decisions based on spontaneity and immediate perception of an environment, to factor in environmental and social context, and to consider past training and learned stylistic rules in the creative process. This understanding will be applicable to similar complex problems, such as economic systems and semantic web services, as well as other problems where creative and timely solutions and/or natural human/computer communication are essential.
The scope of this project was to develop CAIRA, a Creative Artificially-Intuitive and Reasoning Agent. CAIRA is a computer-based system that listens to other musical performers and improvises music based on what the other musicians play. Our goal was to better understand human creativity and to apply this understanding to machines. To achieve this goal, we simulated several functional brain stages related to auditory processing and decision making. To this purpose, we developed an architecture with four functional layers. At the bottom layer, CAIRA performs an auditory scene analysis to extract basic musical features including pitch, tempo, and loudness. In the second layer, machine learning tools analyze sequences of events so CAIRA can make use of the time-based structure of music to, for example, capture melodies. Twotechniques, Emprical Mode Decomposition and Hidden Markov Models, were used for this process. It was important to us that CAIRA analyzed music from actual sound and not through a symbolic notation like a music score. We also incorporated features of Pauline Oliveros’ Deep Listening Practice into the learning algorithm to enable the CAIRA to discriminate between global and focal listening. CAIRA’s third layer resembles aspects of human cognition. Here, it was important to us to enable CAIRA to select between different strategies. CAIRA can reason using the logic-based musical calculus of HANDLE to make decisions, but also employs artificial intuition in its musical performance. CAIRA can trade both techniques off each other to perform in different ways, giving insight into how humans develop different performance styles based on internal strategies. We could demonstrate that CAIRA benefitted from using artificial intuition in cases where it had insufficient time for reasoning to respond to other musicians. A sub version of CAIRA, FILTER – the Freely Improvising, Learning and Transforming Evolutionary Recombination system – simulates a highly intuitive approach to music improvisation building on Pauline Oliveros' Deep Listening Practice. In instances CAIRA uses the fundamentals of jazz theory to accompany a human soloist or perform with two musicians in a jazz. The fourth layer is the output stage of CAIRA. The system’s performance is based on a database of audio samples that CAIRA uses to create its own compositions to perform with others. CAIRA can also conduct an ensemble using an adaptive video score. CAIRA and FILTER have been featured in concerts and recordings including New Interfaces for the Musical Expression (NIME) 2012, Eyebeam, New York City, 2014, and the Deep Listening Conference, Troy, New York 2013.
