This project entails the design of a new computational framework based on music compositional process and sketching to map and identify patterns in large-scale complex scientific data sets. This research project will be undertaken at the California NanoSystems Institute AlloSphere, University of California, Santa Barbara. The Allosphere is one of the largest 3D immersive display devices in the world for scientific visualization and artistic instrumentation. Leveraging mathematical concepts and constructs for binding structure and information flow in scientific and artistic research, the computational framework will incorporate a three-dimensional hierarchical sketching system that will be the basis for structural representation. Sonic marking of patterns in large datasets and techniques in music composition will be explored as ideal methods to identify complex integrated layers of data as music carries meaning on several time-scales, from individual timbres and pitches to short melodies and rhythms all the way up to large-scale form and structure of a work, each engaging distinct perceptual and cognitive processes. Real-time, interactive representations of the data in the AlloSphere will allow researchers to rapidly prototype parametric systems for more time-consuming and resource-demanding simulations and experiments.
Representing complex scientific data through large-scale immersive 3D audiovisual data representation will facilitate understanding to a wide audience, from advanced researchers who will be able to communicate across disciplines, to the general public. The AlloSphere will motivate dissemination of engineering and science research to wide audiences in education and society, through this new software platform that will allow a broader public to comprehend science that would be out of their reach of understanding. The AlloSphere Research Facility has its own formal outreach initiative that services the CNSI's Professional Outreach Program, that coordinates research interns, undergraduates and high school students through programs such as; (1) The after-school LEAPS (Let's Explore Physical Science) program works with eighth grade students and plans to start an independent research project option for high school seniors; (2) The INSET (NSF Internships in NanoSystems Science, Engineering and Technology) program recruits students from largely underrepresented groups from California community colleges for eight-week summer internships; (3) The Apprentice Researchers program engages high school juniors in individual laboratories at UCSB.
By applying the creative compositional process of sketching in building our computational language and representing very complex information through our senses, namely visual and audio representations that you can interact with, we are enabling the same right brain/left brain process that artists experience when they create a work of art, for scientists and other researchers. This will facilitate the uncovering of new patterns in complex information, and allow scientists and engineers to work with their information perceptually and intuitively, the way that artists do. We can represent any information as visual and audio frequencies by mapping their vibratory spectrum into the light and sound domain. This is well understood by mapping heat through infrared light. For instance when atoms vibrate they create waveforms that behave as sound waves, scaling these down into the audio domain will allow us to use our hearing to understand their complex behavior. We are thus finding patterns in this complex information that may lead to new scientific discoveries as well as creating new and unique art forms. We believe that one of the most difficult tasks of understanding big and complex data is the ability to quickly find new patterns in voluminous amounts of information and to have the ability to retain in one’s memory the information from pages and pages of numbers. If there is a way to translate this information into tangible visual and audio taxonomies, we may be able to retain this information much more readily. Psychologists state that multimodal representation of information aids to long-term memory. Having a visual and aural language for large-scale and complex data is in our opinion the next step in understanding, and manipulating big data. Statistical data mining functions well when one knows what they are looking for in big data. We believe that knowledge discovery through a visual and aural language that will represent the information through our senses will be especially beneficial when one is not sure what to look for. All information can be represented through mathematical models and all mathematical models can be translated into visual and aural frequencies. These mappings can be very literal when one is mapping scientific information. They become more abstract with data such as financial information, social networks and economic data. For example with scientific data, such as atoms and sub atomic particles, they vibrate at frequencies outside of the visual and audio domain. Through a mathematical transformation, they can be scaled down into the sense domain where they can be perceived through the senses, the same for chemical and biological information. Intellectual Merit- A computational framework based on the creative process and designed to represent complex mathematical data visually and sonically through hierarchical sketching, will stimulate breakthroughs in science and engineering, by giving new and intuitive ways to represent the data perceptually, facilitating the possibility of finding new patterns in information, and in finding new ways of seeing, knowing, and doing computing, science and engineering. By carefully observing artistic/scientific collaborations, one will learn how creative representation of immersive content and interaction can influence scientific discovery. Broader Impact- The scientific data representations resulting from the framework will prove invaluable for intuitive understanding of complex information by a broader, more general audience. Immersive audiovisual representation of this multidimensional data facilitates understanding to a broader public. The AlloSphere Outreach Program offers presentations to graduate, undergraduate and K-12 students as well as the general public. The broader impacts of this proposal are also directly connected to the advancement of discovery and understanding while promoting teaching, training and learning. Through development of a new language of visual and sonic information for scientific computing and engineering research, complex knowledge of science and engineering will be disseminated to a wider audience for education and the sharing of information. Our visual/audio computational framework will motivate dissemination of engineering and science research to wide audiences in education and society, through this new software platform that will allow a broader public to comprehend science that would be out of their reach of understanding. With this new computational framework using visual and audio representation, a wide array of people will comprehend and understand science and mathematics.
