This project will develop a transderivational search engine based on the neurological condition known as synaesthesia in which two or more senses are crossed (e.g., when seeing a color causes one to hear a sound) to help people to discover connections between text, 1D audio, 2D image, 3D geometry and 4D motion data. The project is inspired by the ability of artists and designers to find analogies between diverse artifacts and bring them together to compose a coherent and novel narrative.
The intellectual merit of this research is the development of matching algorithms that suggest analogies across different media forms by looking at structural similarity within media content. The result will be a transformative technology at the intersection of art, computer graphics, machine learning, cognitive psychology, and human-computer interaction (HCI). Transderivational search will enhance the synaesthetic effect in analogy generation and will naturally lend itself to a wide range of brainstorming pursuits. Finding analogies between media of different forms (e.g., audio and 3D shapes) has not been explored, nor has there been much focus on non-literal search engines. Literal searches rely only on explicit meaning (e.g., the word ?three? and an image of the number 3) and categorization to determine similarity. Instead, this project will compare media samples by looking for structural similarity using analytical approaches such as statistical shape distributions, frequency analysis, and machine learning techniques to discover relationships between mixed- (multi-dimensional) media samples.
The broader impacts of this research are in advancing artificial intelligence through transderivational search (essential to language and cognitive processing) and in opening up new research questions on search technology. The educational impacts are in drawing more women and minorities into CS and improving retention in CS programs by showing the relevance of search technology to creative design and to multimedia management. The transderivational search tools will be used by students in introductory level CS courses to build basic media management software. Transderivational search can also serve as a testbed for exploring algorithms in high level CS courses on machine learning, computer vision and graphics.
