When we see someone's lips move consistently with his/her speech, we perceptually assign the sound to that person. It has long been clear that such temporal congruity, or match, between the various sensory inputs is a crucial factor in the integration of the information from the various senses. The role of temporal congruity in generating a unified, multimodal percept has been well studied in many cases, especially in the case of audio-visual cross-modal integration. However, the role of temporal congruity in the cross-modal integration of visual input with the haptic (active touch) input remains poorly understood. The overall goal of the proposed research is to understand the role of temporal congruity in visual-haptic cross-modal integration. To this end, we will use visual-haptic object recognition as a test case, and systematically study the perceptual effects of congruity (or lack thereof) on object recognition in healthy adult human subjects. The subjects'task will be to categorize a given visual-haptic object into one of the two given categories. The categories will be defined with precisely specified, naturalistic distribution of features, so that the underlying information processing required for successful categorization can be defined using a quantitative, probabilistic framework. To help create the well-defined probability distributions required for testing this framework, we will generate novel, 'designer'categories of 3-D visual and haptic objects with desired distributions of features by simulating the biological processes of embryogenesis and natural selection, and/or computer graphical morphing. The stimuli will be presented to the subjects so that they cannot see what they touch, and cannot touch what they see. We will wirelessly synchronize the visual image with its haptic counterpart in real time using 3-D orientation sensors. We will systematically manipulate the synchrony between the visual image and its haptic counterpart, and measure its effect on the cross-modal recognition of learned objects, as well as on cross-modal learning itself. We expect to find that lack of temporal congruity significantly affects both categorization and category learning performance. The proposed study will not only elucidate the precise role of temporal congruity in visual-haptic object recognition, but will also introduce a highly innovative and quantitative methodology with potentially important implications to basic research as well as visual-haptic rehabilitative treatments.
We may not realize it, but our perception of the world is a rich amalgam produced by our various senses acting in concert with each other. When we behold a rose, for instance, what we experience is more than just the sum of what we see, smell and touch. Very little is known about how the brain combines the senses in this fashion. Understanding this process will not only help us better understand how the brain works, but it also has many potentially profound applications for how impairments in one of the senses can, with appropriate rehabilitative treatment, be compensated for by improvements in another.
