By providing a number of channels through which information may pass between user and computer, multimodal interfaces promise to significantly increase the bandwidth and fluidity of the interface between humans and machines. Natural language processing techniques typically focus on processing strings of words within a single dimension. This program of research seeks to develop an approach to multimodal language processing which enables parsing and interpretation of natural human input distributed across two or three spatial dimensions, time, and the acoustic dimension of speech. A declarative unification-based grammar representation is proposed for the description of multimodal integration strategies. This representation is interpreted by a multidimensional parser yielding an architecture which provides the foundation for next generation natural interfaces supporting human-computer interaction through speech, pen, gesture, and other modes. Fundamental advances in multimodal interfaces are essential to the successful migration of human-computer interaction from the desktop to small mobile computing devices, public information kiosks, and other platforms for which complex graphical user interfaces are not feasible. Multimodal interfaces stand to play a critical role both in realizing the vision of universal access to information and in the development of assistive technologies for people with disabilities.
