Pausch 9157583 This is the fourth year of a five year continuing grant. This research aims at integrating multimodal input streams from a human to a computer into higher level semantic actions. This project is important because of the current imbalance of information flow between a human and the computer with computer output being several orders of magnitude greater than user input by gestures, voice and other input modalities by examining new software architectures to deal with probabilistic input and combinations of simultanewous probabilitistic input streams. Higher level semantics of multi-modal inputs will be addressed by research on input specification languages which properly incorporate time, the sharing of semantic information at various levels of synthesis and the development of a general framework to support the combination of probabilistic input streams.
