The goal of this project is to investigate a framework to capture both the semantics and essence of an activity pertaining to moving sensor data streams. Moving sensors refers to an emerging class of data intensive applications that impacts disciplines such as communication, health-care, scientific applications, etc. These applications consist of a fixed number of sensors that move and produce streams of data as a function of time. With communication, for example, a hearing impaired individual might utilize a cyber glove that translates hand signs into written (spoken) words. The glove consists of a sensor for each finger joint that reports its location as a function of time, producing streams of data.
The framework consumes streams of data and constructs their multi level, spatio temporal representation at different levels of abstraction. This multi-layer design ensures extensibility, modularity, and physical data independence from the stream producing devices. While this framework is demonstrated in the context of several applications, its key demonstration is with an application to recognize hand signs (in particular, American Sign Language) using a virtual-reality glove. This shows the extensibility of the framework to recognize new signs and its modularity to handle new input devices. The framework improves the quality of detected patterns by (a) employing buffers for delayed decision making, (b) different layers maintain context to provide hints when input data is noisy, and (c) employs the concept of Envelope of Limits (EoL) to compensate for a user's slight variations when performing a spatio-temporal sign.
More information can be obtained from the project web sites (http://dblab.usc.edu) and (http://infolab.usc.edu). The broader impact of this project is to further the field of multimedia by incorporating hand-gesture as a new mode for conveying information, with applications for the people with disabilities.
