Mobile computing devices offer increasingly rich human-computer interaction opportunities through the use of new sensor technology such as multi-touch surfaces, microphones, and cameras. These advances provide users with richer sets of interaction modalities and motivate new and novel human-centric interfaces. Among the new input modalities, touch-based interaction is the most advanced and widely deployed, while speech is gaining increased attention. However, even with recent breakthroughs in computer vision and depth and motion sensing technology, the imaging modality remains the least developed for mobile devices. And yet image-based sensing (a) would permit richer interaction opportunities for a general user population and (b) holds special promise for individuals with disabilities who find touch-based interfaces very challenging to use.
This project focuses on the development of Head-Activated Technology for off-the- shelf Mobile Devices (HAT-MD). HAT-MD systems will employ the next-generation imaging technology that will be included in a wide array of mobile devices. Advanced algorithms will detect and track user head and facial features, and map specific movements to specific interface controls and application actions. HAT-MD is initially targeted towards individuals with physical disabilities, and will both (a) extend to mobile platforms the head control interaction techniques that are often employed on desktop computers and (b) broaden the set of head and face movements that can be employed.
Project deliverables include: (a) HAT-MD Algorithms: Next-generation imaging sensors will be used for detection, tracking, and 3D reconstruction. Head position and facial features will be detected, tracked, and mapped to interface controls. Computation will be distributed between the mobile devices and cloud computing services. (b) HAT-MD Applications: Head activated interfaces will be developed to address the challenges that individuals with physical disabilities have when using contemporary mobile platforms. (c) HAT-MD Evaluations: Rigorous human subject studies will evaluate the effectiveness and ease-of-use of the general HAT-MD methodology as well as specific interface and applications that are developed. Algorithms will be benchmarked to current state-of-the-art detection, tracking, and reconstruction methods.
Broader Impacts: The increased accessibility provided by the HAT-MD project will create new opportunities for people to interact with mobile devices, especially for individuals with physical disabilities who currently have limited independent control of such devices. The project will also feature direct involvement by individuals with disabilities at the research, development, and evaluation phases in order to focus the intellectual development in a truly useful direction that will complement the broader impact.
