Enabling Audio-Haptic Interaction with Physical Objects for the Visually Impaired Summary CamIO (Camera Input-Output) is a novel camera system designed to make physical objects (including documents, maps and 3D objects such as architectural models) fully accessible to people who are blind or visually impaired. It works by providing real-time audio-haptic feedback in response to the location on an object that the user is pointing to, which is visible to the camera mounted above the workspace. While exploring the object, the user can move it freely; a gesture such as double-tapping with a finger signals for the system to provide audio feedback about the location on the object currently pointed to by the finger (or an enhanced image of the selected location for users with low vision). Compared with other approaches to making objects accessible to people who are blind or visually impaired, CamIO has several advantages: (a) there is no need to modify or augment existing objects (e.g., with Braille labels or special touch-sensitive buttons), requiring only a low- cost camera and laptop computer; (b) CamIO is accessible even to those who are not fluent in Braille; and (c) it permits natural exploration of the object with all fingers (in contrast with approaches that rely on the use of a special stylus). Note also that existing approaches to making graphics on touch-sensitive tablets (such as the iPad) accessible can provide only limited haptic feedback - audio and vibration cues - which is severely impoverished compared with the haptic feedback obtained by exploring a physical object with the fingers. We propose to develop the necessary computer vision algorithms for CamIO to learn and recognize objects, estimate each object's pose to help determine where the fingers are pointing on the object, track the fingers, recognize gestures and perform OCR (optical character recognition) on text printed on the object surface. The system will be designed with special user interface features that enable blind or visually impaired users to freely explore an object of interest and interact with it naturally to access the information they want, which will be presented in a modality appropriate for their needs (e.g., text-to-speech or enhanced images of text on the laptop screen). Additional functions will allow sighted assistants (either in person or remote) to contribute object annotation information. Finally, people who are blind or visually impaired - the target users of the CamIO system - will be involved in all aspects of this proposed research, to maximize the impact of the research effort. At the conclusion of the grant we plan to release CamIO software as a free and open source (FOSS) project that anyone can download and use, and which can be freely built on or modified for use in 3rd-party software.
For people who are blind or visually impaired, a serious barrier to employment, economic self- sufficiency and independence is insufficient access to a wide range of everyday objects needed for daily activities that require visual inspection on the part of the user. Such objects include printed documents, maps, infographics, and 3D models used in science, technology, engineering, and mathematics (STEM), and are abundant in schools, the home and the workplace. The proposed research would result in an inexpensive camera-based assistive technology system to provide increased access to such objects for the approximately 10 million Americans with significant vision impairments or blindness.
| Coughlan, James M; Miele, Joshua (2017) Evaluating Author and User Experience for an Audio-Haptic System for Annotation of Physical Models. ASSETS 2017:369-370 |
| Mascetti, Sergio; D'Acquisto, Silvia; Gerino, Andrea et al. (2017) JustPoint: Identifying Colors with a Natural User Interface. ASSETS 2017:329-330 |
| Coughlan, James M; Miele, Joshua (2017) AR4VI: AR as an Accessibility Tool for People with Visual Impairments. Int Symp Mix Augment Real 2017:288-292 |
