Providing Access to Appliance Displays for Visually Impaired Users Summary The goal of this project is to develop a computer vision system that runs on smartphones and tablets to enable blind and visually impaired persons to read appliance displays. This ability is increasingly necessary to use a variety of household and commercial appliances equipped with displays, such as microwave ovens and thermostats, and is essential for independent living, education and employment. No access to this information is currently afforded by conventional text reading systems such as optical character recognition (OCR), which is intended for reading printed documents. Our proposed system runs as software on a standard, off-the-shelf smartphone or tablet and uses computer vision algorithms to analyze video images taken by the user and to detect and read the text within each image. For blind users, this text is either read aloud using synthesized speech, or for low vision users, presented in magnified, contrast- enhanced form (which is most suited to the use of a tablet). We propose to build on our past work on a prototype display reader using powerful new techniques that will enable the resulting system to read a greater variety of LED/LCD display text fonts, and to better accommodate the conditions that often make reading displays difficult, including glare, reflections and poor contrast. These techniques include novel character recognition techniques adapted specifically to the domain of digital displays; finger detection to allow the user to point to a specific location of interest in the display; the use of display templates as reference images to match to, and thereby help interpret, noisy images of displays; and the integration of multiple views of the display into a single clear view. Special user interface features such as the use of finger detection to help blind users aim the camera towards the display and contrast-enhancement for low vision users address the particular needs of different users. Blind and visually impaired subjects will test the system periodically to provide feedback and performance data, driving the development and continual improvement of the system, which will be assessed with objective performance measures. The resulting display reader system will be released as an app for smartphones and tablets that can be downloaded and used by anybody for free, and also as an open source project that can be freely built on or modified for use in 3rd-party software.
For blind and visually impaired persons, one of the most serious barriers to employment, economic self-sufficiency and independence is insufficient access to the ever-increasing variety of devices and appliances in the home, workplace, school or university that incorporate visual LED/LCD displays. The proposed research would result in a smartphone/tablet-based assistive technology system (available for free) to provide increased access to such display information for the approximately 10 million Americans with significant vision impairments or blindness.
| Coughlan, James M; Miele, Joshua (2017) Evaluating Author and User Experience for an Audio-Haptic System for Annotation of Physical Models. ASSETS 2017:369-370 |
| Fusco, Giovanni; Tekin, Ender; Giudice, Nicholas A et al. (2015) Appliance Displays: Accessibility Challenges and Proposed Solutions. ASSETS 2015:405-406 |
| Fusco, Giovanni; Tekin, Ender; Ladner, Richard E et al. (2014) Using Computer Vision to Access Appliance Displays. ASSETS 2014:281-282 |
| Tekin, Ender; Vásquez, David; Coughlan, James M (2013) S-K Smartphone Barcode Reader for the Blind. J Technol Pers Disabil 28:230-239 |
| Manduchi, Roberto; Coughlan, James (2012) (Computer) Vision without Sight. Commun ACM 55:96-104 |
| Tekin, Ender; Coughlan, James M; Shen, Huiying (2011) Real-Time Detection and Reading of LED/LCD Displays for Visually Impaired Persons. Proc IEEE Workshop Appl Comput Vis :491-496 |
| Tekin, Ender; Coughlan, James M (2010) A Mobile Phone Application Enabling Visually Impaired Users to Find and Read Product Barcodes. Comput Help People Spec Needs 6180:290-295 |
| Tekin, Ender; Coughlan, James M (2010) A Mobile Phone Application Enabling Visually Impaired Users to Find and Read Product Barcodes. Lect Notes Comput Sci 6180:290-295 |
| Tekin, Ender; Coughlan, James (2009) A Bayesian Algorithm for Reading 1D Barcodes. Proc Can Conf Comput Robot Vis 2009:61-67 |
| Tekin, Ender; Coughlan, James M (2009) An Algorithm Enabling Blind Users to Find and Read Barcodes. Proc IEEE Workshop Appl Comput Vis 2009:1-8 |
