The lack of access to visual information like text labels, icons, and colors frustrates blind people and severely decreases their independence. Current access technology uses fully-automatic approaches to address some problems in this space but is error-prone, limited in scope, and expensive. Blind people who can afford to do so must carry multiple special-purpose portable devices with different audio and tactile interfaces in order to access critical data about their environment such as product information from bar codes and location information via GPS. These devices would likely be used more often if they had greater functionality and failed less often. Providing a fallback by making it easy to consult a human assistant could be part of the solution. The PI's talking VizWiz application for mobile phones is one such prototype that connects blind people to remote human workers who answer general questions about the users' visual environments. VizWiz currently allows blind users to take a picture, speak a question, and receive answers back quickly. Preliminary findings have demonstrated the potential advantages of including humans in the loop to help overcome visual problems that are still too difficult to be solved by automatic approaches alone, but questions remain about the efficacy, privacy, speed, and cost of these approaches. In this project the PI will seek answers to some of these questions, by conducting a longitudinal study of VizWiz with blind people to better understand how the application might fit into their everyday lives. He will endeavor to determine how users' existing social networks might be employed as a source of answers using applications for Facebook and Twitter. And he will seek to define new services with appropriate interfaces that let users mediate between automatic and human-powered remote sources for answers. A mobile accessibility solution using both automated and human Web services represents a significant advance in accessibility but presents challenging user interface questions. Understanding issues such as those enumerated above is necessary for human-powered services to be accepted as part of assistive technology.
Broader Impacts: This exploratory research represents a new paradigm in human-computer interaction in which humans are both clients and providers. VizWiz has the potential to improve the independence of blind people, and may be both less expensive and more sustainable than current accessibility solutions. This project will improve our understanding of the types of tools that would be useful for blind people regardless of what is possible today with automatic computer vision, and will help us better understand how to recruit people to answer questions while respecting the asker's values. The research will involve blind people throughout; the resulting interfaces and functionality will be evaluated by blind people in the world going about their everyday lives. The interfaces, applications, and framework created and improved as part of this project will be released as open source so other researchers may build on the PI's results. Project outcomes will be broadly applicable to other problems where automated solutions may occasionally need human intervention.
This project was a scientific investigation into VizWiz, is a mobile application that we created that lets blind people take a picture, speak a question about it, and receive an answer from people out on the web. We conducted a large-scale longitudinal field study of how blind people use VizWiz. To facilitate this we released VizWiz on the Apple App Store. Since then, nearly 1500 blind participants have used VizWiz to ask nearly 20,000 questions. The submitted questions provide an unprecedented look into what blind people actually want to know about their visual environments, helping to set a roadmap for computer vision. This is in addition to the near-term benefit of providing a valuable service to blind people. We also used our experience developing and releasing a human-powered technology to develop and publish design guidelines for technology in this area that may help future developers and researchers better articulate the advantages and disadvantages of future technology. Unsurprisingly, the primary difficulty that blind people have using VizWiz is taking good pictures – taking picture is difficult if you’re unable to see the viewfinder. We have worked with local blind photographers to motivate new automatic computer vision techniques that help blind people take better pictures in common scenarios. For instance, in one of our prototypes, blind people first take a close-up picture of an object of interest, their phone then tracks that object, and helps them frame it in a picture from farther away. Our techniques for supporting blind photography will be useful in a variety of ways. First, blind people want to take pictures for many of the same reasons that sighted people do, e.g. to share experiences, or capture useful information. Second, tools like VizWiz require good pictures to work their best. This award funded a graduate student researcher for one year, who worked on all research aspects of this project – inventing new technology, designing experiments, and writing and presenting results. In addition, three undergraduate researchers contributed to this project, including one student who is blind. The research products they developed in collaboration with the PI are some of the first examples of real-time human computation, and may be applied in a diversity of tools.
