Regardless of advances in technology, a continual challenge to science, math, and engineering (SMET) education for blind students, and for their careers, is the task of examining and interpreting data, typically accomplished with visual graphs by sighted users. While there have been many projects aimed at enhancing SMET education and scientific access for the blind by adding sound in some manner, there has been very little systematic study of how, exactly, to create effective auditory graphs. Without such auditory display tools (in combination with visual graphs), there is little chance for visually impaired individuals to participate in science, especially when collaborative discussions are part of the learning or discovery process. In this project, the PI will conduct a program of research and empirical validation into the most effective ways to design and develop auditory graphs, train users to interpret them, and evaluate their effectiveness and appropriateness alongside visual graphs. The PI's preliminary findings indicate that both sighted and visually impaired students and scientists can make use of the auditory display of quantitative information, so long as the displays are sophisticated and well designed, the listeners are trained, and the designer is aware of any differences that may exist in how the various listeners tend to interpret what they hear. A multimodal display should be even more effective, especially when there are both sighted and blind participants in a classroom or research team. Building on these insights, the PI will first perform fundamental experiments aimed at determining the optimal methods for providing contextual cues to auditory graphs, analogous to the axis or tick marks in a visual graph. Once these auditory graph design guidelines have been determined, open source software will be developed to produce both auditory and visual graphs simultaneously, so that teachers or others working with data need not duplicate their efforts. The PI will then conduct experiments to determine the most effective ways to train listeners to interpret the auditory graphs, since it is crucial to overcome the lack of experience that most students, sighted or blind, have with auditory displays. Online training materials will also be developed in order to ensure that the teachers who are likely to create and utilize these sophisticated auditory graphs have the requisite knowledge and skills. The training methods and materials will be evaluated and validated in an experiment and focus group with teachers who work with sighted and visually impaired students. Finally, the efficacy of the enhanced auditory+visual graphs will be validated by comparing the learning outcomes for students who are taught science concepts with the aid of auditory vs. auditory+visual graphs. All of these studies will involve the participation of teachers and both visually impaired and sighted students.
Broader Impact: Well-designed auditory displays should be invaluable for both sighted and visually impaired students. Having multiple formats for the data displays will allow all users to maximize learning, and have a shared basis for collaboration. The ongoing research projects will be tied in to university level classes, so that students can learn by taking part in the research efforts; the results and findings will then be rolled into the materials for subsequent classes in computing, psychology, and assistive technology. Visually impaired participants will be a major constituent in focus groups, experiments, and prototype evaluation; this will encourage them to become involved in future R & D efforts that focus on technologies of use and of interest to them. Project outcomes will be disseminated through the PI's various partner organizations and international research collaborators; web sites will make the software, training, and research materials freely available worldwide.
Our research has led to a better understanding of how to design auditory and multimodal graphs, and how to train people to use them. We have learned about how different people interpret auditory graphs, and how to use this to make better displays, especially for learners with vision loss. We have rolled all these findings into our software, which allows users to make effective and usable auditory graphs. We are now applying this in a practical educational setting (math education for blind students). Our research will make math and science education more accessible, which will enable more visually impaired students to gain STEM skills, and thereby enhance their career opportunities. The software tools we are developing are also relevant and useful for visually impaired individuals who are already working in math, science, and data-intense fields, which opens up the range of careers available to people with vision loss. It is also true that our tools are useful for students and scientists, regardless of visual abilities, and may lead to new science discoveries due to the different analysis methods that are now more available (in particular, multimodal representations of data). Our research has inspired the development of several additional software tools, and has led to the development of enhanced auditory interfaces for other important software, such as popular spreadsheet software programs. These potential new products may have broad impact in the data analysis market, and on the design and development of all kinds of multimodal information displays.
