Objectives: The purpose of this research proposal is to test the hypothesis that present day computer-based Automatic Speech Recognition (ASR) systems can be effective and costefficient alternatives to human-generated text generation systems or sign language interpreters, and that the method of displaying the information plays a crucial role in the learning process. The Rochester Institute of Technology (RIT) and the National Technical Institute for the Deaf (NTID), one of eight colleges of RIT, together with a consultant from the Georgia Tech Research Institute (GTRI), will test this hypothesis.
Intellectual Merit: Working with Deaf and Hard of Hearing (D/HOH) student participants mainstreamed in baccalaureate programs at RIT, we will test three text-generated systems, and three different text presentation systems, under realistic but well-controlled lecture-venue conditions. We will compare these systems for accuracy in rendering the lecturer's narrative, evaluate students' comprehension of lecture content, and analyze ease of use and deployment from both the student participants' and the lecturers' points of view. The proposed research will be conducted in two stages. In Stage 1, we will evaluate three different speech to text systems: (1) CART, a word-for-word real-time speech-to-text closed captioning service driven by a steno machine operator; (2) C-Print, a meaning-for-meaning display and note-taking system driven by a captionist using abbreviation-expanding software; and (3) ASR speech-to-text transcription driven by the most powerful conversational speechrecognition software available on personal computers (Dragon Naturally Speaking 8 Preferred). In Stage 2, we will use the best text-generation system from Stage 1 to perform display tests comparing three different methods of putting text and non-text content into close visual proximity: (1) a heads up display of the text generated by the text generation system; (2) a PCbased system developed at RIT that allows continuous broadcast of the text stream, a PowerPoint display, and/or video stream of the lecturer to adjacent panes of conventional web browsers on the participant's personal computer; (3) the same composite display projected on a screen.
Broader Impacts: Deaf and Hard of Hearing (D/HOH) individuals, as well as people with other disabilities such as learning disabilities (LD), dyslexia or attention deficit hyperactivity (ADHD), are often denied full participation in educational, civic, cultural, medical and employment-based activities. Some venue operators do provide sign language interpreters, real time display systems or note-taking systems, but the practice continues to be rare because logistics are challenging and expenses are significant. Recent advances in ASR systems have the potential to expand availability of near-real-time display and/or transcription, and reduce communication and educational barriers that still distance D/HOH from the rest of society. Our findings will offer expanded strategic options and data-based guidance to people who are D/HOH and to providers and administrators of services to D/HOH people. The results from this study will also be of great benefit to people who have other disabilities, and for people for whom English is a second language.
