American Sign Language (ASL) is used by as many as two million people in the United States, with additional users elsewhere in North America. The purpose of this "planning grant" is to enable the PI and her multi-institutional team to explore the case for a possible future NSF investment in an annotated, publicly available, and easily searchable corpus consisting of terabytes of ASL video data (deriving in part from prior work by the PI and her colleagues), including diverse types of content such as dialogues, narratives, elicited sentences illustrating specific grammatical constructions, and isolated signs. The PI contends such a resource would constitute an important infrastructure that would be exploited by a broad research community to advance the fields of linguistics (the structure of ASL), computer vision (machine recognition of gestures), indexing of visual information (through the expansion of mark up vocabularies), and education. The PI notes that the potential value of the existing corpora remains largely untapped, notwithstanding their extensive and productive use by her team and others, due to hardware and software limitations that make it cumbersome to search, identify, and share data of interest.
Broader Impacts: The new resource would be easily accessible by the research community and the broader public, via a user-friendly Web-based interface. Availability of the resource online would allow ASL teachers and users, and others, to access the data directly. Users would be able to look up an unknown sign by submitting a video example of that sign. Students of ASL would be able to retrieve video showing examples of a specific sign used in actual sentences, or examples of a grammatical construction. ASL instructors and teachers of the Deaf would have easy access to video examples of lexical items and grammatical constructions as used by a variety of native signers, for use in language instruction and evaluation.
This project was built upon multi-year collaborative efforts by Boston University, Rutgers University, and the University of Texas at Arlington, to create large, annotated datasets of American Sign Language (ASL) videos for use in linguistic and computer science research, as well as in education. As a result of these efforts, we now have datasets containing terabytes of annotated ASL videos. These datasets have been publicly shared, and they have been a valuable resource to researchers studying the linguistics of ASL, as well as to researchers designing methods for computer-based recognition of ASL. However, the full potential of these assets had not been realized because of the difficulties in browsing the datasets for materials relevant to specific types of research and downloading the appropriate subsets of materials. The NSF grant we were awarded for this project was a "planning grant", allocating a total of $100,000 to the three collaborating institutions (Boston University, Rutgers University, and the University of Texas at Arlington). The main goals were (1) to develop a proposal for a larger grant intended to establish the necessary infrastructure for enabling these rich datasets to be maximally useful as a community resource, and (2) to lay the foundation for making the data accessible, by developing the prototype of a Web interface for accessing these large datasets, so that researchers can easily locate and download content of interest that is available. The three collaborating institutions have used this planning grant in the following ways: (1) to establish interactions with other sign language researchers, and discuss ways in which they have been using our datasets and ways in which they would like to be using these datasets in the future (so that we could plan on incorporating the most desired functionalities into our Web interface for the datasets). (2) to design a prototype of the interface, incorporating some of the most basic functionalities that we would like the final interface to include. Substantial progress was made on the prototype: The database was redesigned such that it can support the required types of queries, and enforce consistencies across annotations. The database has been populated with annotations, glosses, and their metadata. The following functionalities have been developed: browse functionality for glosses, filtered by various criteria, with statistics by participant; display functionality for viewing utterances by gloss and participant, filtered by various criteria; and sentence search functionality, filtered by various criteria. Gloss images were adapted to the new database schema, and regenerated gloss images were linked into the display of the retrieved ASL utterances. An inventory of videos, associated with each annotation file, with complete metadata, was created. A working prototype of the Web interface is now publicly accessible at http://secrets.rutgers.edu/dai/queryPages/. A full NSF grant proposal, to include Gallaudet University as a partner institution, was submitted and funded with a start date of 8/11/2011. This will provide the infrastructure to enable full implementation of the Web interface and broad public release of the searchable corpora, as a community resource. Further enhancements of the interface, including addition of download functionalities and expansions of the annotated corpora to include new types of data, will be carried out as part of the full grant. The resources made available through this project will have significant applications for the teaching and learning of ASL and for interpreter training. Furthermore, the linguistic and computer science research made possible by these resources holds great promise for leading to technologies that benefit the deaf community. These include tools for language learning, mobile sign language dictionaries and retrieval, and tools for searching for signs by example. Ultimately, this resource also is likely to contribute to automated machine translation systems and improved methods for human-computer interaction.
