Reference, the ability to single out objects in the world using linguistic expressions, is a fundamental part of communication. Cognitive scientists have showed that people collaborate on reference, for example by negotiating about how to describe things. This project lays the groundwork for computer systems that can do the same. Eventual applications range from talking robots that can communicate with people in real physical environments, to tutoring and decision-support systems that can give comprehensible explanations of specialized concepts. These applications are open domains, in the sense that one interlocutor, or both, may have no simple descriptor for an unfamiliar object. So the system has to have flexible strategies for describing things in words and inference mechanisms that acknowledge the possibility of a misunderstanding by either system or user.
This project undertakes exploratory work on the communicative strategies and inference mechanisms required for collaborative reference in open domains. During the academic year, PI Stone works with a computer science student to extend a prototype system with additional methods for collaborating under uncertainty, drawing on the student's ongoing research in cognitive modeling, planning and learning. During the summer, Stone works with a three-member interdisciplinary team doing corpus analysis, grammar development and integration to characterize generic strategies for describing shape, using rough descriptions and part-whole relationships. The results are an open-source prototype with more general collaborative reference abilities along with specific hypotheses, informed by publishable linguistic and conceptual analyses, about how these abilities might affect the system's success in referring to unfamiliar objects.
People increasingly use mobile devices to get information about what's going on nearby. This is a new way to interact with computers, and technologists are looking to develop powerful but natural new interfaces in response. Think of Google's glasses display, or Apple's Siri voice search. As engaging as emerging systems are, they still communicate with users in very limited ways. This project focused on expanding systems' ability to identify objects in the world naturally to people. When people communicate with one another, they identify objects by describing them creatively and flexibly. You can find diversity among people in almost any aspect of language use. For example, this project documented differences in the descriptions people use to pick out objects by color. Such variability is not surprising, but it means that you can't build a system to use color words - or any other vocabulary - in some one right way. Instead, systems have to infer good interpretations from knowledge of language, knowledge of the speaker, and the context of ongoing activity. This project developed important new tools for solving such problems. In general, computer scientists develop intelligent systems using representations - formal data structures that package related information together in a constrained and systematic framework. Onecontribution of this project is a new representation for utterance interpretation in interactive systems that shows how utterances relate to ongoing activity more precisely than previous approaches. Using this representation should make it easier for systems to exploit the context to recognize speakers' intended meanings. These results are published at a 2013 conference on computational semantics. In general, intelligent systems work by instantiating representations in response to reasoning problems using systematic search and empirical heuristics. A second contribution of this project is to describe new inference mechanisms for recognizing utterance interpretations. Given representations that link utterances to context and track the system's own experience, it becomes feasible to hypothesize new meanings in interpretations, rather than just to use the predefined meanings preprogrammed in the system. A preliminary empirical study showed that using this open-ended representation and inference improves the system's ability to accommodate creative and natural language from interlocutors. These results are published at a 2012 conference on computational semantics. The new representations and algorithms from the project offer more precise and flexible computational techniques for systems to understand human language. This is the intellectual merit of the project's results. The expectation is that systems with richer abilities will have important broader impacts in society by enabling natural interfaces for people to access information in a wide variety of contexts. The project has also contributed to the training of scientists to pursue this work in the future. The project brought together teams with a wide range of backgrounds and expertise - including undergraduates, graduate students and senior researchers, with training in computer science, lingustics, philosophy and psychology - for collaborative work. This work not only strengthened team members' expertise in their home discipline but exposed them to the interrelated threads of theoretical, computational and empirical research required for building intelligent interactive systems.
