Nearly every graphical user interface (GUI) is implemented using some form of GUI toolkit. While these toolkits have enabled many successes of the past forty years of research and practice in human-computer interaction (HCI), they have unfortunately also become stifling. The rigidity and fragmentation of current GUI toolkits makes it difficult or impossible to deploy new ideas in complex existing applications, so researchers are instead often limited to demonstrating and evaluating small toy applications. This is a major challenge for the field, as it limits both the progress and impact of HCI research. The PI's goal is to transcend the fragmentation of modern GUIs by building upon their single largest commonality, namely that all GUIs ultimately consist of pixels painted to a display. He argues that if it were possible to quickly and robustly interpret those pixels, without application source code and independent of its underlying toolkit implementation, we could use that understanding to modify any interface. The PI's prior work on the Prefab system has shown promising preliminary results, enabled by a strategy of reverse engineering widget appearance. In the current project the PI will develop methods for pixel-based interpretation of graphical user interfaces, including methods for identifying complex widgets, for extracting widget content, and for recovering widget relationships, and he will build upon these to model widgets state and to address occlusion of portions of interfaces. He will create methods for modifying graphical user interfaces, including semantic views on pixel-based interpretation, methods for manipulating underlying interfaces, methods for translation between presentations, visual programming methods, and end-user tools for interface customization. He will implement new tools for informing, deploying, and evaluating interaction research in the field, and he will partner with leading researchers to apply these tools to target-aware pointing and automatic interface generation. And he will validate his pixel-based methods in real-world datasets collected from broad deployments.
Broader Impacts: In this project the PI will pursue a vision of a transformative democratization of every aspect of interaction, in which instead of being limited to a single provided interface anybody can modify any interface of any application. For example, an HCI researcher might implement and evaluate a new interaction technique in several existing applications; a practitioner or hobbyist who sees the publication might add the technique to their favorite applications. Web communities might organize around causes, such as translating interfaces into new languages, improving the accessibility of existing applications, or updating interfaces to support gestures, speech access, or other interactions. End-users might browse libraries of extensions, vote on their favorites, and use visual programming tools to create their own. Project outcomes, including tools and dataset, will be openly disseminated. The PI will make special efforts to recruit students from under-represented groups, to integrate the research into existing and new courses, and to use a high school design competition as a channel for recruiting high school students for internships.
