Despite the enormous effort that goes into application design, end users invariably encounter workflow scenarios that are poorly supported by even the most developed, commercial-quality applications. Lack of support for end-user extension increases the burden on application developers to exhaustively anticipate the needs of all end users and then to commit enough resources to address their needs, which often includes re-implementing solutions already found in other applications. This project develops the feasibility of externally retrofitting, without altering, running applications with new interfaces and functionality made possible by using a combination of application-independent pixel-level recognition techniques and more specialized techniques for inspecting data structures exposed by the window system or application. Users will be able to make annotations to those running applications, including hand-drawn ink, typed text, diagrams, interactive widgets, or even links to other application user interface components. Registration techniques will then be researched to associate the annotations with specific elements of an application or document so that such an annotation can be made to appear perhaps only in one place in a specific file, or whenever a certain application runs. This supports a variety of practices, including: integrating functionality from different applications, enriched collaboration, task or user customized interfaces, and adding new fine-grained user interface elements to applications.
Broader Impact. Making applications more malleable by embracing customization as a first class and general notion could fundamentally shift application design. Instead of relying solely on inherently slow development of one-size-fits-all applications, end-users could become de facto participants in application design, somewhat analogous to the relationship between traditional print-copy publishing and blogging. Rather than waiting, possibly years, for even simple revisions to fix workflow inefficiencies, users will be empowered to make limited modifications to applications almost as easily as marking up a document, for example by freely rearranging applications? interfaces, or adding diagrammatic annotations displayed over documents which do not natively support them.
Despite the enormous effort that goes into application design, end users invariably encounter workflow scenarios that are poorly supported by even the most developed, commercial-quality applications. In this project, we explored a variety of ways for end-users to modify the behavior of applications without involving technically savvy application programmers to modify the application’s source code. The idea is essentially to model the modification of an application’s behavior from a physically-based perspective. Instead of using programming techniques, we wanted to make an application feel more like a piece of paper that can be cut-up, re-arranged, and blended with pieces of other pieces of paper. Since the process of cutting up applications is not directly supported in existing systems, we show the users a "copy" of the display screen – the real, unmodified display screen (with the running applications) is never shown but still exists. Thus, the copied image can be cut apart and re-assembled since it is just an image – however, all the while we maintain a link between each piece that has been moved or changed and the unmodified "original". If the user tries to interact with their modified, copied version of the screen, we transform all of their actions into corresponding interactions on the original display screen. By keeping the modified copy of the display screen synchronized with the original display screen, end-users can feel as if they have changed the application’s behavior even though the original application is still running, unmodified, "in the background." During this project, we developed techniques for recognizing application elements, techniques for copying and modifying a display screen such that it is still synchronized with the original display, and we developed a set of retrofit applications. Recognizing application elements: There are two classes of techniques for recognizing application elements so that they can be synchronized with changes made by a user: applications can be recognized optically, the way a human recognizes an application in terms of its colors, shapes, and text labels; or applications can be recognized by probing the abstract representation of applications that operating systems may expose. The former techniques are the most general since they can work with any application, but the latter techniques can often access ("see") more of an application since not all aspects of an application can be seen on the display all the time (e.g., the elements of a File menu cannot be seen before clicking on the File menu, although data structures representing elements of the File menu generally will always exist). In this project, we developed a toolkit of techniques which try to unify recognition based approaches with data structure based approaches to provide the best balance of generality and robustness. Modifying application behavior: To modify the way applications work, we allow users to "copy" regions of the screen. These screen copies know where in the original application they came from. Thus, for example, if the user copies a button from an application, moves it somewhere else, makes it bigger, and then clicks on it, we pass that click event back to the original button in the underlying (but not visible) application and if that click modifies the appearance of the underlying application, we copy those changes back to the users modified version of the display. By combining the ability to cut apart user interfaces with the ability to create macros (automated sequences of activities), end-users can generate novel application behaviors on their own. For example, a user can instruct another on how to use an application by writing messages over menus and buttons; when another user runs the application, those messages will show up next to right buttons/menus even if their windows are arranged differently on the screen. Interesting retrofit functionality: To emphasize the utility of modifying application behaviors, we created several retrofit applications, including Code Bubbles, Hands-On Math, and a Self-Binding UI. Code Bubbles is a new user interface for writing code that extends an existing application, Eclipse (www.codebubbles.org). With CodeBubbles, the file-oriented display nature that is common to programming environments is fundamentally changed as users are able to "tear out" parts from a file (e.g., a function) and juxtapose them with other displays of documentation, variables, etc. We found the simple process of creating customized displays made programmers significantly more productive for controlled tasks as they could easily see a working set of just the materials they were interested in. With Hands-On Math, we showed that a pen and multi-touch based interface to Mathematica made people want to use mathematical assistance more frequently as they worked. Finally, with our self-binding UI, we allowed users to schedule personalized interruptions of their own workflow to help them manage complex tasks.
