People with low vision find it challenging to use currently available screen magnifiers, their "go-to" assistive technology for interacting with computing devices. Firstly, these magnifiers indiscriminately magnify screen content, including white space, as a blanket operation, often blocking important contextual information such as visual cues (e.g., borders), and semantic relationships between different user interface elements (e.g., a checkbox and its label) from the user's viewport, which is the visible area on a web page that the user can see at a given time. Mentally reconstructing contextual information from exclusively narrow views dramatically increases the burden on the user. Secondly, low vision users have widely varying needs requiring a range of dynamic customizations for interface elements, which in currently available magnifiers is disruptive as it causes users to shift their focus. Thirdly, navigation aids to help users explore the application, obtain quick overviews, and easily locate elements of interest are lacking. The proposed project will research the design and development of SteeringWheel, a transformative next generation screen magnification technology to rectify these limitations. SteeringWheel will be based on several novel ideas. First, it will magnify the white space and non-white space user interface (UI) elements differently in order to keep the local context in the viewport post-magnification. Second, it will confine the cursor movement to the local context, thereby restricting panning. Third, it will interface with a physical dial, supporting simple rotate-and-press gestures with audio-haptic (sense of touch) feedback, that will enable users to quickly navigate different content sections, easily locate desired content, get a quick overview, and seamlessly customize the interface. A byproduct of the project will be the creation of standardized benchmark data sets to gauge the performance of current and future screen magnification technologies. It is anticipated that SteeringWheel will make it far easier for low vision users to perceive and consume digital information, leading to improved productivity. The project will serve as a launching board for creating project-driven graduate and undergraduate courses in Accessible Computing.
The project will research, design and engineer SteeringWheel, a transformative next generation screen magnification technology that is predicated on the fundamental idea of Semantics-based Locality-Preserving Magnification (SLM). For retaining contextual information, SLM incorporates knowledge about the semantics of different UI elements and inter-element relationships, rooted in the concept of a logical segment, which is a collection of related UI elements exhibiting consistency in presentation style and spatial locality (e.g. a form-field textbox and its associated border and label). SteeringWheel, which overcomes the limitations of extant screen magnification technologies, rests on two scientific ideas, namely, incorporating semantics into the magnification process, complemented by an interaction paradigm based on simple rotate and press gestures with haptic feedback, that serves as an "all-in-one" interface for all magnification-related operations. The Research Plan is organized under six objectives. OBJ 1: Algorithms for locality-preservation targeting different screen-sizes for Desktops and Mobiles. Extraction algorithms will be designed to analyze the application layout, identify the semantically meaningful logical segments, and generate a semantic hierarchy by organizing these segments in an object-oriented fashion. Extraction will be followed by the design of locality-preserving algorithms that differentially magnify different types of content of these segments in the semantic hierarchy. Locality-preserving algorithms will keep most (if not all) of the contextual information within the users' viewport after magnification, to minimize the panning effort and hence the associated cognitive burden. OBJ 2: Mapping and integration of SteeringWheel gestures onto input devices for Desktops. A set of input gestures based on simple actions such as rotation and press will be designed for the SteeringWheel interface targeted at the Desktop platform. The set of gestures will be implemented on a Microsoft's Surface Dial and a Gaming Mouse. These gestures will enable users to quickly navigate and easily explore different segments of the application as well as seamlessly make magnification adjustments as needed. OBJ 3: Mapping and integration of SteeringWheel gestures onto input devices for Mobiles. A set of input gestures that are variations of simple rotation and press will be designed for the SteeringWheel targeted at the Mobile platform. The set of gestures will be implemented on Apple and Android Watches. These gestures will enable the users to easily adjust the magnification settings on-the-fly and conveniently explore the mobile application content without having to go through a frustrating and time-consuming interaction process using the default touch-based gestures (e.g. triple press) available in smart phones. OBJ 4: Infrastructure for semi-automated personalization of magnification settings. Techniques will be designed to support semi-automated personalization that enables users to make customizations only once, and SteeringWheel will remember and automatically apply these users' preferences each time they revisit the application. Customization at the granularity of individual segments will be supported, which will further reduce the user's tedious efforts by automatically applying the customizations made for one segment to all other similar segments. OBJ 5: Spoken dialog assistant for SteeringWheel. The Assistant will be designed to helps users easily locate and shift the navigation focus to segments or UI elements of interest using speech commands such as "take me to menu bar", "move to the first form field", etc. The Assistant will also allow users to specify commands such as "increase brightness", "invert colors", etc., for customizing the interface. Speech commands have the potential to transfer this cognitive burden from users to the magnification interface, thereby letting users focus on their tasks instead of expending needless time and effort locating and manually configuring individual segments and UI elements. OBJ 6: Infrastructure for porting users' magnification profile across different devices. Different methods will be designed to facilitate porting of users' profiles containing their preferred magnification settings for different applications to different devices, so that the low vision users need not make the same magnification adjustments for the same applications on different devices. Mechanisms will also be designed for letting users securely share their settings with each other, which will further reduce their interaction overload, as different users with similar eye conditions may need similar magnification settings for the same application.
This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
