The human visual system is characterized by a large spatial bandwidth and concurrent processing capabilities, enabling individuals to learn from individual components, groups of items, and patterns in complex visual layouts. The enabling capabilities of visual information processing are vital in many areas the understanding of documents with mathematical content is one of such areas. The traditional representation of mathematics is inherently spatial and multi-dimensional (e.g., matrices, fractions), and it often combines analytical presentation (e.g., formulae, definitions) with intuitive graphical descriptions (e.g., diagrams). This creates a fundamental obstacle for people with visual disabilities; their lack of visual access prevents them from accessing traditional representations of mathematical content, even when encoded in digital format (e.g., on-line mathematics textbooks). The traditional assistive technologies employed by students with visual disabilities (e.g., Braille displays, screen readers) are inherently linear, removing contextual references and imposing very high load on the reader's working memory. Effectively, accessibility of mathematics has become a digital barriers to information and learning opportunities. This is particularly severe in the educational context, where the emphasis on distance education and on-line material has not been properly matched by the adoption of accessibility and universal usability principles. The goal of this project is to develop a body of knowledge and tools that will enable students with visual disabilities to access on-line educational material with significant mathematical content. The aim of this work is to enable visually impaired students to access and learn mathematics with a level of efficiency and following cognitive processes that are at par with those used by sighted students. In particular, we aim at creating a comprehensive framework which provides access to mathematical content at the analytical level (e.g., formulae, theorems, proofs), at the exploratory/intuitive level (e.g., content exploration, access to diagrams and charts), and at the practical level (e.g., problem solving, exercises). The guiding principle of this work is to understand and formalize the underlying semantic structure present in mathematical documents, starting from its basic components (e.g., individual formulae, components of a graph) and moving towards more complex organizations (e.g., logical steps, proofs, theorems, diagrams). The semantic structure is employed to guide multi-modal presentation schemes, which provide an automatically tuned balance between presentation (i.e., continuous access to the content) and navigation (i.e., user-directed traversal of the content), presentation adaptation based on user models, and combination of aural and haptic presentation. Haptic presentation relies on a novel hardware device (the haptic finger) to be designed and developed as part of this project. The framework we propose will also enable visually impaired students to interleave eading" mathematics with practicing" it, through an accessible workbook, which directly interacts with the reading process. Different levels of evaluation will be applied, involving visually impaired students early on in the design, and relying on formal cognitive models to assess progress. The project assembles a team of experienced researchers with expertise in the areas of applied cognitive/perceptual research, assistive technology software and hardware research, and usability design and evaluation. The researchers have collaborated previously, e.g., Tran and Pontelli conducted NSF-sponsored research in accessibility of the web for visually-impaired students. Intellectual Merit: (1) Observation and analysis of human-computer interaction by students with visual disabilities with mathematical content presented in electronic form, (2) Development of methodologies to extract semantic representations of mathematical content; (3) Design of a haptic device and control software for the presentation of graphical representations of mathematical concepts; (4) Study of approaches to aural presentation and navigation of mathematical content; (4) Integration of haptic and aural presentation and navigation of mathematics, driven by user models, and composed with workspaces for accessible practice of mathematical concepts; (5) Usability evaluation with sighted and visually-impaired students. Broader Impact: (1) promote teaching, training, and learning, with focus on diversity due to visual disabilities; (2) broaden the representation of students with visual disabilities in their ability to access and navigate documents with heavy mathematical content (e.g., math textbooks), thereby increasing their educational opportunities in all areas, but especially in the sciences and engineering; (3) enhance infrastructure for education by improving the ability of usability experts to evaluate haptic and auditory interfaces in an educational setting. Ultimately, society will benefit from the greater inclusion in education and improved training opportunities for students with visual disabilities and generally by making information more widely accessible to these users. All PIs are actively involved in initiatives to advance the educational opportunities for a diverse student population - e.g., NMSU is a Minority and Hispanic-serving Institution, and students from traditionally underrepresented groups will be involved in the research and educational activities. Keywords: accessibility of mathematics, haptic navigation, visual disabilities, usability evaluation methods
The traditional representation of mathematics is inherently spatial and multi-dimensional (e.g., matrices, fractions), and it often combines analytical presentation (e.g., formulae, problems) with intuitive graphical descriptions (e.g., diagrams). This creates a fundamental obstacle for people with visual disabilities; their lack of visual access prevents them from accessing traditional representations of mathematical content, even when encoded in digital format (e.g., on-line mathematics textbooks). The traditional assistive technologies employed by students with visual disabilities (e.g., Braille displays, screen readers) are inherently linear, removing contextual references and imposing very high load on the reader's working memory. Effectively, accessibility of mathematics has become a digital barrier to information and learning opportunities. This is particularly severe in the educational context, where the emphasis on distance education and on-line material has not been properly matched by the adoption of accessibility and universal usability principles. The goal of this project is to develop a body of knowledge and tools that will enable students with visual disabilities to access on-line educational material with significant mathematical content. The aim of this work is to enable visually impaired students to access and learn mathematics with a level of efficiency and following cognitive processes that are at par with those used by sighted students. In particular, we aim at creating a comprehensive framework that provides access to mathematical content at two levels: 1. the analytical level, by providing students with the ability to access and manipulate mathematical content. 2. the abstract level, by providing students with the ability to access non-textual representations of mathematical content (e.g., diagrams and charts). The project articulated the following main research outcomes. The first outcome focused on the development of new multimodal solutions for presenting both analytical as well as graphical representations of mathematical content. We decided to focus on MS Excel as the target of our work – as it provides instances of all the complexities faced by visually impaired. Excel includes traditional mathematical formulae, tabular organization of numbers and other types of data, and graphical representations (as charts). The research led to the creation of a plugin for Excel, connecting it to specialized environments for (1) the non-visual navigation and presentation of mathematical formulae; (2) the non-visual detection, navigation and presentation of complex tables; (3) the non-visual navigation of two dimensional charts. Several challenges had to be addressed to achieve these objectives. First of all, tabular structures are implicit in an Excel worksheet – requiring the development of novel algorithms to automatically identify tables. The second, more significant challenge, is the development of effective methodologies to present 2-dimensional charts. We developed a novel methodology that combines aural, speech, and haptic components in presenting a chart. The core instrument is an off-the-shelf and affordable haptic device (Novint Falcon), used to allow students to explore the content of a chart (receiving haptic feedback for the main components) and ask questions (receiving speech feedback). The specific components as well as the overall infrastructure has been successfully validated with students from the New Mexico School for the Blind and the Visually Impaired. We also investigated methodologies to enable visually impaired students to solve mathematical problems, from basic arithmetic to fundamental algebra. Informed by observations of existing practices in teaching arithmetic and algebra to visually impaired students, we developed a learning management system (LMS) for non-visual mathematical practice. The system provides students with an infrastructure to set up problems and exercises (e.g., arithmetic problems, algebraic problems) and support for solving problems. The LMS is accessible to visually impaired students and provides non-visual access to the operations required to solve arithmetic and algebraic problems. The LMS is adaptive – it changes the level of automation and support provided to the student based on the student’s progress in solving problems. As the student gains increased competency, a higher degree of automation is offered, to allow the student to focus on the more challenging and higher-level aspects of the problem solving. The LMS has been designed following a user-centered perspective, and assessed in several classroom applications. Through its reports, software and hardware solutions, and publications, the project has advanced the state-of-the-art in several areas of accessibility solutions for visually impaired students – e.g., presentation and manipulation of mathematical content, navigation of 2-dimensional charts. The solutions have already been tested and deployed in several classrooms, enabling students and teachers to avail of these new technologies and improve understanding of mathematics.
