Currently there is a significant need for increasing mathematical literacy in the United States. Among other benefits, this would permit more Americans to pursue careers in STEM (Science, Technology, Engineering and Mathematics) disciplines. The most extensive resource for finding introductory information on math, the internet, primarily supports text-based search. Similarly, current Portable Document Format (.pdf) viewers support search for text, but not math. The goal of this research is to develop a query-by-expression mechanism, where users enter expressions using images, stylus/finger, mouse and keyboard. Users may then search using a combination of the expression appearance, symbols, structure, and mathematical semantics. Expression properties may be combined with text-based search, allowing queries for mathematical information to be more precise than current text-based methods.
For query-by-expression methods to be viable, improvements in math recognition are needed, along with the development of efficient methods for indexing and retrieving mathematical expressions. In particular, the project seeks to improve optical character recognition (OCR) for handwritten and typeset mathematics, along with methods for parsing expression structure. Approach based on Graph Transformer Networks (GTN) and adaptations of boosting techniques is applied to intelligent combination of modules that locate, recognize and relate mathematical symbols with an aim to further improve recognition. Research focuses on identifying appropriate features, distance metrics, indexing and search methods for expression retrieval. Developed techniques are evaluated via user studies both in-lab settings and through the internet. Additional user studies to identify appropriate use cases for query-by-expression are also planned.
This project is expected to produce new query-by-expression methods usable by both math experts and (perhaps more importantly) non-experts. These methods might be adapted to retrieving other non-textual document elements such as chemical diagrams, tables, and figures. Source code and experimental data developed for the project will be made public via the project web site (www.cs.rit.edu/~dprl/msearch.html). To promote mathematical literacy, the principal investigator and graduate students working on the project will visit middle schools and talk about the history, recognition and retrieval of mathematical notation. The PI also plans to participate in the McNair Scholars program at RIT, which seeks to provide research experiences to low-income, first-generation college students that are interested in pursuing doctoral studies.
