n many technical disciplines, from electrical to mechanical engineering, the simplest and most straightforward way to communicate an idea is to draw a picture. Diagrams are particularly powerful in education when they are combined with computer simulation programs because, unlike paper, simulation programs allow students to explore the behavior of a physical system, such as a circuit or a mechanical device. Unfortunately, the mouse and keyboard interface to these programs prevent students from drawing their diagrams freely, forcing them continually to consult menus to choose pieces of the diagram. The hardware to draw on the computer exists; the bottleneck is the computer's inability to understand diagrams.
The PI's central research goal is to construct and deploy computer simulation tools capable of understanding students' hand-drawn diagrams. The results of free-sketch recognition research cannot yetbe incorporated into end-user applications for two reasons. First, free-sketch recognition is not sufficiently robust to incorporate into useful tools. Second, little is known about how to build usable interfaces that incorporate free-sketch recognition. This work will bridge the gap between free-sketch recognition technology and its end-users by focusing on its application to undergraduate engineering design. The outcome of this work will be improved techniques for free-sketch recognition, guidelines for incorporating free-sketch recognition into usable interfaces, and educational sketch-based simulation tools. This project also includes a significant outreach component aimed at establishing a community of educational and technological researchers with the combined expertise necessary to evaluate the effectiveness of these tools on learning.
This research project directly supports the PI's educational goal of increasing the number and diversity of students who use and who develop technology. First, technology that can understand hand-drawn diagrams will provide students with a more familiar interface to simulation programs, lowering the barrier to using technology in the classroom, particularly for students with less computer experience including women and underrepresented minorities. Second, this research will be performed at Harvey Mudd College, an exclusively undergraduate institution. The small size of the college and low faculty to student ratio (1:9) will enable the PI to work closely with undergraduates, teaching them the basics of performing research and preparing many for graduate work.
Even in today's technology-enabled classrooms, drawing remains a central activity for students in design-oriented classes. Drawing allows students to think about their designs and to focus on learning the discipline rather than on learning to use a potentially complex tool. Of course, drawing on paper does not allow students to see the behavior of their designs, so students usually transfer their designs to a computer tool after drawing them on paper. This two-stage process is cumbersome and does not allow the computer tool to provide assistance to the student as they are initially creating their designs, when they are likely the most confused. This project developed a tool named LogiSketch, which is a system that allows students to freely draw and then simulate digital logic circuit diagrams. LogiSketch makes four central contributions to the field of pen-based educational tools. First, it is the first pen-based simulation tool for the domain of digital circuit design. This complex domain presents many new challenges: sketches can be quite large, many of the symbols in the domain are visually similar, and drawing styles vary significantly between users. Second, LogiSketch implements delayed recognition of unconstrained sketches in a complex domain, and provides novel recognition feedback and adaptive error correction. Third, LogiSketch incorporates behind-the-scenes user-targeted learning that improves recognition as a student uses the system. Finally, LogiSketch’s user interface incorporates state-of-the-art pen-based drawing and editing techniques to create a seamless drawing and editing interface that uses only the pen. We deployed LogiSketch in a pilot study in an introductory computer science course at Harvey Mudd College during a lab on digital circuit design. Our pilot study upheld our central premise that students would find value in being able to freely draw and then simulate their circuits. But it also showed that LogiSketch's recognition rate, though significantly higher than previous techniques for free-sketch recognition, is not yet sufficiently high to make LogiSketch a suitable replacement for point-and-click menu-based simulation tools. Our work has provided a foundation and direction in which to refine these free-sketch recognition algorithms. LogiSketch along with several labeled sketch data sets are freely available for download from our project site: https://sites.google.com/site/logisketchucsd/
