This project develops and validates interaction design patterns, structured definitions of high-quality design solutions that can be applied at scale, that can be used to design more effective and engaging online mathematics problems. This research is conducted in the context of ASSISTments, free online mathematics software used by middle school students nationwide. The patterns are designed using data from 20,000 mathematics problems previously used by thousands of students (generating millions of data points), and validated through a set of forty small-scale randomized controlled trials conducted via automated experimentation, conducted online in American classrooms nationwide. The project is a partnership among Teachers College, Columbia University, Carnegie Mellon University, and Worcester Polytechnic Institute. The project makes two types of contributions. The first is are basic discoveries as to which types of content lead to the best learning and engagement in online problem-solving. The second is a set of guidelines for the design of online learning that can be applied by teachers and other content creators to produce educationally effective and engaging online mathematics problems.
The project is accomplishing these goals using the following procedure. First, they are studying the design features present in existing ASSISTments mathematics problems, by hand-labeling design features on a small sub-set of problems and then using educational data mining to replicate the hand-labels at scale. These design features include features relevant to interface design, domain content, and pedagogical strategies. They then apply previously developed and validated automated detectors of student learning, engagement, and affect to log files of students solving mathematics problems in ASSISTments. They use association rule mining to determine which combinations of design features lead to better learning, engagement, and affect, and build on these findings to develop interaction design patterns that communicate effective solutions which combine these data features. The design patterns are validated through a set of forty small-scale randomized controlled trials conducted via automated experimentation, where forty mathematics problems are improved using the design patterns. Each improved problem is studied in a random sample of 200 students (drawn from the full population of students currently using ASSISTments as part of their regular curriculum), who receive the problem as part of their regular classroom activities. They statistically assess the impact of these modified problems on learning and engagement using the outputs of automated detectors as dependent measures.
This project results in increasing the effectiveness and engagingness of the mathematics problems in the ASSISTments system, and identifies design patterns that can be applied to improve all of the content in ASSISTments. 50,000 students a year use the ASSISTments system, including large numbers of students from traditionally under-represented populations. More broadly, the proposed project is producin a generalizable and precise approach for the creation of more effective and engaging online learning. The design patterns developed are likely to be useful for improving the design of the range of online problem-solving systems used increasingly in American mathematics education.
