For nearly every major topic in an elementary science course, student assignments include many practice problems. Repeated practice helps students hone their skills. The project explores if software can simulate authentic interaction giving students a more effective experience than traditional rote drill-and-practice. Investigators are developing courseware, called Syrus, to help computer science students develop their skills with predicate logic and proof argument. Syrus allows student interactions that are more flexible than can be achieved with practice problems from a textbook. Students are not given a selection of answers to choose from and are not simply told if their answers are right or wrong. Instead, open-ended responses to practice problems are checked by sophisticated mathematical software in real-time. The system generates an almost endless supply of problems meeting instructor-specified criteria. Syrus is a web application available to anyone, anywhere, using a standard web browser.
"Practice makes perfect." Is is true? The primary goal of this project was to evaluate the impacts on learning outcomes that could be obtained for certain topics in discrete mathematics and logic, by creating an on-line tool based on several observations related to STEM education in general: Introductory courses require students to practice skills and test their understanding on homework problems. Typical human grading of such assignments severely limits the number of such problems that can be attempted by students with useful and timely feedback. State-of-the-art tools used by researchers in the field can sometimes solve such problems automatically. Large numbers of problems with similar (pedagogically useful) structure can be generated automatically. The idea of this pilot project was to build a tool, Syrus, to generate a large number of random problems related to logic and reasoning in a discrete math course for computer science majors; to solve these problems automatically using automated theorem-proving tools available to logic researchers, so the answers would be known to the system; to provide these problems as homework problems to students in an introductory course; and to evaluate the extent to which learning outcomes achieved were related to various measures of student engagement with, and success in solving, these practice problems. The Syrus project showed some promise in improved outcomes in a discrete mathematics course, when adjusted for expectations based on grades in previous math courses. This result was consistent among different subpopulations. However, no effect of the use of Syrus was seen in a question embedded in the final exam that was specifically designed to gauge the knowledge that Syrus most directly covered. Further study in an experimental setting where heavy use of Syrus is required would seem to be needed to better understand its value. It has become especially clear in the last few months that the key ideas behind Syrus will have increasing impact as higher education struggles with the potential of "massive open on-line courses" (MOOCs). MOOC students in STEM courses in particular will need to practice solving problems, some of which are of a sort that can be solved automatically with sophisticated research tools used by researchers in the same discipline. Students should receive, because of their "massive" numbers, automated feedback; and, because it is possible with automation, instantaneous feedback. The key ideas behind the Syrus project are already appearing in MOOCs. For example, Jim Fowler and Bart Snapp of the Ohio State University Mathematics Department are developing their own practice-problem platform, called MOOCulus, to support the Coursera "Calculus I" course that they are currently teaching to over 30,000 students worldwide. MOOCulus supports interactive calculus problem-solving for students: it generates random practice problems, solves them using various sophisticated tools available to researchers, provides real-time hints and feedback to students based on hidden knowledge of how these tools solve the problems at hand, checks student answers against those produced by the tools, and collects copious amounts of data about student usage and performance that eventually will be used to evaluate how much practice-with-instant-feedback impacts learning outcomes. In fact, the primary difference between MOOCulus and Syrus is a subject-matter difference. The application of Syrus ideas to calculus courses was one of the possible futures mentioned in our proposal. Our vision was that Syrus ultimately would help math professors address classical epsilon-delta proofs in a calculus course. Unfortunately, support for this kind of interaction remains beyond the reach of current research tools in logic. By contrast, technical results regarding which problems can be solved automatically by tools for introductory calculus problems date back as far as 50 years. The point is that MOOCulus is another incarnation of the ideas introduced in the Syrus project, but in a subject-matter domain in which theoretical issues that could impact success have been settled and can be incorporated into student-practice tools immediately. We can hardly claim any impact from MOOCulus because we did not build it, but we do feel confident it will soon provide some independent confirmation that the ideas underlying Syrus were sound even though our choice of subject matter for the Syrus project proved to be ambitious for current research tools. It is likely that results regarding the use of technologies such as those developed and evaluated in this project will become important to the future of on-line education in general and MOOCs in particular. At this point, no one really knows whether such classes "work". Do on-line courses and the tools used in them result in student learning outcomes comparable to live classes, traditional homework, etc.? Are they better, or are they simply so much hype? After many, many more attempts at developing on-line courses and support tools like Syrus Lite and MOOCulus, and after careful evaluations of data as in the Syrus project, we may (as a community) find out.
