This engineering education research project seeks to better understand the problem framing stage of complex problem solving; a key aspect of innovation. The research team will study students? problem framing skills and establish the extent to which individualized, just-in-time feedback during the problem framing stage can help students develop the metacognitive skills needed to start solving complex problems. A key aspect of this project focuses on personalized learning by extending the capabilities of online learning systems to address the broader analysis skills required in complex problem solving. Specifically the team is interested in what student behaviors should trigger real-time feedback and determining how much real-time feedback helps students frame complex problems.

This research has broad applicability to many engineering programs. Complex problem solving is a key skill that is very difficult to teach, and often requires significant investment of institutional resources. The real-time feedback system will be developed as an open-source project, which will enhance the ability of other programs to build from and adapt this work.

Project Report

This project extended the scope of real-time feedback by targeting the broader analysis skills that are required to solve more complex problems, namely, problem framing skills that are necessary to structure engineering problems. Model-tracing tutors have been developed successfully for more well-defined problem-solving procedures, but our research addressed more open-ended engineering problems in thermodynamics and statics. We designed and implemented an online tutoring system for drawing diagrams in statics and thermodynamics and assessed the extent to which our real-time feedback helped students frame the problems and improve their problem solving skills. We also designed and tested an equation tutor for thermodynamics. The central premise guiding our approach is that students gain and apply new conceptual knowledge best when they build it themselves from prior knowledge and feedback tied to their understanding while being challenged by unknown problems. Our online tutors provided guidance to the degree that students needed it based on an assessment of their understanding (i.e., personalized learning). Based on instructor and student assessments of problem complexity for a set of increasingly complex problems, we identified factors that had high correlation with the complexity ratings, namely, students are unfamiliar with a problem context, the length of a problem description, the need to simplify a problem, the need to graphically organize the information, and the number of concepts used from a course. These results were used to design a set of problems with varying levels of complexity that we used in the next phase of our study. To understand student reasoning during problem framing, a coding scheme was formulated for protocol analysis of student problem solving. Students used Smart pens to record their problem solving session while "thinking aloud." The students tried to solve problems with varying levels of complexity. The sessions (including both verbal and written information) were encoded and analyzed to understand student reasoning during problem framing. Not surprisingly, for simple problems, students did not search for any information related to the problem because all the information was provided in the description. Metacognitive activities were minimal and were observed early in the problem framing stage as would be expected. They were closely followed by the necessary reasoning activities to set up the problem such that it led to the correct solution. This behavior was consistent across all the students. For complex problems, the behavior was markedly different as very few of the students were able to frame the problems well. As compared to a simple problem, a much more diverse set of activities occurred and there were noticeable differences between the poor performing students and better performing students. Not surprisingly, there was a significant difference in the total time as the activities for the good performer were much more compact. What is interesting to note is that the metacognitive activities for both types of students were shifted towards the end of the sequence. This indicates that we need to provide more formal instruction in problem framing so that students learn how to perform metacognitive activities early on in the process. Another stark contrast between the two behaviors was the relative high amount and frequency of information seeking behavior for the poor performer versus relatively low usage of information by the better performer. This may be a good metric for identifying the poor performer and providing them with feedback (e.g., a short quiz on key concepts) that would be more beneficial to their learning process than trying to complete the problem. The good performing students exhibited a significant percentage of the time with frequent reasoning activities as they tried to address the problem. Most of those activities were focused on relating the different phenomena in the problem. The protocol analysis results guided the design of the online diagram tutors and equation tutor (see Figure 1). Assessing students’ understanding during problem framing is essential for constructing feedback that is personalized. The tutors’ assessment of student work corresponded to a structured set of questions similar to what an instructor would ask students during office hours to assess their understanding. In thermodynamics, we assessed student learning using the diagram tutor for 500 students across multiple semesters and multiple instructors. All students benefited from the problem solving activity, regardless of their background, instructor, or the type of help they received. Students showed improvement regardless of their initial understanding. Similar results were obtained for students in statics but with a much smaller sample size (10 students). The web-based tutors are easily scalable to large numbers of students and are available 24/7.

Project Start
Project End
Budget Start
2010-09-01
Budget End
2014-08-31
Support Year
Fiscal Year
2010
Total Cost
$400,000
Indirect Cost
Name
Iowa State University
Department
Type
DUNS #
City
Ames
State
IA
Country
United States
Zip Code
50011