This pedagogical qorkflow research project will create a novel hybrid-workflow framework that supports efficient assessment of student learning through interactive generation and execution of various assessment workflows. Unlike in many existing workflow systems, the task of student assessment includes steps that cannot be fully automated, such as obtaining grade, background and student survey information. The system will provide assistance in executing and integrating the results of the manual steps. Research steps will include (a) knowledge-based modeling of computational and non-computational assessment tools as workflow components; (b) interactive generation of assessment workflows while propagating and combining constraints from both computational and non-computational components; and (c) interactive execution of hybrid workflows that incorporates new constraints that are inferred from execution of non-computational components. Evaluations will focus on the effects of Pedagogical Workflow technology on learning assessment performance, especially the assessment of pedagogical discourse in undergraduate engineering courses.
Educational technology to support online learning is now centrally supported by many colleges and universities. The perceived mandate to use technology for instruction, in addition to the enormous amount of information available for consumption on the Web, places a considerable burden on instructors who must learn to integrate appropriate student practices and learning assessment via the new media. Pedagogical workflows will allow instructors with little or no training in educational assessment to perform large-scale complex diagnosis and assessment of student learning in ongoing lessons. Facilitating the integration of personal student information into assessment will point to directions to improve STEM participation, learning, and retention. The finding will provide benefits to society by sharing results and technology with instructors and educational experts. The proposed work also has the potential to lead to a new research field on e-Learning workflows, similar to the way in which workflow technology transformed e-Science research with e-Science workflows.
The goal of the project is to create a novel workflow environment that supports efficient assessment of student learning through development and execution of computational workflows. We have developed Pedagogical Assessment Workflow System (PAWS) that enables the efficient and robust integration of diverse datasets for the purposes of student assessment. The workflows were designed to produce results that answer assessment questions relevant to student discussions and provide timely feedback to instructors to facilitate "just in time" instructional adaptation to students learning and needs. Example assessments supported include analyzing answer wait time for individual students, correlating student performance, as measured by project grades, with different dialogue roles such as information seeking and information providing, analyzing distributions of question topics within student discussions, summarizing discussions with key questions and answers, and identifying unanswered questions. To increase accessibility of the assessment results, weekly reports of the workflow-processed results were sent to the instructors. The workflow development makes use of natural language processing and machine learning techniques, making the processing and analysis, both efficient and robust. We handle noisy student data and model subject topics were through a combination of data cleaning and semi-automatic domain term extraction from textbook. Interviews with instructors indicate that the workflows support uncovering useful information for instructors. The results also demonstrated the necessity of working collaboratively with stakeholders that facilitates feedback and analysis to develop graphical results that were meaningful to a different kind of user. Combined with traditional cognitive assessment methods such as assignment and exam grades, the workflow-based approach can be a powerful tool for assessing impact of online learning.
