This project uses new psychometric techniques to create a technological tool that could evaluate how well students in the 4th-8th mathematics and science classrooms respond to complex performance tasks. It will extend a research direction begun many years ago on classroom assessment initiated by the investigator.
The purpose of this tool is to improve the instruction of teachers in mathematics and science. It will produce real-time individualized diagnoses of instructional needs to help teachers plan instruction that specifically addresses the learning needs of each student in that class. Wilson has been working on related techniques for a number of years and in this project will work with two experimental curricula, the IQWST science curriculum from the University of Michigan and Northwestern University, and the CDMW data analysis project from Vanderbilt University.
Overview The three main goals of fads are to create an integrated assessment system that (1) focuses on robust evidence with sound measurement qualities, (2) employs tools and reports of high interest and usability for teachers, and (3) supports the use of rich and engaging content designs for assessment. fads is based on generalizing and integrating a range of assessment tools previously developed by the Berkeley Evaluation and Assessment Research (bear) Center for other projects. In addition to the fads software development, validity and usability trials took place with teacher-developed content in innovative item forms. This paper describes the activities and products that were developed during the fads project. Summary of Project Activities The fads project was developed in collaboration with two partner curriculums that served as prototype, providing the content that was used to populate the system. The first partner was Investigating and Questioning our World through Science and Technology (iqwst), developed at the University of Michigan and Northwestern University, and designed for elementary and middle school grades. The second partner was Constructing Data, Modeling Worlds (cdmw), developed at Vanderbilt University and the bear Center, and it is designed for middle-school grades. The first year of the fads project was dedicated to two main strands of work. On the one hand was the analysis of the types of questions used in the curriculums, while on the other hand was the design of the fads software architecture. Accordingly, on the content side the first year of the project was dedicated to reverse engineering of as many different item types as we can identify in the two partner curricula. This involved the selection of progress variables, modules, and sample items from each curriculum for prototyping in computer-deliverable forms. Meanwhile, in relation to the design of the software architecture, the team worked during the first year on the development of initial use cases for fads, the design of the fads database, and finally, in the development of prototypes for one or more aspects of user interface to the fads database. During the second year of the project, the separation of the initial strands of work was blurred when the project activities concentrated in the gathering teacher input into the design of the student and teacher interfaces for item taking, as well as their feedback on the development of prototypes for computerized item delivery and prototypes for scoring facilities for constructed responses. Moreover, the input from teachers working in these curriculums became central for the collection of usability feedback from initial computer-delivered item prototypes. In addition to the work directly related with the content analysis and programming tasks of fads, the project included a series of field trials to examine the quality and usability of the system that were conducted during the third and fourth year of the project. These trials contributed to collect evidence about the actual use of the system in the context of the two partner curriculums, which provided the opportunity to collect feedback directly from teachers and students and modify different aspects of the application accordingly. For the first round of field testing, which is referred to as the alpha trials, the goal was to implement and test single on-line assessments for both iqwst sixth grade physics and cdmw middle-school mathematics. The aim was not to examine the usefulness of the fads system for formative assessment per se, but rather to use the software in order to find bugs or difficulties with using the software and to assess initial student and teacher reactions to the system. Science teachers using the iqwst physics curriculum and their students participated in one arm of the alpha trials, and mathematics teachers using the cdmw curriculum participated in the other arm. Both arms of the alpha trials were conducted in November 2009. The second round of field testing, referred to as the beta trials, was conducted between February and June 2010, and involved science teachers using the iqwst sixth grade biology curriculum and their students. Three on-line assessments of biology knowledge were implemented, based on items designed by iqwst teachers at a January 2010 workshop, and individual and group reports were made available to teachers. The types of items included in the on-line assessments in the alpha and beta trials were drawn from the Intermediate Constraint Taxonomy for E-Learning Assessment Questions and Tasks (Scalise & Gifford, 2006). The field trials constituted successful examples of the deployment and use of fads in the context of the two partner curriculums, and yielded a generally positive response from the participating teachers and students, with both groups reporting increased engagement in the assessments activities.
