Diagnostic modeling, also known as cognitive diagnosis, is a field of psychometrics that has seen increased research activity in recent years due to the potential for diagnosing the mental states of individuals, such as in the social sciences (e.g., criteria indicating psychological disorders) or in education (e.g., the skills examinees may possess). To date, such models have been developed and expanded for use in a broad spectrum of research areas and have been shown to be effective at classifying individuals on a set of categorical latent variables. Although the development of diagnostic models has focused on more comprehensive and flexible measurement models for data collected at a single occasion, there have been no systematic efforts to extend these models into the measurement of individual change. Simply put, the researcher who wishes to study changes in diagnostic status over time currently cannot do so. This research project will focus on the development, evaluation, and utilization of longitudinal extensions of diagnostic models for the measurement of change over time. In developing longitudinal diagnostic models, advances must be made that balance the model demands of potentially large numbers of parameters with practical and accurate approximations. The investigators will develop practical and feasible longitudinal methods for diagnostic measurement, modeling, and assessment, and they will examine their accuracy, sensitivity, and comparative efficiency. The investigators also will examine the practical considerations and statistical issues in applying these methods in real-world longitudinal diagnostic analyses. Freely available software will be provided for researchers and practitioners to utilize these methods in their own studies.
This project will expand the practicality of diagnostic models and add to the number of research and real-world scenarios where such models can be used. The models and software developed from this project will fill an existing void that has limited the study of diagnostic phenomena over time. Furthermore, guidelines about the structure of data needed for each model (i.e., sample size or number of measurement occasions) will allow for application of such models across a wide spectrum of empirical research areas studying how behavioral phenomena change over time. The methods developed from this project will enable a more rigorous examination of how mental states change over time. The results of this research have the potential to greatly advance basic understanding of how people develop and change with respect to diagnostic attributes and clinical criteria.
Traditional methods used to a score an exam typically provide a single score for each student. It is assumed that this score will be higher for students that know more of the content when compared to students who know less of the content that is being tested. One advantage of this traditional approach is that a teacher can quickly identify those students that are doing well. However, if a student is not doing well, then one natural question that might be asked is, "Why?" As an alternative to the traditional method of scoring an exam, diagnostic classification models can be used. Instead of providing a single score for each student, Diagnostic Classification Models (DCMs) score an exam so that a student’s mastery (versus nonmastery) is determined on a set of skills. For example, using a basic math test, a traditional scoring method could identify those students who received an A on the exam (or passed), whereas DCMs would attempt to indicate whether a student had mastered addition, subtraction, multiplication, and division. Notice that using DCMs a student would receive a profile indicating which of these she had mastered. One argument for the benefit of DCMs is that, when a student is struggling, tailored lesson plans can be developed to focus on the nonmastered skills (e.g., multiplication), which would save teachers time and resources when compared to reviewing all math content for those with low scores based on a traditional approach. Of course, this argument implies that knowing what skills are mastered or not mastered would be useful. Even more so, we assume that once this is known, we could better instruct a student so that the rate of learning the material would be faster and the instruction would be more efficient. This grant focuses on a method of measuring how mastery or nonmastery of these skills (or attributes) change over time. As a result, the methodology in this grant provides both the theory and will provide computer software to explore the rate of change of students’ mastery of skills across time, which can then be used to validate the usefulness of DCMs as an alternative focus of exams in the classroom. Notice that the rate of learning using a traditional scoring approach and intervention versus a DCM scoring approach and intervention could actually be quantified. In addition, learning progressions could be assessed showing that particular skills are learned first where others closely follow. Because these changes are modeled across time, developmental benchmarks could also be studied and when these skills are acquired. The benefit of this research also extends beyond education. For example, most psychological disorders are assessed based on a set of criteria that are either met or not met (similar to mastery of skills in education). These models can be used to determine the developmental path of particular disorders.
