Cognitive diagnosis models have received increasing attention within educational and psychological measurement. The popularity of these models largely may be due to their perceived ability to provide useful information concerning both examinees and test items. However, the validity of such information may be undermined when diagnostic models are misspecified. This project focuses on one aspect of model misspecification: violations of the local item independence assumption. It examines potential causes and consequences of such dependence, with particular attention to those causes unrelated to the attributes a diagnostic test is intended to measure. The project proposes and evaluates a hierarchical diagnosis model as an alternative to traditional diagnosis models in which nuisance dependence is ignored. This model maintains the desirable properties of existing models while allowing for greater complexity in the underlying response process. Importantly, the model may be estimated efficiently, even for models with a large number of nuisance latent variables, using an analytical dimension reduction technique described by Gibbons and Hedeker (1992).
There is growing interest in extracting model-based diagnostic information from assessments in order to provide more useful feedback to stakeholders. Up to this point, however, the question of whether traditional cognitive diagnosis models fit real test data has been somewhat neglected. This project examines the issue of model fit and presents a model that may better account for certain causes of misfit than the traditional diagnosis models. To the extent that the proposed framework better accounts for the structure of real test data, its application will contribute to improved test development and lead to more valid model-based diagnostic inferences, such as the classification of test takers according to cognitive attributes or skills. This, in turn, is expected to enhance decision making, as better diagnostic information may allow for more effective (or better targeted) delivery of instructional strategies or clinical interventions. The results from this project will benefit the psychometric practice in any social and behavior science discipline that involves testing and measurement. As a Doctoral Dissertation Research Improvement award, support is provided to enable a promising student to establish a strong, independent research career.
The research conducted in this project deals with a family of statistical methods used in educational and psychological testing known as diagnostic classification modeling. These models relate test item responses to examinees’ possession or mastery of a set of unobservable attributes (such as abilities related to proficiency in some academic discipline). Because these models can potentially provide quite informative and fine-grained examinee profiles, they have emerged as a promising alternative to more traditional methods of modeling item response data, in which examinees are typically characterized by their standing on a single trait or dimension. Despite the appeal of diagnostic classification models, there has been only limited attention to the question of whether these models fit actual test data. This is an important question, given that poor model fit could lead to faulty interpretations, including misclassification of examinees. In this research, the investigators focused on one particular cause of model misfit: the failure to account for systematic influences on item responses that are unrelated to the attributes that one intends to measure (i.e., what might be described as "construct-irrelevant" or "nuisance"). Such factors may represent features of test construction, mode-of-administration, or individual styles of response (to name but a few possibilities). The impact of such factors has been well studied in other modeling frameworks (including item response theory) but not in the context of diagnostic classification models. The work was motivated by a belief that the failure to attend to such factors may be attributable to (1) limited understanding of the consequences of this cause of model misfit, (2) the lack of readily available tools for evaluating model goodness-of-fit, and (3) the inability of existing diagnostic classification models to account for "nuisance" influences (even when they are expected to influence item responses). Accordingly, the goals of this project were to examine the impact of "nuisance" factors on the accuracy of examinee classifications, to develop indices for detecting and characterizing model misfit (including misfit due to "nuisance" factors), and to propose a flexible diagnostic classification modeling framework capable of accounting for the influence of "nuisance" factors. Major study findings are as follows. First, the failure to account for the influence of "nuisance" factors can indeed reduce the accuracy of examinee classifications. In other words, when such factors are ignored, classifications may be less trustworthy. Researchers and practitioners who use diagnostic classification models should thus consider this cause of model misfit. Second, the alternative modeling framework—which acknowledges the influence of both the attributes of interest and "nuisance" factors on item responses—was developed and shown to lead to improved classification accuracy when "nuisance" factors are present. Although this model is more complex than traditional models, certain constraints on its structure ensure that it can be estimated efficiently. In this way, the research provides a viable alternative to the traditional modeling approaches when the presence of "nuisance" factors renders those approaches ineffective. Finally, tests of model fit based on discrepancies between observed and expected distributions of responses for individual items and item pairs were found to be quite useful. These indices provide a sound basis for judging the fit of diagnostic classification models and should be considered whenever these models are applied.
