Since 1993, we have been engaged in developing a practical method to evaluate the impact of Decision Support Systems (DSSs) on the diagnostic reasoning of clinicians as end-users of these systems. Our research examined the effects of four independent factors: level of subject's clinical experience, geographic site, clinical cases assigned, and specific DSS employed. Several papers addressing the questions outlined in our original proposal have been presented or published; others are under review. As this work has unfolded, we have become increasingly aware of the potential broader value of the research materials themselves and the data set generated by our research subjects. We developed a library of 36 cases, covering a broad spectrum of internal medicine. In our study, each subject evaluated nine cases, first without and then with DSS consultation. At each stage the subject was asked to list up to six diagnostic hypotheses and up to three next steps in the evaluation of the patient presented. The resulting database is one of the largest assembled to address clinical reasoning in medicine encompassing 216 clinicians, at three levels of experience across three geographic sites, who in aggregate completed 1994 cases. Our proposed work is expressed as two aims that together derive full benefit from the research already completed. Via the first aim, we propose to employ our existing data set to explore a collection of research questions at the intersection of cognitive psychology and medical informatics. These questions address issues of basic psychological import, not addressed in our original study, that will also inform the design of clinical information resources. These issues include: errors in data interpretation and problem formulation, effects of clinician confidence on judgment, the dependence of DSS consultation effects on clinician characteristics, consultation effects and case characteristics, and a set of measurement and methodology issues. Under the second aim of our proposed work, we will create and maintain our set of 36 clinical cases as a reusable, calibrated resource for other researchers.
