We propose to develop (1) a method for testing diagnostic decision support software (DDSS); (2) a set of test specifications for evaluation of DDSS whose domain is Internal Medicine; and, (3) a sample test consisting of cases for which the DDSS systems will produce a list of possible diagnoses. The test will be designed for mature systems that are ready to be used in clinical practice. In this project, the test will be used to compare the performance of three DDSS systems (DXPLAIN, QMR, MEDITEL). The method that will be followed in this development process will be based upon standard test development procedures used in examinations to certify physicians' competence. Because these three DDSS systems have had extensive developmental work, we hypothesize that they will perform similarly and that all of them will score above the passing score for the test. The method that will be used involves (1) the use of a test committee composed of general internists and subspecialists in Internal Medicine to plan the content of the examination; (2) design of a set of content specifications for the types of items the examination should contain; (3) construction of examination items; (4) review and revision of test items by the test committee; (5) systematic determination of passing standards; (6) review of a detailed item analysis after an initial scoring of the examination; (6) performance of a final scoring on the examination; and, (7) determination of the reliability and validity of the examination. The outcomes will be (1) a methodology that can be used to develop tests for other types of medical decision support software, (2) a set of content specifications and a pool of case protocols that can be used with other DDSS; and, (3) data on the comparative performance of three DDSS systems on a systematically designed test.
