We propose to develop (1) a method for testing diagnostic decision support software (DDSS); (2) a set of test specifications for evaluation of DDSS whose domain is Internal Medicine; and, (3) a sample test consisting of cases for which the DDSS systems will produce a list of possible diagnoses. The test will be designed for mature systems that are ready to be used in clinical practice. In this project, the test will be used to compare the performance of three DDSS systems (DXPLAIN, QMR, MEDITEL). The method that will be followed in this development process will be based upon standard test development procedures used in examinations to certify physicians' competence. Because these three DDSS systems have had extensive developmental work, we hypothesize that they will perform similarly and that all of them will score above the passing score for the test. The method that will be used involves (1) the use of a test committee composed of general internists and subspecialists in Internal Medicine to plan the content of the examination; (2) design of a set of content specifications for the types of items the examination should contain; (3) construction of examination items; (4) review and revision of test items by the test committee; (5) systematic determination of passing standards; (6) review of a detailed item analysis after an initial scoring of the examination; (6) performance of a final scoring on the examination; and, (7) determination of the reliability and validity of the examination. The outcomes will be (1) a methodology that can be used to develop tests for other types of medical decision support software, (2) a set of content specifications and a pool of case protocols that can be used with other DDSS; and, (3) data on the comparative performance of three DDSS systems on a systematically designed test.

Agency
National Institute of Health (NIH)
Institute
National Library of Medicine (NLM)
Type
Research Project (R01)
Project #
1R01LM005125-01
Application #
3374226
Study Section
Biomedical Library and Informatics Review Committee (BLR)
Project Start
1990-04-01
Project End
1993-03-31
Budget Start
1990-04-01
Budget End
1991-03-31
Support Year
1
Fiscal Year
1990
Total Cost
Indirect Cost
Name
University of Alabama Birmingham
Department
Type
Schools of Medicine
DUNS #
004514360
City
Birmingham
State
AL
Country
United States
Zip Code
35294
Berner, Eta S; Maisiak, Richard S; Heuderbert, Gustavo R et al. (2003) Clinician performance and prominence of diagnoses displayed by a clinical diagnostic decision support system. AMIA Annu Symp Proc :76-80
Berner, Eta S (2003) Diagnostic decision support systems: how to determine the gold standard? J Am Med Inform Assoc 10:608-10
Maisiak, R S; Berner, E S (2000) Comparison of measures to assess change in diagnostic performance due to a decision support system. Proc AMIA Symp :532-6
Berner, E S; Maisiak, R S; Cobbs, C G et al. (1999) Effects of a decision support system on physicians' diagnostic performance. J Am Med Inform Assoc 6:420-7
Berner, E S; Maisiak, R S (1999) Influence of case and physician characteristics on perceptions of decision support systems. J Am Med Inform Assoc 6:428-34
Berner, E S; Jackson, J R; Algina, J (1996) Relationships among performance scores of four diagnostic decision support systems. J Am Med Inform Assoc 3:208-15
Berner, E S; Webster, G D; Shugerman, A A et al. (1994) Performance of four computer-based diagnostic systems. N Engl J Med 330:1792-6
Berner, E S (1993) The problem with software reviews of decision support systems. MD Comput 10:8-12
Berner, E S; Shugerman, A A (1991) Needs assessment for diagnostic decision support systems (DDSS). Proc Annu Symp Comput Appl Med Care :605-8