This study will develop a detailed, written experimental design for evaluating knowledge-based expert systems (KBESs), based on artificial intelligence (AI) techniques, for indexing biomedical information at the National Library of Medicine (NLM). In 1956, Public Law 83-941 gave NLM a congressional mandate to aid the dissemination and exchange of information important to the progress of medicine and public health, identifying indexing as a function to carry it out. NLM currently indexes about 400,000 documents a year for its world-renowned, publicly- available MEDLINE database. As the KBES, this study will use a unique, advanced prototype known as MedIndExTM (Medical Indexing Expert) developed at NLM. The design developed by this study will be based on the research goal of developing systems to improve indexers' performance. The general research question the evaluation should be designed to answer is, does a KBES affect indexers' performance; specific research questions correlate system features (knowledge content/organization and user interface) with performance measures of quality and efficiency. Based on these goals and questions, the study will develop tasks and measures to be used in the evaluation, in particular structured indexing tasks to simulate representative indexing problems. The design developed by this study will be the basis for an evaluation to be conducted as an early assessment of the feasibility of adopting KBESs in a production environment. Since such systems have not existed previously, this design will establish a methodology for evaluating KBESs for indexing. Information from this study will be reported via traditional mechanisms (proceedings, journals, agency reports).
