We show first that the pooling of multiple human judgments of relevance provides a predictor of relevance that is superior to that obtained from a single humans relevance judgments. A learning algorithm applied to a set of relevance judgments obtained from a single human would be expected to perform on new material at a level somewhat below that human. However, we examine two learning methods which when trained on the superior source of pooled human relevance judgments are able to perform at the level of a single human on new material. All performance comparisons are based on an independent human judge. Both algorithms function by producing term weights; one by a log odds calculation and the other by producing a least squares fit to human relevance ratings. Some characteristics of the algorithms are examined. An article has been published and further work remains to be done in examining learning methods applied to this system. - relevance judgments, human judges, group performance, learning algorithms - Human Subjects: Interview, Questionaires, or Surveys Only

Agency
National Institute of Health (NIH)
Institute
National Library of Medicine (NLM)
Type
Intramural Research (Z01)
Project #
1Z01LM000083-03
Application #
6290495
Study Section
Special Emphasis Panel (CBB)
Project Start
Project End
Budget Start
Budget End
Support Year
3
Fiscal Year
1999
Total Cost
Indirect Cost
Name
National Library of Medicine
Department
Type
DUNS #
City
State
Country
United States
Zip Code