We show first that the pooling of multiple human judgments of relevance provides a predictor of relevance that is superior to that obtained from a single human's relevance judgments. A learning algorithm applied to a set of relevance judgments obtained from a single human would be expected to perform on new material at a level somewhat below that human. However, we examine two learning methods which when trained on the superior source of pooled human relevance judgments are able to perform at the level of a single human on new material. All performance comparisons are based on an independent human judge. Both algorithms function by producing term weights; one by a log odds calculation and the other by producing a least squares fit to human relevance ratings. Some characteristics of the algorithms are examined. An article has been published and further work remains to be done in examining learning methods applied to this system.
