This project defines meaningful notions of confidence in prediction, designs procedures for computing such notions, and applies these procedures to core machine learning tasks such as active learning, crowd-sourced learning, and tracking. In many applications it is helpful to have classifiers that output, together with each prediction, a rating of the confidence that the prediction is in fact correct. Existing literature either provides various ad-hoc ways for computing such ratings which typically lack a rigorous mathematical footing, or provides mathematically consistent methods (in the Bayesian framework) for computing confidence ratings under very strong assumptions that are unlikely to hold in practice. The research team investigates methods of computing measures of confidence that are mathematically rigorous while making minimal assumptions on the way data is generated, and use these measures to further develop solutions to core machine learning tasks.
Defining and computing mathematically sound measures of confidence lies at the heart of machine learning, pattern recognition and uncertainty in AI. Confidence-rated prediction, active learning, and tracking are fundamental tasks of machine learning and statistics that arise repeatedly in large-scale problems; this project will develop rigorous solutions to these problems. The algorithms developed in this work are tested and used in the Automatic Cameraman project, an interactive, audio-visual installation in the UCSD Computer Science department. The interactive Automatic Cameraman system are used an educational tool to be extended in many different directions, by teams of students at a variety of skill levels.
