This INSPIRE award is partially funded by the Information Integration and Informatics program in the Division of Information and Intelligent Systems in the Directorate for Computer & Information Science & Engineering, the Perception, Action & Cognition program in the Division of Behavioral and Cognitive Sciences in the Directorate for Social, Behavioral & Economic Sciences, and the Office of Integrative Activities in the Office of the Director.
One of the most common uses of machine learning is to learn to replicate human decisions, a common example is recommender systems. In these systems, computers are trained to replicate the recommendation a collaboration of hundreds or thousands of humans would give, if that were possible. Most of the data used to train these systems are not from a controlled random sample, but are obtained from users based on outputs of algorithms (e.g., which search engine results do users click on?), which introduces bias into the process and ultimately impacts the quality of the results. This project addresses this problem by examining how the human decision process that creates these data in the first place is affected by the data coming from machine algorithms, how this in turn impacts the algorithms themselves, and how to ultimately adjust for human bias in the machine learning process. Specific areas tackled are filtering (e.g., web search) and recommender systems. The deep research into how the human decision process affects machine learning, and how machine learning impacts the human decision process, can provide significant advances in the accuracy and utility of systems using machine learning.
The project builds on analysis of machine learning algorithms based on Hidden Markov Models (HMMs). The formal analysis initially looks at "blind spots" - the impact of bias from users not getting complete (or a random sample) of data. Further analysis will be based on the outcome of two human experiments: Two category recommendation (labeling items, with items to be labelled chosen by random, active learning, and filter-based algorithms), and movie recommendation. The results will be used to develop improved machine learning approaches based on antidotes (altering learned models to reduce bias) and reactive learning (active learning that takes into account the human and machine biases). The PIs also have plans to capitalize on the lessons learned by providing examples of the use of cognitive science in a Web Mining course, and of the impact of machine learning in Data Science for Psychologists courses.
