The PI will organize a machine learning competition on the theme of model selection and organize workshops at major conferences where top-ranking participants will present their results. The proceedings will be published as a special issue of the Journal of Machine Learning Research. The platform used for the competition will remain available for further method benchmarking as an on-line service.
The competition will benchmark classification and regression methods on sizeable real-world data sets that are presently of interest to the industry datasets with either many input variables (features) or many entries (examples, patterns), or both. Many means of the order of 10,000, 100,000 or even one million features and/or patterns. Such problems are encountered, for instance, in vision, text processing, speech processing, bioinformatics, astronomy, and combinatorial chemistry (e.g. for high throughput drug screening). These pose new challenges because many techniques were developed for smaller size data sets.
People have long identified the need for benchmarks in machine learning, not with toy examples but with real-world data. With the advent of the Internet, data exchange has become very easy and it is now very common that people publish their data on-line when they publish results. Data repositories allow researcher to find easily a variety of data sets from which to choose. But, without organized competitions, comparisons can hardly be made, for various reasons. Researchers choose different data sets on which they test their method, sometimes reporting only successes. The results are often not statistically significant. There are sometimes fundamental experimental design flaws that invalidate the results (e.g. training on the test set, in a subtle way). This project will develop and implement competent testing procedures to overcome these and related problems.
