Machine Learning is the science of building hardware or software that can achieve tasks by learning from examples. The examples often come as {input, output} pairs. Given new inputs a trained machine can make predictions of the unknown output.

Examples of machine learning tasks include:

• automatic reading of handwriting

• assisted medical diagnosis

• automatic text classification (classification of web pages; spam filtering)

• stock exchange predictions

We organize challenges to stimulate research in this field. The web sites of past challenges remain open for post-challenge submission as even-going benchmarks.

Feature selection (NIPS 2003): Seventy five participants competed on five classification problems to make best predictions and select the smallest possible subset of relevant input variables (features). The tasks include: cancer diagnosis from mass-spectrometry data, handwritten digit recognition, text classification, and drug discovery.	Challenge web site Workshop page Result page Book edited (CD with data and code) Matlab software and course material
Performance prediction (WCCI 2006): One hundred and forty-five five participants competed on five classification problems to make best predictions and predict their generalization performance on new unseen data. The tasks include: marketing, drug discovery, text classification, handwritten digit recognition, and ecology.	Challenge web site Workshop page Result page Matlab software Call for papers
Agnostic learning vs. prior knowledge (NIPS 2006 and IJCNN 2007): This challenge has two tracks: the agnostic learning track and the prior knowledge track, corresponding to two versions of five datasets. The “agnostic track” version of the data is ready-to-use data preprocessed in a feature-based representation suitable for off-the-shelf machine learning packages. The identity of the features is not revealed. The “prior knowledge track” version of the data is just raw data, not always in a feature representation, coming with information about the nature and source of the data. Can you do better with the raw data and prior knowledge about the task? How far can you get with pure “black box learning”?	Challenge web site NIPS 2006 workshop page IJCNN 2007 workshop page Results Book in preparation
Learning causal dependencies (WCCI 2008): A new competition is in preparation. What affects your health? What affects the economy? What affects climate changes? and… Which actions will have beneficial effects? This new competition will challenge the participants to discover the causes of given effects, based on observational data. We will use re-simulation data from models closely resembling real systems and real data for which the causal dependencies are known from experimental evidence.	Challenge web site Workshop page

We are very grateful to our sponsors:

This project is supported by the National Science Foundation under Grants N0. ECCS-0424142, ECCS-0736687 and ECCS-0725746. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.