IEEE CIS competition program

• contributing to the solution of difficult scientific questions and important real world problems,
• identifying techniques that really work through fair evaluations,
• demonstrating and facilitating progress in computational intelligence,
  
• raising the awareness of computational techniques in academia and Industry,
• helping to disseminate the work of yet unknown talented people.

Past competition programs of the IEEE CIS:

• CEC 2003:
• Checkers contest
• Computer controlled car racing
• WCCI 2004:
• IJCNN 2004: Time series competition, the CATS benchmark.
• CEC 2004: Computer controlled car racing
  
• CEC 2005: Computer controlled car racing
  
• WCCI 2006:
• IJCNN 2006 "model selection / performance prediction" challenge
• IJCNN 2006 "predictive uncertainty in environmental modelling" challenge
• CEC 2006: Competitions in Othello, Binary Series Prediction, Evolved Art, Nonlinear Programming, Huygens Probe.
  
• IJCNN 2007 competition program:
• "Agnostic Learning vs. Prior Knowledge" competition
• NN3 forecasting competition
• CEC 2007: Ms. Pac-Man, Othello, X-Pilot AI, Simulated Car Racing, Multi-Objective Optimization
• WCCI 2008: 10 competitions: games, optimization, times series prediction, causal discovery, and more...
• CEC 2009 (multi-objective optimization, car racing, pac-man, and more...)
• IEEE FUZZ 2009: Fuzzy control of a simplified magnetic suspension system
  

Do you want to organize a competition?

1. IMPACT: Choose a problem with high impact (economical, humanitarian, societal, etc.)
2. DATA: Make sure that
• you have access to large enough datasets to make the challenge interesting and draw conclusive results [how much is enough?];
• the data can be made freely available; 
• the ground truth has been kept confidential.
  
3. RELEVANCE: The problem should be relevant to the computational intelligence community and should be solvable without extensive domain knowledge.
   
4. CHALLENGE: The problem posed should be scientifically or technically challenging. However, it should not be impossible to solve; the organizers should provide baseline results. Think of illustrating the same scientific problem using several datasets from various application domains.
5. APPEAL: The competition setup should be attractive with preferably on-line submission and feed-back and/or an on-site contest at the conference. 
6. EVALUATION: It should be possible to evaluate the results objectively  (provide a metric of significance in performance differences). 
7. RESOURCES: The organizers should make sure to have enough resources (team member availability, computers, support staff, other equipment, sponsors).

Other recently organized competitions

Agnostic learning vs. Prior knowledge challenge. “When everything fails, ask for additional domain knowledge” is the current motto of machine learning. Therefore, assessing the real added value of prior/domain knowledge is a both deep and practical question.The participants competed in two track: the “prior knowledge track” for which they had access to the raw data and information about the data, and the “agnostic learning track” for which they had access to preprocessed data with no knowledge of the identity of the features.

Performance prediction challenge. “How good are you at predicting how good you are? 145 participants tried to answer that question. Cross-validation came very strong. Can you do better? Measure yourself against the winners by participating to the model selection game.

Feature selection challenge. We organized a competition on five data sets in which hundreds of entries were made. The web site of the challenge is still available for post challenge submissions. Measure yourself against the winners! See the book we published with a CD containing the datasets, tutorials, papers on s.o.a. methods.

Pascal challenges and Pascal2 challenges: The Pascal network is sponsoring several challenges in Machine learning.

AIPR TC5 on benchmarking and software

Data mining competitions:
A list of data mining competitions maintained by KDnuggets, including the well known KDD cup.

UCSD student data mining contest. A yearly competition reserved to students sponsored by FICO.

PACKDD 2009: Credit Risk Assessment on a Private Label Credit Card Application.

UCI machine learning repository: A great collection of datasets for machine learning research.

DELVE: A platform developed at University of Torontoto benchmark machine learning algorithms.

ICDAR
International Conference on Document Analysis and Recognition, a bi-annual conference proposing a contest in printed text recognition. Feature extraction/selection is a key component to win such a contest.

TREC
Text Retrieval conference, organized every year by NIST. The conference is organized around the result of a competition. Past winners have had to address feature extraction/selection effectively.

ICPR
In conjunction with the International Conference on Pattern Recognition, ICPR 2004, a face recognition contest is being organized.

CASP
An important competition in protein structure prediction called Critical Assessment of Techniques for Protein Structure Prediction.