Agnostic Learning vs. Prior Knowledge Challenge
Isabelle Guyon, Amir Saffari, Gideon Dror, and Gavin Cawley
isabelle@clopinet.com
“When everything fails, ask for additional domain knowledge” is the current
motto of machine learning. Therefore, assessing the real added value of prior/domain
knowledge is a both deep and practical question. Most commercial data mining
programs accept data pre-formatted as a table, each example being encoded
as a fixed set of features. Is it worth spending time engineering elaborate
features incorporating domain knowledge and/or designing ad hoc algorithms?
Or else, can off-the-shelf programs working on simple features encoding the
raw data without much domain knowledge do as well or better than skilled
data analysts? To answer these questions, we organized a challenge for IJCNN
2007. The participants were allowed to compete in two tracks: The “prior
knowledge” track, for which they had access to the original raw data representation
and as much knowledge as possible about the data, and the “agnostic learning”
track for which they were forced to use data pre-formatted as a table with
dummy features. The challenge web site remains open for post-challenge submissions:
http://www.agnostic.inf.ethz.ch/.