Data Representation Discovery FAQ

Data Representation Discovery
Workshop FAQ

What is the scope of the workshop?
- Methods incorporating "domain knowledge"/"prior knowledge" in machine learning, including preprocessing and feature construction, special purpose kernels (for kernel methods), special architectures (for neural networks, tree classifiers), etc. Must show an advantage over using "black box"/"agnostic learning" methods.
- "Black box"/"agnostic learning" methods closely matching the performance of methods using "prior knowledge". Must show an advantage over using "prior knowledge" methods.
- Discussion of the results of the Agnostic Learning vs. Prior Knowledge Challenge.

What do you mean by "agnostic learning"?
We consider supervised learning problems in which input-output pairs are provided for training and output predictions must be made on new test input. Agnostic learning refers to the capability of learning from data without any a priori or prior knowledge about the task: no knowledge of the nature of the task, of the nature of the features used for data representation, or geometrical or semantic relationships between features, etc. The data set is the sole source of knowledge.
In the Agnostic Learning vs. Prior Knowledge Challenge, "truly agnostic" baseline results are provided by the results of our previous challenge. The same datasets were used but no knowledge about the tasks was disclosed before the end of the challenge. In the present challenge, participants have access to a lot of information about the data and to the raw data. Additionally, they have access to the preprocessed representation used in the previous challenge, so the can try to match or outperform the performances of the previous challenge either using "prior knowledge" or in an "agnostic" manner. We are aware that since information was disclosed about the data, the agnostic results will not be "truly" agnostic, but still it will be a challenge to outperform the "prior knowledge competitors using as little prior knowledge as possible.

What do you mean by "prior knowledge"?
Prior knowledge means any kind of knowledge that may help making predictions and is obtained in addition to the training data. This includes "domain knowledge" about the task gained from working with similar data in the past, additional data, knowledge of the nature of the data representation (topological or semantic relationships between features). Prior knowledge may be incorporated in various ways, including as preprocessing and feature construction, special purpose kernels (for kernel methods), special architectures (for neural networks, tree classifiers), etc.
For the datasets of the challenge, we provide a document describing the data that constitutes a good source of prior knowledge.

What is the "Agnostic Learning vs. Prior Knowledge Challenge"?
We are organizing a competition on five 2-class classification problems, the results of which will be discussed at the workshop. To learn more about the competition, see the challenge web site, the challenge FAQ, and the learning object FAQ.

How do I participate in the workshop?
Just register to the IJCNN 2007 workshops. The workshops are part of the IJCNN 2007 conference, Orlando, Florida, August 12-17, 2007. Our workshop will be held on Thursday August 16, 2007. The participants are encouraged (but not obliged) to also attend the conference. The workshop papers will be published in the conference proceedings.

How do I contribute a paper to the workshop?
Submit a paper to the IJCNN 2007 conference to be published in the proceedings (please use the category of 'Special Competitions').
January 31st, 2007, Submission deadline.
March 15th, 2007, notification of acceptance.
April 15th, 2007, camera ready copy due.
There will be a best paper awards . Since IJCNN 2007 marks the 20 year anniversary of the event, a special issue of Neural Networks, the official journal of the INNS, will be published to include selected outstanding papers from the conference.

Do I need to participate in the workshop to enter the challenge?
No. Limited travel support will be provided to help deserving challenge participants attend the workshop. Send requests to agnostic@clopinet.com.

Do I need to enter the challenge to participate to the workshop?
No. The workshop participants are not obliged (but encouraged) to enter the challenge. All results must be supported by strong experimental evidence.

I am cluless, what is predictive modeling?
We are interested in problems of predicting unknown outcomes from observations. We limit ourselves to the setting in which the number of observations is fixed and can be formatted as a vector of input variables x. The prediction y will be a scalar. For problems of classification, y will take discrete values, whereas for problems of regression, it will take continuous values. Examples of classification problems include:

Handwritten digit recognition: the predictor is presented with images of digits and must label them 0-9.
Spam categorization: the predictor is presented with desirable emails and spam and must decide which ones to discard.
Medical diagnosis: the predictor is presented with sets of symptoms or medical analyses and must determine the disease from which the patient suffers.

Examples of regression problems include:

Medical prognosis: Predicting the time of recurrence of a disease from medical analyses.
Drug screening: Predicting the binding or solubility of a molecule from chemical and physical descriptors.
Marketing: Predicting how much a prospective customer will buy based on census data.

All the problems we are interested in can be cast into the framework of inference or “learning from examples”. A function y=f(x; a) or "model", which may have a parameter vector a, is used to make predictions. The components of its parameter vector a are adjusted during training using a set of input-output pairs examples {(x₁, y₁), (x₂, y₂), … (x_p, y_p)}, called training examples. For non-parametric methods like the “nearest neighbor” method, the training examples are directly used, without parameter adjustment per se, but, in our setting, training examples will be considered parameters in a broader sense. The goal is not to optimize predictions on the training examples, but to optimize predictions on a set of test examples, for which the target values y are not available during training. Such optimization must be carried out, however, using training input-output pairs only, and, when available, the unlabeled test set. Since the true objective involves unknown information (the test set target values), objective functions that approximate the goal are derived. The simplest one is the prediction error on training examples (training error). But other objective functions may be better estimators of the true objective.
Note that, in the challenge, we will have have two test sets: a validation set and a final test set. The validation set will be used to provide the competitors with feed-back during the development period, reserving the final test set for the real test.

Can a participant give an arbitrary hard time to the organizers?
DISCLAIMER: ALL INFORMATION, SOFTWARE, DOCUMENTATION, AND DATA ARE PROVIDED "AS-IS". ISABELLE GUYON AND/OR OTHER ORGANIZERS DISCLAIM ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR ANY PARTICULAR PURPOSE, AND THE WARRANTY OF NON-INFRIGEMENT OF ANY THIRD PARTY'S INTELLECTUAL PROPERTY RIGHTS. IN NO EVENT SHALL ISABELLE GUYON AND/OR OTHER ORGANIZERS BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF SOFTWARE, DOCUMENTS, MATERIALS, PUBLICATIONS, OR INFORMATION MADE AVAILABLE FOR THE CHALLENGE.

Who can I ask for more help?
For all other questions, email agnostic@clopinet.com.