How to format and ship results

Results File Format

You must return the results in separate files for the Toy, Churn, Appetency or Up-selling problems, and for either the small or the large dataset, but the results may be grouped in a single archive. The file format is one number per line corresponding to classification prediction scores for training or test examples, in the order of the data files.

The prediction values should be real numbers corresponding to a score, small for the negative class and large for the positive class. Particular cases of valid scores include:

• Binary {-1, +1} values indicating class membership.
• Discriminant values, negative for the negative class and positive for the positive class.
• A score between 0 and 1 interpretable as the probability of membership of the example to the positive class.
• A rank, smallest values representing examples classified with highest confidence as members of the negative class.

To be eligible for prizes, entries (submitted in a single archive) have to be "complete". A complete entry in either the small or the large challenge consists of 6 files (results on the three tasks for both training and test data). One may submit results on a subset of tasks during development to get on-line feed-back. One may submit simultaneously results on the small and large challenge. You may download below examples of formatted results. Please follow the same naming convention to identify your result files:

Small and large challenge

Required files	Small challenge	Large challenge
Training set	orange_small_train_churn.resu orange_small_train_appetency.resu orange_small_train_upselling.resu	orange_large_train_churn.resu orange_large_train_appetency.resu orange_large_train_upselling.resu
Test set	orange_small_test_churn.resu orange_small_test_appetency.resu orange_small_test_upselling.resu	orange_large_test_churn.resu orange_large_test_appetency.resu orange_large_test_upselling.resu

Toy problem

These files may be submitted for practice purpose but the results on the toy problem are not part of the challenge.

Practice files	Small challenge	Large challenge
Training set	orange_small_train_toy.resu	orange_large_train_toy.resu
Test set	orange_small_test_toy.resu	orange_large_test_toy.resu

Results Archive Format

Submitted files must be in either a .zip or .tar.gz archive format. You can download the example zip archive to help familiarize yourself with the archive structures and contents. If you use tar.gz archives please do not include any leading directory names for the files. Use

zip results.zip *resu 

or

tar cvf results.tar *resu; gzip results.tar

to create valid archives. You may also download sample Matlab code to format your results.

Submission method

The method of submission is via the form on the Submit page. Please limit yourself to 5 submissions per day maximum. If you encounter problems with submission, please contact the Challenge Webmaster. When you submit your results you get immediate feed-back on a small fraction of the test set (10%) and your results are indicated on the Results page. Note that it is always the same 10%, but a different subset for the small and the large challenges. After the challenge is over, we will re-evaluate your entries with the entire test set.