Conformal prediction for top-k-rankings (Ba/Ma)

Topic for a bachelor/master's thesis

Short Description:

Conformal Prediction (CP) is a statistical approach to predictive modeling that allows for producing "reliable" predictions in the form of sets, which cover the true outcome with high probability [1]. One of its appealing properties is its aptitude for online learning settings, in which outcomes are predicted one by one. For this purpose, CP uses past experience (training examples) to obtain precise levels of confidence for new predictions. By now, the CP framework has mainly been studied for "classical" scenarios, such as classification, regression, and clustering. The goal of the thesis is to investigate how the learning-to-rank problem can be tackled by means of the CP paradigm. For this purpose, a suitable nonconformity measure has to be defined, which is at the core of CP-related methods. As a nonconformity measure requires a notion of distance on the underlying data space, it seems to be more reasonable to focus on top-k-distances of the symmetric group to avoid high computational costs. The other ingredient of the nonconformity measure is a suitable point predictor, i.e., a rank aggregation procedure for the training data, for which several manifestations are conceivable [2, 3].

Requirements

The requirements for the thesis include the following: Implementation of a conformal predictor for the learning-to-rank problem, and the (experimental) study of its performance for different nonconformity measures.

Prerequisites

Expertise in machine learning, programming skills.

Contact

Prof. Eyke Hüllermeier or Dr. Viktor Bengs

References

• [1] G. Shafer, and V. Vovk. A tutorial on conformal prediction. In Journal of Machine Learning Research, vol. 9: 371–421, 2008.
• [2] J. Fürnkranz and E. Hüllermeier. Preference learning. Springer, 2010.
• [3] A. Korba. Learning from ranking data: theory and methods. Phd thesis, Université Paris-Saclay, 2018.