CENTRE@CLEF 2019

CLEF   NTCIR   TREC
REproducibility

Aims and Scope

The goal of CENTRE@CLEF 2019 is to run a joint CLEF/NTCIR/TREC task on challenging participants to reproduce best results of the most interesting systems submitted in previous editions of CLEF/NTCIR/TREC and to contribute back to the community the additional components and resources developed to reproduce the results.

The CENTRE@CLEF 2019 lab will offer two pilot tasks:

  • Task 1 - Replicability: the task will focus on the replicability of selected methods on the same experimental collections;
  • Task 2 - Reproducibility: the task will focus on the reproducibility of selected methods on the different experimental collections;
  • Task 3 - Generalizability: the task will focus on collection performance prediction and the goal is to rank (sub-)collections on the basis of the expected performance over them.

Since CENTRE is a joint CLEF/NTCIR/TREC activity, participants in Tasks 1 and 2 will be challenged to reproduce methods and systems developed in all the evaluation campaigns, i.e. CLEF will challenge against NTCIR and TREC results in addition to CLEF results.

To participate in the lab, the groups need to register (by TBA) at the following link:

TBA

Important Dates

Registration closes: TAB

Runs due from participants: TBA

Submission of participant papers: TBA

Notification of acceptance: TBA

Camera ready due: TBA

CLEF 2019 conference: September 09-12, 2019

Tasks Description

TBA

Submission Guidelines

TBA

Organizers

Nicola Ferro, University of Padua, Italy
ferrodei.unipd.it

Norbert Fuhr, University of Duisburg-Essen, Germany
norbert.fuhruni-due.de

Maria Maistro, University of Padua, Italy
maistrodei.unipd.it

Tetsuya Sakai, Waseda University, Japan
tetsuyasakaiacm.org

Ian Soboroff, National Institute of Standards and Technology (NIST), US
ian.soboroffnist.gov