CENTRE@CLEF 2018

CLEF   NTCIR   TREC
REproducibility

Aims and Scope

The goal of CENTRE@CLEF 2018 is to run a joint CLEF/NTCIR/TREC task on challenging participants to reproduce best results of the most interesting systems in previous editions of CLEF/NTCIR/TREC and to contribute back to the community the additional components and resources developed to reproduce the results.
The CENTRE@CLEF 2018 lab will offer three pilot tasks:

  • Task 1 - Replicability: the task will focus on the replicability of selected methods on the same experimental collections;
  • Task 2 - Reproducibility: the task will focus on the reproducibility of selected methods on the different experimental collections;
  • Task 3 - Re-reproducibility: Re-reproducibility: the task will focus on using the components developed in T1 and T2 and made available by the other participants to replicate/reproduce their results.
Since CENTRE is a joint CLEF/NTCIR/TREC activity, participants in Tasks 1 and 2 will be challenged to reproduce methods and systems developed in the other two evaluation activities, i.e. CLEF will challenge against NTCIR and TREC results. Participants in Task 3 will be challenged to re-reproduce the results using the components developed in any of the three evaluation activities.

Sign Up

Important Dates

Task 1 runs due from participants: TBA

Task 2 runs due from participants: TBA

CLEF 2018 conference: September 10-14, 2018

Submission Guidelines

TBA

Organizers

Nicola Ferro, University of Padua, Italy
ferrodei.unipd.it

Maria Maistro, University of Padua, Italy
maistrodei.unipd.it

Tetsuya Sakai, Waseda University, Japan
tetsuyasakaiacm.org

Ian Soboroff, National Institute of Standards and Technology (NIST), US
ian.soboroffnist.gov