Aims and Scope
CENTRE aims at developing and tuning a reproducibility evaluation protocol. In particular, we target two specific objectives:
- Replicability: different team, same experimental setup;
- Reproducibility: different team, different experimental setup.
Organizers will select, among the methods/systems submitted to the CLEF/NTCIR/TREC ad-hoc tasks over the years, the top performing and most impacting ones.
Each participating group will be challenged to replicate and/or reproduce one or more of the selected systems by only using standard open source IR systems, like Lucene, Terrier, and others.
Each participating group will submit one or more runs representing the output of their reproduced systems. They will also develop and integrate into the open source IR systems all the missing components and resources needed to replicate/reproduce the selected systems. Finally, they will be asked to contribute back to open source all the developed components, resources, and configuration via a common repository.
Therefore, the goal of CENTRE is to run a joint CLEF/NTCIR/TREC task on challenging participants:
- to reproduce best results of best/most interesting systems in previous editions of CLEF/NTCIR/TREC by using standard open source IR systems;
- to contribute back to the community the additional components and resources developed to reproduce the results in order to improve existing open source systems.
