CENTRE@CLEF 2018

Tasks Description

Among the systems submitted to the CLEF/NTCIR/TREC ad-hoc tasks we selected the following papers:

• [Guyot et al, 2005] Guyot, J., Radhouani, S., and Falquet, G. (2005). Ontology-Based Multilingual Information Retrieval. In Working Notes for the CLEF 2005 Workshop.
• [Nguyen et al, 2008] Nguyen, D., Overwijk, A., Hauff, C., Trieschnigg, D. R., Hiemstra, D., and De Jong, F. (2008). WikiTranslate: Query Translation for Cross-lingual Information Retrieval Using only Wikipedia. In Working Notes for the CLEF 2008 Workshop.
• [Sorg et al, 2008] Sorg, P., and Cimiano, P. (2008). Cross-lingual Information Retrieval with Explicit Semantic Analysis. In Working Notes for the CLEF 2008 Workshop.
• [Yang et al, 2013] Yang, P., and Fang, H. (2013). Evaluating the Effectiveness of Axiomatic Approaches in Web Track. In TREC 2013.
• [McCreadie et al, 2014] McCreadie, R., Deveaud, R., Albakour, M., Mackie, S., Limsopatham, N., Macdonald, C., Ounis, I., and Thonet, T. (2014). University of Glasgow at TREC 2014: Experiments with Terrier in Contextual Suggestion, Temporal Summarisation and Web Tracks. In TREC 2014.
• [Gallagher et al, 2017] Gallagher, L., Mackenzie, J., Benham, R., Chen, R. C., Scholer, F., and Culpepper, J. S. (2017). RMIT at the NTCIR-13 We Want Web Task. Proceedings of NTCIR-13.

The following table reports, for each paper, the name of the run to be replicated and/or reproduced and the datasets and topics to be used for the replicability and reproducibility tasks.

Corpora:

CLEF datasets can be downloaded here. To get the username and password necessary to download the data, the participants need to follow the instruction written on the end user agreement.

TREC datasets are distributed by CMU at the link: http://lemurproject.org/.

Topics and Qrels:

CLEF topics and qrels are available here. To get the username and password necessary to download the data, the participants need to follow the instruction written on the end user agreement.

Topics and qrels needed for the Replicability Task:

• [Yang et al, 2013], TREC 2013, Web Track, Ad Hoc Task: topics, qrels.
• [McCreadie et al, 2014], TREC 2014, Web Track, Ad Hoc Task: topics, qrels.
• [Gallagher et al, 2017], NTCIR-13, We Want Web Track: topics, qrels.

Topics and qrels needed for the Reproducibility Task:

• [Yang et al, 2013]:
  • TREC 2012, Web Track, Ad Hoc Task: topics, qrels.
  • TREC 2014, Web Track, Ad Hoc Task: topics, qrels.
• [McCreadie et al, 2014]:
  • TREC 2012, Web Track, Ad Hoc Task, topics, topics.
  • TREC 2013, Web Track, Ad Hoc Task: topics, qrels.
• [Gallagher et al, 2017]: NTCIR-13, We Want Web Track: topics, qrels.

Submission Guidelines

Groups should satisfy the following guidelines to participate in this lab:

• They need to use an open source IR system, e.g. Lucene, Terrier, Indri, Atire;
• The runs should be submitted in TREC format;
• For the replicability task, each group can submit a maximum of three runs replicating the given original run;
• For the reproducibility task, each group can submit just one run reproducing the given original run;
• The code used to produce the runs should be uploaded in a Bitbucket repository provided by the organizers upon the registration to CENTRE@CLEF.

Trec Format:

Runs should be submitted with the following format:

30 Q0 ZF08-175-870  0 4238 prise1
30 Q0 ZF08-306-044  1 4223 prise1
30 Q0 ZF09-477-757  2 4207 prise1
30 Q0 ZF08-312-422  3 4194 prise1
30 Q0 ZF08-013-262  4 4189 prise1
...
   						

where:

• Columns are separated by a white space;
• The first column is the topic number;
• The second column is the query number within that topic, which is currently unused and should always be Q0;
• The third column is the official document number of the retrieved document, which is the number found in the "docno" field of the document;
• The fourth column is the rank at which the document is retrieved;
• The fifth column shows the score (integer or floating point) that generated the ranking. This score must be in descending (non-increasing) order. It is important to include the score so that we can handle tied scores (for a given run) in a uniform fashion (trec_eval sorts documents by these scores, not your ranks);
• The sixth column is called the "run tag" and should be an unique identifier for your group AND for the method used. That is, each run should have a different tag that identifies the group and the method that produced the run.

It is important to include all the columns and have a white space delimiter between the columns.

Runs should be submitted at the following link:

TBA

Evaluation:

The quality of the submitted runs will be evaluated from two point of views:

• Effectiveness: how close are the performance scores of the reproduced systems to those of the original ones. This will be measured using the RMSE between the new and original MAP scores;
• Ranked result lists: since different result lists may produce the same effectiveness score, we will also measure how close are the ranked results list of the reproduced systems to those of the original ones. This will be measured using Kendall’s tau.
• Improvement over the baseline: since for the reproducibility task we do not have the original run, we will compare the reproduced run score with a baseline score to see whether the improvement over the baseline is comparable between the original and the new dataset.