LongEval CLEF 2023 Lab
Longitudinal Evaluation of Model Performance
| Description | Dates | Organizers | Tasks | Data | Submissions |
Link: https://github.com/clef-longeval/IR-Participants.git
Submissions will be done using git. Each team will have a private repository created by the organizers, which will be used to submit the runs to the task. The information about the repository is for each team provided by the organizers. Please contact the organizers if you did not receive this information.
All the participants can either participate in Sub-task A, short-term persistence, Sub-task B, long-term persistence, or both these sub-tasks. Participants can submit up to 5 systems applied to either one or both tasks. For each system submitted on either sub-task, the participants also need to submit the results acquired by this system on Train heldout queries which represent within a time benchmark estimate. This allows the organizers to acquire the information about the change of the system’s performance. Participants also need to provide a short description of each of the submitted systems.
We further denote the submission of a single system on a single sub-task or on the within a time query set as the run. The individual runs need to be submitted in the TREC format. For each query in each run, it is allowed to return up to 1000 documents.
Each system should be submitted in a single zipped file consisting of a following tuple:
team_system.WT contains a run (a single TREC file) of the system on the Train heldout queries, used to measure within a time (WT) performance.
team_system.ST contains a run (a single TREC file) of the system acquired on the Short Term Test queries.
team_system.LT contains a run (a single TREC file) of the system acquired on the Long Term Test queries.
Please note, that if the participants decide to only participate in the short-term or only in the long-term sub-task, then team_system.LT or team_system.ST respectively might be missing. However, the participants in all cases need to include team_system.WT and team_system.description files.
team_system.meta contains a short description of the approach. This file should contain the information which indexing and ranking methods were applied, what type of training was applied and which training data were used. Please specify if you used statistical or neural approaches and if you used sparse or dense retrieval methods. Also, please include the information if the approach uses a single ranking approach, multiple-stages of rankers or any (and what) other type of fusion. Participants also need to describe if they used French data, provided English translations or their own translations and the resources (memory, GPUs, CPUs) used. Participants should use this provided form for filling all the system details.
All the files in a single zipped document should thus correspond to a single system. The name of this system should contain the name of the team and an unique identifier of this system. The suffix of each file should either be WT, ST, LT or meta. For example, if the file contains the submission of the BM25 system of the UGA team applied on the long-term task, the file name can be UGA_BM25.LT.
Each system might be either run on French or English data (or their combination). The participants might also opt to use their own translations systems or even manual translations. However, if any manual intervention is used, even for the translation, participants need to clearly state this in the system description.