Building High-Quality Datasets for Information Retrieval Evaluation at a Reduced Cost

• David Otero ^[1] ; Daniel Valcarce ; Javier Parapar ^[1] ; Álvaro Barreiro ^[1]
  1. [1] Universidade da Coruña

     Universidade da Coruña

     A Coruña, España

     
• Localización: XoveTIC 2019: The 2nd XoveTIC Conference (XoveTIC 2019), A Coruña, Spain, 5–6 September / Alberto Alvarellos González (ed. lit.), Joaquim de Moura (ed. lit.), Beatriz Botana Barreiro (ed. lit.), Javier Pereira-Loureiro (ed. lit.), Manuel Francisco González Penedo (ed. lit.), 2019, ISBN 978-3-03921-444-0
• Idioma: inglés
• Enlaces
  • Texto Completo Libro (pdf)
• Resumen

  • Information Retrieval is not any more exclusively about document ranking. Continuously new tasks are proposed on this and sibling fields. With this proliferation of tasks, it becomes crucial to have a cheap way of constructing test collections to evaluate the new developments. Building test collections is time and resource consuming: it requires time to obtain the documents, to define theuser needs and it requires the assessors to judge a lot of documents. To reduce the latest, pooling strategies aim to decrease the assessment effort by presenting to the assessors a sample of documents in the corpus with the maximum number of relevant documents in it. In this paper, we propose the preliminary design of different techniques to easily and cheapily build high-quality test collections without the need of having participants systems.