Evaluating Information Retrieval and Access Tasks

NTCIR was the first large-scale IR evaluation conference series to construct test collections with graded relevance assessments: the NTCIR-1 test collections from 1998 already featured relevant and partially relevant documents. In this
chapter, I provide a survey on the use of graded relevance assessments and of graded
relevance measures in the past NTCIR tasks which primarily tackled ranked retrieval.
My survey shows that the majority of the past tasks fully utilised graded relevance
by means of graded evaluation measures, but not all of them; interestingly, even a
few relatively recent tasks chose to adhere to binary relevance measures. I conclude
the chapter by a summary of my survey in table form.
[2021]

Compartir: