Although, most of the recent studies within the IR domain tend to target how users behave while addressing their information needs. However, even though the collection of document sets and user pro ling is a top research problem, it holds inherent di culties for the establishment of a comparative task to evaluate various approaches. Also nding a comprehensive metrics to evaluate di erent a ects of various aspects, that play a signi cant role in satisfaction of users of personalized IR systems, seems to be another noticeable issue. With the review of related tasks in well-known IR evaluation communities, this paper will discuss the way of gathering users pro les and objects of their interest in last 5 years of IR evaluation campaigns.
Evaluation tasks are well-known and innovative ways of
providing the infrastructure necessary for stimulating,
demonstrating and evaluating substantial improvements of
information retrieval methodologies. \Future information retrieval
systems must anticipate user needs and respond with
information appropriate to the current context."[
Considering the daily-growing tendency to user-centred IR systems, evaluation campaigns are promoted to the exploration of new evaluation methodologies for such systems. Nevertheless, due to the complexity of the evaluation of personalized IR systems and the involved potential costs of evaluation, some IR evaluation forums such as NII Testbeds and Community for Information access Research (NTCIR)1 and Forum for Information Retrieval Evaluation (FIRE)2 have not focused on user-centred tasks yet.So, this section describes a user-oriented and context-based task approach which has been provided in other IR evaluation communities. 2.1
Starting in 1992, the Text REtrieval Conference (TREC)3, co-sponsored by the National Institute of Standards and Technology (NIST) and U.S. Department of Defense, is the most popular IR evaluation campaign which has also provided the rst large-scale evaluations of cross-lingual and multilingual document retrieval tasks. TREC has also introduced evaluations for open-domain question answering, content-based retrieval of digital video and retrieval of recordings of speech.
The goal of the contextual suggestion track [
Conference and Labs of the Evaluation Forum, formerly known as Cross-Language Evaluation Forum (CLEF)5 promotes development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure.
The Social Book Search [
As a continuation of the INEX SBS Track that ran from 2011 up to 2014, the task is targeted to returning a list of recommended books in reply to a user request posted on a LibraryThing 6(LT) discussion forums by matching the user's information need. A set of book requests and a set of user pro les have been assumed as inputs of the task and a submitted ranked list of recommended books has been evaluated as the result of participant's system.
The test collection [
The topic set is focused on requests which are provided as a narrative description of the information need of a user and one or more example books to guide the suggestions. Users typically describe what they are looking for, give examples of what they like and do not like, indicate which books they already know and ask other members for recommendations. There are also annotated elds by crowdsourcing workers to indicate whether the example book had been read by requester and to judge his/her attitude about the book.
The books suggested by members, which are directly linked to their corresponding records on Amazon, have been used as initial relevance judgements for evaluation of participated systems in the Suggestion Track.
The rich user pro les of the topic creators and other LT users have been used as valuable resources of User pro les and personal catalogues. These pro les generally contain information on which books they have in their personal catalogue on LT, which ratings and tags they assigned to them and a social network of friendship relations, interesting library relations and group memberships.
The o cial evaluation measure for this task is nDCG@10. It takes graded relevance values into account and is designed for evaluation based on the top retrieved results. In addition, P@10, MAP and MRR scores will also be reported, with the evaluation results. 3.
Although de ning tasks and scenarios for evaluation purposes in IR domain is one of the most common ways for the exploration of new methodologies and innovative ways in using and discussing experimental data, it has to be noted that the comparison of approaches, the exchange of ideas and transfer of knowledge has been considered a valuable contribution to evaluation tasks during last decades. However, the tendency of modern user-centred IR system for taking user preferences and interests into account through information seeking process also has changed the identi cation, setting and evaluation of shared tasks. This paper reviewed and compared existing user tasks and described their way of collecting task resources, methods and metrics with hope to help improving them with combining/ summarising them to propose new user centered tasks in future. 4.
"The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund." 5.