Accounting for the user's cognitive aspects in the information retrieval field is still considered a challenge up until our days. Knowing that recent frameworks are trying to fill this gap, the bigger challenge remains to evaluate those frameworks and to measure the results' relevance in view of the user cognition. The majority of existing evaluation measures often consider isolated document-query environments. Traditional evaluation measures, for example, precision and recall, are not suitable to evaluate the quality of such IR algorithms. Gofman et al. recognised that the relevance of a document must be determined with respect to the documents appearing before it while Boyce et al. claimed that the change a document makes in the knowledge state must be reflected in the choice of document for the second position. The few measures that account for the user's cognitive aspects when evaluating the “relevance" of a result or ranking are limited to one search session, one query, or one search goal. The evaluation metric proposed by Clarke et al. for example systematically rewarded novelty and diversity; however, only one interaction session was considered. Also, the existing evaluation methods do not consider that the user can submit diferent queries. They neither track nor update the user's knowledge. Filling this gap is the main aim of our work. We present our ongoing work on an evaluation framework for cognitive information retrieval systems that considers the user's previous knowledge and his/her search goal. The user's knowledge is a dynamic representation that changes after reading any result. Similarly, the progress towards the goal is re-assessed. The mentioned representations are not limited to one query or one search session but rather constructed throughout all the user-system interactions. This paper focuses on learning-related search tasks where the user's need (or goal) is to learn about some topic . We adopted the user's learning model named vocabulary learning, which takes place at the lower level of Bloom's taxonomy. During a vocabulary learning task, in an information retrieval context, users interact with the search system to acquire keywords related to a topic T.
Search and Learning Goals: This model considers a user’s information need as achieved when he/she learns about the keywords related to . The learning goal is represented as a weighted set of keywords = {1, . . . , }, which we call vocabulary keyword need; is the number of keywords representing the need. For example, the need to know about the topic “deep learning" might involve learning about machine learning, programming, and calculus. The keywords needed to learn a topic could be extracted implicitly from the query expressing the user’s need, or explicitly from the search task text. A search task, for example, could be a school assignment, or a set of questions to answer. Once the keywords to be learned are defined, the number of occurrences the user has to read must also be decided. We adopt the hypothesis that to satisfy a search need or learning goal, any user should read a given number of occurrences of those keywords. We define the corresponding distribution vector vocabulary occurrence need = {1, . . . , }, ( ∈ N), noting how many instances of every keyword of a user has to read to achieve the learning goal . To calculate the number of instances to be read, we suppose is the total number of keywords the user had to read, then its distribution among the keywords will be proportional to the weight of each keyword in a corpus. We consider in this paper a personalised approach that accounts for the user’s previous knowledge. Indeed, the user might already have some previous knowledge about those keywords, for example, having already read them in another document in the previous session.
User’s Knowledge The user’s knowledge is represented by the set of keywords he/she is knowledgeable about, stored in a knowledge base = {1, . . . , }. The amount of the user’s knowledge about a word is measured by the cumulative count of the number of times the user has seen the word before; this measure is stored in = {1, . . . , }. We also assume that the user’s knowledge of a keyword monotonically increases with each instance of it that the user reads. We made the hypothesis that users acquire their knowledge from the documents they read. When the user reads a document = {1, . . . , }, the keywords contained in the document are taken into account in the user’s knowledge - added to the user’s knowledge base KB. The keywords occurrence is represented by = {1, . . . , }. Several methods can be used to extract the keywords from the documents, for example, RAKE—Rapid Automatic Keyword Extraction Algorithm, TF-IDF, or KeyBERT. The user’s previous knowledge will impact the number of occurrences he/she still has to read. Notice that, in order to solve the “cold start problem”, we assume that the user’s knowledge base is empty at the very ifrst session, i.e. we suppose that the user did not read any document before the first search session ( = ∅). However, unlike in the literature, we relax this hypothesis for the subsequent search sessions and suppose that the user can have some previous knowledge acquired in the previous sessions. In this case, we have that ̸= ∅ i.e. = {1, . . . , }. The iℎ element of the user’s vocabulary occurrence need, must consider the previous user’s knowledge. The vocabulary need at the beginning of one session is calculated using the following equation, with = .
= ⎧ , ⎨ 0, ⎩ − , if = 0; if > ; otherwise .
(1) The intuitive meaning of this formula is that if the user does not have any previous knowledge then the amount of information he/she has still to read corresponds to the original content of the vocabulary occurrence need i.e. = . On the contrary, if the user has already some previous knowledge that is relevant to the information need, the number of occurrences he/she has still to read should decrease with respect to .
The Evaluation Framework We present the features that our cognitive evaluation frameworks will take into account: (I) Measuring the gain brought by one document to a user. Let () denote the learning gain brought by a document to a user (user is represented here by his need ). It is considered to be the sum of gains brought by the keywords inside it (). Calculating the gain of every document will allow determining the most “relevant” document or the one to be returned first. In the ideal situation, the document should contain the exact number of occurrences needed, i.e. ∀, such that = , we should have = . In this case, indeed, the gain would be equal to its maximum value. When the number of occurrences in the document is diferent (greater or lower) from the number of occurrences needed, which is very probable, a gain function must be applied. The ongoing work is studying the relevance of a document with respect to the user’s need. The proportion of and will be taken into account. The results will allow us to come up with the gain formula accounting for the conditions above.
(II) Measuring the Gain brought by a document at rank r. Applying the gain function to each document allows us to determine the one to be ranked first. For the second and for every subsequent document, the gain measure should be reapplied with the user’s updated knowledge state. The documents scoring higher will be ranked first. More precisely, before calculating the gain brought by a document, the must be updated according to the information proposed in the document before it (at rank r-1). For example, the knowledge state of a user after reading document at rank = 1, gets updated by adding all the keywords of document and their occurrences. We can define the gain at rank , [], as the gain provided by the document at rank : [] = ().
(III) Calculating the cumulative and discounted Gains We calculate the cumulative gain vector at rank , [], as follows: [] = ∑︀=1 []
In order to take into consideration the rank of the documents, before computing the cumulative gain vector, a discount may be applied at each rank to penalize documents lower in the ranking. We use the classical discount: 2(1 + ) to calculate the discounted cumulative gain at rank . Finally, we normalize the calculated DCG by the ideal discounted cumulative gain vector. ′: − = ′ .
We discussed in this paper an ongoing work of an evaluation framework for retrieval algorithms that account for the user’s cognition, especially the user’s search goal and knowledge. The framework penalises the redundancy of information not only with respect to the previously proposed documents, but also to the user’s knowledge, and helps in the direction of achieving a search goal. In our framework, the user’s search need is represented by a defined number of occurrences of keywords the user needs to read. A document is considered relevant as long as it contains some needed keywords. After that point, we penalize for redundancy. We also consider the “changing characteristics” of the cognitive aspects. We can say that the to be proposed measure will be a first step toward evaluating the ranking and retrieval algorithms with respect to the user’s cognition.