User knowledge and Search Goals in Information Retrieval: A benchmark and study on the evolution of users' knowledge gain

Abstract

This abstract presents an Information Retrieval framework that personalises results based on the user's knowledge and search goals. The framework utilises the content of the pages visited by the user to represent his/her knowledge, and a set of questions/statements the user wishes to answer to represent his/her search goals. In the absence of related datasets and benchmarks, we propose a methodology to evaluate the framework.

Keywords

• eol>Information Retrieval
• User Knowledge
• User Search Goals
• Search Personalisation

-

1. FRAMEWORK AND

EVALUATION query, it is expected to receive a ranked list of documents that are the least similar to the user’s knowledge and the most similar to his/her goals. The decision of returnThe consideration of the user’s cognitive components in ing a document or not is based on three elements: the the domain of Information Retrieval IR was set as one of knowledge, the goal, and the document to be proposed. the “major challenges” by the IR community in 2018 [ 1 ]. All three elements are supposed to have a textual format; To our knowledge, there is no research dealing with the we propose 3 methods to represent them: (1) Keyword content of the documents read by the user as his/her ac- representation using RAKE - Rapid Automatic keyword quired knowledge. In general, such content has been used extraction [ 2 ] (2) Vector representation using GloVe to construct the user profile from which the user prefer- Global Vectors for Word Representation [ 3 ] (3) Vector ences could be obtained, for example. Those profiles are representation using BERT - Bidirectional Transformers usually either static or not frequently updated, therefore for Language Understanding - embedding [ 4 ]. Finally, cannot help in representing the user’s knowledge, which the similarity between those elements’ representation is is constantly evolving. The constant evolution of the calculated and documents are returned accordingly. user’s knowledge is an important aspect to be considered Evaluation Challenges : The challenge to evaluate when proposing documents that are supposed to have the framework is the lack of adequate datasets or related novel content and/or help him/her achieve a goal not yet benchmarks. Numerous existing datasets logged search achieved. sessions’ activities, however to the best of our knowledge,

The IR Framework: We propose a cognitive agent none did track the user’s knowledge and its change after that is “aware” about its user’s knowledge and goals; reading a document. Our idea is to obtain such informathose information are set as the agent’s beliefs. The tion by adapting a public dataset [ 5 ] that measured the user’s knowledge is represented by the content of the user’s knowledge gain during a search session. That will documents he/she reads; the agent will update its beliefs allow us to evaluate the framework. about the user’s knowledge after every document read. Dataset’s Experiment : The dataset’s experiment The goals are represented by the set of questions the user quantified the user’s knowledge gain about a topic after wishes to answer at the end of a search session. The pro- a search session. The participants were provided an inposed agent employs its beliefs to provide the user with formation need sentence for a specific topic, then were documents that contain novel information in respect to invited to search the web about it; their behaviour was what he/she already knows and that also help to reach getting logged meanwhile. They also had to respond to his/her search goals. Therefore, in response to a user pre- and post-session tests that consisted of statements related to the topic. The tests assessed the participants knowledge regarding the topics and were scored based on the correctness of the answers. A user’s knowledge gain was measured as the diference between the postand pre- tests’ scores.

Benchmark Creation : To estimate the page knowledge gain brought by each page , we perform a linear regression analysis of the user knowledge gain against visited pages which are binary values- visited or not visited. would then be the regression coeficient. We could hence understand and predict a user’s knowledge gain after visiting a set of pages P. As the user visits one page after the other, we track the cumulative evolution of the knowledge gain. We construct a benchmark containing for each user, the set of submitted queries, the related visited pages and the associated evolution of knowledge gain.

Framework Evaluation : The evaluation’s idea is to submit to the framework, the set of queries submitted by every user and suppose the user read the document returned by the agent. We consider the study population to be the set of users who scored zero in the pre-session test, representing those having no previous knowledge about the searched topic; the agent’s beliefs about the user’s knowledge are then still empty. They will get updated as the user starts visiting pages. The user goals consisted of the information need and the test statements.

For the first query submitted by a user, since the agent has no information yet about the user’s knowledge, we return the same page visited in the benchmark. The agent builds its initial beliefs about its user’s knowledge and starts its personalising task. For the following queries, the agent compares the content of the pages to be proposed to the agent’s beliefs (both the user knowledge and goal) and decides which document to return. We track the evolution of the user knowledge gain and compare it to the benchmark.

References

[1] J. S. Culpepper , F. Diaz , M. D. Smucker, Research frontiers in information retrieval: Report from the third strategic workshop on information retrieval in lorne (SWIRL 2018 ), SIGIR Forum 52 ( 2018 ) 34 - 90 .

[2] S. Rose , D. Engel , N. Cramer , W. Cowley , Automatic keyword extraction from individual documents , Text mining: applications and theory 1 ( 2010 ) 1 - 20 .

[3] J. Pennington , R. Socher , C. D. Manning , Glove: Global vectors for word representation , in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , 2014 , pp. 1532 - 1543 .

[4] J. Devlin , M.- W. Chang , K. Lee , K. Toutanova , Bert: Pre-training of deep bidirectional transformers for language understanding , arXiv preprint arXiv: 1810 . 04805 ( 2018 ).

[5] U. Gadiraju , R. Yu , S. Dietze , P. Holtz , Analyzing knowledge gain of users in informational search sessions on the web , in: Proceedings of the 2018 Conference on Human Information Interaction & Retrieval , 2018 , pp. 2 - 11 .