A Realistic Approach Towards Users' Simulation

Abstract

Simulation has been proposed and utilised widely in the eld of the evaluation of information retrieval (IR) and interactive IR (IIR) systems. It can signi cantly reduce costs, make experiments easier to reproduce and save time to users and researchers. The question of how realistic these simulations are remains, to a great extent, unexplored. This is due to the fact that searching for information is a self-directed activity, and varies among users in terms of their information seeking behaviours (ISB) and their relevance judgments. Such variations are affected by a number of attributes describing users, tasks, and systems and their interactions. By identifying these attributes researchers could design more e ective user models and realistic simulations. This paper presents a user-centric evaluation methodology based on user pro les and ISBs.

Keywords

• Interactive information retrieval
• evaluation
• simulation
• user modelling
• user pro le
• information seeking behaviour

-

Evaluation is a vital activity that can not be ignored in designing IIR systems. User-centric evaluation approaches are based on user studies and laboratory experiments which are costly and time-consuming. Simulation has been proposed and utilised widely as a resource saving solution. The current users' simulations are criticised for not being based on real user studies therefore this research will try to build more realistic user simulation based on user pro les. The paper starts with an overview of the eld of the IIR and evaluation. Next, it presents key attributes that distinguish speci c users on the ground of well-established informational behaviour models. Then, it summarises methods for constructing user pro les in order to produce realistic models, and concludes with a brief description of the proposed evaluation methodology. The core activities of IIR eld is to study users' interaction with IR systems and evaluate the users' satisfactions with the retrieved information [ 8 ]. IIR systems are speci cally de ned by Borlund [ 6 ] as "those where the user dynamically conducts searching tasks and correspondingly reacts to systems responses over session time.". Thus, users' behaviours, experiences and interactions with systems or information are the main focus of IIR's studies [ 18 ]. Three ingredients are essential in IIR studies [ 7 ]. 1. The involvement of potential users as test participants. 2. The use of dynamic and individual information needs. 3. The employment of multidimensional and dynamic relevance judgements.

Belkin claims that research on IR algorithms is much more popular due to the complexity of studying and measuring the human perspective [ 4 ]. The human perspective includes "information processing, changes in goals in the strategies of users, e ective and contextual elements of information seeking, and the in uence of individual characteristics or behaviour patterns" [ 12 ]. 3

Evaluation

Evaluation is a fundamental aspect of both IR and IIR research [ 19, 29 ].

Generally, IIR evaluation aims to involve real users in the evaluation process. Given the fact that the user's interaction is the focus of IIR evaluation, it is essential to evaluate the system in relation to interactive information searching and retrieval processes [ 6 ].

The main concern of the IIR evaluation is to study the ability of the users to engage with a system in order to retrieve relevant documents [ 18 ].

Over all, the user-centred evaluation approach is costly and time consuming. Thus, simulating users has been proposed as a resource saving solution. 4

Simulation of Users

Recently, simulation has become a preferable tool for evaluating IR and IIR systems due to its ability to reduce the expenses and time of conducting users' studies. However, its credibility is still under investigation. Most of the simulations have been built on theoretical bases instead of on real users' studies. In order to be accurate and realistic simulation, the simulation should be seeded on real data and real interactions [ 1 ]. The existing simulation models can be classi ed into: Conceptual and descriptive models such as Bates' Berry Picking Model [ 3 ] and Ingwersen and Jarvelin model of information seeking research [ 10 ]. Predictive and explanatory models for example Information Foraging Theory [ 22 ], and the interactive probability ranking principle model [ 13 ]. Formal models such as modelling user variance in time-biased gain [ 26 ], modelling the interaction of the users with the topic summaries and predict the probability of clicking on a result [ 11 ], and Complex Searcher Model (CSM) and User State Model (USM) [ 21 ]. The main focus of the existing simulations is on the users' interactions in particular simulating search behaviour including formulating queries, scanning snippets, clicking links, reading documents, judging document relevance and deciding stopping. They exclude the individual di erences between searchers and their link with the users' ISB. Therefore, our proposed methodology will consider the most in uential factors of ISB and try to personalise the simulations by extracting data from the users' pro les. 5

Information Seeking Behaviour

In order to satisfy their information needs users tend to search with IR systems. Thus, the e ectiveness of such systems can be evaluated in terms of their support to achieve users' goals or tasks. Understanding the engagement of users in the information seeking process and their behaviour is vital in order to build and design e ective IR systems [ 5 ]. Many studies have been conducted to investigate what are the most in uential factors in ISB. However, here the most cited and wellestablished models will be considered including Wilson models [ 27, 28 ], Leckie model [ 20 ], Savolainen model [ 23 ], Johnson model [ 17 ], Bystrom and Jarvelin model [ 9 ], and Ingwersen and Jarvelin model [ 16 ].

According to the literature, the ISB factors can be categorised into seven main categories: 1. information needs; 2. roles and tasks (frequency, predictability, importance, and complexity); 3. information sources and awareness (familiarity, trustworthiness, packaging, timeliness,cost, quality, accessibility); 4. context(cultural, organisational, social, sector's type); 5. socioeconomic (wealth, contact networks, occupation); 6. personal (education, attitude, experience, motivation, values physiological, a ective or emotional needs, cognitive needs, demographics, environmental variable, personal style of seeking and personal relevance, person's degree of knowledge); 7. situational (situation speci c need, available time, state of health).

Each of these can be personalised by considering individual user's pro le. 6

Users' Pro les

User's pro le is "a digital representation of the unique data concerning a particular user" [ 25 ] where essential information about individuals are presented [ 24 ] User's pro le is used to collect users' interests, improve quality of information access and infer user's intentions [ 25 ]. The most common contents of user pro les are: { Users' interests topics. They can be either short-term interests i.e the user's current interests or long-term interests that do not change frequently [ 14,25 ]. { Knowledge, background and skills include goals, user's behaviour, characteristics, and contextual information [ 24 ].

There are three di erent methods to construct the user pro les: explicit, implicit and hybrid [ 15, 24 ] these are summarised in Table 1. We propose a novel approach for simulating users which does not merely focus on simulating the search behaviours but also considers attributes describing users, system, and tasks [ 2 ]. Our proposed methodology is grounded on real user studies where user data are based on real users' pro les. In contrast to the existing approaches where the simulated users are grounded on surrogate subjects performing simulated tasks [ 21 ], we study how real searchers use a IIR system in order to ful l their genuine information needs. We choose RERO Doc which is a Swiss digital library1 to conduct the user study. The rst phase starts by collecting data of real users while they are searching in RERO Doc. The aim of this phase is to build the user pro les. To capture the implicit and explicit data of the users, the hybrid method of constructing user pro le is used. Based on the literature the most signi cant attributes are [ 2 ]: { Demographic data: age, gender, academic status,satisfaction. { Task data: description of the task, di culty, urgency, initial queries, su ciency of information gathering. { System data: search experience, degree of familiarity, frequency of using the system, e orts to locate useful information and relevance judgment. { Sessions: changes in queries during a session, the duration of a session, number of pages viewed and number of documents downloaded. { Queries: queries issued, queries modi cations, mean length of search queries, the use of logic and modi ers, and the types of queries i.e. unique, modi ed, identical, and repeat query. 1 https://doc.rero.ch { Terms: the most highly used search terms and the number of terms.

The second phase will produce personas based on data gathered in the rst phase. Multiple personas for each type of RERO Doc's users will be created to act as placeholders for the real users.

In the third phase, we will design simulated tasks extracted from genuine information needs expressed by real users during the rst phase. After that, we will recruit a small number of human subjects and have them and the personas to perform the simulated tasks. The aim of this phase is to evaluate the performance of the personas compared with the human subjects.

The fourth phase is to evaluate the proposed methodology by comparing the results of the rst and the third phases. 8

Conclusion

This paper reviews the state of the art in IIR and proposes a new approach to simulate users when running evaluations. The proposed method is original in two aspects: it is grounded on a user study where real users conduct their own searches and it includes the most in uential attributes of ISB. We expect this combination to produce more realistic simulations by accounting for individual di erences among searchers and focusing on real tasks.

References

1. Azzopardi , L. , Jarvelin, K. , Kamps , J. , Smucker , M.D. : Report on the sigir 2010 workshop on the simulation of interaction. SIGIR Forum ( 2011 )

2. Barifah , M. : Automatic simulation of users for interactive information retrieval . In: Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval (March 2017 )

3. Bates , M.J.: The design of browsing and berrypicking techniques for the online search interface . Online review 13(5) , 407 { 424 ( 1989 )

4. Belkin , N.: On the evaluation of interactive information retrieval systems . Rutgers University Community Repository document ( 2010 )

5. Belkin , N.: People, interacting with information . SIGIR Forum 49 ( 2 ), 13 { 27 ( 2015 )

6. Borlund , P. : Experimental components for the evaluation of interactive information retrieval systems . Journal of Documentation 56 ( 1 ) ( 2000 )

7. Borlund , P.: The iir evaluation model: a framework for evaluation of interactive information retrieval systems . Information Research 8 ( 3 ) ( 2003 )

8. Borlund , P. : Interactive information retrieval: An introduction . Journal of Information Science Theory and Practice 1 ( 3 ), 12 { 32 ( 2013 )

9. Bystrom, K. , Jarvelin, K.: Task complexity a ects information seeking and use . Inf. Process. Manage . 31 ( 2 ), 191 {213 (mar 1995 )

10. Catarci , T. , Kimani , S. : Human-computer interaction view on information retrieval evaluation . In: Information Retrieval Meets Information Visualization . Springer ( 2012 )

11. Clarke , C.A. , Freund , L. , Smucker , M. , Yilmaz , E.: Sigir 2013 workshop on modeling user behavior for information retrieval evaluation . In: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval . pp. 1134 { 1134 . ACM ( 2013 )

12. Daz , A. , Garca , A. , Gervs , P.: User-centred versus system-centred evaluation of a personalization system . Information Processing and Management 44 ( 3 ), 1293 { 1307 ( 2008 )

13. Fuhr , N.: A probability ranking principle for interactive information retrieval . Information Retrieval 11 ( 3 ), 251 { 265 ( 2008 )

14. Gauch , S. , Speretta , M. , Chandramouli , A. , Micarelli , A. : User pro les for personalized information access . In: The adaptive web . Springer ( 2007 )

15. Godoy , D. , Amandi , A. : User pro ling for web page ltering . IEEE Internet Computing ( 2005 )

16. Ingwersen , P. , Jarvelin, K. : The Turn: Integration of Information Seeking and Retrieval in Context (The Information Retrieval Series ). Springer-Verlag New York, Inc. ( 2005 )

17. Johnson , J.D., Donohue , W.A. , Atkin , C.K. , Johnson , S.: A comprehensive model of information seeking tests focusing on a technical organization . Science Communication ( 1995 )

18. Kelly , D. : Methods for evaluating interactive information retrieval systems with users . Foundations and Trends in Information Retrieval ( 2009 )

19. Kelly , D. , Sugimoto , C. : A systematic review of interactive information retrieval evaluation studies, 1967{2006 . Journal Of The American Society For Information Science And Technology 64 (( 4 )) ( 2013 )

20. Leckie , G.J. , Pettigrew , K.E. , Sylvain , C. : Modeling the information-seeking of professionals: A general model derived from research on engineers, health care professionals and lawyers . Library Quarterly 66 ( 2 ), 161 { 193 ( 1996 )

21. Maxwell , D. , Azzopardi , L. : Agents, simulated users and humans: An analysis of performance and behaviour . In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management series, CIKM '16 . pp. 731 { 740 . No. 10 , ACM ( 2016 )

22. Pirolli , P. , Card , S. : Information foraging . Psychological review 106(4) , 643 ( 1999 )

23. Savolainen , R.: Everyday life information seeking: approaching information seeking in the context of "way of life" . Library & information science research ( 1995 )

24. Schia no, S. , Amandi , A. : Intelligent user pro ling . In: Arti cial Intelligence An International Perspective , pp. 193 { 216 . Springer ( 2009 )

25. Skillen , K.L. , Chen , L. , Nugent , C.D. , Donnelly , M.P. , Burns , W. , Solheim , I. : Ontological user pro le modeling for context-aware application personalization . In: Ubiquitous Computing and Ambient Intelligence , pp. 261 { 268 . Springer ( 2012 )

26. Smucker , M.D. , Clarke , C.L. : Modeling user variance in time-biased gain . In: Proceedings of the Symposium on Human-Computer Interaction and Information Retrieval . p. 3 . ACM ( 2012 )

27. Wilson, T.: Models in information behaviour research . Journal of Documentation pp. 249 { 270 ( 1999 )

28. Wilson, T.D.: On user studies and information needs . Journal of documentation ( 1981 )

29. Zuva , K. , Zuva , T. : Evaluation of information retrieval systems . International Journal of Computer Science and Information Technology (IJCSIT) 4 ( 3 ) ( 2012 )