Evaluation is a vital activity that can not be ignored in designing IIR systems. User-centric evaluation approaches are based on user studies and laboratory experiments which are costly and time-consuming. Simulation has been proposed and utilised widely as a resource saving solution. The current users' simulations are criticised for not being based on real user studies therefore this research will try to build more realistic user simulation based on user profiles. The paper starts with an overview of the field of the IIR and evaluation. Next, it presents key attributes that distinguish specific users on the ground of well-established informational behaviour models. Then, it summarises methods for constructing user profiles in order to produce realistic models, and concludes with a brief description of the proposed evaluation methodology.
The core activities of IIR field is to study users' interaction with IR systems and evaluate the users' satisfactions with the retrieved information [8]. IIR systems are specifically defined by Borlund [6] as "those where the user dynamically conducts searching tasks and correspondingly reacts to systems responses over session time.". Thus, users' behaviours, experiences and interactions with systems or information are the main focus of IIR's studies [18]. Three ingredients are essential in IIR studies [7].
1. The involvement of potential users as test participants. 2. The use of dynamic and individual information needs. 3. The employment of multidimensional and dynamic relevance judgements.
Belkin claims that research on IR algorithms is much more popular due to the complexity of studying and measuring the human perspective [4]. The human perspective includes "information processing, changes in goals in the strategies of users, effective and contextual elements of information seeking, and the influence of individual characteristics or behaviour patterns" [12].
Evaluation is a fundamental aspect of both IR and IIR research [19,29].
Generally, IIR evaluation aims to involve real users in the evaluation process. Given the fact that the user's interaction is the focus of IIR evaluation, it is essential to evaluate the system in relation to interactive information searching and retrieval processes [6].
The main concern of the IIR evaluation is to study the ability of the users to engage with a system in order to retrieve relevant documents [18].
Over all, the user-centred evaluation approach is costly and time consuming. Thus, simulating users has been proposed as a resource saving solution.
Recently, simulation has become a preferable tool for evaluating IR and IIR systems due to its ability to reduce the expenses and time of conducting users' studies. However, its credibility is still under investigation. Most of the simulations have been built on theoretical bases instead of on real users' studies. In order to be accurate and realistic simulation, the simulation should be seeded on real data and real interactions [1]. The existing simulation models can be classified into: Conceptual and descriptive models such as Bates' Berry Picking Model [3] and Ingwersen and Järvelin model of information seeking research [10]. Predictive and explanatory models for example Information Foraging Theory [22], and the interactive probability ranking principle model [13]. Formal models such as modelling user variance in time-biased gain [26], modelling the interaction of the users with the topic summaries and predict the probability of clicking on a result [11], and Complex Searcher Model (CSM) and User State Model (USM) [21]. The main focus of the existing simulations is on the users' interactions in particular simulating search behaviour including formulating queries, scanning snippets, clicking links, reading documents, judging document relevance and deciding stopping. They exclude the individual differences between searchers and their link with the users' ISB. Therefore, our proposed methodology will consider the most influential factors of ISB and try to personalise the simulations by extracting data from the users' profiles.
In order to satisfy their information needs users tend to search with IR systems. Thus, the effectiveness of such systems can be evaluated in terms of their support to achieve users' goals or tasks. Understanding the engagement of users in the information seeking process and their behaviour is vital in order to build and design effective IR systems [5]. Many studies have been conducted to investigate what are the most influential factors in ISB. However, here the most cited and wellestablished models will be considered including Wilson models [27,28], Leckie model [20], Savolainen model [23], Johnson model [17], Byström and Järvelin model [9], and Ingwersen and Jarvelin model [16].
According to the literature, the ISB factors can be categorised into seven main categories: 1. information needs; 2. roles and tasks (frequency, predictability, importance, and complexity); 3. information sources and awareness (familiarity, trustworthiness, packaging, timeliness,cost, quality, accessibility); 4. context(cultural, organisational, social, sector's type); 5. socioeconomic (wealth, contact networks, occupation); 6. personal (education, attitude, experience, motivation, values physiological, affective or emotional needs, cognitive needs, demographics, environmental variable, personal style of seeking and personal relevance, person's degree of knowledge); 7. situational (situation specific need, available time, state of health).
Each of these can be personalised by considering individual user's profile.
User's profile is "a digital representation of the unique data concerning a particular user" [25] where essential information about individuals are presented [24] User's profile is used to collect users' interests, improve quality of information access and infer user's intentions [25]. The most common contents of user profiles are:
-Users' interests topics. They can be either short-term interests i.e the user's current interests or long-term interests that do not change frequently [14,25]. -Knowledge, background and skills include goals, user's behaviour, characteristics, and contextual information [24].
There are three different methods to construct the user profiles: explicit, implicit and hybrid [15,24] these are summarised in Table 1.
Table 1. Methods of User Profiling Construction.
We propose a novel approach for simulating users which does not merely focus on simulating the search behaviours but also considers attributes describing users, system, and tasks [2]. Our proposed methodology is grounded on real user studies where user data are based on real users' profiles. In contrast to the existing approaches where the simulated users are grounded on surrogate subjects performing simulated tasks [21], we study how real searchers use a IIR system in order to fulfil their genuine information needs. We choose RERO Doc which is a Swiss digital library1 to conduct the user study. The first phase starts by collecting data of real users while they are searching in RERO Doc. The aim of this phase is to build the user profiles. To capture the implicit and explicit data of the users, the hybrid method of constructing user profile is used. Based on the literature the most significant attributes are [2]:
-Demographic data: age, gender, academic status,satisfaction.
-Task data: description of the task, difficulty, urgency, initial queries, sufficiency of information gathering. -System data: search experience, degree of familiarity, frequency of using the system, efforts to locate useful information and relevance judgment. -Sessions: changes in queries during a session, the duration of a session, number of pages viewed and number of documents downloaded. -Queries: queries issued, queries modifications, mean length of search queries, the use of logic and modifiers, and the types of queries i.e. unique, modified, identical, and repeat query.
-Terms: the most highly used search terms and the number of terms.
The second phase will produce personas based on data gathered in the first phase. Multiple personas for each type of RERO Doc's users will be created to act as placeholders for the real users.
In the third phase, we will design simulated tasks extracted from genuine information needs expressed by real users during the first phase. After that, we will recruit a small number of human subjects and have them and the personas to perform the simulated tasks. The aim of this phase is to evaluate the performance of the personas compared with the human subjects.
The fourth phase is to evaluate the proposed methodology by comparing the results of the first and the third phases.
This paper reviews the state of the art in IIR and proposes a new approach to simulate users when running evaluations. The proposed method is original in two aspects: it is grounded on a user study where real users conduct their own searches and it includes the most influential attributes of ISB. We expect this combination to produce more realistic simulations by accounting for individual differences among searchers and focusing on real tasks.