How does Domain Expertise Affect Users' Search Processes in Exploratory Searches?

Abstract

Huge amount of users use Web search engines to learn new skills and knowledge everyday. Understanding how the users search to learn is essential for making search engines support these learning-related searches more e ectively. Previous researches categorize these learning-related searches as exploratory searches, because they are often open-ended and multi-faceted, in which the user usually submits multiple queries iteratively to explore a large information space. In this position paper, we propose to conduct a user study to investigate whether and how users' domain expertise a ect their search processes in exploratory searches. We also set up a preliminary research framework, design the experiment protocol of the user study, and discuss about the limitations of this study and the potential implications for improving Web search engines.

-

Expertise; Query

1. INTRODUCTION

Web search engines help people e ciently access information on the Web, and fundamentally change the way we learn new skills and knowledge [ 11 ]. When search engine users search to learn new knowledge, their initial information needs are usually multi-faceted and open-ended. While they digest new information by reading the search results, their knowledge structures in mind and their immediate information needs are evolving simultaneously, which leads to highly interactive search sessions with multiple iterative query reformulations. These characters match the de nition of exploratory search adopted by White and Roth [ 15 ]: \Exploratory search can be used to describe an information-seeking problem context that is open-ended, persistent, and multi-faceted; and to Search as Learning (SAL), July 21, 2016, Pisa, Italy The copyright for this paper remains with its authors. Copying permitted for private and academic purposes describe information-seeking processes that are opportunistic, iterative, and multi-tactical ".

While modern search engines are extremely good at helping users locate speci c facts and information, how to better support exploratory search is still a challenging problem. One of the reasons that make supporting exploratory search harder is that the search user plays an even more important role in the interactive exploratory search process. Therefore, the search system needs to go beyond locating information relevant to the query, and provide further help and guidance in exploring unfamiliar information space for users.

To make web search engines more e ective in supporting such tasks, we need to study and understand the process of exploratory search from the user's perspective. In particular, we want to know which user factors a ect the search outcomes of the exploratory search. In this position paper, we focus to study one of the most important factors, domain expertise, and design a user study to investigate whether and how the domain expertise of search users a ects the search outcomes.

In the following of the paper, we will further discuss the research framework and propose research questions in Section 2, present the design of the user study in Section 3, and nally discuss the limitations and potential implications of this study in Section 4. 2.

RESEARCH FRAMEWORK

In this section, we introduce the research framework and the research questions.

The overall research framework is demonstrated in a concept map [ 9 ] shown in Figure 1. A closely related conceptual framework was proposed by Vakkari [ 13 ]. Through a longitudinal empirical study, in which the subjects were college students who attended a 4-month seminar on preparing a research proposal for a master's thesis, Vakkari studied the systematic relationship between the stages of the task performance process, the information sought for, the search tactics adopted by the search users, and the usefulness of the information retrieved. He di erentiated the task performance process into 3 stages: pre-focus, formulation, and post-focus, and analyzed subjects' searching behavior in each stage during the 4-month period. He showed that as the subjects' domain knowledge developed across these stages, the information sought for became more speci c, the number of search terms increased, as well as the search tactics became more diverse. Our work di ers from and further extends Vakkari's study in two ways: 1) while Vakkari's study and ndings are associated to an speci c academic IR system, the LISA data-base, we build an experiment Web search engine to study users' search-to-learn behaviors on general-purpose Web search engines; and 2) in Vakkari's study, the domain knowledge is a longitudinal, within-subject variable determined by the stages in task performance process, but in our study, besides measuring the within-subject learning process over a session, we set the domain knowledge level as a cross-subject independent variable (see Section 3 for how we design the experiment search system to simulate Web search scenarios and how we manipulate domain expertise levels). 2.1

Search Outcome

The search outcomes can be decomposed into two parts: knowledge gain and user satisfaction. They can be measured independently.

The most direct way to measure the knowledge gain is to ask the user to answer questions about the search after she nishes searching. In the user study, we will ask subjects to use search engine to nd answers about a set of pre-de ned questions from di erent domains, and let the domain expert assessors with pro cient domain knowledge grade their answers.

User satisfaction is a measure that \attempts to gauge subjects' feelings about their interactions with the system" [ 6 ]. We plan to use a post-task questionnaire to get explicit satisfaction feedbacks from subjects as well as use implicit user behavior metrics to estimate subjects' satisfaction [ 8 ].

Measuring both knowledge gain and user satisfaction will provide us with a more comprehensive view of the search outcome. For example, domain experts are expected to be more successful in answering in-domain questions [ 14 ]; however, they may be more sensitive to the non-relevant results [ 13 ], and therefore, more likely to feel unsatis ed. 2.2

Domain Expertise

The user's background knowledge about the search task (i.e. the domain expertise) is the rst user factor that we want to investigate in this study.

Previous research suggests that compared to users with little domain knowledge, domain experts search di erently and are generally more successful in in-domain search tasks [ 14 ]. Because the exploratory search is a learning process, search users' domain knowledge and expertise also change simultaneously during search sessions. In previous studies, Eickho et al. [ 3 ] use a few implicit search behavior metrics as evidences of users' knowledge acquisition during searching, and Egusa et al.[ 1 ] use Concept Map to explicitly measure the changes in users' knowledge structures after search. These previous studies developed methods to measure knowledge development during exploratory search sessions. However, they did not investigate the e ects of users' initial domain expertise on the search processes and outcomes, which we will investigate by setting domain expertise as an independent variable in the user study. While domain experts are expected to be more successful in in-domain search tasks, their success may be due to their background knowledge or their expertise in searching for pertinent information. To investigate which is the case, in addition to question answering, we will adopt the implicit behavior metrics and the explicit concept map method to measure the changes of users' knowledge. 2.3

Query Reformulation Strategy

Because the user mainly relies on query reformulations to convey her changing information needs to the search engines, the query reformulation strategy may be another vital factor for the success of exploratory search. Previous works on query reformulation strategy study the reformulation patterns [ 4 ], why the user adds or removes terms in query reformulation [ 5 ], the sources of query terms [ 2, 12 ], and the relationship between query reformulations and search success in struggling search tasks [ 10 ]. These previous works establish methodologies and measures to characterize and model users' query reformulation strategies. In this work, we will adopt these methods to characterize the query reformulation strategy in exploratory search.

Previous study also shows that domain expertise will in uence users' querying behaviors [ 14 ]. In this work, we will study this in uence for the learning-related search tasks, too. On the one hand, the feedback of search outcome is usually hard to collect outside the laboratory user study environment. Therefore, the relationship between query reformulation strategies and domain expertise may be more important in identifying domain experts in practice. On the other hand, understanding how the domain experts query di erently than other users helps us understand how the domain expertise in uences the search processes and outcomes. In a recent study, Odijk et al. [ 10 ] show that in struggling search sessions, the pivotal query to a great extent determines whether the search will succeed or not. We are interested in how the users come up with such pivotal queries. Are the query terms mainly from users' background knowledge (i.e. the domain expertise), or are they read and collected from the SERPs and landing pages during the search processes? To answer these questions, we will investigate the sources of the query terms, and their relationships with both domain expertise and search outcomes. 2.4

Research Questions

To summarize, in this study, we want to investigate the relationship between the domain expertise, query reformulation strategy, and search outcome in exploratory search. Therefore, we propose the following research questions: RQ1 Whether and how does users' domain expertise in uence the search outcomes in exploratory search? RQ2 How does users' query reformulation strategy in uence the search outcomes of the exploratory search? RQ3 Do domain experts have a di erent query reformulation strategy in exploratory search?

USER STUDY DESIGN

The procedure of the user study is shown in Figure 2. We choose 3 domains in this work: environment, medicine, and politics. For each domain, we hired senior graduate students in related majors as domain expert assessors. They are responsible for designing the knowledge learning search tasks and assessing the answers submitted by experiment subjects. With the help of the domain expert assessors, 6 search tasks, 2 for each domain, were designed. Each search task is an open-ended question that can be answered in about 60-100 words. The descriptions for the search tasks are shown in Table 1. The domain expert assessors also provided a reference answer for each task. These answers will be used to access the subjects' answers.

To manipulate the domain expertise level of the subjects, for each domain we will hire 10-15 senior undergraduate students in related majors. Each subject will be asked to complete all 6 search tasks, which means that he or she will complete 2 in-domain tasks and 4 out-of-domain tasks. The order of the tasks will be rotated using the Latin square method. Before the experiment starts, each subject will go through a pre-experiment training stage (I.1), a pre-experiment questionnaire stage (I.2), and an eye-tracking device calibration stage (I.3). In I.1 stage, we will use an example search task, which is not from environment, medicine, or politics domain, to teach the

Subjects  w/   different   domain   knowledge

level I.1 P re-­‐experiment Training I.2 P re-­‐experiment Questionnaire I.3 E ye-­‐tracking Device   Calibration   Each s ubject  needs  to  complete  2  in-­‐domain   &  

4  out-­‐of-­‐domain   tasks II.1 T ask Description Reading and Rehearsal II.2 P re-­‐task Q uestionnaire II.3 P re-­‐task C oncept  Map  Drawing II.4 T ask C ompletion   w/  the  Experiment  Search  Engine   II.5 Q uestion  Answering II.6 P ost-­‐task  Concept  Map  Drawing II.7 P ost-­‐task  Questionnaire subject how to use the experiment search engine. We will also teach the subject how to use concept map in I.1 stage. In I.2 stage, we will collect the subject's basic information, such as age, gender, and experience in using Web search engines. In previous study, Eickho et al. [ 2 ] use an eye-tracking device to study the sources of query terms. In this work, we will also use a Tobii X2-30 eye-tracker to log subjects' eye xations. Therefore, for each subject, we need to calibrate the eye-tracker for her in I.3 stage.

For each search task, the subject will rst read and memorize the task description (i.e. an open-ended question) in II.1 stage. After that, she will complete a pre-task questionnaire (II.2) about the current domain knowledge level, the expected di culty, and the interest level of the task [ 7 ]. The questions in pre-task questionnaire are shown in Table 2. The subject will be required to answer these questions in a 5-point Likert scale (1: not at all, 2: slightly, 3: somewhat, 4: moderately, 5: very). Then, in II.3 stage, the subject will draw a pre-task concept map on paper. This concept map is expected to measure the subject's background knowledge about the current task. In II.4 stage, the subject will use an experiment search engine to complete the search task. When the experiment search engine receives a query, it will forward the query to a commercial Web search engine and retrieve the corresponding SERP. To control the variability in the SERPs, we will lter all the query suggestions, sponsor search results, knowledge graph results, and vertical results out, and only return the organic results to the subject. We will inject JavaScript into this ltered SERP to log all the query reformulations along with other user behaviors such as clicks, tab-switchings, scrolls, and mouse-movements. After completing the search task, the subject will answer the task-related question in II.5 stage and draw a post-task concept map on paper in II.6 stage. The answer and the concept maps will be assessed by the domain expert assessors to measure the subject's knowledge gain. Finally in II.7 stage, the subject will complete a post-task questionnaire about the knowledge level after search, the perceived di culty as well as interest of the task, and the overall user satisfaction, in the same 5-point Likert scale used for the pre-task questionnaire. The post-task questionnaire, which is shown in Table 3, is expected to measure subjects' satisfaction and perceived knowledge gain. 4.

DISCUSION

In this section, we discuss the limitations of this study as well as the potential implications for the design of Web search engines. 4.1

Limitations

We plan to collect data from a laboratory user study. Compared to a naturalistic log-based study (e.g. [ 14 ]), the laboratory user study has limitations in its relative small scale and the questionable ecological validity of the collected data. To address the ecological validity problem, we carefully design the experiment search system and user study protocol to simulate a practical Web search scenario.

The only independent variable in this work is the domain expertise of users. We plan to control it by hiring subjects among senior undergraduate students from the corresponding majors. However, whether this manipulation can e ectively control the domain expertise variable needs to be veri ed by the collected data. The reported domain expertise, measured by the pre-task questionnaire, can be used to test the e ectiveness of our manipulation. 4.2

Potential Implications for System Design

The investigations of the proposed research questions may lead to useful implications for improving the search engines. For example: for RQ1, if the domain experts indeed have a higher knowledge gain during the search, the results read by them are more likely to be of high quality, and the search engine can identify these high-quality results based on domain experts' click logs; and if the domain experts are more likely to feel unsatis ed during the search, then maybe we should consider providing more specialized and authoritative information in the SERPs to make them satis ed. For RQ2, if we can nd most e ective query reformulation strategies for knowledge learning task, we can teach users how to adopt these strategies or make search engines provide better guidance during the search session via query suggestions. And for RQ3, if the domain experts have a di erent query reformulation strategy, we can identify them by observing their query logs in exploratory search sessions, and then provide personalized results for them; furthermore, understanding the relationship between the developing domain expertise and the changing querying strategy will help us understand how information needs emerge and evolve during exploratory searches, which may provide new insights for constructing a better session-level user behavioral model.

ACKNOWLEDGMENTS

This work was supported by Tsinghua University Initiative Scienti c Research Program(2014Z21032), National Key Basic Research Program (2015CB358700), Natural Science Foundation (61532011, 61472206) of China and Tsinghua-Samsung Joint Laboratory for Intelligent Media Computing.

References

[1] Y. Egusa , H. Saito , M. Takaku , H. Terai , M. Miwa , and N. Kando . Using a concept map to evaluate exploratory search . In Proceedings of the third symposium on Information interaction in context , pages 175 { 184 . ACM, 2010 .

[2] C. Eickho , S. Dungs , and V. Tran . An eye-tracking study of query reformulation . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 13 { 22 . ACM, 2015 .

[3] C. Eickho , J. Teevan , R. White , and S. Dumais . Lessons from the journey: A query log analysis of within-session learning . In Proceedings of the 7th ACM international conference on Web search and data mining , pages 223 { 232 . ACM, 2014 .

[4] J. Huang and E. N. Efthimiadis . Analyzing and evaluating query reformulation strategies in web search logs . In Proceedings of the 18th ACM conference on Information and knowledge management , pages 77 { 86 . ACM, 2009 .

[5] J. Jiang and C. Ni . What a ects word changes in query reformulation during a task-based search session ? In Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval , pages 111 { 120 . ACM, 2016 .

[6] D. Kelly . Methods for evaluating interactive information retrieval systems with users . Foundations and Trends in Information Retrieval , 3 ( 1 |2):1{ 224 , 2009 .

[7] D. Kelly , J. Arguello , A. Edwards , and W.-c. Wu. Development and evaluation of search tasks for iir experiments using a cognitive complexity framework . In Proceedings of the 2015 International Conference on The Theory of Information Retrieval , pages 101 { 110 . ACM, 2015 .

[8] Y. Liu , Y. Chen , J. Tang , J. Sun , M. Zhang , S. Ma, and X. Zhu . Di erent users, di erent opinions: Predicting search satisfaction with mouse movement information . In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 493 { 502 . ACM, 2015 .

[9] J. D. Novak and D. B. Gowin . Learning how to learn . Cambridge University Press, 1984 .

[10] D. Odijk , R. W. White , A. Hassan Awadallah , and S. T. Dumais . Struggling and success in web search . In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management , pages 1551 { 1560 . ACM, 2015 .

[11] D. M. Russell . What do you need to know to use a search engine? why we still need to teach research skills . AI Magazine , 36 ( 4 ), 2015 .

[12] M. Sloan , H. Yang , and J. Wang . A term-based methodology for query reformulation understanding . Information Retrieval Journal , 18 ( 2 ): 145 { 165 , 2015 .

[13] P. Vakkari . Changes in search tactics and relevance judgments in preparing a research proposal: A summary of ndings of a longitudinal study . Information Retrieval , 4 : 295 { 310 , 2001 .

[14] R. W. White , S. T. Dumais , and J. Teevan . Characterizing the in uence of domain expertise on web search behavior . In Proceedings of the Second ACM International Conference on Web Search and Data Mining , pages 132 { 141 . ACM, 2009 .

[15] R. W. White and R. A. Roth . Exploratory search: Beyond the query-response paradigm . Synthesis Lectures on Information Concepts , Retrieval, and Services , 1 ( 1 ):1{ 98 , 2009 .