This study focused on the impact of prior domain knowledge on the resolution of search tasks. More precisely, the study looked at the efect of procedural and semantic support on search strategies and performances during information search activity comparing diferent levels of learning tasks. Eighteen students with prior domain knowledge, fourteen unguided students without prior domain knowledge and fifteen guided students without prior domain knowledge had to solve six learning tasks (two “remember” tasks, two “understand” tasks, two “evaluate” tasks) related to psychology. Main results showed that procedural and semantic support improved the navigation of users without prior domain knowledge (i.e. fewer links opened from SERP, less time spent on URL and globally less time to find information, longer queries) and they got close to users having higher prior domain knowledge.
1. Introduction users with a higher level of prior domain knowledge if we provided them with procedural (i.e. regarding Information systems are no longer seen simply as a the procedure for optimal task solving) and semantic tool for retrieving content to meet a specific informa- (i.e. regarding the specific vocabulary used in a dotion need but as a tool for acquiring new knowledge in main) support. More precisely, when they are solvthe course of searching, i.e. searching as learning [1]. ing learning tasks at diferent levels. In this paper, we According to [2], Searching as Learning aims to deter- present related work on information search, prior domine the relationships between the information search main knowledge and learning search tasks. We review activity (e.g., formulation of queries, search strategies, the methodology used to test our hypotheses and then etc.) and learning activities (e.g., reading, note-taking, describe our results. organizing information collected, etc.). The tasks in SAL can have diferent levels of learning goals, ranging from simple fact-finding task (i.e remember) to the 2. Related Work production of a new set of information (i.e create), [3, 4]. According to [5], search tasks in general can be In the cognitive model of information search, [6] demodulated by other factors, such as prior knowledge scribe the role of cognitive abilities (e.g., verbal and related to the search domain, knowledge of the tasks vocabulary abilities, selective attention, etc.) on the procedures and knowledge in information search. This three stages of this cyclical activity: (1) during planknowledge have been widely studied, but not in the ning and formulating the query stage, (2) the stage of context of Searching as Learning. Therefore, this study evaluating and selecting the information provided by aimed to understand how these types of knowledge the search engine, and (3) the stage of deep processing could support search activity when users dealt with of the information contained in the web pages. Among search tasks of diferent levels of learning. In partic- these abilities, verbal and vocabulary abilities are imular, we want to know whether users who have no portant on all stages. These ones are directly related to or little knowledge in a domain could get closer to the prior domain knowledge [7]. Users with domainspecific vocabulary knowledge generally construct a more consistent mental representation of the task than IPrreolaceneddings of the CIKM 2020 Workshops, October 19–20, Galway, users with lower prior domain knowledge, making it email: cheyenne.dosso@univ-tlse2.fr (C. Dosso); easier for them to assess the relevance of the SERP and aline.chevalier@univ-tlse2.fr (A. Chevalier); lynda.lechani@irit.fr to select more relevant sources [8]. Specifically, users (L. Tamine) with a high level of prior domain knowledge are able orcid: to focus their attention on relevant elements and inhibit others [9]. With regard to query (re)formulation © 2020 Copyright for this paper by its authors. Use permitted under Creative CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CCoEmUmoRns WLiceonrsekAsthtriobuptioPnr4o.0cIneteerdnaitniognasl ((CCC EBYU4R.0)-.WS.org) strategies, those of users having higher prior domain parallel task, users are supported in their search acknowledge are often longer (i.e., composed of more tivity. Based to [3], parallel and dependent tasks have words) [10] and these users are faster and perform bet- been called "Evaluate" in the present study because the ter [6]. Beyond prior domain knowledge related to main objective of these tasks is to compare a set of elvocabulary, other type of knowledge that can impact ements. Parallel task corresponds to guided evaluate information search activity is procedural knowledge task and dependent task to unguided evaluate task. In related to the task [11]. If a task is well known and this way,the current study aimed to understand how routine, the resolution procedure is easy to perform; semantic and procedural support can help users withusers have less dificulty understanding the structure out prior domain knowledge to solve learning tasks of the task [12]. According to [13], the interaction of compared to users with prior domain knowledge. Specifthese two types of knowledge (i.e., domain and pro- ically, the present study’s objective is to determine the cedure) supports search activity when users are solv- efect of knowledge of a resolution procedure (“undering search tasks. In this study, we focus on the res- stand” task) and a specific vocabulary (“evaluate” task) olution of learning tasks at diferent levels. The first on search activity, with respect to the level of prior dolevel is "Remember" [3]. It is a simple fact-finding task main knowledge (with vs without in psychology). where the learning objective is unique. The key words provided in the statement are clear, consistent, welldefined, and achievement of the learning objective does 3. Method not require high cognitive efort [14]. For this tasks, a plateau efect appeared between experts and novices 3.1. Hypotheses on search performance [7]. The second level is "Un- • Hypothesis 1: Guided users without domain knowledge derstand" [3]. This is a task that will require the un- should spend less time to solve tasks and to have better scores derstanding and clarification of certain terms in order (correct answers) than unguided users without domain knowlto access the answer. The statement here is poorly de- edge so that the first ones should be close to users with high ifned because the terms used are not clear but also be- level of domain knowledge. cause they are linked to specific high vocabulary. The • Hypothesis 2: Guided users without domain knowledge production of new search terms is requisite [14]. These should formulate more and longer queries than unguided tasks can be solved by following a specific procedure users without domain knowledge so that the first ones should (e.g. understanding the definition and terminology of be close to users with high level of domain knowledge. the proposed terms so that inferences can be made to • Hypothesis 3: Guided users without domain knowledge more relevant terms, understanding each search crite- should open fewer links from SERPs and spend less time ria and finding an answer that satisfies them). If users to explore webpages than unguided users without domain know this procedure, they could solve the task easier knowledge so that the first ones should be close to users with and obtain higher performance. For this task, users high level of domain knowledge. with a high level of prior domain knowledge formulated more queries than users without prior domain knowledge [7] and had less dificulty adding new search 3.2. Independent Variables terms [8]. The third level used is "Evaluate" [3]. It re- • IV1: Level of prior domain knowledge as between-subject quires a comparison of elements as proposed by [15]. factor (high/without) These tasks involve text production but vary in struc- • IV2: Level of support as between-subject factor (guided/ unture: "parallel task" and "dependent task". For the par- guided) – only for users without prior domain knowledge allel task, the elements to be compared are clearly de- • IV3: Level of learning task as a within-subject factor (Reifned and the specific vocabulary to be used to learn member/Understand/Evaluate) new information is provided in the task statement. Semantic support is therefore high. For the dependent 3.3. Dependent Variables task, the elements to be compared are not given in statement and have to be inferred by users during their • DV1: Total time (in sec.) of search session. For each search. The dependent task requires a higher level of task (included total time spent on SERP and total time spent prior knowledge than the parallel task since the se- on webpages). mantic support(i.e. specific vocabulary level related • DV2: Total score in percent. For remember and underto the elements to be compared) is lower in the state- stand tasks, when one answer was correct and possible, scores ment; users have to define and to infer them. These are “1” (correct) and “0” (wrong). For evaluate tasks, which diferences are important because in the case of the were open-ended tasks (several answers were acceptable but specific elements had to be found), 1 point was assigned for
each expected element contained in the answer with a score this task, you have to produce new keywords and it is necessary that varying from 0 to 9 by task.
research method integrates the set of given criteria. • DV3: Queries (number and length). For each search task, Evaluate task - unguided: You have an interest about social the total number of queries submitted to the search engine
psychology domain and you want to write an article about social per search session was computed. The mean length of queries perceptions, in particular on ones which contribute to discriminaper search corresponds to the total sum of keywords number
tion. To do that, you have to know what are the elements included used during a search session divided by the number of total in social perceptions, how they work, how they build themselves, queries submitted to the system during this search session. • DV4: Number of links opened up from SERP. For each search task, total number of selected and opened links by users from the search results pages.
task during the search session.
• DV5: Total spent time (in sec.) on webpages. For each
Eighteen users with high level of domain knowledge what the sub-processes are and how they influence the discrimination. Specifically, you have to carry out these following activities: 1) to retrieve information about social perception elements, which contribute to discrimination. 2) To select three elements on which you are going to concentrate in this article. You want to present their specific characteristics which encourage you to select them among others elements. 3) To compare their functioning at the level of sub-processes.
Evaluate task - guided: You have [...] discrimination. You want to focus on three elements about social perception, which allow ex= 24.6 aged from 22 to 30 years old ( iffteen guided users without domain knowledge aged = 2.20), plaining the functioning of discrimination. These three elements are: 1) social categorization, 2) Stereotypes, 3) Prejudices. You wish from 22 to 28 years old ( = 24 = 1.77) and four- to describe the set of these three elements, particularly: to present
The study took place at the University of Toulouse. Before starting search sessions, participants had to complete four online questionnaires: demographic information; habits with internet, self-eficacy scale in inedge (16 questions). Once the pre-questionnaires were completed, the main instructions were presented and participants started to perform the six search tasks in randomized order. Participants had to provide a written response. Users with domain knowledge and a part of users without domain knowledge saw the unguided tasks and the other part of users without do( = 3.39 = 0.74 teen unguided users without domain knowledge aged from 22 to 27 years old ( = 24.1 part in the experiment. All of them were French native speakers. The sample was composed of 12 males and 35 females, all in master degree (16 females with psychology knowledge, 8 guided and 11 unguided females without psychology knowledge). Concerning the selfassessment scale of psychology knowledge (4-p Likert scale), scores were significantly diferent ( 7.69, > . 001) between users with domain knowledge = 1.44) took = 0.5) and users without ( ). In addition, the scores obtained through (45) = significant ( (45) = 7.71, < . 001). Users with domain than users without ( knowledge had better scores ( = 3.34 = 8.94 = 2.39).
external mouse. To record data, we used an ad-hoc software, which recorded time, clicks, visited SERPs and documents. To test our hypotheses, we created related to psychology: of generative theory?
Understand task - unguided: As part of his researches, Lionel conducts observations in various circumstances that he extrapolates to make previsions. In your view, what is the research method used six search tasks in guided and unguided version, all formation search (10 items), MCQ of psychology knowltheir functioning, how they build themselves, the sub-processes and how they influence discrimination. You have to integrate in the article the completeness of three descriptions, which correspond to the set of analysis criteria.
Understand and evaluate tasks test the support variable. In the unguided version, participants saw only the task statement. In the guided condition, the procedural support (Understand) took the form of an additional instruction that informed the participant about the procedure to follow to succeed in the task. The semantic support (Evaluate) informed participants on the items to compare. For evaluate task non-guided, the items on which to perform evaluation and comparison work were not indicated in the statement. For = 2.46) the remember tasks, no support was provided because these were control tasks where the literature does not show any significant diference in their resolution. main knowledge performed the tasks in their guided 6031.29, < . 001. No significant diferences were obversion. tained between users with knowledge and guided users without, nor between the two groups of users without knowledge ( > . 05). The second part of H2 was not 4. Results completely verified.
No significant efect of support appeared for links For all the dependent variables, we carried out ANOVA (repeated measures) on two independent variables and ionpdeincaedteudpafrsoigmnSificEaRnPt sd(ife(r2e,n4c4e) b=e0tw.4e5e,n>u.se0r5s).wCoitnhtrasts contrasts to identify if the support helps non-experts.
We mixed the "level of prior domain knowledge" (IV1) knowledge ( = 8.85 = 1.49) and unguided users and the "support" (IV2) to obtain the independent vari- without ( = 10.76 = 1.7), with (2, 43) = 19232.32, able "Group" with three modalities (with domain knowl- < . 001. Users with knowledge opened fewer links from SERPs. A significant diference appeared between edge unguided, without domain knowledge guided and unguided).The independent variable "level of learning guided ( = 9.96 = 1.64) and unguided users withtask" stayed the same. oopuetnkendowfelwedegrelin( k(s2f,r4o3m)=SE1R5P43s..3N1,o s<ig.ni0fic0a1n)t. dGifeuri-ded
Regarding the total time on search session, statisti- ences were obtained between users with knowledge cal analyzes did not reveal any significant efect of supand guided users without ( > . 05). This part of hyport ( (2, 44) = 1.10, > . 05). Nevertheless, contrasts pothesis 3 was validated. indicated that users with domain knowledge ( = 340 Regarding the total time spent on web pages, ANOVA guid=ed 6u0s.e1r6s) wnietehdouletsdsotmimaien tkonosowlvleedgtaesk( s t=ha4n64u.5n8- 1d.i2d0n, o t >sho.w05a)n.y Csiognntificraasnttseifencdtiocfastuedpptohratt( user(s2, w44it)h= knnificoa=wntl6e8dd.ig4fe2er)eanwncidethwgua(is3d,eo4db2t)uais=neer9sd99wb5e1itt0hw6o,euet<n(.us0>e0r1s.. 0wN5oi)thsniogr- kwneobwplaegdegse t(han=un3g16uided =wi5t8h.o4u7)t s(pen=t 4le4s1s.7t2im e o=n between the two groups of users without knowledge 66.26) with (3, 42) = 11520492, < . 001. Guided users ( > . 05). This part of hypothesis 1 was only partially without ( = 347.64 = 64.02) spent less time than verified. suinggnuificidanedt dwifeitrheonucte (be(t3w,4e2e)n=us5e8r5s6w71i.t6h, k<no.w0l0e1d)g.eNo
Concerning the scores of correct answers, ANOVA and guided without appeared.The second part of hydid not show any significant efect of the support with pothesis 3 was confirmed. (2, 44) = 2.76, > . 05. Contrasts indicated that users with knowledge ( = 0.53 = 0.02) have better scores than guided users without ( = 0.49 = 5. Conclusion 0.03) with (2, 43) = 487.71, < . 001.No significant diferences were observed between users with knowl- Users with knowledge and guided users without opened edge and unguided users without, nor between the two up fewer links and spent less time on web pages than groups of users without knowledge ( > . 05). The sec- unguided users without knowledge. These results sugond part of hypothesis 1 was not confirmed. gest that support used was able to allow users without
Concerning the efect of support, the ANOVA did knowledge who benefited from it to focus more on not show any significant diference on the total num- the relevant information contained in the SERPs and ber of queries ( (2, 44) = 0.62, > . 05). Contrasts web pages. Users with knowledge scored better than showed that users with knowledge ( = 7.12 = guided users without. This result may in part raise 1.05) produce fewer queries than unguided users with- questions about the relevance of the support used. First, out ( = 7.83 = 1.20) with (2, 43) = 5657.23, for the understand task, procedural support had to help < . 001. No significant diferences were obtained users when they were formulating queries. However, between users with knowledge guided users without although this instruction was handled by the partici( > . 05), nor between the two groups of users with- pants, guided users may have experienced dificulties out knowledge ( > . 05). The first part of H2 was not completing this activity and understanding the inforvalidated. mation from the web content. As for the semantic
The ANOVA did not reveal a significant efect of the support for the evaluate tasks, it made the task more support on queries length ( (2, 44) = 1.78, > . 05). closed than the non-guided version. Guided users had Contrast indicated that users with knowledge ( = to compare specific items, while the other two groups 4.09 = 0.35) produced longer queries than unguided had more freedom in the items to be selected. To furusers without ( = 3.75 = 0.38) with (2, 43) = ther understand the correct answer scores, qualitative