In this paper, we examine the roles children play when using web search engines in the classroom context by revisiting, not replicating, a seminal work set in the home context. In particular, we describe how we juxtaposed performance indicators inferred from a combination of search logs (collected over two years) and expert grading of completed inquiry assignments to discern emerging search roles among children in primary four and five (aged 9 to 11). In light of the COVID-19 pandemic, we also explore diferences when a traditional classroom is replaced by online instruction at home. Lastly, we discuss future research directions that we see as pivotal to advance research in Information Retrieval to and for children.
task without completing it [11]; visual searchers look for non-textual materials to quench their thirst for inMainstream search engines are designed to serve “large formation; rule-bound searchers follow strict steps to commercial masses”, i.e., adult searchers. Still, children compensate for a lack of confidence in searching; develare known to favour popular search engines [1, 2] even oping searchers make an efort to learn how to search if that causes them to face well-studied barriers, for in- but are not yet able to deal with complex searches; constance formulating succinct keyword queries or swiftly tent searchers go back to familiar websites; and power navigating search engine result pages (SERP) to identify searchers are confident and competent in using search relevant resources [3, 1, 4, 5]. tools across leisure and education-related searches.
The obvious need for search engines to more efec- As stated in [10], these roles showcase the range of tively serve children, has motivated researchers to study skills and aptitudes that children exhibit when searching children’s search behaviour from a system perspective at home [12, 3]. Searching, however, is not limited to this in addition to design algorithmic and interface function- context as it is commonplace to embed search engines in ality tailored to children [6, 7, 8, 9]. Research from a the classroom–they are convenient and valuable assets user perspective, however, is more limited. Seminal work for children’s education [6, 13]. At home “children have in this area is the one by Druin et al. [10], who define a freer access to the computer and encounter a wider seven roles that children play when searching at home: and more incidental array of search topics, as opposed non-motivated searchers stop at the first result and use to the constrained topics [...] in the school” [3]. In the SERP snippets instead of exploring retrieved resources; classroom, however, information seeking involves locatdistracted searchers are easily attracted by other ac- ing online information in order to learn [14, 15]. This tivities around them and quickly abandon the search is not a trivial task, especially if we consider children’s widespread developing abilities to search, or lack thereof, due to limited search literacy instruction [13]. Further, in the classroom context, there are factors that define the search experience that difers from those at home.
Distinguishing factors include (i) the fact that the search task is set by teachers, (ii) there is an extrinsic motivation provided by the grade assigned to outcomes of the search task, (iii) there are fixed deadlines for search task completion, and (iv) teachers and peers can ofer varying levels of direct and proactive support.
The departure from the home context leads us to ques- Table 1 tion whether the skills and aptitudes for search remain Details of the studies conducted for data collection purposes, the same in the classroom. We seek to better understand resulting in the search logs and teachers’ observations leversearch roles children play in the classroom context and aged in the exploration presented in Section 3. recognise any new roles that spontaneously manifest as Study 1 Study 2 Study 3 a result of scrutinising context-dependent data. Unlike Source [17] [16] [9] Druin et al. [10], who use interviews and observations Participants 75 66 31 while engaging children in simple, as well as multi-step Age group Tor9n-ad11oes, 9 to 11 10-11 sdeisacrecrhntraoslkess,awnde thaeknecae qtaukaentaitaqtuivaelitaaptpivreoaacphp.rWoaechretloy Topic vpoylrcaamnoidess,, Environment Ancient Rome toenacpheerfrosr’mobasnecreviantdioicnastocorslliencfteerdrevdiafruosmerssetaurdchiesloignsvaonlvd- Interface endTarnagdevitroieoidcneaalnvims.als Traditional eTmraodjii-teionnraiclhvesd. icnlagss1r7o2omchcioldnrteenxt.wDhaotaccoonldleucctitoend twooekb psleaacrecohveesriantwthoe- Context Tcrlaadssitriooonmal Tcrlaadssitriooonmal clOasnslrionoem year period at two Italian-speaking European schools, enabling us to gather evidence of children’s natural interactions with search tools. Midway through this data- broad expertise using search tools to support learning; gathering period, the COVID-19 pandemic caused class- making them a representative population for primary rooms worldwide to move to the home context. This four and five classrooms. aforded us a unique opportunity to explore children’s search roles in traditional vs. online classrooms. 2.2. Protocol
Our exploration describes a prospective approach for identifying search roles based on quantitative indicators, In each study, teachers presented their class with an init also ofers insights on other roles potentially played quiry assignment related to a topic aligned with regular by children in the classroom context and ways in which curriculum instruction. Children had to respond to questechnology could better support information seeking. tions (4 to 12, depending on the study) on subjects inLessons learned can inform the design of web search cluding science, geography, and history. These questions technology for the classroom that can ofer the scafold- ranged from a description of political life in ancient Rome ing young searchers need to successfully complete search to ways to prevent ecological disasters and how to recogtasks for learning regardless of their in-development nise diferent types of volcanoes. Further, some questions (search) skills. were fact-based (e.g., “What is the island of plastic?”) and others open-ended (e.g., “How were the pyramids built?”) to capture user interactions when addressing inquiries 2. Data Collection of increased complexity.
To locate information to answer said questions, children used a search tool resembling a popular search engine (powered by Bing’s API; language set to Italian).
The interface of the search tool varied across studies, allowing us to more closely look into the wide-ranging roles that materialise while children search. In Study 1, in addition to a traditional text-based interface, children used a vocal search assistant, an interaction medium now en-vogue. In Study 2, children interacted with a traditional interface. In Study 3, children engaged with a traditional interface, as well as emoji-enriched interfaces, where emojis served as relevance clues for SERP results.
We base our exploration–aimed at understanding how children search for information in a classroom context– on data collected via several related user studies we conducted between September 2018-2020 [16, 17, 9]. Given the longitudinal nature of the collected data (summarised in Table 1), to enable analysis across studies we followed the framework proposed in [17], which establishes four pillars to study and evaluate information retrieval systems for children: user group, task, context, and search strategy. In our case, children in primary four/five, looking for information to answer questions related to the school curriculum, in a classroom context, using the same search tool altered with diverse interfaces. 2.3. Data 2.1. Participants We examine search logs generated on each study, as they capture user interactions with search tools. If availStudy participants included 172 children (ages 9 to 11) in able, we consider direct observations and teachers’ asprimary four and five classrooms in two Italian-speaking sessments (i.e., grade) of submitted inquiry assignments. schools in two European countries (Italy and Switzer- These provide additional insights on the performance land). Children had varied exposure to digital tools and of each student, as well as the mood and attitude of the # of Clicks/session # Queries/session Avg. query terms Avg. click position Session length (s)
We scrutinise well-known performance indicators: session length, number of queries, number of query terms, number of clicks, rank position of clicked resources, number of positive/negative clicks (clicks on known 3.1. Can we capture search roles using relevant/non-relevant results as defined by teachers), quantitative indicators? click accuracy (proportion of positive clicks over total We discuss whether, how, and to what extent the seven clicks), and assignments’ grade–a score between 0 and search roles presented in [10] for the home context are 100 experts (teachers) assigned to the responses children mimicked in the classroom context. In Table 2, we sumsubmitted after using search tools to locate relevant infor- marise performance indicators across the studies. In Tamation related to the corresponding inquiry assignment. ble 3, we report performance variations across searchers
We look into indicators’ distributions and, when befit- in diferent groups (low, medium, and upper) using the ting, cluster searchers portraying similar behaviour into most distinguishing indicators as lenses for investigation. three groups we denote: upper, medium, and lower. For To illustrate search roles, we plot in Figure 2 the distribuinstance, consider the click accuracy depicted in Figure 1. tion of performance indicators inferred from the study In this case, searchers in the lower quartile belong to the with the least number of participants (i.e., Study 3). lower group; those in the upper quartile are in the upper The power searcher role is directly linked to schoolgroup; remaining searchers are in the medium group. related searches in [10], making it an ideal candidate to start our investigation. This role is the most straightforward to recognise and describe via implicit indicators. Children playing this role can search most autonomously, with minimal support from teachers and/or custom-designed search tools; children can perform all the necessary steps leading to a successful search: from translating their information need into a query to identifying useful (relevant) results. We look at the distribution of the number and position of clicks, as well as Figure 1: Segmentation of click accuracy distribution to the number of clicks on known relevant SERP resources; group searchers. we consider any above-average combination of these indicators as a clue for this role. We assume that power searchers would extensively click on relevant resources and be focused on task completion, thus we situate power 3. Analysis and Discussion searchers in the upper group emerging when considering click accuracy distribution (Figure 1). Samples power In this section, we discuss how we map, whenever possi- searchers are users 22 and 23 in Figure 2. In their case, ble, quantitative indicators with each of the roles origi- the number of positive clicks is very close to the numnally defined in [ 10]. Note that search roles in [10] are ber of clicks (high click accuracy), yet they deviate from not mutually exclusive, i.e., they do not set a partition on the expectation that children favour resources higher in young searchers and the role they play while searching the ranking by clicking on resources on lower-ranked as a child can play more than one role in their many positions, displaying their savviness and confidence with interactions with search engines. Further, some roles are conducting online information-seeking tasks. more closely related to others. For example, the develop- We associate the developing searcher role with chiling searcher role is often combined with the rule-bound, dren in the medium group for click accuracy (users 13 and content, and distracted searcher roles. By changing the 15 in Figure 2). From analysis of direct observations and context (from home to classroom), and the data source grades, we note that click accuracy varies depending on (explicit in the original study to primarily implicit here), the complexity of the task evidencing that children lack we anticipate variations on the emerging roles. sophisticated search strategies, especially for recognising relevant results. to complete the assignment. Consider users 26 and 27
We interpret unnecessary long search sessions as a on Figure 2. The former with a short session, few clicks, sign of a distracted searcher role, particularly among and even fewer clicks on positive results exemplifies the children in the medium group for click accuracy who pose non-motivated searcher role. The latter, with a higher few clicks and obtain lower-than-average grades, which ratio of clicks on relevant results, serves as an example we attribute to them forgoing task completion (e.g., user of a content searcher. 7 in Figure 2). We equate a very limited number of clicks It became apparent that it would not be possible to iden(practically none) coupled with no depth in SERP ex- tify visual searcher and rule-bound searcher roles in ploration with the non-motivated searcher role. User the classroom context from performance indicators. To behaviour portrayed by non-motivated searchers par- quantitatively define visual searchers we would need to tially overlaps with that of children assuming the con- analyse more closely the nature of the clicked results and tent searcher role. Here, children only look for the an- define a heuristic to discern visual from textual content. swer to the assigned search task but are not interested in A predefined list of resources known to provide support any further exploration, thus they do not take full advan- to children when running school-related searches (such tage of the natural opportunity for learning by searching. as the educational digital content provided by Wizenoze It is possible, however, to distinguish between content [18]) or turning to existing approaches to automatically and non-motivated searchers by considering teachers’ as- detect sites satisfying classroom requirements [19, 20], sessment of task completion: content searchers try hard would be useful to recognise rule-bound searchers. and most likely succeed in finding answers to their in- We noticed emerging trends that, while not aligning quiries (high grades in Table 3), whereas non-motivated with any of the original roles in [10], ofered insights searchers tend to minimise clicks and instead rely on into what we believe to be roles specific to the classSERP snippets to come up with the information needed room context. Among the most prominent ones, we find (a) Click distribution. Significant diference between Study 3 and the other distributions; Study 1 and 2 are not significantly diferent (ANOVA with LSD post-hoc pairwise comparison ≪ 0.001). the stimulated searcher role, which accounts for the importance of emotional factors motivating the search.
Figure 3a captures that children in Study 2 clicked on more results than those in remaining studies. All studies followed the same protocol difering only on the topic of the search task. For Study 1 and Study 3, we used impartial curriculum topics, e.g., tornadoes. The topic for Study 2 was the environment–specifically, we asked children to find information about ecological disasters.
From direct teacher’s observations, we learned that children responded with great enthusiasm to search tasks expressing emotions even fear and rage. Children tried their best to complete the assigned task as a means to help the planet get better by understanding the causes for these calamities and prevent them from happening again. (A sample stimulated searcher is user 5 in Figure 2, with high assignment grade, above-average session length, and many clicks). From performance indicators– particularly the number of clicks generated–we notice the importance to set motivational tasks as these positively afect the behaviour of young searchers and their overall will to complete the task successfully.
Another role we noticed merges the characteristics of the original power searcher and content searcher roles to yield the answer searcher. This role picks up on the search behaviour of a child interested in locating answers to the posed questions but not necessarily in the exploration of related information, which would be expected in the learning-by-searching experience. We recognise this behaviour by looking at the limited amount of clicks among top-ranked results when compared with that of power searchers who have the right combination of accuracy, depth, and width when clicking SERP results. We also see how answer searchers have an extrinsic motivation (the grade assigned by their teachers) as opposed to an intrinsic one (a genuine interest and curiosity for the topic of the search tasks). Indicators for user 25 in Figure 2 signal an answer searcher. Given that teachers can influence and motivate children when searching, we wonder whether the teaching mode has an impact on the roles taken by searchers.
Traditional Classroom Online Classroom (b) Session length in diferent classroom contexts.
changes dramatically from the traditional to the online classroom context. This could be caused by several factors influencing children’s transition to online classes.
Together with the cognitive overload from conducting learning tasks completely online, children miss the aid provided by teachers/peers in the form of natural exchanges taking place when they are all physically together sharing the same space. These considerations align with those in [3], in light of the importance of 3.2. Does teaching mode impact search the social dimension of searching among adolescents. A roles? classroom is a social place with specific rules and dynamics. From teachers’ feedback, we envisage children Exploring whether search roles are impacted if classes longed for exchanges with their peers as well as fortake place remotely is not only a necessity during the mal/informal guidance provided impromptu by teachers, COVID-19 pandemic [21] but it is also a common medium if and when necessary. Indeed, the online classroom conamong families who chose to home-school children sup- text prevented teachers from spotting critical situations ported by online education [22] and online distance ed- when they could have otherwise intervened. For instance, ucation (among rural populations) [23]. We again turn teachers could not ofer struggling children extra help in to quantitative indicators, but rather than considering deconstructing complex search tasks or ofering clarifithe classroom context as a whole, we explore search logs cations to make them better understand the search, i.e., generated on traditional classrooms (i.e., Study 1 and what information needed to be found among SERP results. 2) vs. those from online ones (i.e., Study 3). This lack of spontaneous scafolding surely afected the exploration evidenced why identifying a strong connecweakest searchers. It emerges from comparisons among tion between the presence of emotions in a task and the lower and upper user groups in Table 3 that although motivation it generates among young searches can have weaker searchers exhibit similar behaviour (in terms of implications on the design of innovative search tools clicks and session length) to that of most successful ones, for the classroom. Context comparison also brought to they are not able to recognise they have found enough light the social side of searching in the classroom and information to provide the right answer to the proposed the role teachers and peers play in the search process. search task (evidenced on lower grade scores). These takeaways highlight our main contribution: better
We see an increase in the average number of query understanding of young searchers’ behaviour. terms in the online classroom context (Table 2); we at- We discovered gaps in how to define visual and ruletribute this to the lack of peers and/or teacher guidance bound searchers; we also identified other factors that that could advise on efective query formulation. These should be considered for analysis purposes when explordiscoveries prompt us to define a new role, the assisted ing search roles. This lead to another contribution: betsearcher representing children who depend on guid- ter understanding of how to design future studies to go ance to boost their overall search experience and success. deeper in our exploration of diferent search behaviours (Among users with similar total clicks, session length, and factors impacting them. and positive clicks, user 17–a student with adequate Overall, we surmise that the development of heurissearch skills as per teacher’s observation in a traditional tics for reusing and interpreting user data, as we have classroom–obtained the lowest grade, exemplifying in done in this exploration, can prove a valid alternative to Figure 2 an assisted searcher.) expensive, and often hard to conduct, user studies while saving time to researchers, and more importantly, users.
searching in the classroom by relying on well-studied search performance indicators, teachers’ analysis of students’ outcomes after conducting inquiry tasks in a class- Search engines are widely used to support learning. As room context, observations, and search behaviour trends children learn in their own way, it is natural to think that emerging from scrutiny of indicators in-tandem. young searchers would have diferent search behaviours
From preliminary analysis of a reasonably limited sam- based on their search skills, experience, and ability. In ple, we were able to discern most of the original search their 2010 seminal work, Druin et al. [10] examined qualroles [10] and even emerging new roles that we put for- itative data and search observations to determine the ward, based on performance indicators, which we sum- roles that children play when seeking information at marise in Table 4. Outcomes from our context-related home. Grounded on their findings, we hypothesised that
children’s search roles could be inferred as a result of diferent relevance cues help children perform better in a quantitative analysis. Consequently, in this paper, we search session. We assume that search cues are a form of instead focused on the classroom context and attempted scafolding, and can be translated into an agent’s actions to ‘capture’ these roles using quantitative indicators esti- in a conversation (e.g., a relevance cue in a search sesmated from data collected across longitudinal user studies sion can be considered as an agent’s action in which it involving children ages 9-11. mentions a certain document might be useful). Based on
By relying on search logs generated by children in the the searcher profile and the required cue, an agent can classroom context, in addition to teachers’ observations, decide when and how to intervene. we could study children’s engagement with search tasks The discussions presented in this paper can set the in a natural environment. This resulted in an initial pic- foundation on how to recognise and forecast the search ture of children’s search roles in the classroom, as well roles children play while searching for learning specifas inspiration for the design of future studies that enable ically in the classroom setting (traditional or online). the collection and analysis of a combination of implicit While preliminary, outcomes from our analyses evidence and explicit data using non-artificial tasks and settings. the need for a long-term commitment from the InformaEncouraged by the outcomes of our analysis of search tion Retrieval community to advance theoretical and pracroles among children aged 9-11, we plan to extend our tical knowledge regarding the design of (multi-modal) work to include children of broader age ranges, as exist- search tools for children–tools that ofer voice, text, and ing research also suggests that diferent searchers need conversation as means to best address searchers’ needs a diferent kind of support [ 24, 25]. Children naturally while minimising classroom distractions [24, 31]. Detake on diferent roles as they grow, learn, and experience sign is naturally coupled with evaluation, which when varied search contexts. The degree to which context influ- it comes to children and information retrieval systems ences their ability to search is still an open question, and is not an easy feat [32]. We attribute this to (i) the lack one we are currently exploring with children in physical of dedicated datasets and events like TREC or CLEF that vs. remote classrooms. It would be of interest to examine could ease development and comparison across proposed how diferent types of tasks require and benefit from algorithmic solutions for which children are the main diferent roles played by the young searchers. This, in stakeholders, in addition to (ii) the need to explore asturn, will advance knowledge regarding the relationship sessment metrics that go beyond the traditional NDCG, between roles and tasks. precision, or mean reciprocal rank [33], in order to si
More research is also needed to understand how in- multaneously account for the complex demands imposed novative search tools can substitute or at least alleviate by the goal of the search task (learning) [34] and needs the lack of “just-when-needed” personalised guidance and abilities of the target audience (children of broad age provided by teachers and supportive interventions by ranges) [35, 36]. peers. This aligns with our findings on if, when, and A good starting point in this transition from theory to how children take advantage of visual cues for relevant practice is the exploration reported in [27], where chilresources in their quest for online resources that can help dren participated in co-design activities to define their them satisfy their classroom-related information needs natural sense for relevance. Outcomes revealed that rel[26, 27], while evidencing other factors that contribute evant results for children were those that would act as to a deeper understanding of young searchers’ needs and an open window enabling them to look outside and go their many facets according to the diferent roles they and explore further, explain obscure concepts, and/or can play. Accordingly, scafolding strategies should be highlight material suitable for children in the classroom. designed for each role. In particular, our study can inform Therefore, a relevant result should be stimulating, explithe development of conversational search agents that aim cable, and understandable. Even if we often saw a consento aid children throughout their search experience. Rel- sus on what relevant means for children, we also noted evant work [28] proposes a conversational system that that when engaging with search engines, not all children can interact with users to clarify their information need. would recognise which results on a SERP were, in fact, However, as argued in the literature, the level of trust and relevant. This could also be due to teachers and children bias that a system can have on a user is very diferent be- having a diferent sense of relevance [ 26], a mismatch to tween adults and children [29, 30]. Therefore, our study account for when tuning into children’s perspectives as and the derived search roles shed light on which young determined by their search roles. users need more help while in a search session. Moreover, We posit that leveraging findings from our appraisal we hypothesise that a conversational agent’s actions can of search roles we can advance knowledge towards algogo beyond clarification or requesting for feedback, as a rithmically determining who needs support for relevance system can guide children in a search session (just like detection–whether that be in the form of visual clues augtheir teacher) towards the right set of actions. This aligns menting traditional SERP or dedicated interfaces– and with our previous works [9, 25] where we studied how when that support is indeed needed. M. S. Pera, Say it with emojis: Co-designing rele- reports, volume 7, Schloss Dagstuhl-Leibnizvance cues for searching in the classroom, in: Pro- Zentrum fuer Informatik, 2017. ceedings of the Joint Conference of the Information [35] E. Murgia, M. Landoni, T. Huibers, J. A. Fails, M. S. Retrieval Communities in Europe (CIRCLE 2020), Pera, The seven layers of complexity of recom2020, pp. http://ceur--ws.org/Vol--2621/CIRCLE20_ mender systems for children in educational con30.pdf . texts, in: roceedings of the Workshop on Recom[26] M. Landoni, M. Aliannejadi, T. Huibers, E. Mur- mendation in Complex Scenarios co-located with gia, M. S. Pera, Right way, right time: Towards 13th ACM Conference on Recommender Systems a better comprehension of young students’ needs (RecSys 2019), 2019, pp. CEUR Workshop Proceedwhen looking for relevant search results, in: Pro- ings, 2449, 5–9. ceedings of the 29th Conference on User Modeling, [36] M. Rothschild, T. Horiuchi, M. Maxey, EvaluatAdaptation and Personalization (UMAP), 2021, pp. ing “just right" in edtech recommendation systems, 256–261. in: 3rd International and Interdisciplinary Perspec[27] M. Landoni, T. Huibers, E. Murgia, M. Aliannejadi, tives on Children & Recommender and InformaM. S. Pera, Somewhere over the rainbow: Exploring tion Retrieval Systems (KidRec) What does good the sense for relevance in children, in: European look like?, 2019, p. https://kidrec.github.io/papers/ Conference on Cognitive Ergonomics 2021, 2021, KidRec_2019_paper_6.pdf .
pp. 1–5. [28] M. Aliannejadi, H. Zamani, F. Crestani, W. B.
Croft, Asking clarifying questions in open-domain information-seeking conversations, in: SIGIR,
ACM, 2019, pp. 475–484. [29] M. S. Pera, E. Murgia, M. Landoni, T. Huibers,
With a little help from my friends: Use of recommendations at school, in: Proceedings of ACM RecSys 2019 Late-breaking Results, 2019, pp. http: //ceur--ws.org/Vol--2431/. [30] T. Hassan, B. Edmison, T. Stelter, D. S. McCrickard,
Learning to trust: Understanding editorial authority and trust in recommender systems for education, in: Proceedings of the 29th Conference on User Modeling, Adaptation and Personalization (UMAP), 2021, pp. 24–32. [31] D. Mak, D. Nathan-Roberts, Design considerations for educational mobile apps for young children, in: Proceedings of the human factors and ergonomics society annual meeting, volume 61, SAGE Publications Sage CA: Los Angeles, CA, 2017, pp. 1156– 1160. [32] T. Huibers, M. Landoni, M. S. Pera, J. A. Fails, E. Murgia, N. Kucirkova, What does good look like? report on the 3rd international and interdisciplinary perspectives on children & recommender and information retrieval systems (kidrec) at idc 2019, in: ACM SIGIR Forum, volume 53, ACM New York, NY,
USA, 2019, pp. 76–81. [33] A. Milton, M. S. Pera, Evaluating information retrieval systems for kids, in: 4th International and Interdisciplinary Perspectives on Children& Recommender and Information Retrieval Systems (KidRec) What does good look like: From design, research, and practice to policy, 2020, p. arXiv preprint arXiv:2005.12992. [34] K. Collins-Thompson, P. Hansen, C. Hauf, Search as learning (dagstuhl seminar 17092), in: Dagstuhl