Getting to Know You: Search Logs and Expert Grading to Define Children’s Search Roles in the Classroom Monica Landoni1 , Theo Huibers2 , Mohammad Aliannejadi3 , Emiliana Murgia4 and Maria Soledad Pera5 1 Università della Svizzera Italiana, Switzerland 2 University of Twente & Wizenoze, The Netherlands 3 University of Amsterdam, Amsterdam, The Netherlands 4 Università degli Studi di Milano-Bicocca, Italy 5 People and Information Research team (PIReT)–Boise State University,USA Abstract In this paper, we examine the roles children play when using web search engines in the classroom context by revisiting, not replicating, a seminal work set in the home context. In particular, we describe how we juxtaposed performance indicators inferred from a combination of search logs (collected over two years) and expert grading of completed inquiry assignments to discern emerging search roles among children in primary four and five (aged 9 to 11). In light of the COVID-19 pandemic, we also explore differences when a traditional classroom is replaced by online instruction at home. Lastly, we discuss future research directions that we see as pivotal to advance research in Information Retrieval to and for children. Keywords Children, Search Engines, Search Roles, Information Retrieval 1. Introduction task without completing it [11]; visual searchers look for non-textual materials to quench their thirst for in- Mainstream search engines are designed to serve “large formation; rule-bound searchers follow strict steps to commercial masses”, i.e., adult searchers. Still, children compensate for a lack of confidence in searching; devel- are known to favour popular search engines [1, 2] even oping searchers make an effort to learn how to search if that causes them to face well-studied barriers, for in- but are not yet able to deal with complex searches; con- stance formulating succinct keyword queries or swiftly tent searchers go back to familiar websites; and power navigating search engine result pages (SERP) to identify searchers are confident and competent in using search relevant resources [3, 1, 4, 5]. tools across leisure and education-related searches. The obvious need for search engines to more effec- As stated in [10], these roles showcase the range of tively serve children, has motivated researchers to study skills and aptitudes that children exhibit when searching children’s search behaviour from a system perspective at home [12, 3]. Searching, however, is not limited to this in addition to design algorithmic and interface function- context as it is commonplace to embed search engines in ality tailored to children [6, 7, 8, 9]. Research from a the classroom–they are convenient and valuable assets user perspective, however, is more limited. Seminal work for children’s education [6, 13]. At home “children have in this area is the one by Druin et al. [10], who define a freer access to the computer and encounter a wider seven roles that children play when searching at home: and more incidental array of search topics, as opposed non-motivated searchers stop at the first result and use to the constrained topics [...] in the school” [3]. In the SERP snippets instead of exploring retrieved resources; classroom, however, information seeking involves locat- distracted searchers are easily attracted by other ac- ing online information in order to learn [14, 15]. This tivities around them and quickly abandon the search is not a trivial task, especially if we consider children’s DESIRES 2021 – 2nd International Conference on Design of widespread developing abilities to search, or lack thereof, Experimental Search & Information REtrieval Systems, September due to limited search literacy instruction [13]. Further, 15–18, 2021, Padua, Italy in the classroom context, there are factors that define " monica.landoni@usi.ch (M. Landoni); the search experience that differs from those at home. t.w.c.huibers@utwente.nl (T. Huibers); m.aliannejadi@uva.nl Distinguishing factors include (i) the fact that the search (M. Aliannejadi); emiliana.murgia@unimib.it (E. Murgia); solepera@boisestate.edu (M. S. Pera) task is set by teachers, (ii) there is an extrinsic motiva-  0000-0003-1414-6329 (M. Landoni); 0000-0002-9837-8639 tion provided by the grade assigned to outcomes of the (T. Huibers); 0000-0001-5134-5234 (M. Aliannejadi); search task, (iii) there are fixed deadlines for search task 0000-0002-2008-9204 (M. S. Pera) completion, and (iv) teachers and peers can offer varying © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). levels of direct and proactive support. CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) The departure from the home context leads us to ques- Table 1 tion whether the skills and aptitudes for search remain Details of the studies conducted for data collection purposes, the same in the classroom. We seek to better understand resulting in the search logs and teachers’ observations lever- search roles children play in the classroom context and aged in the exploration presented in Section 3. recognise any new roles that spontaneously manifest as Study 1 Study 2 Study 3 a result of scrutinising context-dependent data. Unlike Source [17] [16] [9] Druin et al. [10], who use interviews and observations Participants 75 66 31 Age group 9 - 11 9 to 11 10-11 while engaging children in simple, as well as multi-step Tornadoes, search tasks and hence take a qualitative approach to volcanoes, Topic Environment Ancient Rome discern roles, we take a quantitative approach. We rely pyramids, endangered animals on performance indicators inferred from search logs and Traditional vs. Traditional vs. teachers’ observations collected via user studies involv- Interface voice Traditional emoji-enriched ing 172 children who conducted web searches in the Context Traditional Traditional Online classroom classroom classroom classroom context. Data collection took place over a two- year period at two Italian-speaking European schools, enabling us to gather evidence of children’s natural in- teractions with search tools. Midway through this data- broad expertise using search tools to support learning; gathering period, the COVID-19 pandemic caused class- making them a representative population for primary rooms worldwide to move to the home context. This four and five classrooms. afforded us a unique opportunity to explore children’s search roles in traditional vs. online classrooms. 2.2. Protocol Our exploration describes a prospective approach for In each study, teachers presented their class with an in- identifying search roles based on quantitative indicators, quiry assignment related to a topic aligned with regular it also offers insights on other roles potentially played curriculum instruction. Children had to respond to ques- by children in the classroom context and ways in which tions (4 to 12, depending on the study) on subjects in- technology could better support information seeking. cluding science, geography, and history. These questions Lessons learned can inform the design of web search ranged from a description of political life in ancient Rome technology for the classroom that can offer the scaffold- to ways to prevent ecological disasters and how to recog- ing young searchers need to successfully complete search nise different types of volcanoes. Further, some questions tasks for learning regardless of their in-development were fact-based (e.g., “What is the island of plastic?”) and (search) skills. others open-ended (e.g., “How were the pyramids built?”) to capture user interactions when addressing inquiries 2. Data Collection of increased complexity. To locate information to answer said questions, chil- We base our exploration–aimed at understanding how dren used a search tool resembling a popular search en- children search for information in a classroom context– gine (powered by Bing’s API; language set to Italian). on data collected via several related user studies we con- The interface of the search tool varied across studies, ducted between September 2018-2020 [16, 17, 9]. Given allowing us to more closely look into the wide-ranging the longitudinal nature of the collected data (summarised roles that materialise while children search. In Study 1, in Table 1), to enable analysis across studies we followed in addition to a traditional text-based interface, children the framework proposed in [17], which establishes four used a vocal search assistant, an interaction medium now pillars to study and evaluate information retrieval sys- en-vogue. In Study 2, children interacted with a tradi- tems for children: user group, task, context, and search tional interface. In Study 3, children engaged with a strategy. In our case, children in primary four/five, look- traditional interface, as well as emoji-enriched interfaces, ing for information to answer questions related to the where emojis served as relevance clues for SERP results. school curriculum, in a classroom context, using the same search tool altered with diverse interfaces. 2.3. Data 2.1. Participants We examine search logs generated on each study, as they capture user interactions with search tools. If avail- Study participants included 172 children (ages 9 to 11) in able, we consider direct observations and teachers’ as- primary four and five classrooms in two Italian-speaking sessments (i.e., grade) of submitted inquiry assignments. schools in two European countries (Italy and Switzer- These provide additional insights on the performance land). Children had varied exposure to digital tools and of each student, as well as the mood and attitude of the children while engaging with the proposed search tasks. Table 2 Whenever possible, teachers monitored the nature and Study performance indicators. * indicates statistically signif- timing of children’s requests for guidance and support. icant difference with Study 3 (t-test, 𝑝 < 0.001). All essential elements for better understanding how chil- Indicators Study 1 Study 2 Study 3 dren relate to search tools in the classroom. # of Clicks/session 4.70 (3.73)* 5.91 (5.93)* 2.91 (2.60) # Queries/session 4.51 (6.68)* 6.88 (11.90)* 2.15 (1.47) Avg. query terms 5.77 (3.29)* 5.68 (3.02)* 6.98 (2.98) 2.4. Exploration Avg. click position 5.77 (3.29) – 5.68 (3.02) Session length (s) 1357 (1062)* – 106 (184) We scrutinise well-known performance indicators: ses- sion length, number of queries, number of query terms, number of clicks, rank position of clicked resources, number of positive/negative clicks (clicks on known 3.1. Can we capture search roles using relevant/non-relevant results as defined by teachers), quantitative indicators? click accuracy (proportion of positive clicks over total We discuss whether, how, and to what extent the seven clicks), and assignments’ grade–a score between 0 and search roles presented in [10] for the home context are 100 experts (teachers) assigned to the responses children mimicked in the classroom context. In Table 2, we sum- submitted after using search tools to locate relevant infor- marise performance indicators across the studies. In Ta- mation related to the corresponding inquiry assignment. ble 3, we report performance variations across searchers We look into indicators’ distributions and, when befit- in different groups (low, medium, and upper) using the ting, cluster searchers portraying similar behaviour into most distinguishing indicators as lenses for investigation. three groups we denote: upper, medium, and lower. For To illustrate search roles, we plot in Figure 2 the distribu- instance, consider the click accuracy depicted in Figure 1. tion of performance indicators inferred from the study In this case, searchers in the lower quartile belong to the with the least number of participants (i.e., Study 3). lower group; those in the upper quartile are in the upper The power searcher role is directly linked to school- group; remaining searchers are in the medium group. related searches in [10], making it an ideal candidate to start our investigation. This role is the most straight- forward to recognise and describe via implicit indica- tors. Children playing this role can search most au- tonomously, with minimal support from teachers and/or custom-designed search tools; children can perform all the necessary steps leading to a successful search: from translating their information need into a query to iden- tifying useful (relevant) results. We look at the distri- bution of the number and position of clicks, as well as Figure 1: Segmentation of click accuracy distribution to the number of clicks on known relevant SERP resources; group searchers. we consider any above-average combination of these in- dicators as a clue for this role. We assume that power searchers would extensively click on relevant resources and be focused on task completion, thus we situate power 3. Analysis and Discussion searchers in the upper group emerging when consider- ing click accuracy distribution (Figure 1). Samples power In this section, we discuss how we map, whenever possi- searchers are users 22 and 23 in Figure 2. In their case, ble, quantitative indicators with each of the roles origi- the number of positive clicks is very close to the num- nally defined in [10]. Note that search roles in [10] are ber of clicks (high click accuracy), yet they deviate from not mutually exclusive, i.e., they do not set a partition on the expectation that children favour resources higher in young searchers and the role they play while searching the ranking by clicking on resources on lower-ranked as a child can play more than one role in their many positions, displaying their savviness and confidence with interactions with search engines. Further, some roles are conducting online information-seeking tasks. more closely related to others. For example, the develop- We associate the developing searcher role with chil- ing searcher role is often combined with the rule-bound, dren in the medium group for click accuracy (users 13 and content, and distracted searcher roles. By changing the 15 in Figure 2). From analysis of direct observations and context (from home to classroom), and the data source grades, we note that click accuracy varies depending on (explicit in the original study to primarily implicit here), the complexity of the task evidencing that children lack we anticipate variations on the emerging roles. sophisticated search strategies, especially for recognising Figure 2: Performance indicators inferred from data collected during Study 3. Table 3 Variations on performance indicators across user groups in Study 3. Group by Group Session Length Click Accuracy # Positive Clicks # Negative Clicks Grade # Query Terms upper 161.87 0.50 3.77 1.47 87 7.12 Click Count medium 104.62 0.58 1.39 0.41 83.75 6.88 lower – – – – – – upper 366.50 0.48 2.30 1.03 82.61 7.13 Session Length medium 69.74 0.56 2.79 1.03 82.97 6.54 lower 17.08 0.57 1.93 0.42 78.61 7.95 upper 183.16 0.49 3.24 1.46 80.19 7.22 Click Position medium 120.11 0.58 1.66 0.52 83.45 7.13 lower 56.21 0.57 0.57 0.06 84.66 6.12 upper 118.45 0.61 1.90 0.80 95.69 6.25 Grade medium 139.40 0.51 2.24 0.99 86.44 6.65 lower 113.02 0.59 3.45 0.66 62.52 8.28 relevant results. to complete the assignment. Consider users 26 and 27 We interpret unnecessary long search sessions as a on Figure 2. The former with a short session, few clicks, sign of a distracted searcher role, particularly among and even fewer clicks on positive results exemplifies the children in the medium group for click accuracy who pose non-motivated searcher role. The latter, with a higher few clicks and obtain lower-than-average grades, which ratio of clicks on relevant results, serves as an example we attribute to them forgoing task completion (e.g., user of a content searcher. 7 in Figure 2). We equate a very limited number of clicks It became apparent that it would not be possible to iden- (practically none) coupled with no depth in SERP ex- tify visual searcher and rule-bound searcher roles in ploration with the non-motivated searcher role. User the classroom context from performance indicators. To behaviour portrayed by non-motivated searchers par- quantitatively define visual searchers we would need to tially overlaps with that of children assuming the con- analyse more closely the nature of the clicked results and tent searcher role. Here, children only look for the an- define a heuristic to discern visual from textual content. swer to the assigned search task but are not interested in A predefined list of resources known to provide support any further exploration, thus they do not take full advan- to children when running school-related searches (such tage of the natural opportunity for learning by searching. as the educational digital content provided by Wizenoze It is possible, however, to distinguish between content [18]) or turning to existing approaches to automatically and non-motivated searchers by considering teachers’ as- detect sites satisfying classroom requirements [19, 20], sessment of task completion: content searchers try hard would be useful to recognise rule-bound searchers. and most likely succeed in finding answers to their in- We noticed emerging trends that, while not aligning quiries (high grades in Table 3), whereas non-motivated with any of the original roles in [10], offered insights searchers tend to minimise clicks and instead rely on into what we believe to be roles specific to the class- SERP snippets to come up with the information needed room context. Among the most prominent ones, we find the stimulated searcher role, which accounts for the 20 importance of emotional factors motivating the search. Figure 3a captures that children in Study 2 clicked on 15 more results than those in remaining studies. All studies # Clicks followed the same protocol differing only on the topic 10 of the search task. For Study 1 and Study 3, we used impartial curriculum topics, e.g., tornadoes. The topic for Study 2 was the environment–specifically, we asked 5 children to find information about ecological disasters. From direct teacher’s observations, we learned that chil- 0 Study 1 Study 2 Study 3 dren responded with great enthusiasm to search tasks expressing emotions even fear and rage. Children tried (a) Click distribution. Significant difference be- their best to complete the assigned task as a means to tween Study 3 and the other distributions; help the planet get better by understanding the causes Study 1 and 2 are not significantly different for these calamities and prevent them from happening (ANOVA with LSD post-hoc pairwise compari- again. (A sample stimulated searcher is user 5 in Figure son 𝑝 ≪ 0.001). 2, with high assignment grade, above-average session 4000 length, and many clicks). From performance indicators– particularly the number of clicks generated–we notice Session Length (s) 3000 the importance to set motivational tasks as these posi- tively affect the behaviour of young searchers and their 2000 overall will to complete the task successfully. Another role we noticed merges the characteristics of 1000 the original power searcher and content searcher roles to yield the answer searcher. This role picks up on the search behaviour of a child interested in locating answers 0 to the posed questions but not necessarily in the explo- Traditional Classroom Online Classroom ration of related information, which would be expected (b) Session length in different classroom contexts. in the learning-by-searching experience. We recognise this behaviour by looking at the limited amount of clicks Figure 3: Explorations across studies and contexts. among top-ranked results when compared with that of power searchers who have the right combination of ac- curacy, depth, and width when clicking SERP results. We From Figure 3b, it is apparent that session length also see how answer searchers have an extrinsic moti- changes dramatically from the traditional to the online vation (the grade assigned by their teachers) as opposed classroom context. This could be caused by several fac- to an intrinsic one (a genuine interest and curiosity fortors influencing children’s transition to online classes. the topic of the search tasks). Indicators for user 25 inTogether with the cognitive overload from conducting Figure 2 signal an answer searcher. Given that teachers learning tasks completely online, children miss the aid can influence and motivate children when searching, we provided by teachers/peers in the form of natural ex- wonder whether the teaching mode has an impact on the changes taking place when they are all physically to- roles taken by searchers. gether sharing the same space. These considerations align with those in [3], in light of the importance of 3.2. Does teaching mode impact search the social dimension of searching among adolescents. A roles? classroom is a social place with specific rules and dy- namics. From teachers’ feedback, we envisage children Exploring whether search roles are impacted if classes longed for exchanges with their peers as well as for- take place remotely is not only a necessity during the mal/informal guidance provided impromptu by teachers, COVID-19 pandemic [21] but it is also a common medium if and when necessary. Indeed, the online classroom con- among families who chose to home-school children sup- text prevented teachers from spotting critical situations ported by online education [22] and online distance ed- when they could have otherwise intervened. For instance, ucation (among rural populations) [23]. We again turn teachers could not offer struggling children extra help in to quantitative indicators, but rather than considering deconstructing complex search tasks or offering clarifi- the classroom context as a whole, we explore search logs cations to make them better understand the search, i.e., generated on traditional classrooms (i.e., Study 1 and what information needed to be found among SERP results. 2) vs. those from online ones (i.e., Study 3). Table 4 Overview of children’s search roles in the home and classroom contexts. Classroom Role Home Main Indicators Observations Traditional Online Based on available indicators could not Visual ✓ — be determined for the classroom context Based on available indicators could not Rule-Bound ✓ — be determined for the classroom context Developing ✓ Click accuracy Content ✓ Total clicks Non-motivated ✓ Total clicks Distracted ✓ ✓ ✓ Session length, grade Power ✓ Click accuracy Searchers inspired to search due to emotion Stimulated ✓ ✓ Session length, total clicks associated with the topic of the search emerged as a new role inherent to the classroom context Expands the content searcher role to account for searchers who only strive to locate an Answer ✓ ✓ Total clicks answer to a posed inquiry, as opposed to learning as a result of searching Searchers that depend on peer/teacher assistance Guided ✓ Query length to enhance their overall search experience This lack of spontaneous scaffolding surely affected the exploration evidenced why identifying a strong connec- weakest searchers. It emerges from comparisons among tion between the presence of emotions in a task and the lower and upper user groups in Table 3 that although motivation it generates among young searches can have weaker searchers exhibit similar behaviour (in terms of implications on the design of innovative search tools clicks and session length) to that of most successful ones, for the classroom. Context comparison also brought to they are not able to recognise they have found enough light the social side of searching in the classroom and information to provide the right answer to the proposed the role teachers and peers play in the search process. search task (evidenced on lower grade scores). These takeaways highlight our main contribution: better We see an increase in the average number of query understanding of young searchers’ behaviour. terms in the online classroom context (Table 2); we at- We discovered gaps in how to define visual and rule- tribute this to the lack of peers and/or teacher guidance bound searchers; we also identified other factors that that could advise on effective query formulation. These should be considered for analysis purposes when explor- discoveries prompt us to define a new role, the assisted ing search roles. This lead to another contribution: bet- searcher representing children who depend on guid- ter understanding of how to design future studies to go ance to boost their overall search experience and success. deeper in our exploration of different search behaviours (Among users with similar total clicks, session length, and factors impacting them. and positive clicks, user 17–a student with adequate Overall, we surmise that the development of heuris- search skills as per teacher’s observation in a traditional tics for reusing and interpreting user data, as we have classroom–obtained the lowest grade, exemplifying in done in this exploration, can prove a valid alternative to Figure 2 an assisted searcher.) expensive, and often hard to conduct, user studies while saving time to researchers, and more importantly, users. 3.3. Getting to know young searchers We aimed to recognise search roles children play while 4. Conclusions, Limitations and searching in the classroom by relying on well-studied Further Research search performance indicators, teachers’ analysis of stu- dents’ outcomes after conducting inquiry tasks in a class- Search engines are widely used to support learning. As room context, observations, and search behaviour trends children learn in their own way, it is natural to think that emerging from scrutiny of indicators in-tandem. young searchers would have different search behaviours From preliminary analysis of a reasonably limited sam- based on their search skills, experience, and ability. In ple, we were able to discern most of the original search their 2010 seminal work, Druin et al. [10] examined qual- roles [10] and even emerging new roles that we put for- itative data and search observations to determine the ward, based on performance indicators, which we sum- roles that children play when seeking information at marise in Table 4. Outcomes from our context-related home. Grounded on their findings, we hypothesised that children’s search roles could be inferred as a result of different relevance cues help children perform better in a quantitative analysis. Consequently, in this paper, we search session. We assume that search cues are a form of instead focused on the classroom context and attempted scaffolding, and can be translated into an agent’s actions to ‘capture’ these roles using quantitative indicators esti- in a conversation (e.g., a relevance cue in a search ses- mated from data collected across longitudinal user studies sion can be considered as an agent’s action in which it involving children ages 9-11. mentions a certain document might be useful). Based on By relying on search logs generated by children in the the searcher profile and the required cue, an agent can classroom context, in addition to teachers’ observations, decide when and how to intervene. we could study children’s engagement with search tasks The discussions presented in this paper can set the in a natural environment. This resulted in an initial pic- foundation on how to recognise and forecast the search ture of children’s search roles in the classroom, as well roles children play while searching for learning specif- as inspiration for the design of future studies that enable ically in the classroom setting (traditional or online). the collection and analysis of a combination of implicit While preliminary, outcomes from our analyses evidence and explicit data using non-artificial tasks and settings. the need for a long-term commitment from the Informa- Encouraged by the outcomes of our analysis of search tion Retrieval community to advance theoretical and prac- roles among children aged 9-11, we plan to extend our tical knowledge regarding the design of (multi-modal) work to include children of broader age ranges, as exist- search tools for children–tools that offer voice, text, and ing research also suggests that different searchers need conversation as means to best address searchers’ needs a different kind of support [24, 25]. Children naturally while minimising classroom distractions [24, 31]. De- take on different roles as they grow, learn, and experience sign is naturally coupled with evaluation, which when varied search contexts. The degree to which context influ- it comes to children and information retrieval systems ences their ability to search is still an open question, and is not an easy feat [32]. We attribute this to (i) the lack one we are currently exploring with children in physical of dedicated datasets and events like TREC or CLEF that vs. remote classrooms. It would be of interest to examine could ease development and comparison across proposed how different types of tasks require and benefit from algorithmic solutions for which children are the main different roles played by the young searchers. This, in stakeholders, in addition to (ii) the need to explore as- turn, will advance knowledge regarding the relationship sessment metrics that go beyond the traditional NDCG, between roles and tasks. precision, or mean reciprocal rank [33], in order to si- More research is also needed to understand how in- multaneously account for the complex demands imposed novative search tools can substitute or at least alleviate by the goal of the search task (learning) [34] and needs the lack of “just-when-needed” personalised guidance and abilities of the target audience (children of broad age provided by teachers and supportive interventions by ranges) [35, 36]. peers. This aligns with our findings on if, when, and A good starting point in this transition from theory to how children take advantage of visual cues for relevant practice is the exploration reported in [27], where chil- resources in their quest for online resources that can help dren participated in co-design activities to define their them satisfy their classroom-related information needs natural sense for relevance. Outcomes revealed that rel- [26, 27], while evidencing other factors that contribute evant results for children were those that would act as to a deeper understanding of young searchers’ needs and an open window enabling them to look outside and go their many facets according to the different roles they and explore further, explain obscure concepts, and/or can play. Accordingly, scaffolding strategies should be highlight material suitable for children in the classroom. designed for each role. In particular, our study can inform Therefore, a relevant result should be stimulating, expli- the development of conversational search agents that aim cable, and understandable. Even if we often saw a consen- to aid children throughout their search experience. Rel- sus on what relevant means for children, we also noted evant work [28] proposes a conversational system that that when engaging with search engines, not all children can interact with users to clarify their information need. would recognise which results on a SERP were, in fact, However, as argued in the literature, the level of trust and relevant. This could also be due to teachers and children bias that a system can have on a user is very different be- having a different sense of relevance [26], a mismatch to tween adults and children [29, 30]. Therefore, our study account for when tuning into children’s perspectives as and the derived search roles shed light on which young determined by their search roles. users need more help while in a search session. Moreover, We posit that leveraging findings from our appraisal we hypothesise that a conversational agent’s actions can of search roles we can advance knowledge towards algo- go beyond clarification or requesting for feedback, as a rithmically determining who needs support for relevance system can guide children in a search session (just like detection–whether that be in the form of visual clues aug- their teacher) towards the right set of actions. This aligns menting traditional SERP or dedicated interfaces– and with our previous works [9, 25] where we studied how when that support is indeed needed. References sources, ACM Inroads 4 (2013) 16–17. [13] J. Pilgrim, Are we preparing students for the web [1] D. Bilal, L.-M. Huang, Readability and word com- in the wild? an analysis of features of websites for plexity of serps snippets and web pages on chil- children, The Journal of Literacy and Technology dren’s search queries, Aslib Journal of Information 20 (2019) 97–124. Management (2019). [14] S. Y. Rieh, K. Collins-Thompson, P. Hansen, H.-J. [2] J. Gwizdka, D. Bilal, Analysis of children’s queries Lee, Towards searching as a learning process: A and click behavior on ranked results and their review of current perspectives and future directions, thought processes in google search, in: Proceedings Journal of Information Science 42 (2016) 19–34. of the 2017 conference on conference human infor- [15] A. Usta, I. S. Altingovde, I. B. Vidinli, R. Ozcan, mation interaction and retrieval, 2017, pp. 377–380. Ö. Ulusoy, How k-12 students search for learning? [3] E. Foss, A. Druin, R. Brewer, P. Lo, L. Sanchez, analysis of an educational search engine log, in: E. Golub, H. Hutchinson, Children’s search roles Proceedings of the 37th international ACM SIGIR at home: Implications for designers, researchers, conference on Research & development in informa- educators, and parents, Journal of the American tion retrieval, 2014, pp. 1151–1154. Society for Information Science and Technology 63 [16] M. Landoni, M. S. Pera, E. Murgia, T. Huibers, In- (2012) 558–573. side out: Exploring the emotional side of search [4] N. Dragovic, I. Madrazo Azpiazu, M. S. Pera, " is engines in the classroom, in: Proceedings of the sven seven?" a search intent module for children, 28th ACM Conference on User Modeling, Adapta- in: Proceedings of the 39th International ACM SI- tion and Personalization, 2020, pp. 136–144. GIR conference on Research and Development in [17] M. Landoni, D. Matteri, E. Murgia, T. Huibers, M. S. Information Retrieval, 2016, pp. 885–888. Pera, Sonny, cerca! evaluating the impact of using a [5] A. Druin, E. Foss, L. Hatley, E. Golub, M. L. Guha, vocal assistant to search at school, in: International J. Fails, H. Hutchinson, How children search the Conference of the Cross-Language Evaluation Fo- internet with keyword interfaces, in: Proceedings rum for European Languages, Springer, 2019, pp. of the 8th International conference on interaction 101–113. design and children, 2009, pp. 89–96. [18] Wizenoze, It’s time to wizeup, 2021. URL: https:// [6] I. M. Azpiazu, N. Dragovic, M. S. Pera, J. A. Fails, On- www.wizenoze.com/en/solutions/wizeup, accessed line searching and learning: Yum and other search June 14, 2021. tools for children and teachers, Information Re- [19] R. Rajalakshmi, H. Tiwari, J. Patel, R. Rameshkan- trieval Journal 20 (2017) 524–545. nan, R. Karthik, Bidirectional gru-based attention [7] T. Gossen, Search engines for children: search model for kid-specific url classification, in: Deep user interfaces and information-seeking behaviour, Learning Techniques and Optimization Strategies Springer, 2016. in Big Data Analytics, IGI Global, 2020, pp. 78–90. [8] D. Bilal, Children’s use of the yahooligans! web [20] G. Allen, B. Downs, A. Shukla, C. Kennington, J. A. search engine: I. cognitive, physical, and affective Fails, K. L. Wright, M. S. Pera, Bigbert: Classifying behaviors on fact-based search tasks, Journal of the educational web resources for kindergarten-12th American Society for information Science 51 (2000) grades., in: European Conference on Information 646–665. Retrieval (ECIR), 2021, pp. 176–184. [9] M. Aliannejadi, M. Landoni, T. Huibers, E. Murgia, [21] J. Kim, Learning and teaching online during covid- M. S. Pera, Children’s perspective on how emojis 19: Experiences of student teachers in an early child- help them to recognise relevant results: Do actions hood education practicum, International Journal of speak louder than words?, in: Proceedings of the Early Childhood 52 (2020) 145–158. 2021 Conference on Human Information Interac- [22] A. Davis, Evolution of homeschooling, Distance tion and Retrieval, 2021, pp. 301–305. learning 8 (2011) 29. [10] A. Druin, E. Foss, H. Hutchinson, E. Golub, L. Hat- [23] C. de la Varre, M. J. Irvin, A. W. Jordan, W. H. Han- ley, Children’s roles using keyword search inter- num, T. W. Farmer, Reasons for student dropout in faces at home, in: Proceedings of the SIGCHI Con- an online course in a rural k–12 setting, Distance ference on Human Factors in Computing Systems, Education 35 (2014) 324–344. 2010, pp. 413–422. [24] M. Landoni, E. Murgia, T. Huibers, M. S. Pera, [11] M. Aliannejadi, M. Harvey, L. Costa, M. Pointon, You’ve got a friend in me: Children and search F. Crestani, Understanding mobile search task rel- agents, in: Adjunct Publication of the 28th ACM evance and user behaviour in context, in: CHIIR, Conference on User Modeling, Adaptation and Per- ACM, 2019, pp. 143–151. sonalization, 2020, pp. 89–94. [12] H. M. Walker, Homework assignments and internet [25] M. Aliannejadi, M. Landoni, T. Huibers, E. Murgia, M. S. Pera, Say it with emojis: Co-designing rele- reports, volume 7, Schloss Dagstuhl-Leibniz- vance cues for searching in the classroom, in: Pro- Zentrum fuer Informatik, 2017. ceedings of the Joint Conference of the Information [35] E. Murgia, M. Landoni, T. Huibers, J. A. Fails, M. S. Retrieval Communities in Europe (CIRCLE 2020), Pera, The seven layers of complexity of recom- 2020, pp. http://ceur--ws.org/Vol--2621/CIRCLE20_ mender systems for children in educational con- 30.pdf. texts, in: roceedings of the Workshop on Recom- [26] M. Landoni, M. Aliannejadi, T. Huibers, E. Mur- mendation in Complex Scenarios co-located with gia, M. S. Pera, Right way, right time: Towards 13th ACM Conference on Recommender Systems a better comprehension of young students’ needs (RecSys 2019), 2019, pp. CEUR Workshop Proceed- when looking for relevant search results, in: Pro- ings, 2449, 5–9. ceedings of the 29th Conference on User Modeling, [36] M. Rothschild, T. Horiuchi, M. Maxey, Evaluat- Adaptation and Personalization (UMAP), 2021, pp. ing “just right" in edtech recommendation systems, 256–261. in: 3rd International and Interdisciplinary Perspec- [27] M. Landoni, T. Huibers, E. Murgia, M. Aliannejadi, tives on Children & Recommender and Informa- M. S. Pera, Somewhere over the rainbow: Exploring tion Retrieval Systems (KidRec) What does good the sense for relevance in children, in: European look like?, 2019, p. https://kidrec.github.io/papers/ Conference on Cognitive Ergonomics 2021, 2021, KidRec_2019_paper_6.pdf. pp. 1–5. [28] M. Aliannejadi, H. Zamani, F. Crestani, W. B. Croft, Asking clarifying questions in open-domain information-seeking conversations, in: SIGIR, ACM, 2019, pp. 475–484. [29] M. S. Pera, E. Murgia, M. Landoni, T. Huibers, With a little help from my friends: Use of recom- mendations at school, in: Proceedings of ACM RecSys 2019 Late-breaking Results, 2019, pp. http: //ceur--ws.org/Vol--2431/. [30] T. Hassan, B. Edmison, T. Stelter, D. S. McCrickard, Learning to trust: Understanding editorial authority and trust in recommender systems for education, in: Proceedings of the 29th Conference on User Modeling, Adaptation and Personalization (UMAP), 2021, pp. 24–32. [31] D. Mak, D. Nathan-Roberts, Design considerations for educational mobile apps for young children, in: Proceedings of the human factors and ergonomics society annual meeting, volume 61, SAGE Publica- tions Sage CA: Los Angeles, CA, 2017, pp. 1156– 1160. [32] T. Huibers, M. Landoni, M. S. Pera, J. A. Fails, E. Mur- gia, N. Kucirkova, What does good look like? re- port on the 3rd international and interdisciplinary perspectives on children & recommender and in- formation retrieval systems (kidrec) at idc 2019, in: ACM SIGIR Forum, volume 53, ACM New York, NY, USA, 2019, pp. 76–81. [33] A. Milton, M. S. Pera, Evaluating information retrieval systems for kids, in: 4th International and Interdisciplinary Perspectives on Children& Recommender and Information Retrieval Systems (KidRec) What does good look like: From design, research, and practice to policy, 2020, p. arXiv preprint arXiv:2005.12992. [34] K. Collins-Thompson, P. Hansen, C. Hauff, Search as learning (dagstuhl seminar 17092), in: Dagstuhl