With a Little Help from my Friends: Use of Recommendations at School Maria Soledad Pera∗ Emiliana Murgia PIReT – Dept. of CS – Boise State University Università degli Studi di Milano-Bicocca Boise, Idaho Milano, Italy solepera@boisestate.edu emiliana.murgia@unimib.it Monica Landoni Theo Huibers Università della Svizzera Italiana University of Twente Lugano, Switzerland Enschede, The Netherlands monica.landoni@usi.ch t.w.c.huibers@utwente.nl ∗ Corresponding author. ABSTRACT In this exploratory paper, we study the usage of recommendations by and for children (ages 9 to 11) in an educational setting. From our preliminary analysis, it becomes apparent that recommender systems (RS) could provide extra support to and help children successfully complete inquiry tasks. Nonetheless, children have difficulty in recognizing the role of RS, in terms of aiding information discovery for classroom assignments. Findings from our study set a foundation that can inform future design and development of RS for children that support classroom-related work. CCS CONCEPTS • Social and professional topics → Children; • Information systems → Recommender systems. KEYWORDS children, recommender systems, peer suggestions, classroom, trust, authority INTRODUCTION From serious games [2] to robots [1], technology is rapidly making its entry into education. Technology changes the classroom, as it offers new opportunities to teachers and redefines how students learn. One ACM RecSys 2019 Late-breaking Results, 16th-20th September 2019, Copenhagen, Denmark Copyright ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). With a Little Help from my Friends: Use of Recommendations at School ACM RecSys 2019 Late-breaking Results, 16th-20th September 2019, Copenhagen, Denmark of the most important tasks for children is learning to develop their knowledge, for which availability of resources at their own reading level is a must. Traditionally, the teacher was the one pointing students to resources they could gather from the library. Nowadays, children spend considerable time online and rely on materials from various websites and apps to help with their homework. In their quest for educational materials, school children directly (and indirectly) rely on search and recommender systems to identify resources that might be useful for them. Recommender systems (RS) are designed to ease decision-making and reduce information overload. These characteristics make them particularly useful for children, who are known to struggle with Figure 1: The paradigm. judging relevance and quality of content presented to them in online environments [4]. RS are also useful for teachers, offering assistance as teachers can no longer oversee each resource to determine its potential relevance for their students. Unfortunately, the majority of the literature pertaining to RS focuses on a general audience and thus directly responds to their needs and expectations [6]. While a few RS have been specially designed to better serve young users [5, 7], children need to be willing to accept and rely on the recommendations for them to be of any value. We are interested in studying multiple aspects of the paradigm using search and recommendation technology in the classroom (see Figure 1). In this article, we explore (i) the added value of a RS to assist children in their autonomous quest for information and (ii) the degree to which children take advantage of RS. To control scope, and in order to fit in our framework, we focus on recommendations generated for the purpose of facilitating completion of inquiry assignments in the classroom setting. We Figure 2: Participants’ experience with in- depend upon a simple popularity-like strategy to generate recommendations. In this case, suggested formation discovery. resources are chosen based on their frequency of occurrence in the dataset that resulted from the study presented in [3], which examined how children ages 9 to 11 completed search tasks pertaining to school subjects using known search engines. Children have difficulties with selecting relevant and reliable sources, while ignoring irrelevant ones. They could benefit from some guidance, and it is typically the teacher who undertakes this role. Unfortunately, in a digital environment this is not very practical. RS could play a pivot role. Thus, the question we tackle is “to what extend do children use recommendations and how to improve this usage”. Lessons learned (i) offer insights on children’ perceptions of RS when used to solve classroom-related tasks and (ii) provide foundations for RS that take into account content and context characteristics to enhance the recommendation process. More importantly, findings prompt us to acknowledge factors that can directly impact RS and their effectiveness: from expert involvement (to provide instruction Figure 3: Participants’ frequency of inter- on what RS are and their potential benefits within the classroom setting) and interface design (to action with information discovery envi- make recommendations visible and accessible), to the level of detail needed on the source of the ronments. recommendations to inspire trust in and define authority of the generated suggestions. With a Little Help from my Friends: Use of Recommendations at School ACM RecSys 2019 Late-breaking Results, 16th-20th September 2019, Copenhagen, Denmark CONTEXT & STUDY SETUP To better understand if children would choose to bypass browsing resources in favor of directly turning to resources curated and recommended for them, we recruited 33 child participants in the 4th and 5th grades (ages 9 to 11; 19 males and 14 females; demographic data that contextualizes participants’ experience with online environments is shown in Figures 2 and 3). Following the study framework Table 1: Sample questions presented to presented in [3], we asked children to complete an assignment that required them to respond to four participants. questions about common subjects in the 4th and 5th grades: volcanoes, tornadoes, pyramids, and endangered animals. Two questions were fact-based, one open-ended, and one multi-step, to explore Type Example user interaction when addressing inquiries of increased complexity (see sample questions on Table 1). Fact What is the highest Egyptian To locate information to answer the aforementioned questions, children used a simple interface pyramid? that resembles a popular search engine: Google. Instead of the common “I’m feeling lucky” option, Open How were the Egyptian pyra- they were presented with an option that, if selected, would lead them to a suggested resource mids built? that they could use to complete the question sheet. For recommendation purposes, we simulated Multi- How tall is the tomb of Myceri- a popularity-based strategy. For each of the 16 prompts (4 topics, 4 questions), we treated the step nus’ father? resource that was most often selected as relevant by participants in the search study presented in [3] as the “popular” peer-suggestion.1 By presenting recommendations based on the selections made by children of the same age when accomplishing the same tasks, we aim at providing information both relevant and at the right level of complexity for our target age group. There exists a number of RS for educational resources [8] we could have considered for recommendation purposes. Yet, they tend to serve traditional populations and rely on historical data–rarely available for children due to ethical constraints. This motivated our choice for a popularity-based RS as a starting point, with the caveat that popularity explicitly relates to our target audience. To understand how children perceived recommendations, we conducted two different experi- ments using two disjoint groups of children. The 15 children in S1 could either search for resources, Figure 4: Environment study participants or instead access recommendations when selecting the “Suggestions by other children” button. For interacted with to complete the proposed the remaining 18 children in S2, the button instead read “Suggestions for you” (see Figure 4 for an classroom tasks. illustration of the interface children used for study purposes). 1 Tasks in our study are a subset of those out- lined in [3]. RESULTS & ANALYSIS As reported in Table 2, based on the query logs generated as a result of the aforementioned experi- ments, children in S1 used recommendations to address 15% of their assigned questions, a percentage that decreased to 10% in S2. At the same time, the number of search sessions is comparable for S1 and S2, but children in S2 used more queries per search session to locate the needed information (average 5.8 queries per search sessions for S1 vs. 8.9 for S2). From these results, we believe recommendations With a Little Help from my Friends: Use of Recommendations at School ACM RecSys 2019 Late-breaking Results, 16th-20th September 2019, Copenhagen, Denmark were a contributing factor towards minimizing the time and effort required from children to locate relevant resources that contained the data necessary to complete the tasks presented to them. Table 2: Statistics inferred from generated We noticed that 32% of the search sessions in S1 included the use of the recommendation alternative query log. at least once; a proportion that decreased to 16% for S2. We attribute this to distrust, as the source of the recommendations was not explicitly mentioned to the children in S2, unlike participants in S1 Description S1 S2 who were made aware that suggestions were based on children’s use of resources. No. of search sessions 25 18 Another insight that emerged from query log analysis refers to frequency of query repetition: Total number of queries 145 161 children repeated their queries more often when using the search option than when using the Avg. queries per session 5.8 8.9 recommendation option (85 total queries, as opposed to 12 queries). This, compounded with the fact No. of queries for which rec- 22 17 that more often a query was repeated when search was used first followed by a recommendation than ommendations were selected vice versa (10 vs. 6), lead us to believe that when the recommendation option was used, participants No. of sessions that used the 8 3 successfully completed their inquiry, i.e., no further action was needed. recommendation option at In addition to query logs, we examined the results of a short survey presented to the child partici- least once pants. Survey questions were meant to capture children’s perception on RS and their willingness to Transition from search to rec- 5 5 use them as a proxy of search engines to complete class assignments. As shown in Table 3, during S1 ommendation on same query the use of the recommendation option was uniformly distributed. That was not the case for S2, when Transition from recommenda- 3 3 tion to search on same query the majority of the children favored searching over depending on recommendations. Furthermore, Transition from search to 34 51 85% of the children in S1 stated that they would use the recommendation option in the future, a search on same query percentage that decreased to 50% among S2 participants. Yet, more children in S2 than S1 would be Transition from recommenda- 1 11 open to trying the recommendation option, if presented with that choice in the future. This could tion to recommendation on suggest that children, in retrospect, see the value of recommendations and are willing to use them, same query just not in this iteration as they lacked information to identify the source of the recommendations. The “Suggested for you” label was too vague for them to trust: not knowing how recommendations Table 3: Survey responses to the question were generated could have influenced the likelihood of relying on them to complete an assignment. “Did you use recommendations to com- plete your assignment?” . The main reasons why children used the recommendation option were because they felt “It showed what I needed” and because it was “Useful and complete: it showed me the right sources of information”, still 2 of the children stated not trusting the recommendations (see Figure 5). Session Yes No [Would use op- [Would try op- CONCLUSIONS & NEXT STEPS tion again] tion] Given the struggles children face with successfully completing information discovery tasks in the S1 7 [6] 8 [3] classroom environment, we argued that RS could help ease this process for them. With that in mind, S2 6 [3] 12 [7] we conducted an initial study to understand children’s perceptions and usages on RS. From the direct observation of the students engaged with the tasks, it emerged that they enjoyed recommendations and expected them to facilitate the completion of their tasks at school. Nonetheless, they also believed that it would be necessary to receive in advance instructions and/or insights on how to use RS. Based on the inferences made from query log analysis, responses to surveys, and completion of the With a Little Help from my Friends: Use of Recommendations at School ACM RecSys 2019 Late-breaking Results, 16th-20th September 2019, Copenhagen, Denmark presented assignments (i.e., responses to prompted questions), we can conclude that when used, recommendations have a positive impact on the achievement of inquiry-related tasks in the classroom setting. It became apparent, however, that children did not really trust and were less willing to turn to suggested resources, if they did not know the source of the recommendation and its authority (i.e., triggered by “Suggestions by other children” vs. the more vague “Suggestions for you”). The study findings prompt us to explore other suggestion-generation strategies that are not only popular among peers, but are also suitable to the target audience and the classroom context. We are also interested in interface design, so that the recommendation option is visible, intuitive to use, and transparent (i.e., declares explicitly the source of the recommendations). For this, participatory design sessions involving children and teachers will be the next step. Lastly, it became apparent during our preliminary studies that new algorithmic and interface developments will not be enough, which is why we will team up with teachers to develop the type of materials that can support classroom instruction, so that children can be more cognizant of the recommendation process and thus take full Figure 5: Survey responses for “If you used advantage of presented suggestions, even beyond completion of schools assignments. the recommendations, please select state- ments that are true for you”. ACKNOWLEDGMENTS We appreciate the participation in our study of the students in the Leonardo da Vinci school in Lugano and Istituto Comprensivo Antonio Stoppani in Milan. We want to especially thank Meis Huibers for her art. Work partially supported by National Science Foundation, award 1565937. REFERENCES [1] Fabiane Barreto Vavassori Benitti. 2012. Exploring the educational potential of robotics in schools: A systematic review. Computers & Education 58, 3 (2012), 978–988. [2] Frederik De Grove, Jeroen Bourgonjon, and Jan Van Looy. 2012. Digital games in the classroom? A contextual approach to teachers’ adoption intention of digital games in formal education. Computers in Human behavior 28, 6 (2012), 2023–2033. [3] Monica Landoni, Davide Matteri, Emiliana Murgia, Theo Huibers, and Maria Soledad Pera. 2019. Sonny, Cerca! Evaluating the Impact of Using a Vocal Assistant to Search at School. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, To appear. [4] Renee Morrison, Georgina Barton, et al. 2018. Search engine use as a literacy in the middle years: The need for explicit instruction and active learners. Literacy Learning: The Middle Years 26, 3 (2018), 37. [5] Maria Soledad Pera and Yiu-Kai Ng. 2014. Automating readers’ advisory to make book recommendations for k-12 readers. In Proceedings of the 8th ACM Conference on Recommender systems. ACM, 9–16. [6] Francesco Ricci, Lior Rokach, and Bracha Shapira. 2011. Recommender systems handbook. Springer. [7] Elisa Rubegni and Monica Landoni. 2017. Aesthetic Relevance When Selecting Multimedia Stories. In 1s t KidRec Workshop co-located with ACM RecSys 2017. Available at: https://drive.google.com/file/d/0B8S_mL8gIjZwdWNLN1R3a3pxUkU/view. [8] Yong Zheng, Nastaran Ghane, and Milad Sabouri. 2019. Personalized Educational Learning with Multi-Stakeholder Optimizations. In Adjunct Proceedings of ACM UMAP.