Towards using pedagogical agents to orchestrate collaborative learning activities combining music and mathematics in K-12 Eric Roldán Roaa, Irene-Angelica Chountab, and Margus Pedastea a University of Tartu, Ülikooli 18, Tartu, 51005, Estonia b University of Duisburg-Essen, Forsthausweg 2, Duisburg, 47057, Germany Abstract Research on pedagogical agents (PAs) focuses on personalizing and adapting content and instruction to students’ diverse needs to support learning. Teachers can use this technology to support individual students’ work. However, it is not clear what could be the impact of a PA that helps teachers to orchestrate collaborative learning activities on the classroom level. Our work explores two dimensions. Firstly, the effects of employing a PA in a technology-enhanced learning setting to promote students’ motivation and learning outcomes. To that end, we will conduct a series of studies employing various methods and data (tests, questionnaires, observations, and students’ performance data). Secondly, this work aims to develop a design framework based on teachers’ expectations and needs when using a PA to orchestrate collaborative learning tasks. To build the PA design framework, we will conduct a study to categorize teachers’ PA expectations and needs, accompanied with findings from the literature. Our hypothesis is that classrooms, where the PA is used to support teachers in the learning activity, will demonstrate high learning gains and students’ perceived motivation. Keywords 1 Pedagogical agent, collaboration, class orchestration, music and mathematics. 1. Introduction support cognitive, metacognitive [2] motivational, or social [8] aspects of learning. 1.1. Pedagogical agents Integrating PAs in learning digital systems goes in line with social learning theory Pedagogical agents (PAs) are lifelike [9]. The main premise of this theory is that virtual characters playing an educational role, learning is a social contextualized process, thus, aiming to facilitate learning in digital learning in digital learning systems, PAs serve as a environments (DLE) [1]. For instance, PAs social entity that can simulate real-life facilitate learning by providing students with interactions, such as role modelling. However, scaffolding [2] and guidance [3]. PAs can be reviews discuss mixed evidence on the benefits combined with the support of various forms, PAs can have on learning [1, 10]. For instance, such as text, voice, 2D or 3D character, and Schroeder et al. [10] meta-analysis reported a human-like appearance [4]. The roles PAs can small but statistically significant (g = .19, p < play in the DLE, may include tutor [5], expert, .001) learning effect in favor for agent-based mentor, motivator [6] student [7] Depending on systems. They found this effect to be prominent the PA system, the behaviour of the agent can in K-12 education and discussed that motivational benefits may be related to this Proceedings of the Doctoral Consortium of Sixteenth European Conference on Technology Enhanced Learning, September 20–21, 2021, Bolzano, Italy (online). EMAIL: eric.roldan.roa@ut.ee (A. 1); irene- angelica.chounta@uni-due.de (A. 2); margus.pedaste@ut.ee (A. 3) ORCID: 0000-0002-7519-4933 (A. 1); 0000-0001-9159-0664(A. 2); 0000-0002-5087-9637 (A. 3) © 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Wor Pr ks hop oceedi ngs ht I tp: // ceur - SSN1613- ws .or 0073 g CEUR Workshop Proceedings (CEUR-WS.org) positive result. Kim & Baylor [6] suggests that learning space. Conversely, when arts are used a single agent design or behaviour can’t fit all as a vehicle for teaching mathematics in the students’ needs. Therefore, they stress the same session instead of a different parallel importance of designing the PA with the activity, it contributes positively to learning appropriate persona and media features to outcomes as it helps to: (i) promote adequately support every student’s learning communication among students; (ii) transform process. learning environments; (iii) reach students that Literature shows that PAs can otherwise may not be reachable; (iv) offer new contribute to the motivation of students at an challenges to successful students; (v) decrease individual level [8]. Our research interest is curricula fragmentation; (vi) connect in-school whether a PA that targets the classroom as a learning with real-world, among others [16] whole, would have a similar impact on [17]. This was also confirmed by a study from students’ motivation, thus, better learning An et al. [18] which demonstrated that outcomes. integrated music and math lessons have a positive impact on multiple mathematical 1.2. Collaboration and technology abilities. Dillenbourg [11] defines collaborative 1.4. Classroom orchestration and learning as the situation in which two or more technology people learn or attempt to learn something together. The research field that explores how PAs are typically used to support students on technology impacts and can promote the individual level, while it is not clear how a collaboration is computer supported PA can be used in the classroom as a teacher’s collaborative learning. Stahl et al. [12] have support in classroom orchestration. Dillenbourg defined this as the field to study how learning [19] defined classroom orchestration as a can be scaffolding in computer-supported teacher's ability to manage, in real-time, the collaboration scenarios. We first need to activities and contextual constraints inherent to contextualize the collaborative learning the learning session. This managerial instance scenario to design appropriate scaffolding encompasses the nature of the activity (for towards the reinforcement of domain example, individual, teamwork, class-wide), knowledge acquisition and collaboration. To the pedagogical tools (such as simulations, that end, we will build on a technology- wikis, quizzes), and the distribution channels enhanced method that combines music and (for example laptops, tablets, smartPhones). mathematics in collaborative learning tasks Conversely – and complementary – to [13]. In this case technology is used both on the instructional design and adaptive learning, individual and classroom level. classroom orchestration deals with extrinsic activities (moving chairs, collecting papers, 1.3. Music and mathematics checking on students’ activity status, student log-in problems) and extrinsic constraints According to Tobias [14], teaching and (discipline, limited lesson time, energy learning experiences that are not based on a management, classroom physical space) [19]. traditional mathematics curriculum can bridge Regarding the technological aspects, related the achievement gap and reduce mathematical work has explored teachers’ needs for anxiety. To that end, we argue that combining educational technologies. For example, mathematics with music may potentially help to Holstein et al. [20] showed that teachers bridge the achievement gap and reduce anxiety. expressed their wish to be able to see students However, it is not evident what pedagogical thinking process and being able to adopt strategies need to be considered to ensure this system-like features like monitoring all successful combination. For instance, Vaughn students at the same time. Furthermore, another [15] found that there was a positive association case study by Chounta et al. [21] showed that between the voluntary study of music and teachers would like to receive support to be mathematical achievement. It is important to more efficient and effective in their practice. note that in Vaughn’s study, the academic and The authors convey the message that systems learning activities did not occur in the same including artificial intelligent techniques, could address such teachers’ needs. Amarasinghe et [RQ3] What is the impact, in terms of al. [22], presented the notion of orchestration learning gains and motivation, when agents, which can help teachers by suggesting employing a PA hybrid system at a orchestration actions, thus offloading decision- classroom level? making responsibilities whilst respecting their [RQ4] What benefits, challenges, and agency. They referred to the latter scenario as a constraints can be seen when employing a hybrid human-machine approach. Our work PA at a classroom level? expands on what teachers expect from a PA (in the form of a 2D character) helping them at a 2. Methodology outline classroom level and exploring the impact of a hybrid system solution for K-12 education. In this study, we have selected a mixed- For our PA system, we envision the agent method approach. In terms of qualitative helping teachers with the activities as well as research, we will conduct a literature review orchestration decisions. One example of an and a case study to develop the design activity employed at a classroom level can be framework for our PA. In terms of quantitative found in Chin et al. [23]. In this case, the research, we will conduct a series of studies to feature allowed the teacher to show on a report on the learning outcomes and perceived projected screen students’ teachable agents students’ motivation via tests and with the aim to discuss on agents’ different questionnaires. We elaborate more on the answers, hence, students understanding. planned studies in the next sections. Additionally, we are taking inspiration from existing PA systems targeting mathematics [2] [8]. However, our approach is different from 2.1. Participants the aforementioned studies in that the learning activities combine music and mathematics as For the case study, we plan to carry out focus means to motivate and support students’ groups with teachers (n = 5 to 7) to understand conceptual and procedural knowledge their expectations and needs when using a PA understanding. as orchestration support at a classroom level. The target population are mathematics teachers 1.5. Research questions from primary education. For the pilot study, we will test the PA in one elementary classroom (n To understand how PA technology could = 15-30 students). This will allow us to modify support social dimensions on the classroom and adjust the PA system as well as our planned level rather than the individual level, we further measuring instruments. Finally for the main investigate when and how a PA can help study, we will employ the PA system in teachers in collaborative learning activity while elementary classrooms (n = (4 to 8) including motivating students to learn and be engaged in experimental and control groups) to evaluate the task. To that end, the PA will be used in the the PA design framework based on teachers’ classroom by integrating a virtual character to insights, and to evaluate students’ (100-200) assist the teacher. We are interested to see learning gains and perceived motivation. whether employing a PA in the classroom makes a difference in terms of learning 2.2. Materials outcomes and contribute to students’ motivation in the collaborative activity. In this Pedagogical Agent and learning study, we examine the following research activities. For the PA system, we are using a questions (RQs): face tracking solution, the agent can emulate gestures, eye blinking, lip-synching to a sound [RQ1] Which kind of interventions and source, and head swing. Additionally, by key affordances do the current PA systems in K- commands, the agent can walk, run, wave, point 12 education have for teachers at a out, and trigger special moves (i.e., thinking classroom level? pose, wearing glasses, eating a banana). On the [RQ2] What do teachers expect and need other hand, for the learning activities, we want from a PA helping them to instruct and to build on prior research done with an orchestrate collaborative learning activities? educational game that combines music and mathematics in the format of a board game and a digital version. In the case of the board-game format, two elementary schools in Belgrade, Serbia, played the game for two sessions. Students were randomly assigned to play in small groups and to answer questionnaires targeting their learning experience. The results showed that the educational game supported their cognitive development while boosting their motivation and desire to have success on the learning tasks [24]. Building on the latter study, we used the digital version of the game and focused on group formation strategies and learning outcomes [13]. Using students’ prior knowledge (as assessed by pre-knowledge tests), authors formed homogeneous (high or low performers only) and heterogeneous (high and low performers mixed together) groups and explored whether their game performance would be reflected on students’ learning gains. Conversely to related research [25], the aforementioned study reported students belonging to heterogeneous condition to benefit less than homogeneous groups in terms of learning gains. However, this was not the case for the game score, where HE groups Figure 1: Research timeline and plan. outperformed HO groups playing the game [13]. This article is particularly relevant We will adapt collaborative learning because we are considering the authors’ activities with and without the PA (independent suggestions to better align the educational game variable), as they will serve as the educational with the learning goals and to enhance the context of the experiments. We will divide half collaboration activities. of the participating school groups of students Instruments. Furtermore, we will create an into experimental condition (with PA) and interview protocol and we will carry out control group (without PA). Our dependent teachers’ focus groups. The aim is to develop a variables are, on one hand, students’ design framework based on their expectations performance (as assessed by pre and post and needs when collaborating and employing a knowledge tests) along with their perceived pedagogical agent in the classroom. Regarding motivation when working with or without the the students, they will be asked to answer a agent (questionnaires); and on the other hand, questionnaire (still to be defined) targeting teachers’ evaluation of the PA when facilitating motivation, from which we will analyze and the orchestration of the collaborative tasks. report on PA effects and design challenges. We hypothesize that the experimental group Finally, we will use students’ data to find a will benefit more from having a PA helping the possible correlation between their perceived teacher to orchestrate the collaborative learning motivation and their learning outcomes (pre- activity. The benefits will be reflected in terms post tests evaluation). of students’ learning outcomes and motivation. On the other hand, we expect teachers to 2.3. Study design and procedure evaluate the PA system in the expected dimensions we will be able to find after In the following figure (Figure 1) we present concluding the case study with them. and describe the aim of the planned studies and timeline. We link each study to our research questions. 3. Progress so far Journal of Educational Computing Research 53(2) (2015), 183–204. We are currently in the process of doi:10.1177/0735633115597625.A. conducting a systematic literature review (SLR) [2] N. Matsuda, W. Weng, N. Wall. The effect considering all types of PAs addressing of metacognitive scaffolding for learning mathematics education in K-12 education. The by teaching a teachable agent, Journal of aim is to identify all pedagogical affordances Artificial Intelligence in Education 30(1) these systems offer teachers when PAs are used (2020) 1–37. doi:10.1007/s40593-019- on a classroom level. We envision the findings 00190-2. of the latter will support us in designing and [3] J. Kinnebrew, G. Biswas. Modeling and proposing a theoretical framework for teachers’ measuring self-regulated learning in interventions in PA systems, and to classify PA teachable agent environments, Journal of systems affordances when using them on the e-Learning and Knowledge Society 7(2) classroom level for supporting instruction and (2011), 19–35. doi:10.20368/1971- orchestration activities. In parallel, we are in 8829/518. the process of conducting focus groups to [4] A. S. D. Martha, H. B. Santoso. The design understand teachers’ expectations and needs and impact of the pedagogical agent: a when having a PA helping them to orchestrate systematic literature review, Journal of collaborative activities in the classroom. From Educators Online 16(1) (2019) 1–15. both, the SLR and the case study, we aim to doi:10.9743/jeo.2019.16.1.8. have the pedagogical and orchestral design [5] A. C. Graesser. Conversations with requirements to be met by the PA. AutoTutor help students learn, Technology wise, currently, we are International Journal of Artificial performing tests with the PA system in real- Intelligence in Education 26(1) (2016) 1– time in remote teaching settings (video 15. doi: 10.1007/s40593-015-0086-4. conferencing platforms) and classrooms [6] Y. Kim, A. L. Baylor. Research-based settings by using projection. The aim is to test design of pedagogical agent roles: a and adjust PA social cues, signals, reactions review, progress, and recommendations, [26], and social fidelity contributors (i.e., International Journal of Artificial personalization, slang, politeness, enthusiasm, Intelligence in Education 26(1) (2016) interactivity) [27]. 160–169. doi:10.1007/s40593-015-0055- y. 4. Theoretical and practical [7] N. Matsuda, E. Yarzebinski, V. Keiser, R. Raizada, W. W. Cohen, G. J. Stylianides, contributions K. R. Koedinger. Cognitive anatomy of We envision that this research will tutor learning: lessons learned with contribute to the field of PA by bringing SimStudent, Journal of Educational teachers’ perspectives and needs when having Psychology 105(4) (2013), 1152–1163. the agent in a classroom collaborative setting. doi:10.1037/a0031955. Moreover, by using PA systems technology at [8] B. Sjödén, A. Silvervarg, M. Haake, A. a classroom level, it could create a bond Gulz. Extending an educational math between the agent and student that could impact game with a pedagogical conversational learning outcomes. Finally, this study will agent: facing design challenges, report on an innovative technology-enhanced Interdisciplinary Approaches to Adaptive learning scenario, and future research could Learning: A Look at the Neighbours, ITEC potentially expand on the psychological, (2010), volume 126 of Communications in collaborative, and social effects this technology Computer and Information Science, may bring for both teachers and students. Springer, Berlin, Heidelberg (2011). doi:10.1007/978-3-642-20074-8_10. [9] A. Bandura. Human agency in social 5. References cognitive theory. American Psychologist 44(9) (1989), 1175–1184. doi:10.1037/0003-066X-44-9-1175. [1] N. L. Schroeder, C. M. Gotch. Persisting [10] N. L. Schroeder, O. O. Adesope, R. B. issues in pedagogical agent research, Gilbert. How effective are pedagogical agents for learning? A meta-analytic [20] K. Holstein, B. M. McLaren, V. Aleven. review, Journal of Educational Computing Intelligent tutors as teachers’ aides: Research 49(1) (2013), 1–39. exploring teacher needs for real-time doi:10.2190/EC.49.1.a. analytics in blended classrooms, [11] P. Dillenbourg, What do you mean by Proceedings of the Seventh International collaborative learning?, in: P. Dillenbourg Learning Analytics & Knowledge (Ed.), Collaborative-Learning: Cognitive Conference (2017), 257–266. and Computational Approaches, Elsevier, doi:10.1145/3027385.30.27451 Oxford, 1999, pp. 1–19. [21] I.-A. Chounta, E. Bardone, A. Raudsep, [12] G. Stahl, T. Koschmann, D. Suthers, M. Pedaste, Exploring teachers’ Computer-supported collaborative perceptions of artificial intelligence as a learning: an historical perspective, in: R. tool to support their practice in Estonian K. Sawyer (Ed.), Cambridge Handbook of K-12 education, International Journal of the Learning Sciences, Cambridge Artificial Intelligence in Education (2021). University Press, Cambridge, 2006, pp. doi:10.1007/s40593-021-00243-5. 409–426. [22] I. Amarasinghe, D. Hernández-Leo, K. [13] E. Roldán Roa, É. Roldán Roa, I.-A. Manathunga, A. Jonson. Sustaining Chounta, Learning music and math, Continous Collaborative Leearning Flows together as one: towards a collaborative in MOOCs: Orchestration Agent approach for practicing math skills with Approach. Journal of Universal Computer music, Volume 1232 of Collaboration Science, vol. 24, no 8 (2018), 1034-1051. Technologies and Social Computing, [23] E. D. B. Chin, I. M. Dohmen, B. H. Cheng, Springer International Publishing, 2020, M. A. Oppezzo, C. C. Chase, D. L. pp. 143–156. doi:10.1007/978-3-030- Schwartz. Preparing students for future 58157-2-10. learning with teachable agents, Education [14] S. Tobias, C. Weissbrod, Anxiety and Tech Research Dev (2010). doi: mathematics: an update, Harvard 10.1007/s11423-010-9154-5 Education Review 50(1) (1980), 63–70. [24] S. Rajić. Mathematics and music game in doi:10.17763/haer.50.1.xw483257j60350 the function of child’s cognitive 84. development, motivation, and activity, [15] K. Vaughn, Music and mathematics: Early Child Development and Care modest support for the off-claimed (2019), 1–13. relationship, Journal of Aesthetic doi:10.1080/03004430.2019.1656620. Education 34(3/4) (2000), 1175–1184. [25] S. Manske, T. Hecking, I.-A. Chounta, S- [16] E. B. Fiske, Champions of Change: The Werneburg, H. U. Hoppe. Using Impact of the Arts on Learning, Arts difference to make a difference: a study on Education Partnership and President’s heterogeneity of learning groups Volume Committee on the Arts and Humanities, 1 of Exploring the Material Conditions of Washington, DC, 1999. Learning: The Computer Supported [17] H. L. Erickson, Concept-Based Collaborative Learning (CSCL) Curriculum and Instruction: Teaching Conference, The International Society of Beyond the Facts, Corwin Press, Thousand the Learning Sciences, Gothenburg, 2015. Oaks, CA, 2002. [26] J. Feine, U. Gnewuch, S. Morana, A. [18] S. An, M. M. Capraro, D. A. Tillman, Maedche. A taxonomy of social cues for Elementary teachers integrate music conversational agents, International activities into regular mathematics Journal of Human-Computer Studies 132 lessons: effects on students' mathematical (2019),138–161. abilities, Journal for Learning through the doi:10.1016/j.ijhcs.2019.07.009. Arts, 9(1) (2013). [27] A. M. Sinatra, K. Pollard, B. Files, A. doi:10.21977/D99112867. Oiknine, M. Ericson, P. Khooshabeh. [19] Dillenbourg, P (2013). Design for Social fidelity in virtual agents: impacts on classroom orchestration. Computers & presence and learning, Computers in Education, 69, 485-492. doi: Human Behavior 114 (2021). 10.1016/j.compedu.2013.04.013 doi:10.1016/j.chb.2020.106562.