A systematic narrative review research in K-12 and schools of learning analytics

Abstract

The field of learning analytics emerged in the last decade to take advantage of the increasing availability of data about learners that digital systems generate. Existing research in learning analytics has focused on higher education, as this context often relies heavily on digital platforms such as online learning management systems, making data collection easier. In this paper, we focus on LA research in the context of elementary level teaching. We provide a systematic narrative review in which we analyze the articles that had the most impact in the field. Our results show the existence of some recurring themes such as gamification and multimodal methods. We make a distinction between papers in which learning analytics is the target of the study (e.g., dashboards) and papers in which learning analytics methods were used as a means to study a given behavior/skill/phenomenon (e.g., problem-solving skills). Lastly, we found that most studies lack a strong theoretical foundation on education science and, thus, there is a need to develop more elaborated theoretical perspectives in future research on school-level learning analytics, as well as papers that deliver a real impact on learning and teaching.

1. Introduction

Within the Horizon reports for the K-12 we can see the development trends in the use of educational technology [ 1 ]. Such reports have listed the technologies that have become actively used within the K-12 teaching and created predictions about the technologies that are expected to become actively used within the next few years. Within these currently used and future technologies we are able to see a difference: the current technologies have typically been technologies such as cloud computing and tablet computing, i.e., everyday technologies for teachers. Instead, the technologies anticipated within four to five years have been technologies such as artificial intelligence, internet of things and wearable technology. This distinction aligns with the ideas of Phillips and Harris [ 2 ], who suggest that, from the perspective of teachers, the technologies can be seen as transparent technologies or emerging technologies, i.e., technologies that have become part of teachers’ routines or new technologies that demand special effort, new learning. One trend within the developing educational technologies [ 3 ] is Learning Analytics (LA), which has been listed several times within the Horizon reports for the K-12 and higher education levels [ 1,4 ].

LA is commonly defined —yet not unanimously agreed— as “the measurement, collection, analysis, and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the environments in which it occurs” [ 5 ]. Since the emergence of the discipline, researchers have tried to harness the opportunities inherent in automatically collected data about students’ learning activities within various digital services [ 6 ]. Still, not so long ago, the researchers recognized the deficiencies of the digital data collected by traditional services, e.g., Learning Management Systems (LMS). The main problem with the data was its relatively poor relevance to learning, lack of granularity and contextual details. Several new sources of data were explored such as multimodal and dispositional data [ 7 ]. Multimodal data offers authentic, real-life sources of data about, e.g., students’ movements, physiological functions and interactions within the learning environment [ 8 ]. Offering such authentic information, multimodal LA is expected to offer a more nuanced understanding of real-life learning where it occurs [ 9 ]. Yet, such promises are still open for research and inquiry and therefore, rather distant from practice. Dispositional LA offers a qualitative depth of students’ dispositions, e.g., motivation, self-efficacy and engagement. It is hypothesized that by adding dispositional data to digital data one can get a more holistic view of students’ learning [ 7 ].

Within this paper, we focus on LA research in the context of elementary level classroom teaching where the research related to LA has remained scarce. One of the reasons may lie in that learning environments in the elementary level are usually physical classroom environments with emphasis on the face-to-face teaching and digital devices and online-learning have traditionally had a minor role, making the data collection challenging. Again, within the context of Finland, elementary education is directed by the national core curriculum [ 10 ] aligning with the Finland’s Basic Education Act [ 11 ]. The core curriculum states that the goal of student assessment is to guide and encourage learning as well as to develop the student's prerequisites for self-assessment. The core curriculum resorts to the idea that the focus of assessment is that it promotes learning instead of mere measuring with highstakes assessment scales and strict assessment rubrics. As the national core curriculum was constructed in a collaborative process with various stakeholders and written before 2014, it does not mention LA or the possibilities or the role it might have in the assessment processes.

Thus, it is clear that elementary education learning environments present new challenges for understanding the benefits of integrating LA. Still, according to Weller [ 12 ] the mid-1990s was a turning point within the field of educational technology, starting the rapid development. Since then, new technologies like the Internet, followed by mobile technologies and social networks, have become part of everyday teaching and learning practices at all levels of education [ 13 ]. Development of educational technologies started to provide various tools especially for students. The concepts like One-to-One computing [ 14 ], Bring Your Own Device [ 15 ], Personal Learning Environments [ 16 ] and Cloud Computing [ 17 ] provided students with tools and environments for supporting their personal and self-regulated learning activities. At the same time, these technologies started to provide new opportunities also for integrating LA in the elementary school context. With the technologies used, especially the students’ personal technologies, the possibilities for collecting and analyzing data from students’ learning practices, i.e., taking advantage of LA for learning, became more feasible. Also, according to Kovanovic et al. [ 18 ], the COVID-19 period with distance education has boosted the use of technologies suitable for LA at the elementary level.

Since the facilities for using LA within the elementary school contexts are getting better, there is a need for outlining the research done within that field. According to Lee et al. [ 19 ], the focus of LA research has been strongly on higher education level and within Massive Open Online Courses (MOOCs). MOOCs have provided ideal opportunities for collecting data on students’ activities. Also, the research field has been dominated by computer science. The aim of this research is to provide an overview for the research targeting elementary level education using LA based on the most cited articles. The aim is to focus on the educational science, to outline the pedagogical groundings and pedagogical aims of using LA in the elementary level context.

2. Methods

We searched the Scopus database in order to retrieve relevant articles to LA in the context of elementary level education. The Scopus database includes a large variety of conferences and journals that are relevant to the research questions of this article, in addition to the majority of Web of Science titles [ 20 ]. Scopus is well-maintained in comparison to other databases, and it utilizes strict selection criteria for journals and conference proceedings [21,22]. We performed the following search query on March, 16th 2022 which resulted in 996 articles: TITLE-ABS-KEY ( "educational data mining" OR "learning analytic*" ) AND TITLE-ABS-KEY ( "K12" OR "K-12" OR "school*" OR "child*" OR "kid*" OR primary OR elementary )

Two of the authors screened all of the articles (title, abstract, keywords, and venue) using Rayyan.ai (an online service for conducting systematic literature reviews) and proceeded to exclude those that were not conducted in the elementary and primary level context, corresponding to the Finnish elementary level education. The screening process resulted in 136 articles matching the inclusion criteria (covering LA in K-12) and 860 excluded. Furthermore, to ensure that we only captured those articles that have captured the attention of other researchers, we limited our analysis to those articles that had at least two Scopus citations per year. This filtering resulted in 33 papers selected for analysis. The complete search and screening process is depicted in Figure 1.

3. Results

The deductive approach to code the final list of papers resulted in the categorization of articles according to subthemes: LA methods (including dashboards), LA stakeholders, LA approach (LA as a target or LA as a tool), and pedagogical perspectives. Following is a literature-supported narrative review.

Regarding LA methods, games and gamification were one of most important emerging themes over the last decade in education in general and so were in LA research [23]. Several articles in our data have addressed the games theme, for instance, [24] used a Kinect game to help disabled students improve their motor praxis in an authentic school environment. The collected multimodal data were used to assess students’ performance and to offer teachers a method that could help track students’ progress. Similarly, children’s problem-solving skills of a sorting motion-based task were monitored using multimodal data collection (e.g., eye-trackers, Kinect joint tracking and wristbands). Multimodal data was then used to analyze students’ play and problem-solving behavior [ 9 ]. Other papers with a similar approach using multimodal LA to track students’ learning in real-life were reported by other researchers to capture students’ coordination of motor activities [e.g., 25] or to track and improve children’s motor skills with gaming [26]. Computational thinking (CT) has been a buzz word lately in education aiming to teach students to “think like computers'' [27], and therefore, researchers have tried to find ways that LA can help students improve learning CT with LA and gamification [e.g., 28] and using LA with robotics [e.g., 29]. Dashboards have been one of the main approaches to LA since conceptualization and early prototypes, and therefore, it was natural to offer them to teachers in schools. For instance, [30] offered a dashboard to mathematics teachers and assessed teachers' pedagogical actions and usage. The authors found that dashboard usage has positively impacted teachers practice by helping teachers improve their feedback and activate diverse pedagogical approaches. Similarly, dashboards were used to inform teachers in other papers e.g., [31]. In terms of research related to dashboards, also students’ perspectives have received attention. For example Wang and Lin [32] found that younger students prefer more colorful LA visualizations compared to a little older elementary level students.

From the stakeholders perspective, children have been the main targets of most LA research, e.g., [25,26], followed by teachers e.g., [30] and less so were families. The aim of these studies was to understand the possibilities of the analytics as part of the learning process. This was done using dashboards targeted for teachers [30,33]: the aim was to provide teachers with real time information about the ongoing learning processes within the classrooms. Families are an unusual stakeholder in LA, yet, with the wide scale adoption of LA, research has to address their perspectives. For instance, [34] assessed parents' usage of monitoring their children through mobiles and found teachers' use of the school database helped stimulate parents and children access the information.

Qualitative synthesis of the LA approach in the included articles can be roughly divided into two groups. First are the articles where analytics is the target of the research and second where the analytics can be seen as a means or method for conducting the research. The number of the articles in the first category is smaller. Within the articles in this group the aim was testing and developing the adaptive learning environments for providing students with more personalized materials and assignments for supporting their learning activities [35,36]. Along with adaptive technologies, this area covers studies for developing LA methods and algorithms that are capable of predicting the students’ success [37]. The second category contained studies where the possibilities of the analytics were mainly used as a means for analyzing the research data. Within these studies there were different learning activities conducted using different technologies such as motion-based games [ 9 ] or augmented reality [38] where researchers collected multimodal data sets, in order to understand the effects of the different learning methods and technologies along with students' different behavior patterns. The use of analytics allowed researchers to approach areas of research that would have been challenging or impossible without. These two approaches of research are intended to serve as ways to illustrate the field of LA research, the use of analytics within the studies conducted at the elementary level.

From the perspective of the learning and the pedagogical perspectives used within the studies we can see the emphasis on the gamification and problem-solving along with collaborative approaches. Articles using gamification and problem-solving targeted typically areas of mathematics or CT providing students with various learning activities, typically conducted individually [ 9,39 ]. Within these cases, the role of analytics was built on adaptive learning environments and using analytics for gaining understanding of the different learning patterns and behaviors. Along with these, the articles contained experiments based on collaborative learning activities [40,41]. Within these the role of analytics was built on providing information about the nature of the collaboration process for teachers and students and again the role of analytics was also on making the collaboration process more visible, to understand the nature of collaboration [33]. Many of the articles were conducted in learning situations where students were using tablet computers. Still, the use of blended learning was mentioned rarely. Similarly, within the analyzed studies there were experiments for targeting students self-regulated learning and inquiry-based learning activities. Also, the Bloom taxonomy was used —rather uncritically— as a way to highlight the influence of adaptive e-learning on learning effectiveness [35].

4. Discussion

Since datafication started to target our daily life, interactions with the Internet and digitized services, fear, skepticism, and distrust of the risks and drawbacks to our privacy have been on the rise [42]. Yet, datafication has grown faster and more indiscriminately to capitalize on more aspects of our life and invade more factions of our societies, e.g., children in our case. Extending LA and datafication in general to children —one the most vulnerable groups of our societies— requires far more caution. Yet, the analysis of the group of papers we have addressed here shows that the collected data about students have used more intrusive techniques to collect data about students compared to higher education. Several studies have used multimodal data gathering including sensors to track the eyes, body movements and interactions with the environment. While precautions may have been taken, ethical guidelines may have been followed, it remains to be seen what such obtrusive data collection offers compared to the privacy risks it imposes. Once digital data is recorded, it is a permanent record, and we need to be mindful of the permanent record we would keep about our vulnerable children.

The analysis reveals that the number of studies targeting the elementary level with LA is rather limited, especially the highly cited studies. The analysis reveals that the research area is rather fragmented. The aim of the studies varies, containing different terminology related to topic, the research methods and data used. The following provide insights into the nature of the field.

Results concerning the LA research within the elementary level education poses different perspectives for the field. First there is the difference between whether the analytics is considered as the target of research, a way to support pupils’ learning or as a means for conducting research. Within most of the studies analyzed, the analytics was used as a way for analyzing the data using different data sets. Along with these studies, there are studies where the analytics is used as part of the actual learning process. Again, these articles can be divided for two subcategories, based on the pedagogical goals concerning the purposes of the analytics. First there are studies where the role of analytics is in the form of providing students with personalized learning paths based on their learning activities. Along with these, there are studies where the analytics is used for providing teachers and students information about the learning processes within different pedagogical designs. These two approaches to use analytics for supporting learning reflect different learning theories, aligning with the development paradigms of educational technology [43]. Technologies have been built on more behavioristic learning theories, technology serving as a tireless personal teacher providing new assignments and tasks as long as the pupil has energy to continue. At the other end, development of educational technologies have been grounded on theories that emphasize more collaborative and self-regulative learning approaches [ 3,43 ].

We found the theoretical foundations for using and developing LA for elementary level highly important. In practice, there seems to be a tension whether LA is used as part of formal or more informal assessment of the student or as a support structure through which students are facilitated to take an active role in their learning processes. Instead of using analytics for assessment or building learning paths for pupils’ needs, we see the potential of LA also in supporting pupils’ collaborative and self-regulated learning processes, supporting their metacognitive thinking. Current theoretical paradigms from the field of education along with the 21st century skills that pupils are expected to gain from school, emphasize pupils' role as active learners, capable of regulating their own learning process and collaborating with their peers. For this purpose, we find the role of LA important, especially as a tool for making the learning process more visible. Different visualizations and information about various learning activities conducted can serve as means for developing pupils' readiness for self-monitoring and organizing their learning activities, which are vital skills for selfregulated learners [44]. This area needs more research and development, to understand the ways how LA can serve as a tool for pupils to develop their regulation and learning skills. LA needs to serve as a scaffold, for achieving permanent effect for pupils’ ways to self regulate and learn, for pupils' metacognitive thinking.

This systematic narrative analysis of selected LA research on elementary school level indicates that there is a need to develop more elaborated theoretical perspectives for elementary school context LA research. This is a growing research field, and as such the number of studies is limited and themes seem to be somewhat fragmented. Also, the descriptions of educational theories to which these studies are built, remain to a large extent on a quite superficial level. Thus, more rigorous educational and learning theoretical perspectives should be developed. For example, Bloom’s taxonomy, which was adopted as a key theoretical basis in one of the studies, has received critical remarks from educational science researchers (e.g., [45]). Also, there is a risk of resorting to a behaviorist epistemology and learning conceptions, if we blindly rely on simple behavioral data and intensive testing in educational contexts. Furthermore, even though educational scientists may have a consensus to some extent on the general features of a learning process, elementary school teaching and learning environments are very much regulated by national and contextualized guidelines, with varying levels of for example teacher autonomy and guiding role of the curriculum. Therefore, more rigorous research on LA in various elementary school contexts is highly needed. [21] J. Baas, M. Schotten, A. Plume, G. Côté, Scopus as a curated, high-quality bibliometric data source for academic research in quantitative science studies, Quantitative Science. (2020). https://direct.mit.edu/qss/article-abstract/1/1/377/15571. [22] V.K. Singh, P. Singh, M. Karmakar, J. Leta, P. Mayr, The journal coverage of Web of Science,

Scopus and Dimensions: A comparative analysis, Scientometrics. 126 (2021) 5113–5142. [23] S. Schöbel, M. Saqr, A. Janson, Two decades of game concepts in digital learning environments – A bibliometric study and research agenda, Comput. Educ. 173 (2021) 104296. [24] G. Altanis, M. Boloudakis, S. Retalis, N. Nikou, Children with motor impairments play a kinect learning game: first findings from a pilot case in an authentic classroom environment, J Interact Design Architect. 19 (2013) 91–104. [25] D. Abrahamson, S. Shayan, A. Bakker, M. van der Schaaf, Eye-Tracking Piaget: Capturing the Emergence of Attentional Anchors in the Coordination of Proportional Motor Action, Human Development. 58 (2015) 218–244. https://doi.org/10.1159/000443153. [26] M. Kourakli, I. Altanis, S. Retalis, M. Boloudakis, D. Zbainos, K. Antonopoulou, Towards the improvement of the cognitive, motoric and academic skills of students with special educational needs using Kinect learning games, International Journal of Child-Computer Interaction. 11 (2016) 28–39. [27] M. Saqr, K. Ng, O.S. Sunday, T. Matti, People, Ideas, Milestones: A Scientometric Study of

Computational Thinking, ACM Transactions on Computing Education. 21 (2021) 1–17. [28] R. Israel-Fishelson, A. Hershkovitz, Persistence in a Game-Based Learning Environment: The Case of Elementary School Students Learning Computational Thinking, Journal of Educational Computing Research. 58 (2020) 891–918. [29] D. Scaradozzi, L. Cesaretti, L. Screpanti, E. Mangina, Identification of the Students Learning

Process During Education Robotics Activities, Front Robot AI. 7 (2020) 21. [30] I. Molenaar, C.A.N. Knoop-van Campen, How Teachers Make Dashboard Information

Actionable, IEEE Transactions on Learning Technologies. 12 (2018) 347–355. [31] S. Sergis, D.G. Sampson, M.J. Rodríguez-Triana, D. Gillet, L. Pelliccione, T. de Jong, Using educational data from teaching and learning to inform teachers’ reflective educational design in inquiry-based STEM education, Comput. Human Behav. 92 (2019) 724–738. [32] H.-F. Wang, C.-H. Lin, An investigation into visual complexity and aesthetic preference to facilitate the creation of more appropriate learning analytics systems for children, Comput.

Human Behav. 92 (2019) 706–715. [33] A. van Leeuwen, N. Rummel, T. van Gog, What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations?, International Journal of ComputerSupported Collaborative Learning. 14 (2019) 261–289. [34] I. Blau, M. Hameiri, Ubiquitous mobile educational data management by teachers, students and parents: Does technology change school-family communication and parental involvement?, Education and Information Technologies. 22 (2017) 1231–1247. https://doi.org/10.1007/s10639016-9487-8. [35] S. Hubalovsky, M. Hubalovska, M. Musilek, Assessment of the influence of adaptive E-learning on learning effectiveness of primary school pupils, Comput. Human Behav. 92 (2019) 691–705. [36] I. Molenaar, C. Knoop-van Campen, Teacher Dashboards in Practice: Usage and Impact, in: Data

Driven Approaches in Digital Education, Springer International Publishing, 2017: pp. 125–138. [37] K.I.M. Ramaphosa, T. Zuva, R. Kwuimi, Educational Data Mining to Improve Learner Performance in Gauteng Primary Schools, in: 2018 International Conference on Advances in Big Data, Computing and Data Communication Systems (IcABCD), 2018: pp. 1–6. [38] A.O.R. Rodríguez, M.A. Riaño, P.A.G. García, C.E.M. Marín, R.G. Crespo, X. Wu, Emotional characterization of children through a learning environment using learning analytics and ARSandbox, J. Ambient Intell. Humaniz. Comput. 11 (2020) 5353–5367. [39] J. Asbell-Clarke, E. Rowe, V. Almeda, T. Edwards, E. Bardar, S. Gasca, R.S. Baker, R. Scruggs, The development of students’ computational thinking practices in elementary- and middle-school classes using the learning game, Zoombinis, Comput. Human Behav. 115 (2021) 106587. [40] M. Cukurova, R. Luckin, M. Mavrikis, E. Millán, Machine and human observable differences in groups’ collaborative problem-solving behaviours, in: Data Driven Approaches in Digital Education, Springer International Publishing, Cham, 2017: pp. 17–29. [41] B.B. Schwarz, N. Prusak, O. Swidan, A. Livny, K. Gal, A. Segal, Orchestrating the emergence of conceptual learning: a case study in a geometry class, Int. J. Comput. Support. Collab. Learn. 13 (2018) 189–211. [42] M. Saqr, Big data and the emerging ethical challenges, Int. J. Health Sci. . 11 (2017) 1–2. [43] T. Koschmann, Revisiting the paradigms of instructional technology, in: Meeting at the Crossroads. Proceedings of the 18th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education, Citeseer, 2001: pp. 15–22. [44] B.J. Zimmerman, Self-Regulated Learning and Academic Achievement: An Overview, Educ.

Psychol. 25 (1990) 3–17. [45] M. Murtonen, H. Gruber, E. Lehtinen, The return of behaviourist epistemology: A review of learning outcomes studies, Educational Research Review. 22 (2017) 114–128.

References

[1] A. Freeman , S.A. Becker , M. Cummins , A. Davis , C.H. Giesinger , NMC/CoSN horizon report: 2017 K-- 12 edition. Austin, Texas: The new media consortium, ( 2017 ).

[2] M. Phillips , J. Harris, PCK and TPCK/TPACK: More than etiology , in: Society for Information Technology & Teacher Education International Conference , Association for the Advancement of Computing in Education (AACE ), 2018 : pp. 2109 - 2116 .

[3] T. Valtonen , S. López-Pernas , M. Saqr , H. Vartiainen , E.T. Sointu , M. Tedre , The nature and building blocks of educational technology research , Comput. Human Behav . 128 ( 2022 ) 107123 .

[4] K. Pelletier , M. Brown , D. Christopher Brooks, M. McCormack , J. Reeves , N. Arbino , A. Bozkurt , S. Crawford , L. Czerniewicz , R. Gibson , K. Linder , J. Mason , V. Mondelli , 2021 EDUCAUSE Horizon report teaching and learning edition , ( 2021 ) 2 - 50 .

[5] P. Long , Proceedings of the 1st International Conference on Learning Analytics and Knowledge , Association for Computing Machinery, 2011 .

[6] M. Saqr , S. López-Pernas , The curious case of centrality measures: A large-scale empirical investigation , J. Learn. Anal. 9 ( 2022 ) 13 - 31 .

[7] D. Tempelaar , B. Rienties , J. Mittelmeier , Q. Nguyen , Student profiling in a dispositional learning analytics application using formative assessment , Comput. Human Behav . 78 ( 2018 ) 408 - 420 .

[8] J. Malmberg , M. Saqr , H. Järvenoja , S. Järvelä , How the monitoring events of individual students are associated with phases of regulation , J. Learn. Anal. 9 ( 2022 ) 77 - 92 .

[9] S. Lee-Cultura , K. Sharma , M. Giannakos , Children's play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach , International Journal of Child-Computer Interaction . 31 ( 2022 ) 100355 . https://doi.org/10.1016/j.ijcci. 2021 . 100355 .

[10] F.N. B. of Education, New national core curriculum for basic education: Focus on school culture and integrative approach , ( 2016 ).

[11] B.E. Act , Perusopetuslaki ( 628 / 1998 ), Finnish Parliament. ( 1998 ).

[12] M. Weller , 25 Years of Ed Tech, Athabasca University Press, 2020 .

[13] R.T. Huang , C.L. Yu , Exploring the impact of self-management of learning and personal learning initiative on mobile language learning: A moderated mediation model , Australasian Journal of Educational Technology . ( 2019 ). https://ajet.org.au/index.php/AJET/article/view/4188.

[14] J. Lei , Y. Zhao , One- to-One Computing: What Does it Bring to Schools?, Journal of Educational Computing Research . 39 ( 2008 ) 97 - 122 .

[15] P. Bruder , Gadgets go to school: The benefits and risks of BYOD (bring your own device ), The Education Digest . 80 ( 2014 ) 15 .

[16] G. Attwell , Personal Learning Environments-the future of eLearning , Elearning Papers . 2 ( 2007 ) 1 - 8 .

[17] J.A. González-Martínez , M.L. Bote-Lorenzo , E. Gómez-Sánchez , R. Cano-Parra , Cloud computing and education: A state-of-the-art survey , Comput. Educ . 80 ( 2015 ) 132 - 151 .

[18] V. Kovanovic , C. Mazziotti , J. Lodge , Learning Analytics for Primary and Secondary Schools, Learning Analytics . 8 ( 2021 ) 1 - 5 .

[19] L.-K. Lee , S.K.S. Cheung , L.-F. Kwok , Learning analytics: current trends and innovative practices , Journal of Computers in Education. 7 ( 2020 ) 1 - 6 .

[20] M. Norris , C. Oppenheim , Comparing alternatives to the Web of Science for coverage of the social sciences' literature , J. Informetr. 1 ( 2007 ) 161 - 169 .