Does Personalised Learning Influence Students' Self- Evaluation of Learning in Digital Learning Environments?⋆

Abstract

This study examines the impact of digital personalised learning (DPL) environments on students' selfevaluation of learning. The DPL environment in this study adapts to individual characteristics such as prior knowledge, metacognition, and motivation. While previous research on DPL has produced mixed results, few studies have explored its influence on self-evaluation of learning, which is associated with higher motivation and improved learning outcomes. This study aims to examine the following: (1) the difference in students' self-evaluation between adaptive and non-adaptive learning environments, (2) the moderating effect of the subject area on this difference, and (3) the impact of the amount of adaptivity on self-evaluation of learning. Data from 2,634 secondary school students using the DPL tool in a large-scale innovation project were analysed through multilevel linear regression. Results showed that students in adaptive learning tracks reported significantly higher self-evaluation scores compared to those in non-adaptive environments. While the subject area did not significantly moderate this effect, students reported higher self-evaluations in Science & Technology than in Social Sciences. Finally, no significant association was observed between the amount of adaptivity and students' self-evaluations. Our study highlights that adaptive learning environments positively influence self-evaluation though this influence does not differ by subject area. Furthermore, the extent to which the learning environment is personalised was not associated with self-evaluation of learning. This study demonstrates the benefits of personalised learning in real-world settings, despite the challenges of controlling variables in such environments.

Keywords

• personalised learning
• digital learning technologies
• self-evaluation of learning 1

1. Introduction

1.1. Digital Personalised Learning

Students vary widely in terms of prior knowledge, experiences, motivation, language proficiency, socio-economic background etc. Addressing these differences in traditional learning environments is a significant challenge for educators. However, advances in digital technology and data analytics now allow the creation of personalised learning environments tailored to individual learner characteristics. These personalised learning environments hold promise in different contexts such as K12-education and the corporate sector where learners have less guidance of an educator who takes into account individual differences but are required to learn autonomously. Digital personalised learning (DPL) is defined as “enabling and supporting learning based upon particular characteristics of relevance or importance to students through technology, potentially adapting to students’ needs by teaching at the right level” [1, p. 10]. Personalised learning supports individual needs and goals by tailoring the learning process to each student's unique characteristics, such as their prior knowledge and motivation [ 2 ]. A concept that is often used in research on how personalised learning can be facilitated through digital learning technologies is adaptivity. Adaptivity can be defined as “the ability of a learning system to diagnose a range of learner variables and to accommodate a learner's specific needs by making appropriate adjustments to the learner's experience with the goal of enhancing learning outcomes” [23, p. 276].

1.2. Effectiveness of DPL

It is hypothesized that if we adapt the learning environment to students’ characteristics, they will learn more effectively. However, empirical findings on its effectiveness have been mixed. For example, a meta-analysis by [ 3 ] showed that adaptivity was not a significant moderator of the effectiveness of digital technologies to train reading. Furthermore, a systematic review of analytics for adaptivity revealed mixed results, with only about half of the studies showing positive effects of adaptation on learning outcomes [ 4 ]. A recent meta-analysis by [ 5 ] showed that personalised learning has an overall medium positive effect on learning achievement. [ 6 ] also investigated the effect of DPL on learning perceptions including students’ motivation and attitudes towards learning and found a small effect size. The effectiveness of DPL is typically assessed using pre- and post-test interventions that compare experimental and control conditions. However, these studies often fail to detect significant effects (e.g., [ 7 ]). This could be attributed to the short duration of interventions, which may only yield small differences between adaptive and non-adaptive conditions, especially in the short term.

Analyzing log data presents an opportunity for a more nuanced understanding of student behavior and outcomes in DPL environments [ 8 ]. Log data offers precise and detailed insights into students’ behavior and performance throughout the learning process. Furthermore, compared to traditional pre- and post-tests, analyzing log data is non-intrusive and does not require shifting tasks from the learning environment to a (standardized) testing format [ 7 ]. For instance, [ 9 ] investigated learning effectiveness by analyzing the participants' usage data, such as the number of attempts at testing themselves, and found that those who made more attempts gained higher post-test scores. [ 10 ] measured learning efficiency by dividing the recorded total number of completed tasks by the number of correctly answered problems. Additionally, they used Bayesian knowledge tracing to generate moment-by-moment learning curves, and they concluded that the curves accurately depicted how well pupils could regulate their learning across time. [ 7 ] applied a modeling approach to assess learning efficiency through the extracted log data where the student's average learning rate within a session and over sessions is modeled. The findings revealed that adaptive digital educational games could promote learning efficiency across game-playing sessions compared to the nonadaptive games. Although log data shows the potential of better understanding students' behavioral or learning patterns in DPL environments, students’ log data are underexplored [ 8 ].

1.3. The Effect of DPL on Self-Evaluation of Learning

The effects of DPL on learning have mainly been studied through self-developed or standardized knowledge tests or self-report questionnaires measuring students’ attitudes or motivation [ 5, 6 ]. However, limited attention has been given to how DPL environments influence students' selfevaluation of learning [ 11 ]. Self-evaluating abilities can be defined as "a personal, unguided reflection on performance for the purpose of generating an individually derived summary of one's own level of knowledge, skill, and understanding in a particular area" [12, p.15]. This concept encompasses both quantity estimates (e.g., "How many task requirements have I met?") and quality estimates (e.g., "How well have I done?") [ 13 ]. Building on this, [ 14 ] further distinguishes between formative selfassessment, aimed at fostering learning during the process, and summative self-assessment, which involves post-task evaluations of learning based on performance. Self-evaluation plays a crucial role in students' academic achievement and self-regulated learning, where they set goals, monitor progress, and adjust strategies to achieve those goals [ 14, 15 ]. These are important skills not only within K12 education, but also in corporate settings where employees often need to train autonomously through digital learning platforms. Self-regulation skills are important for learners to estimate their mastery in certain topics and make decisions about the next best actions within the learning environment.

In digital learning environments, self-evaluation gains particular significance. Online selfevaluation methods offer several advantages over traditional pen-and-paper approaches, such as time-efficient scoring, immediate feedback, flexible assessment formats, and enhanced opportunities for students to reflect on their learning [ 16 ]. By leveraging these digital features, DPL environments have the potential to foster more accurate and meaningful self-evaluations, enabling students to better understand their progress and adapt their learning strategies. Exploring the relationship between DPL and students’ self-evaluation is essential to understand how DPL environments influence not only cognitive outcomes but also students' metacognition.

1.4. Moderator Variables of the Effectiveness of DPL

The effectiveness of DPL is likely influenced by multiple factors, including individual learner characteristics, subject area, and the design of the adaptive system. However, research findings on the moderating effects of these variables remain inconsistent [ 5, 6 ]. Subject area appears to play a role in the variability of DPL outcomes. A recent meta-analysis found that the impact of DPL differs across disciplines, showing small effects in subjects like languages, math, and science; medium effects in psychology; and larger effects in technology-based subjects [ 5 ]. In contrast, [ 6 ] found subject domain was not a significant moderator of the effect of DPL. Moreover, self-evaluation scores may also vary by subject area. For example, students in well-defined subjects like mathematics or science may find it easier to estimate their abilities compared to students in more interpretive or subjective areas like history or literature [ 17 ].

The design of the adaptive system itself is another potential moderator. Some DPL environments rely on simple, rule-based adaptations, while others employ more advanced systems that continuously assess students’ abilities and dynamically adjust tasks [ 18, 23 ]. It is hypothesized that more flexible and sophisticated adaptive systems may yield greater improvements in learning outcomes. Despite these theoretical considerations, empirical evidence remains limited. Existing research has yet to draw definitive conclusions about how these variables interact to influence the effectiveness of DPL, underscoring the need for further research in this area.

1.5. Research Questions

Previous research on DPL has primarily been conducted through controlled experiments, often with fixed instructions, to facilitate comparisons between adaptive and non-adaptive learning environments. While these "efficacy" studies [20] are critical for establishing baseline effectiveness, it is equally important to explore the potential of DPL in real-world settings. Such studies can include larger participant groups, cover a broader range of subjects, and account for the variability inherent in authentic educational contexts. Assessing the impact of DPL through self-evaluation of learning offers a valuable approach. Self-evaluation is less intrusive for students, making it easier to integrate into authentic learning settings. The following research questions were investigated: (1) Is there a difference in students’ self-evaluation of learning between adaptive and non-adaptive learning environments?, (2) Is the difference in students’ self-evaluation of learning between adaptive and non-adaptive learning tracks moderated by the subject area?, and (3) Is students’ self-evaluation of learning influenced by the amount of adaptivity in the learning tracks?.

2. Methodology

This study is part of a wider evaluation of an innovation project in which over 550 schools participated. The aim of this project was to foster DPL in Flemish secondary schools (Belgium). The DPL tool enables teachers to design personalised learning pathways for their students. This personalisation is driven by key moments that include cognitive (e.g., "What is 2 x 2?"), motivational (e.g., "Would you like to learn more about this topic by watching a video or reading a text?"), and metacognitive (e.g., "Do you think you have mastered this topic?") questions (see Figure 1 for an example learning track). Based on how students respond to these key moments, the system assigns them a customized pathway within the learning track. An example of the design of a learning track for computational thinking developed with the DPL tool can be found in [24]. Regarding the students’ self-evaluation in the DPL tool, students are shown the expected learning content and goals at the beginning of each learning track. At the end, they are asked to assess their progress by moving a slider from 'Completely not' to 'Definitely yes,' which corresponds to a numerical score from 1 to 10, reflecting their perceived achievement of the learning goals.

This study utilizes log data from December 2021 to April 2024, and entailed 2,511 students from secondary schools actively participating in the DPL project. Of these, 683 students participated in the adaptive learning environments (including at least one key moment) and 1,828 students in the non-adaptive learning tracks. Subject areas were categorized as Science and Technology (e.g. Mathematics, Chemistry, and Biology) and Social Sciences (e.g. English, History, and Politics). The distribution of students in each category was as follows: Adaptive Science and Technology (n = 21), Nonadaptive Science and Technology (n = 238), Adaptive Social Sciences (n = 73), and Nonadaptive Social Sciences (n = 208). The DPL tool was used by teachers for multiple purposes ranging from instruction of new concepts, exercising learning content, formative assessment, and addressing individual learner needs. The DPL tool was mostly used in the classroom, and to a limited extent at home.

A multilevel linear regression analysis was conducted including three levels because the data structure was hierarchical: measurements were nested within students, and students were nested within teachers. The proportion of variance explained at each level was quantified by the Intraclass Correlation Coefficient (ICC). The analyses were conducted using the R package lmerTest [21].

3. Results

3.1. Adaptive Versus Non-Adaptive Learning Tracks

To answer RQ1, a three-level random intercept regression model was tested. The levels considered were measurements (level 1), students (level 2), and teachers (level 3). The majority of the variance in the student self-evaluation of learning was accounted for at the student level (ICC = 0.51), followed by the teacher level (ICC = 0.25). This suggests that the individual differences among students play a more significant role in explaining the variability in students’ self-evaluation of learning than the differences between teachers. The results of this model (see attached Table 1 and Figure 2) showed that self-evaluation of learning was significantly larger in adaptive learning tracks (M = 7.36, SE = 0.12) compared to non-adaptive learning tracks (M = 7.14, SE = 0.12), although the effect size of the difference was small (Cohen's d = 0.07).

3.2. Influence of Subject Area

Regarding RQ2, to test the moderating effects of subject area, a model including an interaction between adaptivity and the subject area was tested (Model 2). It should be noted that these analyses are based on a smaller dataset due to the fact that part of the learning tracks lacked metadata on subject area. As shown in Table 1 and Figure 3, the moderating effect was not significant (Cohen's d = 0.03, small). Yet, there was a significant main effect of ‘Subject Area’, with Social Science subjects (M = 6.88, SE = 0.31) linked to lower self-evaluation scores compared to Science & Technology subjects (M = 7.58, SE = 0.28), with a small effect size (Cohen's d = 0.28).

3.3. Influence of Amount of Adaptivity

A three-level random intercept regression model was tested. The levels considered were measurements (level 1), students (level 2), and teachers (level 3). The amount of adaptivity was operationalized as the total number of key moments in a learning track. It is assumed that the higher the total number of key moments in a learning track, the more adaptive the learning track is. The results of this model (see Table 2, Figure 4) showed that there was no association between the amount of adaptivity and students’ self-evaluation of learning (Cohen's d = 0.10, small).

4. Discussion

This study investigated the effectiveness of DPL environments on students’ self-evaluation of learning. Specifically, we examined: (1) whether students’ self-evaluation of learning differs between adaptive and non-adaptive environments, (2) the moderating effect of subject area on this relationship, and (3) whether the degree of adaptivity influences students’ self-evaluation of learning. Using multilevel regression analysis, we analyzed students' log data with a three-level model encompassing measurement, student, and teacher levels.

Our findings underscore the influence of learning environment structure on students’ selfevaluation of learning. Students in more personalised environments generally rated their learning experiences more positively, aligning with prior research that highlights the benefits of DPL environments on learning outcomes (e.g., [ 6 ], [ 11 ]). However, the dependent variable in this study— self-evaluation of learning—has received limited attention in previous research, making direct comparisons challenging.

Interestingly, the subject area did not moderate the effectiveness of DPL environments on students’ self-evaluations. This suggests that DPL environments can support positive self-evaluation across diverse subjects, corroborating findings from [ 6 ], who observed no significant differences in effect sizes among learning domains. However, a main effect of subject area was noted, with students rating themselves higher in science and technology subjects compared to social sciences.

Contrary to our expectations, higher levels of adaptivity did not significantly enhance students’ self-evaluation of learning. We hypothesized that greater personalization would lead to better outcomes, but the limited variation in adaptivity levels (ranging from 1 to 3) may have been insufficient to reveal significant differences. This finding suggests the need for further exploration of adaptivity operationalization and their impact on learning.

Several limitations should be considered when interpreting these findings. First, the validity of self-evaluation scores could not be assessed due to the absence of an objective criterion. Selfevaluation is prone to errors such as overconfidence, which may inflate students’ self-assessments relative to their actual performance [ 17 ]. Second, teachers had the flexibility to choose, adapt, or create learning tracks, requiring them to label activities and key moments accurately. Regular checks were conducted, however, given the real-life nature of the dataset and the freedom given to teachers, it is possible that a small portion of the dataset was not labelled correctly (e.g., a cognitive key moment labeled wrongly as a motivational key moment). Nevertheless, given the size of our dataset, these inaccuracies are unlikely to have significantly impacted the results.

To enhance the robustness of future studies, we recommend incorporating objective performance measures, such as knowledge tests, to complement self-evaluation data. Research indicates that the validity of self-evaluation improves with students’ experience in self-assessment and can be further refined by questioning approaches (e.g., absolute vs. relative to peers), anonymity, and alignment with objective benchmarks [22]. Additionally, this study employed a rule-based personalization system designed by teachers. Exploring more advanced, AI-driven adaptive systems could provide insights into the effectiveness of different adaptivity operationalizations [ 18 ].

This study examined the effectiveness of a DPL environment within a realistic, curriculumintegrated setting across various subjects. While our findings primarily pertain to secondary education, they have broader implications for lifelong learning—a key educational goal in the 21st century. As digital learning environments increasingly emphasize autonomy and personalization, fostering students' self-assessment skills will be crucial. Effective tools that support reflection and self-monitoring can empower students, paving the way for autonomous, lifelong learning.

5. Acknowledgements

The Research Foundation - Flanders has financed this research. The research has taken place within the following project: 1244424N: Fostering personalized learning in primary schools through the use of learning analytics dashboards. 6. References [20] C. Shawn Green, D. Bavelier, A. F. Kramer, S. Vinogradov, U. Ansorge, K. K. Ball, U. Bingel, J.

M. Chein, L. S. Colzato, J. D. Edwards, et al., Improving Methodological Standards in Behavioral Interventions for Cognitive Enhancement, J. Cogn. Enhanc. 3.1 (2019) 2–29. doi:10.1007/s41465018-0115-y. [21] A. Kuznetsova, P. B. Brockhoff, R. H. B. Christensen, lmerTest Package: Tests in Linear Mixed

Effects Models, J. Stat. Softw. 82.13 (2017). doi:10.18637/jss.v082.i13. [22] P. A. Mabe, S. G. West, Validity of self-evaluation of ability: A review and meta-analysis., J.

Appl. Psychol. 67.3 (1982) 280–296. doi:10.1037/0021-9010.67.3.280. [23] J. L. Plass, S. Pawar, Toward a taxonomy of adaptivity for learning, J. Res. Technol. Educ. 52.3 (2020) 275–300. doi:10.1080/15391523.2020.1719943. [24] R., Van Schoors, R., S. M., Bhatt, J., Elen, A., Raes, W., Van den Noortgate, F., Depaepe, Design and Development of a Digital Personalized Learning Track: Bridging the Gap between Textual and Visual Programming. International Journal of Designs for Learning, 15.1 (2024) 74-95. https://doi.org/10.14434/ijdl.v15i1.35224

References

[1] L. Major , G. A. Francis , M. Tsapali , The effectiveness of technology‐supported personalised learning in low‐ and middle‐income countries: A meta‐analysis , Br. J. Educ. Technol. 52.5 ( 2021 ) 1935 - 1964 . doi: 10 .1111/bjet.13116.

[2] D. Sampson , C. Karagiannidis , Personalised learning: educational, technological and standardisation perspective , Digit. Educ. Rev. No. (4) ( 2002 ) 24 - 39 .

[3] S. Vanbecelaere , F. Cornillie , D. Sasanguie , B. Reynvoet , F. Depaepe , The effectiveness of an adaptive digital educational game for the training of early numerical abilities in terms of cognitive, noncognitive and efficiency outcomes, Br . J. Educ. Technol. ( 2020 ). doi: 10 .1111/bjet.12957.

[4] M. Ninaus , S. Nebel , A Systematic Literature Review of Analytics for Adaptivity Within Educational Video Games, Front . Educ. 5 ( 2021 ). doi: 10 .3389/feduc. 2020 . 611072 .

[5] A. Tlili , S. Salha , H. Wang , R. Huang , J. Rudolph , R. Weidong , Does personalization really help in improving learning achievement? A meta-analysis , in: 2024 IEEE International Conference on Advanced Learning Technologies (ICALT) , IEEE, 2024 , pp. 13 - 17 . doi: 10 .1109/icalt61570. 2024 . 00011 .

[6] L. Zheng , M. Long , L. Zhong , J. F. Gyasi , The effectiveness of technology-facilitated personalized learning on learning achievements and learning perceptions: a meta-analysis , Educ. Inf. Technol . ( 2022 ). doi:10.1007/s10639-022-11092-7.

[7] D. Debeer , S. Vanbecelaere , W. Van Den Noortgate, B. Reynvoet , F. Depaepe , The effect of adaptivity in digital learning technologies. Modelling learning efficiency using data from an educational game , Br. J. Educ. Technol. 52.5 ( 2021 ) 1881 - 1897 . doi: 10 .1111/bjet.13103.

[8] R. Van Schoors , J. Elen , A. Raes , F. Depaepe , An overview of 25 years of research on digital personalised learning in primary and secondary education: A systematic review of conceptual and methodological trends , Br. J. Educ. Technol. 52.5 ( 2021 ) 1798 - 1822 . doi: 10 .1111/bjet.13148.

[9] M. Liu , E. McKelroy , S. B. Corliss , J. Carrigan , Investigating the effect of an adaptive learning intervention on students' learning, Educ. Technol. Res. Dev. 65.6 ( 2017 ) 1605 - 1625 . doi: 10 .1007/s11423-017-9542-1.

[10] I. Molenaar , A. Horvers , R. S. Baker , What can moment-by-moment learning curves tell about students' self-regulated learning?, Learn. Instr. ( 2019 ) 101206 . doi: 10 .1016/j.learninstruc. 2019 . 05 .003.

[11] H. Xie , H.-C. Chu, G.-J. Hwang , C.-C. Wang, Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017, Comput . & Educ . 140 ( 2019 ) 103599 . doi: 10 .1016/j.compedu. 2019 . 103599 .

[12] K. W. Eva , G. Regehr, “ Iʼll never play professional football” and other fallacies of self-assessment , J. Contin. Educ. Health Prof. 28.1 ( 2008 ) 14 - 19 . doi: 10 .1002/chp.150.

[13] SAGE Handbook of Research on Classroom Assessment , SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States , 2013 . doi: 10 .4135/9781452218649.

[14] H. L. Andrade , A Critical Review of Research on Student Self-Assessment, Front . Educ. 4 ( 2019 ). doi: 10 .3389/feduc. 2019 . 00087 .

[15] K. North , R. Maier , O. Haas (Eds.), Knowledge Management in Digital Change , Springer International Publishing, Cham, 2018 . doi: 10 .1007/978-3- 319 -73546-7.

[16] C . -h. Chen , The implementation and evaluation of a mobile self- and peer-assessment system , Comput. & Educ. 55.1 ( 2010 ) 229 - 236 . doi: 10 .1016/j.compedu. 2010 . 01 .008.

[17] D. Dunning , C. Heath , J. M. Suls , Flawed Self-Assessment, Psychol. Sci. Public Interest 5.3 ( 2004 ) 69 - 106 . doi: 10 .1111/j.1529- 1006 . 2004 . 00018 .x.

[18] M. Sailer , M. Ninaus , S. E. Huber , E. Bauer , S. Greiff , The End is the Beginning is the End: The closed-loop learning analytics framework , Comput. Hum. Behav . ( 2024 ) 108305 . doi: 10 .1016/j.chb. 2024 . 108305 .