Generative AI for Learning Analytics (GenAI-LA): Evidence of Impacts on Human Learning

Abstract

The Second GenAI-LA workshop aims to examine the impacts of generative artificial intelligence (GenAI) on human learning. As technological advancements continue to reshape education, GenAI presents new opportunities for various aspects such as personalised learning, automated feedback and so on. However, empirical evidence of GenAI's impacts on human learning remains limited, necessitating the adoption of learning analytics to ofer rigorous and evidence-driven insights on how GenAI afects human learning. This workshop aims to ignite discussions and foster collaboration among a subcommunity of LA researchers and practitioners to scrutinise and envision how LA may shed light on GenAI's impacts on human learning. We received a total of 13 paper submissions. Following a thorough peer review process, we accepted 10 papers. These papers present unique ifndings / directions to the utilisation of LA for enabling empirical evidence regarding how GenAI plays a role in human learning, from the theoretical discussions of concerns in leveraging GenAI, to the practical development and evaluation of GenAI-powered tools in supporting learning.

1. Introduction

Human learning is the dynamic process through which individuals acquire, process and retain knowledge or skills by experience, observation, and more [ 1 ]. It is a lifelong journey that enables personal development of diverse competencies, such as critical thinking and collaboration, which in turn promotes the advancement of our society [ 2 ]. Recently, the innovation in generative artificial intelligence (GenAI) technologies like large language models (LLMs) brings forward a new dilemma for educational stakeholders seeking to integrate these advanced computational tools within learning environments while maintaining instructional integrity and efectiveness. These emerging technologies ofer unprecedented opportunities for personalised learning experiences while raising significant concerns regarding knowledge acquisition authenticity and the development of core cognitive and metacognitive competencies [ 3, 4 ]. The integration of GenAI into authentic pedagogical scenarios necessitates indepth examination of how these tools may transform traditional learning processes, potentially altering the mechanisms underlying knowledge construction and retention [ 1 ]. However, empirical research exploring the diferential impacts of GenAI-mediated learning across diverse student populations and disciplinary contexts remains notably insuficient, creating an urgent need for evidence-based methodological approaches to evaluate these emerging educational paradigms. Learning Analytics (LA), being

CEUR

ceur-ws.org one such approach, ofers a promising avenue to bridge this gap by harnessing individuals’ learning data to enable evidence that facilitates the understanding and the optimisation of human learning and the GenAI-mediated environments in which learning occurs [ 5, 6 ]. However, the adoption of LA to evidence the impacts of GenAI on human learning also comes with its challenges, particularly concerning whether specific measures (e.g., academic performance [ 7 ]) are still appropriate to quantify learning in the era of GenAI. In response to both the promises and the challenges of leveraging LA for educational contexts adopting GenAI, we proposed our second GenAI-LA workshop to ignite discussions among LA researchers and practitioners regarding potential future directions/approaches to enable suitable evidence-based insights of human learning as a result of GenAI-mediated learning.

2. The positive impacts of GenAI on human learning

The promise of GenAI lies in its potential to revolutionise human learning by scaling personalised and timely assistance, diversifying educational resources, and innovating assessment methods [ 1 ].

2.1. Personalised Recommendation for Learning

Inspired by the capability of GenAI technologies to produce contextually relevant responses derived from extensive knowledge base underlying their training data, one of the promising directions frequently discussed in recent literature to benefit human learning from GenAI is the automatic recommendation of tailored content to stakeholders (e.g., learners, educators) [ 1, 8, 9 ]. However, empirical evidence justifying the eficacy of GenAI technologies in generating personalised recommendations tailored to specific educational contexts remains scarce and an ongoing area of research. For instance, Dehbozorgi et al. [ 10 ] proposed a GenAI-powered recommender system based on the Retrieval-Augmented Generation framework, designed to facilitate the implementation of personalised pedagogy in their future research.

To advance this line of research, two accepted studies in our workshop ofered empirical evidence from leveraging GenAI technologies for personalised educational recommendations. Ahmed et al. [11] evaluated ChatGPT’s capability in generating educational recommendations based on predictive learning analytics (e.g., predicting student success and dropout) conducted in pertinent prior studies. Findings from their research highlighted several strengths of recommendations from ChatGPT, including the accuracy, coherence, usefulness, alignment with traditional learning theories (e.g., Constructivism, Cognitivism and Behaviourism), and avoidance of protected student data (e.g., ethnicity, grade band) in the recommendations. In the meantime, the authors suggested future directions to further improve the quality of the recommendations, especially in promoting diversity and inclusion for disadvantaged students as well as fostering higher-order cognitive engagements from learners, potentially by involving ifne-tuned AI models to better align with learning principles. Wang et al. [12] presented LRS4TP, a LLM-based literature recommender system designated to assist higher education students in their early stages of term paper preparation. The system focused on providing personalised feedback and literature recommendation to stimulate students’ refinement of their research scopes, thereby fostering their critical thinking skills. The authors conducted a case study in authentic curriculum settings to underscore the systems’ abilities to reduce teacher workload while maintaining high-quality supervision of student learning.

2.2. Timely Assistance during Learning

The constant accessibility of GenAI technologies to respond to student queries in a timely manner presents significant potential for mitigating existing challenges in education, such as the disproportionate student-teacher ratios, that impede optimal instructional eficacy and learning outcomes [ 13]. As a result, educational researchers are increasingly harnessing GenAI technologies to implement educational tools that ofer real-time assistance to students in specific curriculum settings (e.g., foundational programming [13], database and information systems [14]).

To extend existing evidence to a broader spectrum of educational contexts, two accepted work in our workshop implemented GenAI-powered tools to ofer scalable and timely assistance during student learning. Tashkovska et al. [15] presented Memoire, a GenAI-powered writing assistant leveraging Retrieval-Augmented Generation to support students in reflective writing by integrating their prior reflections with new insights. Their tool ofered three types of GenAI-powered suggestions during student reflections, including critical questions, auto-complete suggestions, and summarising feedback. The initial findings from their piloting evaluation with 17 participants indicated that auto-complete suggestions and critical questions were preferred over summarising feedback, with users finding them more relevant and helpful in overcoming writer’s block (i.e., a condition where a writer struggles to produce new content or experiences a creative slowdown). Likewise, Soliman et al. [16] leveraged Retrieval-Augmented Generation to implement a course acronym tool, BiWi AI Tutor, which responded to student questions regarding course content and the organisation while ofering feedback to students’ submitted written products to scafold students’ metacognitive behaviours (e.g., planning, monitoring) during learning. Their initial evaluations ofered promising evidence regarding the response accuracy achieved by the tool.

2.3. Resource Compilation for Learning

The simplicity of using GenAI technologies to produce textual and digital content (e.g., pictorial illustrations of complex data insights [17]) for instructional purposes presents a promising opportunity to substantially scale up the quality of educational resources and the eficiency of learning design [ 18]. For instance, Dickey and Bejarano [19] introduced the GAIDE framework to guide educators’ adoption of GenAI technologies to assist in their development of course content, leading to reduced time and efort in content creation, without compromising on the breadth or depth of the content. Almatrafi [20] suggested that GenAI technologies could, to some extent, support the establishment of course learning outcomes. However, research to date has commonly underscored the necessity of human oversight to evaluate the quality of the AI-generated content [19, 20, 18]. This demand introduces procedural constraints that potentially counteract the scalability advantages initially aforded by GenAI technologies in educational resource compilation workflows.

To bridge this gap, an accepted work in our workshop by Clark et al. [21] explored the development and evaluation of Aila, a GenAI-powered lesson planning tool designed to enhance the quality and safety of AI-generated educational resources. The study employed an auto-evaluation agent using an LLM-as-a-Judge methodology to assess lesson quality against predefined benchmarks, focusing on multiple-choice quiz dificulty. Through a case study, the researchers compared human and GenAI evaluations, finding that the GenAI evaluator initially imposed stricter standards than human teachers, but improved alignment after refining prompts based on thematic analysis. Their work demonstrates the potential of GenAI technologies to serve evaluation purposes for AI-generated educational content, pointing to a promising direction where the time-consuming human evaluation may be ofloaded to GenAI.

2.4. Automated Assessment of Learning

Several research to date has explored the efectiveness of GenAI technologies in assessing student responses for assessment tasks, reporting substantial alignment in assessment results with educators under certain pedagogical contexts [22, 23]. Dai et al. [24] indicated that feedback generated by GPT-4 for students’ written submissions was more readable and consistent than that by educators. However, it is worth emphasising that the adoption of GenAI technologies to deliver high-quality assessment results (e.g., scores, feedback) autonomously remains a work-in-progress due to issues such as inaccuracy [25] and hallucination [26], which can be detrimental to human learning [27].

Two of the accepted studies from our workshop explored novel methodological approaches to contribute empirical evidence towards bridging the gaps of inaccuracy. The first study by Borchers et al. [28] proposed a hybrid method for improving text classification in open-response assessments by augmenting human-coded datasets with synthetic data generated by GPT-4o, then distilling both into a smaller Bidirectional Encoder Representations from Transformers (BERT) model. Their findings demonstrated that the model for assessing student responses performed best when 80% of the training data was synthetic and 20% was human-coded, with lower temperature settings (0.3) improving stability but limiting model learning, while higher temperature (0.7 and above) introduced variability and occasional performance drops. Unlike most existing research that engages GenAI technologies to directly assess student responses [e.g., 22], evidence from this work presents an intriguing future direction to enable more scalable and efective automatic assessment leveraging GenAI. The study by Zhong et al. [29] examined the efectiveness of knowledge-empowered fine-tuning (KEFT) of GPT models in assessing interdisciplinary learning quality from students’ online posts and essay sections, and reported assessment accuracy comparable to that of human researchers. The achieved performance further motivated the incorporation of the fine-tuned GPT models into their learning analytics platform TopicWise to continue their ongoing research in authentic pedagogical settings.

Another accepted study from our workshop by Ruijten-Dodoiu et al. [30] focused on the evaluation of GenAI-produced feedback. The authors aimed to explore the use of GenAI to provide scalable, iterative feedback on student reflections by designing a Turing-test-inspired experiment to examine whether students can distinguish AI-generated feedback from human feedback and whether students find the feedback meaningful and actionable. Their ongoing empirical investigations are anticipated to yield evidence that potentially catalyses the establishment of more efective and scalable reflective practices within education contexts.

2.5. Summary

Collectively, the empirical findings presented across these studies reinforce the potential of GenAI to augment human learning through tailored recommendations, on-demand assistance, reliable resource creation, and scalable assessments. Yet, it remains critical to approach this rapidly evolving space with both optimism and caution. While GenAI tools may reduce teacher workload, enhance learning experiences, and broaden access to educational resources, their efectiveness hinges on careful implementation and ongoing human oversight. Issues such as fairness, data privacy, model hallucination, and overreliance on AI-powered insights persist as key challenges [ 1 ]. Further, the alignment of AI-generated content with established pedagogical principles across diverse educational contexts, potentially through context-specific fine-tuning, will be paramount for ensuring that GenAI can benefit human learning validly and sustainably. Moving forward, more rigorous, large-scale, and longitudinal evaluations of GenAI’s educational impact are essential. These eforts, combined with interdisciplinary collaborations among educators, technologists, and policymakers, will help chart a path towards harnessing GenAI’s transformative promise while preserving the authenticity of human learning.

3. The challenges in GenAI-mediated context

While educational institutions are increasingly integrating GenAI technologies in teaching and learning (e.g., Cogniti by The University of Sydney [31]), the readily available nature of diverse GenAI-powered tools during learning complicates the validity of traditional assessment [ 1 ]. By focusing solely on the end product of learning (e.g., students’ submitted assessment responses), conventional evaluation methods risk overlooking the extent to which GenAI tools may have shaped, or even fully produced, that output [32]. This raises critical questions about how best to measure learning in an era of pervasive AI assistance.

Two accepted work in our workshop contributed to the discussion of evidencing learning in the era of GenAI. Shah [33] proposed the utilisation of the ICAP (Interactive, Constructive, Active, Passive) framework coping with students’ engagement data (e.g., chat logs, system interaction logs) within the Sherpath AI platform to holistically evidence nursing students’ learning. The author presented an ongoing work aiming to identify the distribution of ICAP engagement modes and their correlation with learning performance so as to inform the design of more efective AI-supported learning environments that ultimately foster students’ knowledge acquisition and critical thinking skills. Brandl et al. [34] discussed the potential of GenAI as a collaborator in problem-solving within learning environments. The authors highlighted that, despite GenAI’s ability to simulate certain aspects of collaborative problem solving (CPS) by ofering structured interactions and responses, it lacks essential human attributes like shared intentionality, empathy, and emotional engagement, crucial features for true collaborators. As a result, the authors argued that interacting with GenAI required AI literacy rather than traditional CPS skills, posing the validity of assessing students’ CPS skills under GenAI-mediated context problematic. They further highlighted that while GenAI technologies could support skill development, they hardly replicate the complexity of human collaboration. It is therefore pivotal to ensure that the adoption of GenAI in authentic pedagogical settings does not replace the critical human elements necessary for human learning.

Conclusively, in an era where GenAI tools increasingly permeate learning, the traditional emphasis on final products risks obscuring genuine evidence of learners’ engagement and critical thinking. Moving forward, educators and educational researchers should explore critical and holistic approaches to designing assessment activities and assessing learning. These approaches should focus not only on the outcomes of AI-supported activities but also on the processes behind them (e.g., harnessing multi-modal data produced during learning [18]) to ensure that technology remains a catalyst for deeper learning rather than a substitute for it. Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Springer Nature Switzerland, Cham, 2024, pp. 63–70. [11] H. Ahmed, H. Kayaduman, S. López-Pernas, M. Tukiainen, M. Saqr, User-centric evaluation of genai alignment and recommendations based on predictive learning analtycis, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [12] X. Wang, N. Duong-Trung, R. R. Bhoyar, A. M. Jose, Llm-based literature recommender system in higher education – a case study of supervising students’ term papers, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [13] S. Vemula, Enriching python programming education with generative ai: Leveraging large language models for personalized support and interactive learning, in: 2024 IEEE Frontiers in Education Conference (FIE), 2024, pp. 1–8. doi:10.1109/FIE61694.2024.10893561. [14] A. T. Neumann, Y. Yin, S. Sowe, S. Decker, M. Jarke, An llm-driven chatbot in higher education for databases and information systems, IEEE Transactions on Education 68 (2025) 103–116. doi:10.1109/TE.2024.3467912. [15] M. Tashkovska, S. P. Neshaei, P. Mejia-Domenzain, T. Käser, Memoire: Harnessing generative ai to bridge the metacognitive gap in reflective writing, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [16] H. Soliman, H. Kotte, M. Kravčík, N. Pengel, N. Duong-Trung, Retrieval-augmented chatbots for scalable educational support in higher education, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [17] M. E. Milesi, R. Alfredo, V. Echeverria, L. Yan, L. Zhao, Y.-S. Tsai, R. Martinez-Maldonado, ”it’s really enjoyable to see me solve the problem like a hero”: Genai-enhanced data comics as a learning analytics tool, in: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, CHI EA ’24, Association for Computing Machinery, New York, NY, USA, 2024. URL: https://doi.org/10.1145/3613905.3651111. doi:10.1145/3613905.3651111. [18] M. Giannakos, R. Azevedo, P. Brusilovsky, M. Cukurova, Y. Dimitriadis, D. HernandezLeo, S. Järvelä, M. Mavrikis, B. R. and, The promise and challenges of generative ai in education, Behaviour & Information Technology 0 (2024) 1–27. URL: https://doi.org/10.1080/0144929X.2024.2394886. doi:10.1080/0144929X.2024.2394886. arXiv:https://doi.org/10.1080/0144929X.2024.2394886. [19] E. Dickey, A. Bejarano, GAIDE: A Framework for Using Generative AI to Assist in Course Content Development , in: 2024 IEEE Frontiers in Education Conference (FIE), IEEE Computer Society, Los Alamitos, CA, USA, 2024, pp. 1–9. URL: https://doi.ieeecomputersociety.org/10.1109/FIE61694. 2024.10893132. doi:10.1109/FIE61694.2024.10893132. [20] O. Almatrafi, Assessing chatgpt’s capability to generate course learning outcomes, in: 2024 7th International Conference on Information and Computer Technologies (ICICT), 2024, pp. 527–531. doi:10.1109/ICICT62343.2024.00092. [21] H.-B. Clark, O. Henkel, L. Benton, M. Dowland, R. Budai, I. K. Keskin, E. Searle, M. Gregory, M. Hodierne, W. Gayne, J. Roberts, Improving quality and safety in ai-generated lessons, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [22] O. Henkel, L. Hills, A. Boxer, B. Roberts, Z. Levonian, Can large language models make the grade? an empirical study evaluating llms ability to mark short answer questions in k-12 education, in: Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 300–304. URL: https://doi.org/10.1145/3657604. 3664693. doi:10.1145/3657604.3664693. [23] Z. Yan, R. Zhang, F. Jia, Exploring the potential of large language models as a grading tool for conceptual short-answer questions in introductory physics, in: Proceedings of the 2024 9th International Conference on Distance Education and Learning, ICDEL ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 308–314. URL: https://doi.org/10.1145/3675812. 3675837. doi:10.1145/3675812.3675837. [24] W. Dai, Y.-S. Tsai, J. Lin, A. Aldino, H. Jin, T. Li, D. Gašević, G. Chen, Assessing the proficiency of large language models in automatic feedback generation: An evaluation study, Computers and Education: Artificial Intelligence 7 (2024) 100299. URL: https://www.sciencedirect.com/science/ article/pii/S2666920X24001024. doi:https://doi.org/10.1016/j.caeai.2024.100299. [25] I. Chamieh, T. Zesch, K. Giebermann, LLMs in short answer scoring: Limitations and promise of zero-shot and few-shot approaches, in: E. Kochmar, M. Bexte, J. Burstein, A. Horbach, R. LaarmannQuante, A. Tack, V. Yaneva, Z. Yuan (Eds.), Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 309–315. URL: https://aclanthology.org/2024.bea-1.25. [26] Q. Jia, J. Cui, R. Xi, C. Liu, P. Rashid, R. Li, E. Gehringer, On assessing the faithfulness of llmgenerated feedback on student assignments, in: B. Paaßen, C. D. Epp (Eds.), Proceedings of the 17th International Conference on Educational Data Mining, International Educational Data Mining Society, Atlanta, Georgia, USA, 2024, pp. 491–499. doi:10.5281/zenodo.12729868. [27] C. Figueras, C. Rossitto, T. Cerratto Pargman, Doing responsibilities with automated grading systems: An empirical multi-stakeholder exploration, in: Proceedings of the 13th Nordic Conference on Human-Computer Interaction, NordiCHI ’24, Association for Computing Machinery, New York, NY, USA, 2024. URL: https://doi.org/10.1145/3679318.3685334. doi:10.1145/3679318.3685334. [28] C. Borchers, D. R. Thomas, J. Lin, R. Abboud, K. R. Koedinger, Augmenting human-annotated training data with large language model generation and distillation in open-response assessment, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [29] T. Zhong, G. Zhu, S. C. Low, S. Liu, Towards learning analytics for interdisciplinary learning: Leveraging knowledge-empowered fine-tuned gpt models, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [30] P. A. M. Ruijten-Dodoiu, M. Oliveira, E. Ventura-Medina, Towards scalable ai feedback systems: Preparing a turing-test-inspired experiment, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [31] D. Liu, Letting educators take control of generative ai to improve learning, teaching, and assessment, 2023. URL: https://educational-innovation.sydney.edu.au/teaching@sydney/ letting-educators-take-control-of-generative-ai-to-improve-learning-teaching-and-assessment/. [32] Y. Li, L. Sha, L. Yan, J. Lin, M. Raković, K. Galbraith, K. Lyons, D. Gašević, G. Chen, Can large language models write reflectively, Computers and Education: Artificial Intelligence 4 (2023) 100140. URL: https://www.sciencedirect.com/science/article/pii/S2666920X2300019X. doi:https: //doi.org/10.1016/j.caeai.2023.100140. [33] M. Shah, Learning analytics and generative ai: Mapping cognitive engagement in nursing education, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025. [34] L. Brandl, C. Richters, N. Kolb, M. Stadler, Can generative artificial intelligence ever be a true collaborator? rethinking the nature of collaborative problem-solving, in: Proceedings of the 2nd Workshop on Generative AI for Learning Analytics (GenAI-LA), 2025.

References

[1] L. Yan , S. Greif , Z. Teuber , D. Gašević , Promises and challenges of generative artificial intelligence for human learning , Nature Human Behaviour 8 ( 2024 ) 1839 - 1850 .

[2] P. Jarvis , Towards a comprehensive theory of human learning , Routledge , 2012 .

[3] Y. Fan , L. Tang , H. Le , K. Shen , S. Tan , Y. Zhao , Y. Shen , X. Li , D. Gašević , Beware of metacognitive laziness: Efects of generative artificial intelligence on learning motivation, processes, and performance , British Journal of Educational Technology ( 2024 ).

[4] M. Stadler , M. Bannert , M. Sailer , Cognitive ease at a cost: Llms reduce mental efort but compromise depth in student scientific inquiry , Computers in Human Behavior 160 ( 2024 ) 108386 . URL: https://www.sciencedirect.com/science/article/pii/S0747563224002541. doi:https://doi.org/10. 1016/j.chb. 2024 . 108386 .

[5] H. Khosravi , O. Viberg , V. Kovanovic , R. Ferguson , Generative ai and learning analytics , Journal of Learning Analytics 10 ( 2023 ) 1 - 6 .

[6] L. Yan , R. Martinez-Maldonado , D. Gasevic , Generative artificial intelligence in learning analytics: Contextualising opportunities and challenges through the learning analytics cycle , in: Proceedings of the 14th Learning Analytics and Knowledge Conference , LAK '24, Association for Computing Machinery, New York, NY, USA, 2024 , p. 101 - 111 . URL: https://doi.org/10.1145/3636555.3636856. doi: 10 .1145/3636555.3636856.

[7] R. Deng , M. Jiang , X. Yu , Y. Lu , S. Liu, Does chatgpt enhance student learning? a systematic review and meta-analysis of experimental studies , Computers & Education 227 ( 2025 ) 105224 . URL: https://www.sciencedirect.com/science/article/pii/S0360131524002380. doi:https://doi.org/10. 1016/j.compedu. 2024 . 105224 .

[8] Q. Lang , M. Wang , M. Yin , S. Liang , W. Song, Transforming education with generative ai (gai): Key insights and future prospects , IEEE Transactions on Learning Technologies 18 ( 2025 ) 230 - 242 . doi: 10 .1109/TLT. 2025 . 3537618 .

[9] T. K. Chiu , Future research recommendations for transforming higher education with generative ai , Computers and Education: Artificial Intelligence 6 ( 2024 ) 100197 . URL: https:// www.sciencedirect.com/science/article/pii/S2666920X23000760. doi:https://doi.org/10.1016/ j.caeai. 2023 . 100197 .

[10] N. Dehbozorgi , M. T. Kunuku , S. Pouriyeh , Personalized pedagogy through a llm-based recommender system , in: A. M. Olney , I.-A. Chounta , Z. Liu , O. C. Santos , I. I. Bittencourt (Eds.),