Data plays a crucial role in learning analytics research. The analysis of student digital traces has the potential to not only provide new insights into authentic student learning but also to inform interventions and give students feedback on their performance. However, there are many challenges to implementing learning analytics, such as access to data and difficulties with aligning learning activities with appropriate data collection. This paper describes some of these obstacles encountered during a European project that focused on collaborative learning in virtual labs with help of learning analytics and presents recommendations for future learning analytics initiatives.
Access to data is crucial in the field of learning analytics, which focuses on analysis, sense-making
and providing actionable insights from educational digital traces [
The accessibility of data for research purposes varies. It can be limited by creating institutional
barriers ensuring the authorship of publications for a specific group of researchers [
The issues of limited data access for research purposes are common in the learning analytics field
but are rarely discussed in the literature. Leitner et al. [
This paper reports on challenges to accessing and collecting educational data for learning analytics in a case study of a European project focused on collaborative learning in virtual labs. In addition, recommendations for future learning analytics initiatives for more collaborative virtual labs are presented.
The underlying assumption in learning analytics is that “generating more information about how
learning processes unfold can help us better improve them” [16, p. 120]. To improve and better
understand learning processes, educational data is collected, transformed from the raw data into
indicators and analysed and, finally, interventions are developed based on the insights [
The insights from the educational data can be useful for both learners and instructors. Learners
can better understand their learning processes and performance, reflect more, and improve their
selfregulated learning skills [
The pandemic of the Covid-19 has considerably changed teaching and learning in higher education, and laboratories are not an exception. Despite the restrictions of physical contact becoming easier following 2022, post-pandemic practices in education and laboratories have undergone significant changes compared to pre-pandemic times. For virtual laboratories, the exploration has marked a surge in terms of usage, adoption, and research [23]. Although physical interaction with equipment is crucial to get a better sense of experiments, virtual labs may grant students an acceptable level of practical skills they need [24]. For example, students can get a good overall understanding of the practical guidelines, safety procedures, and equipment operation.
From a data ecosystem perspective, virtual labs have generated novel categories of data, creating possibilities to enable learning analytics for further supporting students and optimising the environment of running online labs. Learning analytics is an inter/multi-disciplinary field that has been inspired and guided by diverse disciplines including but not limited to computer science, psychology, and education sciences. The field sustains its definition in 2011 to be the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimising learning and the environments in which it occurs [25].
In exploring the potential and challenges of learning analytics in virtual labs, a recent study by Elmoazen et al. [23] provided an overview of this intersection via a systematic literature review. Elmoazen et al. [23] stated that despite the emergence of learning analytics over a decade ago, the number of articles specifically devoted to virtual laboratories remains few. The authors found out that the 21 studies included in their analysis generally remained in a descriptive research stage rather than explanatory and conclusive. For example, learning analytics was utilised to study the perceptions via self-reported feedback [26]. Learning analytics was also used to explain behaviour patterns [27] and assess inquiry-based educational designs by teachers [28]. Elmoazen et al. [23] discussed that while learning analytics should provide actionable insights to relevant stakeholders (i.e., dashboards, collaborative prompts, recommendations etc.), the current state of the field together with virtual labs is still in its early stages. That is, for instance, little is known about dashboards introduced to both students and teachers in virtual labs, collaboration incentives, and insights to lab scientists/facilitators who moderate the virtual experiments.
Despite the fact that collaboration is a key component of virtual laboratories, teamwork incentives are found to be lacking when it comes to learning analytics and virtual labs, hindering further research and development in this area [29, 23].
The main goal of the European Network for Virtual lab & Interactive SImulated ONline learning 2027 (ENVISION_2027) project is to enhance conventional classroom teaching with virtual laboratories and other digital tools in Bioscience studies. Specifically, the project aims at supporting student engagement and collaboration. In the project, virtual labs were integrated using software from a commercial vendor into multiple biology courses at four partner universities in four countries over three years. The courses were evaluated after each semester and improved for the new iteration. Overall, students reported a positive impact of virtual labs on their learning.
There were some specific goals connected to learning analytics and virtual labs in the project. First, the project aimed to explore ways to implement learning analytics in virtual labs, also real-time learning analytics. Second, learning analytics should have been integrated into the courses to support both student learning and collaborative learning. In addition, learning analytics insights should have informed future course development and teaching.
The commercial virtual lab software, Labster2, selected for this project features a catalogue of over 300 simulations in multiple disciplines and allows embedding quizzes. The instructors are supported with a teacher dashboard and supplemental resources. Previous studies reported that Labster had a positive effect on student motivation, confidence and learning [30, 31].
It was only possible to obtain data about student attempts at solving a simulation with a timestamp and if they were successful. Student activity data, such as click data, were not shared with the project partners. This made the analysis of data using learning analytics techniques challenging, as it was not fine-grained enough to enhance the virtual lab learning environment. In addition, this data was not detailed enough to inform future course design or teaching. Hence, the evaluation of courses was conducted through student and instructor surveys.
As it was not possible to analyse the data from the virtual lab activities, the focus shifted to the other project goals, i.e. collaborative learning. Typically, virtual lab learning activities in the project were not designed as collaborative tasks. Students were required to solve the simulations as homework individually. In addition, Labster does not support collaboration tasks without the use of third party software [32].
To collect student collaboration data, an open-source instant messaging tool, Discord, was chosen. A recent literature review by Craig and Kay [33] on Discord use in Higher Education reported that it is easy to use and access, as well as increases social presence which can lead to increased student learning outcomes. At the same time, using Discord was found to lead to higher distraction levels. Another study found that students using Discord interacted more, and their discussion was less centralized compared to students using a discussion forum in a learning management system [34]. 2 https://www.labster.com/ (last accessed March 2023)
Integrating Discord with Labster was a challenging task. Communicating while working on a simulation would require students to have both tools open on their devices, providing an inconvenient lab/course structure. Alternatively, they would have to switch between tools leading to distraction. In addition, since the virtual lab simulations were designed to be individual tasks, there was little incentive for students to collaborate while solving them other than help-seeking when they got stuck. As virtual lab activities were assigned as homeworks, students would need to coordinate to meet in Discord at the same time to collaborate. A more natural setting would be for students to collaborate on solving the virtual lab task in a face-to-face setting since most classes take place on campus. However, then there would be no need to use an online messaging tool. This would also require additional data collection through video and audio of student collaboration sessions and would be difficult to scale. Finally, Discord allows for both chat and audio/video conferencing. Text data from a chat is less challenging to analyse, but using video or audio for communication would be more convenient for students while solving a task with another digital tool. Although Discord enables downloading of video and audio using additional software, this would require students to participate in the recording and sharing of their sessions with the researchers. As a result of these constraints, the text data collected from Discord was limited, and student collaboration was assessed through surveys.
Digital traces can help discovering more authentic insights into student behaviour in comparison
to, for example, surveys that rely on student self-assessment and self-reporting [
The specific goals for learning analytics implementation in this project faced challenges. The lack of data access limited the analysis options substantially, as student behaviour in the virtual labs could not be analysed. In addition, integration of learning analytics into the courses was not possible since the alternative data collection avenues, such as video and audio, were not scalable. Finally, collaborative learning requires more than “simply placing learners in a group and assigning them a task” [43, p. 1]. The virtual labs in this project were designed to be individual tasks, which resulted in many obstacles in redesigning them to be collaborative learning activities.
This paper describes difficulties and challenges that may be important to consider in other learning analytics initiatives. Although we cannot report on the successful implementation of learning analytics, here are some recommendations based on the experiences from the case study: 1. To map the data available and its granularity with the commercial vendor before deciding on a tool or, if possible, to ask other researchers who worked with a specific vendor about the type and format of data shared. 2. To consider alternative activity data sources if the data from a specific tool is not sufficient, such as multimodal data of students interacting with a tool. 3. To supplement the activity data collection with surveys and other conventional data sources. 4. To align the tools used with the main pedagogical goals, such as supporting collaborative learning, to ensure that alternative data collection scenarios are possible.
This study is co-funded by the Erasmus+ program of the European Union within the project ENVISION_2027, grant number (2020-1-FI01-KA226-HE- 092653). 6. References [38] T. C. Pargman, C. McGrath, O. Viberg, S. Knight, New vistas on responsible learning analytics,
Journal of Learning Analytics 10 (2023) 133–148. doi:10.18608/jla.2023.7781. [39] A. F. Wise, J. P. Sarmiento, M. B. Jr., Subversive learning analytics, in: LAK21: Proceedings of the 11th International Learning Analytics and Knowledge Conference, ACM, 2021. doi:10.1145/ 3448139.3448210. [40] J. Samuelsen, W. Chen, B. Wasson, Integrating multiple data sources for learning analytics— review of literature, Research and Practice in Technology Enhanced Learning 14 (2019). doi:10.1186/s41039-019-0105-4. [41] M. Y. Amare, S. Šimonová, Learning analytics for higher education: Proposal of big data ingestion architecture, SHS Web of Conferences 92 (2021) 02002. doi:10.1051/shsconf/20219202002. [42] C. Guzmán-Valenzuela, C. Gómez-González, A. R.-M. Tagle, A. Lorca-Vyhmeister, Learning analytics in higher education: a preponderance of analytics but very little learning?, International Journal of Educational Technology in Higher Education 18 (2021). doi:10. 1186/s41239-021-00258x. [43] P. A. Kirschner, G. Erkens, Toward a framework for CSCL research, Educational Psychologist (2013) 1–8. doi:10.1080/00461520.2012.750227.