Learning Analytics is currently undergoing a self-reflection process, with calls to bring the learning back into Learning Analytics (LA). At the same time, there is increasing interest in using temporal and sequential analysis to describe and understand learner behaviour, but current work is mostly exploratory, with promising potential for educational interventions. Obtaining sequential patterns from learner data can provide insights into processes that can be missed by descriptive analytics, where the order and timing of actions make a diference. A current scenario is in practical exercises in Data Science Education where teachers cannot access the learners' development process and are therefore unable to properly assess them or provide detailed feedback. Furthermore, there are phases of data science that may not generate data records, such as setting an objective, or intermediate evaluation and reflection. The introduction of Large Language Models (ChatGPT, Github Copilot) as coding assistants has the potential to obtain such data in the form of conversation. By using learning sequence analytics on the interactions between LLM and learners, integrated with the data traces in their programming and analytics environments, we can identify and understand behavioural patterns that can be used to assess, monitor and support the learning process in
Models
sequences.
Learning analytics is working on measuring and optimis- sequence analysis will be implemented in tasks of Data
process, while students work on a task [
Proceedings of the Doctoral Consortium of the 18th European Confer- timely intervention, where the student can be guided tasks. The environment will be based on interactive note- landscape is evolving to meet these demands, equipping books, with a chatbot interface, recording both the pro- students with the necessary data analytics skills that will gramming and conversational interactions. First, a lab become a necessity in their professional lives. experiment will be conducted as a pilot to observe the Data science education (DSE) is the process of teachlearner’s actions in a data science environment, advanc- ing and learning data science skills and concepts, such ing our understanding of behaviours that are often missed as programming, statistics, machine learning, data visuin final assessments. This understanding will be used alization, and data ethics. DSE can be delivered through for research using the learning environment in tasks of various formats, such as online courses, university prodata analytics and data science courses to assess poten- grams, bootcamps, and competitions; with a variety of tial interventions, and their impact on learning outcomes. programming languages and tools [5]. Some of the main In the following, I describe the results of the literature challenges for DSE include the lack of standard topics, review and explain current and future work. practical activities, and methods, which makes it dificult to the scope of a course, and the need to adapt to a wide variety of students’ backgrounds. Furthermore, the 2. Background assessment of activities is reduced to a final product, or automated grading systems on clearly defined tasks, as they depend on pre-written test codes [6].
Learning Sequences, in the context of this work, refer
to the sequences of actions that learners execute during
a learning task, such as solving a programming
problem or creating a concept map from reading materials 2.2. Generative AI in Education
[
Learning analytics, however, can use the data produced While there are several ways educators and instituby the systems in modern online education to scale mon- tions are dealing with AI in the classroom, from banning itoring and assessment capabilities [12]. Furthermore, it to embracing it as a mandatory part of the curriculum, by analysing the sequence of learning actions, an ed- the popularity of LLMs has brought important change ucational system can use all the information available in education [19]. In consequence, there has been an about a learner throughout their current process, pro- explosion of research on Generative AI and education, vide timely support and enable a formative assessment focusing on large-scale risk and opportunity analyses approach for teachers [13]. A growing field of education [20, 19], on the ability of LLMs to assist teachers [21, 22] where many processes are lost in a computer system, but or solve existing homework and exams [23, 24]. The last where learners would benefit from such timely support, ones mention the risks and potentials for students, for is Data Science [6]. instance, that it can facilitate cheating or explain dificult concepts in many diferent ways. 2.1. Data Science Education The focus of this work lies in the overlap of these topics: to use learning sequence analysis to understand and support practical work in data science education, including activities that traditionally happen outside of the learning environment.
Data has revolutionized various scientific disciplines,
paving the way for unprecedented data collection and
analysis capabilities [14]. This data-driven
transformation extends from engineering to computational and
natural science studies, where data analytics techniques can
support processes such as scientific hypothesis
evaluation, system modelling and simulation, and
decisionmaking [
is how can learning analytics be used to support learners when solving complex problems in technology-enhanced learning environments? To address this, I seek to answer
sequential analytics, making the first question how are learning sequences analyzed to describe and support learners’ actions in computer systems?
gramming is involved, integrating a chatbot as a tutor/assistant and analysing the interactions might provide the information we need. This leads to the third question, how are learners using LLMs when dealing with a data science problem? Furthermore, I will use Data Science education as the complex problem to use for evaluation, an exciting and relevant domain, which together with current developments in Artificial Intelligence technology leads to the second question: how do learners work on Data Science problems in a technology-enhanced environment? To answer the first questions in Figure 1, a systematic literature review on learning sequence analytics has been executed and submitted for review. The selected records • How are data records transformed into sequence were published between 2010 and 2023, from a search units for analysis? performed in SCOPUS and Web of Science, including 74 • Which methods are used to analyze learning se- works that were full-research articles, in English, where quences? For which purposes? sequence analysis methods were used on data produced • Which interventions are designed and imple- during a learning task. The analysis considered 5 main mented using the insights from learning sequence sections: the types of tasks to be studied, the purpose analysis? of using sequences for the main analysis, the learning actions that build the sequences, the analysis methods implemented, and the educational interventions designed.
The literature review is currently under review and the main takeaways are the following: The transformation from raw data to sequence units is essential and will define the analysis and interpretability of the results.
Domain-specific knowledge is required to identify the sequence units as learning actions, it facilitates the interpretation of patterns but limits generalizeability. Using • Which are the main actions and phases of a data theory to define the sequence units, such as self-regulated science task? learning actions, can be used to generalize the results, but • How to describe it as a sequence to determine it is not always possible. Sequence analysis methods behaviour patterns and support the learning pro- can be grouped by their scope, and the purpose of the cess? sequence, shown in Figure 2. For example, the scope of the analysis can be to identify strategies in the whole sequence, common when using Process Mining or Probability Models. On the other hand, methods such as Sequential Pattern Mining can be used to find contextspecific tactics, which can be as short as 2-3 learning actions. Support methods, such as machine learning classifiers, are often used to determine learner characteristics • How can we identify the students’ prompting or from the obtained sequential patterns. Most reviewed questioning tactics? articles mention potential educational interventions with • How can we leverage the interactions between the obtained insights, but only a few implement such learners and LLMs to understand the complete interventions, with 2 of the articles evaluating them. data science learning process?
To evaluate the trends in Section 3.1, the next step of my project is to analyse a learning task using learning sequence analysis, with an educational intervention grounded both in the obtained insights and in learning theory. The idea is to use Data Science tasks to understand the tactics and strategies that users implement when working on them. These tasks require both analytical skills and domain knowledge, as well as a certain proficiency in programming and statistics. As such, they can be conceptualized as creative problem solving, with intermediate evaluations and reflections, which can be dificult to observe and assess by teachers [ 5]. This makes them a perfect use case for learning sequence analysis.
A customised Jupyter Notebook system is currently in development, integrating a chat interface and detailed logging capabilities. The system will track the development and execution of Python code used for all data processing and visualization, as well as the outputs and errors from the interpreter. The chatbot interface is able to use diferent LLMs in the background, currently running on OpenAI’s API, but capable of working with a local model as well. A logger on the chatbot interface will be used to store the learner-agent interactions as a sequence of periodic text edits, executed requests and agent responses.
For a pilot experiment, participants will be asked to
work on a data science problem using a Large Language
Model (LLM) as an assistant. This pilot will be performed
in a controlled lab setting, to evaluate the system and
generate a general overview of data science actions and LLM
interactions. Phases, subphases and actions of the data
science process will be identified using examples from
existing Jupyter notebook repositories [25]. LLM
interactions will be coded using problem decomposition, from
a prompt engineering perspective [26]. The integrated
sequence will be analysed using Learning Sequence
Analytics on two scopes: micro-level pattern mining [
For the second study, the system will be used as optional practice for diferent phases of data science, part of a real course. Finally, the presence of certain tactics or the use of problem-solving strategies identified can be used to classify learners, for instance, by their proficiency in programming, statistics, analytics, or any of the disciplines within data science. In future work, we plan to implement a middle layer in the interface between the LLM chatbot and the learner to provide support within the assistant, with interventions based on sequential patterns to optimise the learning support.
The first contribution of this work is the literature review on learning sequence analytics, including the lessons learned on learning tasks, design, methods, and educational interventions in the context of sequence analysis. The second will be a better understanding of the processes that students execute when working on data science tasks. This would also serve as an evaluation of the findings from the literature review on learning sequence analytics. The third is an exploration of the interactions between learners and LLMs, what they ask and how they use them, bringing us closer to understanding the potential of LLMs in education.
for Education and Learning.