As e-Learning systems have become gradually prevalent, forcing a (sometimes needed) physical distance between lecturers and their students, new methods need to emerge to fill this enlarging gap. Educators need, more than ever, systems capable of warning them (and the students) of situations that might create future problems for the learning process. The capacity to give and get feedback is naturally the best way to overcome this problem. However, in e-learning contexts, with dozens or hundreds of students, the solution becomes less simple. In this work we propose a system capable of continuously giving feedback on the performance of the students based on the interaction sequences they undertake with the LMS. This work innovates in what concerns the sequences of activity accesses together with the computation of the duration of these online learning activities, which are then encoded and fed into machine learning algorithms. We used a longitudinal experiment from five academic years. From our set of classifiers, the Random Forest obtained the best results for preventing low grades, with an accuracy of nearly 87%.
The current pandemic situation changed how students throughout the world are learning. With education leaving the classrooms and entering students’ homes, e-learning platforms such as Moodle have become popular not only as mere content distribution platforms, but as virtual spaces where students can interact with each other and their professors. However, with the transition to digital learning, face-to-face interactions have been replaced with e-mails and video conferencing. The loss of these kinds of interactions can lead to professors not having a full understanding of their students’ performance, and students might not feel as motivated, resulting in lower grades or even in a complete lack of interest in the course. With these constraints, learning analytics are relevant as the educators are reduced in their capability of continuous feedback that is helpful in their activities. As such, we aim to provide a platform to ease the burden on the educators as to whether a student is performing well in their course. With the introduction of a machine learning and automated statistical analysis tool to the educators, we are in essence aiming to provide them with a smart learning environment focused on a particular online educational content distribution platform. Our research focus on Moodle as the LMS platform, due to its usage in our institution, and consequently to the ease of access to the relevant data. Moodle stores user interaction information in a log file in each course. The fields in the log files are shown in Table 1. The collected information allows for a wider capacity to help educators in understanding the areas their students have the most dificulties in. Moodle allows for the distribution of Resources and Activities by the educators. Resources are defined as static material, used for exposition or consultation, while activities are defined as any item that requires user interaction, such as tests, forums or work submissions.
Thus, a tool capable of analysing the students’ interactions with the platforms and comparing it to a perceived optimal access pattern might provide insights to both professors and students about their potential shortcomings. Such a tool should be able to adapt itself to any course and be able to warn both professors and students should the latter’s actions result in them failing the course. Ideally, the tool should also be explainable so that the people involved can understand why it raised the alerts. The optimal medium for this tool’s analysis is the activity logs of the platforms in use, since they contain both time, student and activity information. It also stands to reason that an educator may design their courses with an optimal track (the order in which resources are accessed) in mind. Comparing this track to a student’s activity records allows for a measurement on how distant the student is from the intended course consumption.
Other works in this field also attempt to create a tool such as the one we envision, with the
more relevant being: Ademi et al.[
In a previous work [
This section describes a paradigm shift from the classification model such that the predictions
can be: 1) extended to any number of resources/activities being undertaken during a course in
an LMS; 2) the computation of the time taken by each student in each activity is enhanced and
ifne-tuned, 3) and, especially, we revise how to compare the optimal track with the student’s
path. In our perspective the time a student spends in an activity or in a resource is also important,
but what we consider to be fundamental is the order by which the resources and activities are
accessed, which is not considered in [
Moodle’s logs contain a number of identifying fields, such as students’ full names, unique identifying IDs and IP addresses. IP addresses are discarded. Names and numeric IDs are used to generate and are replaced with SHA-256-based hashes. The logs can also contain entries pertaining to professors and other faculty staf. It is possible to filter which kinds of users appear in the anonymized logs: students, professors, and admins. This distinction is based on blacklists for professors, educators, assistants and administrators, meaning if a name is not present on any of those lists, then it is categorised as a student.
In order to achieve better generalisation capabilities, our system adapts itself to accommodate all activities provided via Moodle, so that our system does not need to be tailored for each course. For this purpose, the original logs are traversed to find occurrences of existing activities, prefix them with the relevant course identifier and academic year, as well as assign it a single letter to be used in activity strings if it’s deemed to be relevant for analysis. An example of the outcome of this process is shown in Table 2. Activities can be left out of this analysis by means of a blocklist which in the case below, includes the course’s landing page as well as forums.
While it is fair to assume that one student might navigate from one resource/activity to another in minutes or even seconds (being the diferential the time spent in the former one), that is not always the case. That poses a number of issues. For example: if there is no further access from that student in the log; if there is further activity but only on another day, or simply if it would be unreasonable to assume that the student spend that much time on that precise activity. These situations need to rely on a threshold, after which it is can be assumed the student has left the activity. Our approach difers from others in the sense that we do not rely on a pre-established threshold limit equal to all Moodle activities. To handle these issues, as an initial step we take the same approach, and an initial set of times, considered to be reasonable thresholds, are defined for each activity. But then, for each entry we check the activity duration in the data. The data distribution and outliers are considered for extrapolation against our established thresholds and, should the duration be greater than the thresholds, flag them as “doubtful”. The number of interactions that were flagged as doubtful in the our test data (a ”Technical Communication” course) across five academic years were: 11692 in 2015/16 (37.67% of total); 9193 in 2016/17 (35.14% of total), 7689 in 2017/18 (36.64% of total), 4187 in 2018/19 (23.11% of total), and 13079 in 2019/20 (36.75% of total) which makes them fairly consistent along the years. Although one third of the duration times raises concerns, it is interesting to have a system that can assure (up to a good degree) that the duration session of two-thirds of the activities are correctly computed.
Once doubtful duration times are identified, or if they are missing, they are then handled via the
imputation of new values that are taken to be a reasonable approximation of a typical access
duration for the particular activity. The handling of these doubtful times is their replacement
with a more reasonable value depends on the current distribution of the non doubtful duration
values. The average duration and the k-nearest neighbours are the most used methods, which
depend on the size of the sample for that particular activity or resource [
The authors in [
The student activity strings are compared to a string we consider to be optimal, reflecting the way that the educator considers that the course’s information should be ’consumed’. The optimal path will naturally vary by course and by year, and is obtained by asking the educator running the tool to input it. It should be noted that what we call the optimal path is not optimal in a pedagogical sense, because diferent students will have diferent educational needs so what can be considered optimal for one student may not be optimal for another. However, we consider it to be ’scientifically optimal’ because this provides a single point of comparison based on the educator’s project. Also, it is preferable to be warned that a student is in danger of failing when they end up passing the course (false positives) than not receiving any warnings because the system considers that a student will pass the course, then end up failing it (false negatives). To keep the generated strings to reasonable lengths, only activities deemed relevant (and not in a blocklist) are used for generating these strings. These student activity strings are space-separated single characters, or if the activity is a test, the ’Tn’ combination counts as a single character. An example of such string containing 5 activities is: ”A T1 B A T2”. This representation allows us to extract bi- and tri-grams, which are sequences of two or three consecutive words, which in this case represent online activities (or accessed resources). The presence of bi-grams in this context means that there is a pair of activities that are performed in an order, in an analogous way for tri-grams there is an order between three activities. The motivation is that there are coherent sequences to access pedagogic materials. For example, accesses to handouts before undertaking quizzes; reading posts before posting; obtaining templates before submitting papers, etc. These strings allow us to generate new features to complement the features already present in the previous work. These being: a) The String Length: the total activity count on a given access string. Note: it is not the character count of that string, but the activity count; b) Distance: the Levenshtein distance of a student’s access string to the optimal access string; c) Similarity: a score of the best fitting partial substring regarding the optimal string; d) number of 2- and 3-grams (unique) that are common to the optimal string.
First we divide the student grades (originally in a scale from 0 up to 20) into three categories: A (ranging from 0 to 8), B (9 to 11), C (12 to 20). Due to the imbalanced nature of the data, each classifier was tested using diferent sampling techniques to address this imbalance, such as oversampling, undersampling, and class weights. The dataset of initially 150k interactions was reduced to 93k (just student interactions). For the testing we had 625 rows, each agglomerating all features and corresponding to a single student attending the course from school years 2015/16 up to 2018/19. They were split 70/30 for training and testing sets, meaning 437 rows for training and 188 for testing before any class balancing efort. The training set consists of 83 samples (class A), 81 samples (class B) and 273 samples (class C). The testing set consists of 38 samples of class A, 33 samples of class B and 117 samples of class C.
Three types of classification algorithms were tested: Decision Trees (DT), random forests (RF),
and k-Nearest Neighbours (KNN). DT tests were carried out with various maximum depths for
these trees: 6, 7, and 8. The best performing DT had a depth of 8 and utilised class weights. For
RF, tests were carried out in a grid with diferent numbers of trees (100, 150, 200), and maximum
depths (6, 7, 8). The best performing RF used maximum depth 8, with 200 trees, and utilised
oversampling. For KNN, tests were carried out with 2, 4, 8, 10, and 20 neighbours. Together
with oversampling, lower numbers of neighbours produced better results in this category as
opposed to the higher numbers tested. The best KNN run with neighbours=2. The results of
each of the classifiers can be seen in Table 3. Considering the work [
The reported results indicate that the classifiers have a good capability to separate the classes chosen, this is apparent when analysing the achieved metrics. Whilst the data is quite noisy, with each individual feature not being a particularly good indicator on all of their possible scores, the interactions between the features seem to provide the classifiers with suficient data to achieve good results. A potential issue with the system is that the optimal string for one course on a given school year may not yet be applicable to other courses or even diferent instances of the same course. Depending on how long, or short, those strings end up being, it may result in individual features being too clustered around a small range of values being incapable of providing significant discriminatory performance on the classes.
The results achieved in the diferent classifiers show promising accuracy with the best being the random forests classifier. These results indicate there is also a margin for more fine tuning of the classifiers and the narrowing down of the semi-arbitrary characteristics of some of the features, such as the ideal string, which should lead in future research to better results. Furthermore, the ability to choose diferent times for each activity to be considered doubtful, as well as the ability to tune the optimal string to the lecturer’s preferences, allow for customization of the system and adapt to diferent circumstances, courses, and also evaluation schemes.