Virtual Learning Environments (VLEs) are online educational platforms that combine static educational content with interactive tools to support the learning process. Click-based data, reporting the students' interactions with the VLE, are continuously collected, so automated methods able to manage big, nonstationary, and changing data are necessary to extract useful knowledge from them. Moreover, automatic methods able to explain their results are needed, especially in sensitive domains such as the educational one, where users need to understand and trust the process leading to the results. This paper compares two adaptive and interpretable algorithms (Hoefding Decision Tree and its fuzzy version) for predicting exam failure/success of students. Experiments, conducted on a subset of the Open University Learning Analytics (OULAD) dataset, demonstrate the reliability of the adaptive models in accurately classifying the evolving educational data and the efectiveness of the fuzzy methods in returning interpretable results.
Learning Analytics refers to an iterative process aiming at collecting and analyzing educational
data in order to generate new knowledge that can be used as feedback for all the involved
stakeholders, to improve their tasks [
Particularly, the use of automatic techniques to analyze sensitive data, such as the educational
ones, is gaining attention, since regulation is required. Specifically, automatic analyses must be
nEvelop-O
explainable and trustworthy [
Fuzzy Logic plays an important role in explainability, since it is able to represent uncertain
and vague concepts by using natural language. This leads to interpretable or explainable
results, which are easier to understand for the domain experts than those returned by black-box
algorithms [
However, most of the Learning Analytics literature ignores time, which, on the contrary, is a
critical factor, since the learning process is inherently incremental. An exception is the Deep
Knowledge Tracing (DKT) methodology, which models the student’s learning behavior from the
analysis of previously solved tasks [
In the last years, also thanks to the spreading of the COVID-19 pandemic, distance learning and the usage of Virtual Learning Environments (VLEs) have experienced a steep increase, becoming powerful tools to support higher education throughout the world. VLEs allow also one to continuously collect logs and information, non-stationary by nature, regarding how and when students interact with the educational platform.
In this work, we exploit the evolving nature of students’ behaviors on VLEs, by using two
stream-based classifiers, namely Hoefding Decision Tree (HDT) [
In order to test the aforementioned evolving models, the Open University Dataset has been used, which reports click-stream interactions among students with a VLE.
The rest of the paper is structured as follows. Section 2 briefly details the considered subset of data and the adopted algorithms. Section 3 discusses the obtained results, while conclusions and future developments of our research are depicted in Section 4.
In this paper, a subset of the Open University Learning Analytics Dataset (OULAD), referring
to the academic years 2013 and 2014, has been used1 [
A total of 18 features, grouped into three semantic classes, i.e., demographic information, student performance, and the interactions with the VLE, have been used to describe the behavior of a single student for a given course. Moreover, an additional feature is used for the target class to represent the student’s final outcome, which can assume two values: PASS and FAIL.
Regarding the classification models, we adopted HDTs and FHDTs, whose structure can be updated incrementally while new chunks of labeled semester data become available.
Both considered algorithms are trained by an incremental procedure, made of two main
phases: i) the update of the statistics of the classes (binary outcomes of the students) in both
the internal nodes and the leaves, and ii) the expansion of the tree if certain conditions on some
parameters are fulfilled. The considered parameters are the grace period, the tie threshold, the
split confidence , and the minimum fraction [
FHDT difers from traditional HDT in the following two aspects [ 27]: i) the update of the
statistics of a given node, ii) the use of the fuzzy Information Gain to choose the best splitting
attribute. Concerning the statistics, a training instance in the FHDT can reach more than one
node and leaf because of the fuzzy partition defined for each input attribute. The considered
partition is strong and uniform, thus exactly two output branches are initialized at each split.
The computed statistics at each node are the membership degree, the local fuzzy cardinality of
the whole node, and the fuzzy cardinalities per class in a node. As regards the fuzzy Information
Gain, the Hoefding bound has been modified to consider a local fuzzy cardinality instead of
the usual sum of the instances in a given leaf. More details on FHDTs, which ensure a good
trade-of between their classification performance level, the overall model complexity, and their
explainability, in turn, one of the current hot topic in the specialized literature [28, 29], can be
found in [
To evaluate the efectiveness of the proposed approaches, the experiments have been carried out in an incremental way, i.e., using the so-called Test-the-train approach. This implies that the stream of data is subdivided into chunks (4 in our case, each corresponding to a semester) and that each chunk is used as a test set in advance and, subsequently, as a part of the training set of the considered tree models.
Moreover, since the cardinality of the fuzzy sets used to describe each fuzzy variable is a critical parameter and could afect the results of FHDTs, two fuzzy models have been experimented, considering two diferent granularities for the fuzzy partition for each input variable . Particularly, we set the number of fuzzy sets for each input variable to 3 and 5. Indeed, a lower number would have not been suficient for representing the values of the variables, whilst a higher number would have led to more complex and less interpretable models.
The final voting strategy of both HDTs and FHDTs has been the Adaptive Naive Bayes one.
In order to evaluate the considered models, we focused on the following metrics: the Area Under the Curve (AUC), and the number of leaves of the derived trees, to assess the classification performance and model complexity of the adopted predictive methods, respectively.
Table 1 reports the comparison of the HDT and the FHDTs models, in terms of classification performance and model complexity, for each semester (test set). We can observe that both HDT and FHDT have low classification performance for the first chunk, thus suggesting that the models are not able to correctly represent the incoming data.
However, when the third and the fourth chunks arrive the models are able to adapt their structures, thus leading to high and stable AUC values. For these chunks, models based on fuzzy logic return slightly better results than HDT.
Moreover, the FHDT models need a lower number of leaves if compared with the traditional HDT. This suggests that while the fuzzy models outperform the results given by HDT, they are also able to greatly reduce the complexity, thus resulting in higher interpretability.
Figure 1 shows the model obtained with the FHDT and 3 fuzzy sets per feature, on the training set at the end of the processing (i.e., after the third semester).
Each node represents a fuzzy feature, while the branches stand for the 3 values (low, medium, and high) associated with each fuzzy partition. As previously discussed, the model is compact and thus easy to understand. It can be further explained through IF-THEN rules, leading to the two target classes (PASS and FAIL). From the tree, IF-THEN rules can be easily derived, following the paths from the root to the leaves. To this aim, we consider a zero-order Takagi-Sugeno (TS) fuzzy model [30]. In this case, the antecedent of each rule is expressed through fuzzy sets defining the input variables in the nodes and their values on the branches, while the consequent is expressed through fuzzy singletons corresponding to output classes on the leaves. Formally, the TS fuzzy rules can be defined as: ∶ IF 1 1, ,1
AND … AND
, , THEN = (X) (1) where , ∈ [1, ] identifies the index of the fuzzy set of partition of input variable used in the rule . In the case of the zero-order TS model that we adopted in this work, we consider that the consequent part of each rule can assume only two values, namely PASS or FAIL. Furthermore, in our experiments, we used triangular uniform fuzzy partitions.
Some examples of the final extracted rules are reported in Table 2.
Rule 3, for example, suggests that students with a medium number of intermediate assessments will pass the exam. Instead, if the number of intermediate assessments is high, additional criteria must be verified (e.g., if the age band is high then the student will pass the exam). Finally, if the number of intermediate assessments is low, then the attribute Gender plays a critical role in predicting the failure/success.
In domains such as the educational one, where the final stakeholders are no technicians, models that are easy to understand are preferable since their results are meant to be used as feedback to improve the course design, or the student’s learning behavior, for example.
1 IF (No. of assessment is LOW) AND (Gender is Female) AND (Forum is HIGH) THEN PASS 2 IF (No. of assessment is LOW) AND (Gender is Male) THEN FAIL 3 IF (No. of assessment is MEDIUM) THEN PASS 4 IF (No. of assessment is HIGH) AND (Age band is HIGH) THEN PASS 5 IF (No. of assessment is HIGH) AND (No. prev. attempts is HIGH) THEN FAIL Beyond the results presented so far, a feature importance analysis has been carried out on the HDT model and its fuzzy variants. Figure 2 shows the most relevant features for the classification task, returned by the three algorithms. We can observe that three diferent subsets of features have been returned, but all the models identify the feature Number of assessment as one of the most important one. This feature counts the number of intermediate assessments a given student has performed for a given course. It is related to the students’ success/failure since a higher number of intermediate assessments suggests a constant study, that is more likely to lead to passing the exams. Also, it is interesting noticing that the two fuzzy models have identified features related to the demographic information as relevant, whilst the crisp model focused on the student’s interaction with the VLE.
To conclude, in this preliminary work, a student-oriented subset of the Open University dataset has been incrementally analyzed to verify the efectiveness of the Hoefding Decision Trees, and their fuzzy variants, to correctly predict the students’ outcomes in a degree course. To this aim, information related to four semesters has been sequentially analyzed. Results have shown that the fuzzy algorithms are more able to incrementally adapt the structure of the learned model to the new incoming data. Moreover, they have been proven to be more interpretable and thus more suitable for the educational domain.
Finally, a feature importance analysis has been performed to identify the most relevant features for the predictive task. Whilst the tree algorithms do not agree on the set of the most important features, all of them identified the number of intermediate assessments as a discriminant feature.
Further analyses are necessary to better understand the influence of the diferent categories of features on the students’ assessments. Also, a deeper study on the models’ interpretability, and how this characteristic could help in the adoption of automatic techniques in real scenarios, are needed. To this aim, domain experts will be involved in problem definition and analysis.
Gabriella Casalino acknowledges funding from the Italian Ministry of Education, University and Research through the European PON project AIM (Attraction and International Mobility), nr. 1852414, activity 2, line 1. Gabriella Casalino is a member of the INdAM GNCS research group. The contribution of Pietro Ducange has been partly funded by the Italian Ministry of University and Research (MIUR), in the framework of the Cross-Lab project (Departments of Excellence). Riccardo Pecori is a member of the INdAM GNCS research group. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’00, Association for Computing Machinery, New York, NY, USA, 2000, p. 71–80. URL: https://doi.org/10.1145/347090.347107. doi:10.1145/347090.347107. [27] R. Pecori, P. Ducange, F. Marcelloni, Incremental Learning of Fuzzy Decision Trees for Streaming Data Classification, in: Proceedings of the 11th Conference of the European Society for Fuzzy Logic and Technology (EUSFLAT 2019), Atlantis Press, 2019, pp. 748–755. [28] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, et al., Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI, Information fusion 58 (2020) 82–115. [29] K. Kaczmarek-Majer, G. Casalino, G. Castellano, M. Dominiak, O. Hryniewicz, O. Kamińska, G. Vessio, N. Díaz-Rodríguez, Plenary: Explaining black-box models in natural language through fuzzy linguistic summaries, Information Sciences 614 (2022) 374–399. [30] T. Takagi, M. Sugeno, Fuzzy identification of systems and its applications to modeling and control, IEEE transactions on systems, man, and cybernetics (1985) 116–132.