The Prerequisite Relation Learning (PRELEARN) task is the EVALITA 2020 shared task on concept prerequisite learning, which consists of classifying prerequisite relations between pairs of concepts distinguishing between prerequisite pairs and non-prerequisite pairs. Four sub-tasks were defined: two of them define different types of features that participants are allowed to use when training their model, while the other two define the classification scenarios where the proposed models would be tested. In total, 14 runs were submitted by 3 teams comprising 9 total individual participants.
The present paper provides an overview of the systems participating to PRELEARN, the first shared task on automatic prerequisite learning between educational concepts.
In the past decades we have witnessed a great
revolution in the field of Education: advancement
of technologies drastically transformed the
teaching method and the setting of the learning
process thanks to the raise of e-learning platforms
and electronic educational materials. While so far
they’ve been mainly used in lifelong learning, the
current pandemic situation made very clear that
distant learning is a valuable resource at all
educational levels. This new era in education is
commonly referred to as Education 4.0
Copyright c 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). there is still much work to do to develop usable and scalable personalisation systems, much of the attention has been devoted to building and testing the building blocks of such applications.
The massive use of distance learning platforms has shed light on the need of developing intelligent agents able to support both students and teachers by, e.g., automatically identifying educational relations between learning concepts. Educational resources are designed to guide students through learning paths consisting of concepts related to each other. Among all pedagogical relations, prerequisite is the most fundamental since it establishes which sequence of concepts allows students to have a full understanding of the domain. In fact, the order in which concepts are presented to the learner plays a crucial role in avoiding student’s frustration and misunderstandings while approaching a new topic, so teachers are very careful to organise the content of their learning materials accordingly and to highlight relevant connections to their students. Doing this automatically is still challenging from many perspectives.
The NLP community has tackled automatic
prerequisite learning in the past with the goal of
integrating prerequisite relations in systems for, e.g.,
curriculum planning
Based on the works available in the
literature, we distinguish prerequisite learning systems
in two main categories: 1) those based on
relational metrics and 2) those on machine
learning approaches. Relational metrics are designed
to capture the strength of the relation between
co-occurring concepts and identify pairs of
concepts obtaining low values as non-prerequisites.
The RefD metric
Unfortunately, the results obtained by those
systems are not directly comparable: their approaches
are based on different assumptions of what a
concept is and which are the distinctive features
for a prerequisite relation. Moreover, knowledge
structures defined by domain experts are not
always easily available or are missing for some
domains. With PRELEARN, we are proposing the
first shared task on automatic prerequisite
learning, at least to the best of our knowledge.
Located in the context of EVALITA 2020
evaluation campaign
PRELEARN (Prerequisite Relation Learning) is a shared task on concept prerequisite learning which consists of classifying prerequisite relations between pairs of concepts. This is the first time, to the best or our knowledge, that automatic prerequisite learning is addressed in a shared task. PRELEARN challenges participants to test their models for automatic prerequisite learning on four different domains and four training scenarios. 2.1
For the purposes of this task, prerequisite relations learning is proposed as a binary classification problem of concept pairs: given a pair of concepts (A, B), we ask to predict whether or not concept B is a prerequisite of concept A. We define a “concept” as single or multi word domain terms corresponding to the title of a page on the Italian Wikipedia: Prodotto scalare and Aritmetica are both concepts of the precalculus domain and are also the titles of two Italian Wikipedia pages. Prerequisite relations instead are dependency relations that naturally occur between educational concepts determining their learning precedence.
Consider the knowledge structure proposed as an example in Figure 1. Here, nodes represent concepts while links identify the prerequisite relations that connect them. According to the graph, “Aritmetica” is a prerequisite of “Potenza” since, if a student wants to understand what “Potenza” is, he/she has to know “Aritmetica” first. Hence, we formally define a prerequisite relation as a relation connecting a target and a prerequisite concept if the second has to be known in order to understand the first. In other words, the Wikipedia page of the prerequisite concept contains the prior knowledge required to understand the content of the Wikipedia page of the target concept. 2.2
We defined four sub-tasks for addressing automatic concepts prerequisite learning: two of them concern the model used by participants for tackling the task, the other two distinguish different classification scenarios where the proposed model can be tested. In order to make a valid submission, we asked participants to submit at least one model complying with at least one of these settings: i) Raw features setting (RF): a model that acquires information only from raw text (e.g. textual content of the Wikipedia pages offered as training set, corpora for acquiring distributional representations, etc.); ii) Raw and structured features setting (RnS): a model that can rely both on raw text and structured information (e.g. Wikipedia graph structure of a domain and metadata of a Wikipedia page, DBpedia, page hierarchical structure in terms of sections and paragraphs, etc.).
Each submitted model was tested in two evaluation scenarios, defined as follows: i) In-domain scenario: the model(s) can be trained on data belonging to any domain, including the one appearing in the test set; ii) Cross-domain scenario: the model(s) can be trained on data belonging to any domain but the domain of the test set.
Overall, we defined a total of four sub-tasks: 1) RF setting in an in–domain scenario; 2) RF setting in a cross–domain scenario; 3) RnS setting in an in–domain scenario; 4) RnS setting in an cross–domain scenario.
Only few work in the literature test their
systems in a cross-domain scenario: our previous
attempts in this direction
Metrics. Evaluation of participants’ systems outputs was carried out on four balanced datasets, one for each domain, used for both in– and cross– domain evaluation. The size of the test sets is reported in Table 1. Each sub-task (i.e. each model on each scenario) was evaluated independently from the others by using standard metrics, such as Accuracy (A), Precision (P ), Recall (R) and F1-score (F1). Since the test sets are balanced, we used Accuracy metric to rank participants’ submitted runs.
Baseline. We used for all settings a linear SVM classifier trained using two binary features capturing the presence of a mention of concept B/A in the text of the Wikipedia page of concept A/B. Each feature returns 1 if the name of concept B/A is mentioned in the text of the Wikipedia page of concept A/B, while it returns 0 otherwise. 3
We relied on ITA-PREREQ dataset
The construction of ITA-PREREQ was carried
out as follows, as described in
PRELEARN participants were provided, upon request, with five files: a “concept pairs file” for each of the four domains containing the labelled concept pairs and one “Wikipedia pages file” containing the raw text and the link of the Wikipedia pages referring to the concepts appearing in the dataset. Here’s an example of the pairs contained in the “concept pairs file”: Riflessione interna totale,Luce,1 Plasticita’ (fisica),Durezza,0 ...
Campo magnetico,Magnete,1 1https://github.com/attardi/wikiextractor dataset and overall. The number of concepts and pairs varies for each domain: while Geometry and Data Mining have a comparable amount of concepts, the latter shows a significantly smaller number of labelled pairs. It is interesting to note that, although not being the richer domain in terms of concepts, Physics shows the higher number of relations. As can be noted, regardless of the domain the dataset is strongly unbalanced since the majority of concept pairs do not show a prerequisite relation (Non-PR Pairs). For each domain we split the pairs into a portion of training and a portion of test data. For the test portion, we defined a fixed number of pairs to include (i.e. 200 pairs), with the exception of Data Mining where, given the limited number of total pairs, we included only 99 pairs. The distribution of prerequisite and nonprerequisite labels was balanced (50/50) for each domain only in the test datasets. 4
Following a call for interest, 16 teams registered for the task and thus obtained the training data. Eventually, three teams submitted their predictions, for a total of 14 runs, each executed on all four domains of the dataset. Two teams participated in all four sub-tasks while one team submitted results only for the two sub-tasks involving the RF setting. A summary of participants is provided in Table 2.
NLP-CIC
PR Pairs 159 (30.40%) 432 (24.71%) 415 (17.14%) 508 (26.51%) 1,514 (22.91%) editors and readers) over the last year. The second model (Complex+wd) is an improved version of the first one: it takes as input the same set of features along with the Wiki-data embedding of each concept appearing in the concept pairs of ITA-PREREQ dataset.
B4DS
UNIGE SE
In this section we provide both a discussion of the approaches and an analysis of the results reported in Tables 3 and 4.
Participants experimented with more classical machine learning algorithm as well as with Neural Networks (NN): we received results computed exploiting 7 different systems, 4 trained using only raw text features (RF setting) and 3 exploiting also structural information (RnS setting). Considering their average performances across all four domains, all systems outperformed the baseline. In this Section, we describe the results obtained by the submitted models and compare their performances on the official test set based on their average accuracy scores over the four domains (column AVG in the Tables). 5.1
In–Domain Scenario. As shown in Table 3,
overall the model showing the best performances
is Italian BERT, achieving an average accuracy
score of 0.887 in the RF setting. Such result
is not surprising if we consider the
state-of-theart performances obtained by recent Neural
Language Models in the resolution of downstream
NLP tasks. However, results obtained by BERT
show only a small gap with respect to some of
the other models. For instance, B4DS’ Model
1, exploiting a decision tree based on XGBoost
framework and trained using both word
embedding and handcrafted features, achieved 0.866
accuracy thus gaining the second place in the in–
domain scenario. Similar competitive results are
obtained by the Complex+wd model submitted by
NLP-CIC team: this model combines Wiki-data
embedding of each concept with a set of manually
defined features that measure concept
complexity and were designed to solve the task of
complex word identification
Among submitted systems, only three didn’t exploit word embeddings: NLP-CIC team submitted a tree-ensemble learner trained using only complexity features, and UNIGE SE team used two versions of a two-layer NN trained with different sets of handcrafted features to comply with settings requirements. The results obtained by these models provide some interesting insights on the role of raw and structural features for solving the task. First, we observe that exploiting raw textual features based on lexical similarity and topic modelling (UNIGE SE NN in the RF setting) only slightly outperforms the baseline, thus, when no lexical features are available, it seems more useful to rely on structural information. Anyways, complexity-based features exploited by NLP-CIC are more informative for prerequisite learning task than Wikipedia category and link structure. The intuition behind the NLP-CIC team approach is that less complex concepts are prerequisite for the more complex ones and, considering that the results are only slightly below those obtained using word embeddings, the intuition that complexity is involved in the process of defining prerequisite sequences seems confirmed.
domain evaluation scenario (see Table 4), we observe only small variations in the ranking of the submitted systems. In spite of this, we also observe a consistent drop of the accuracies obtained by the submitted systems.
Considering again the average accuracy scores, BERT model proved to be the best performing model also in this scenario. Interestingly, this time NLP CIC’s Complex+wd model outperforms B4DS’s Model 1: both models are trained using both word embeddings and handcrafted features, with the latter being more useful possibly because capturing domain independent properties. The different performances of the two systems could be again due to the higher effectiveness of complexity-based features for identifying prerequisite relations. Consequently, these results suggest that, unlike the in-domain scenario, lexical information are not enough to identify prerequisite relations. Nevertheless, lexical features proved somehow useful since using handcrafted features only, as in the case of Complex NLP-CIC model and the NN models submitted by UNIGE SE team, is outperformed by B4DS’s Model 2 (based solely on word embeddings). 5.2
Focusing on the differences between the four domains, we observe that for almost all submitted systems the results obtained on concept pairs belonging to the Data Mining domain are lower than the others. This is especially true for the cross– domain scenario and seems to corroborate what was already stated in Miaschi et al. (2019), namely that Data Mining is a relatively new and more specialised topic that presents shorter pages and, therefore, that contains less clear prerequisite relationships. Nevertheless, the model submitted by the UNIGE SE team for the RF setting achieved the lowest results when tested on concept pairs belonging to the Physics domain.
With the exception of the UNIGE SE’s RF model in the cross–domain setting, all systems achieved best (and similar) results when classifying Geometry and Precalculus concepts pairs. This might be due to the fact that these two domains are more fundamental and broad subjects and, therefore, present more clear learning dependencies expressed through Wikipedia. Furthermore, since Geometry and Precalculus share more lexicon that the others, we believe that the models can take advantage of this overlap to better classify concept pairs, especially for the cross–domain evaluation setting. 6
Automatic prerequisite learning was for the first time the focus of a dedicated shared task. In particular, PRELEARN task was aimed at comparing the performances of different approaches and models tested within and across the four domains of ITA-PREREQ dataset. Although the results of 14 submitted runs were all above baseline, we observe several differences within the proposed settings and across domains. In particular, results suggests that automatic prerequisite learning is more affected by the predictors rather than by the classification models. Results also confirm that the RF cross–domain setting is the most challenging scenario. Nevertheless, BERT achieved best scores in both RF settings, also outperforming models trained with structural features extracted from the knowledge structure of Wikipedia.
For the future, it would be interesting to test the impact of hand-crafted features combined with a contextual language model such BERT and, considering the effectiveness of complexity–based features, explore the contribution of predictors encoding text readability properties in prerequisite learning systems.