We present our systems and findings for the prerequisite relation learning task (PRELEARN) at EVALITA 2020. The task aims to classify whether a pair of concepts hold a prerequisite relation or not. We model the problem using handcrafted features and embedding representations for in-domain and cross-domain scenarios. Our submissions ranked first place in both scenarios with average F1 score of 0:887 and 0:690 respectively across domains on the test sets. We made our code freely available1.
A prerequisite relation is a pedagogical relation that indicates the order in which concepts can be presented to learners. The relation can be used to guide the presentation sequence of topics and subjects during the design of academic programs, lectures, and curricula or instructional materials.
In this work, we present our systems to
automatically detect prerequisite relations for
Italian language in the context of the PRELEARN
shared task
The combination of these settings defined the four PRELEARN subtasks. Formally, a prerequisite relation exists between two concepts if one has to be known beforehand in order to understand the other. For the PRELEARN task, given a pair of concepts, the relation exists only if the latter concept is a prerequisite for the former. Therefore, the task is a binary classification task.
We approach the problem from two perspectives: handcrafted features based on lexical complexity and pre-trained embeddings. We employed static embeddings from Wikipedia and Wikidata, and contextual embeddings from Italian-BERT model. 2
Prerequisite relation learning has been mostly
studied for the English language
This section describes the data analysis, the features we used to model the task, and the system we finally submitted to the PRELEARN competition. 3.1
The dataset provided by the organizers includes the concept pairs splitted into the following domains: ’data mining’, ’geometry’, ’precalculus’ and ’physics’. The dataset contains the list of concepts with a link to the corresponding Wikipedia article. The first paragraph of such article is named the concept description. All concept descriptions are cleaned in order to facilitate the extraction of information from the text, e.g. the mathematical expressions are already tagged using this pattern formula <number>.
Table 1 displays the number of samples and the distribution over the prerequisite relations (positive samples) across domains for the training set. The test sets in turn exhibits a 50-50 distribution over positive and negative samples.
The only preprocessing we did was lowercase the concept description and remove line-breaks.
Data mining Geometry Precalculus Physics The following are the set of features we experiment with:
Complexity-based: a set of handcrafted
features intended to measure how complex a concept
is. The rationale is that less complex concepts are
prerequisites for the more complex ones. We used
some features that have been found effective for
the task of complex word identification
Concept-to-Concept features: they aim to model the relation between the concept pairs, specifically we evaluate whether a concept appears as a sub-string in the title or description of the other concept. We did this in both directions resulting in two features. We also represent the domain they belong to as a one-hot vector.
Wiki-embeddings: We map each concept
identifier to their corresponding Wikipedia title and
Wikidata identifier using the Wikidata Query
Service3. Then, we obtain the 100 dimensional
vector for each Wikipedia title from a pre-trained
Wikipedia embedding4
Italian-BERT features: We used a pre-trained uncased version of Italian BERT (base model)6 provided by the MDZ Digital Library team (dbmdz) trained on 13GB of text mainly from 2https://pageviews.toolforge.org 3query.wikidata.org 4http://wikipedia2vec.s3.amazonaws. com/models/it/2018-04-20/itwiki_ 20180420_100d.pkl.bz2
5https://dl.fbaipublicfiles.com/ torchbiggraph/wikidata_translation_v1. tsv.gz
6https://huggingface.co/dbmdz/ bert-base-italian-uncased Scenario in-domain in-domain in-domain cross-domain cross-domain cross-domain
Resources raw-text structured structured raw-text structured structured
Italian-BERT
Complex+wd
Complex
Italian-BERT
Complex+wd
Complex
Wikipedia and other text sources. With this model,
we get the 768 dimensional vector representation
for a sequence corresponding to the [CLS] token
as in the original implementation of BERT
Considering the proposed features and our experimental results at Section 5, we proposed the following three systems to address both, in-domain and cross-domain scenarios. For the in-domain scenario we trained with a combination of all the training samples per domain. In the same way, we combined the remaining three domains for each cross-domain experiment (i.e. excluding samples from the target domain).
Complex: a completely handcrafted machine learning system, it uses all the complexity-based and Concept-to-Concept features (except the domain vector for cross-domain scenario), and we normalize the features using Z-score normalization. This system uses a tree-ensemble learner as classifier7 with the default parameters provided by Breiman (2001)8. This system participated under the structured resource setting because the “concept view frequency” feature is structured information.
Complex+wd: an improved version of the Complex system by only concatenating the Wiki7Other classifiers were tested and obtained lower peformance
8https://cran.r-project.org/web/ packages/randomForest/index.html data embedding of each concept in the concept pair to the feature set. This system participated under the structured resource setting as well. We decided to not include the Wikipedia embeddings considering the ablation analysis we present in Table 4.
Italian-BERT: a single layer neural network
mapping the 768 features from the [CLS] to the
output space of dimension 2 as a sequence pair
classification task. In addition, the pre-trained
weights of the base model are fine-tuned on the
training dataset. We fine-tune the base model
using the huggingface transformers library (version
3.1) for Pytorch
Table 2 shows our per-domain results for our systems indicating the kind of scenario and resources they used. We observe the clear superiority of Italian-BERT which only relies on raw-text resources. This suggest that just fine-tuning BERT Scenario in-domain in-domain in-domain in-domain in-domain in-domain in-domain cross-domain cross-domain cross-domain cross-domain cross-domain cross-domain cross-domain
Resources raw raw raw structured structured structured structured raw raw raw structured structured structured structured
complexity wp embedding Italian-BERT complexity +page view wd embedding wd+wp embedding complexity +page view +wd embedding complexity wp embedding Italian-BERT complexity +page view wd embedding wd+wp embedding complexity +page view +wd embedding is enough for gaining a notion of prerequisite relations on concepts. Still, the systems based on handcrafted features and non-contextual embedding exhibit competitive results, with a good enough performance to rank first in the structured resource setting, while being faster, more interpretable and simpler than the Italian-BERT counterpart.
The results showed that there is a huge performance reduction for the cross-domain scenario. The largest performance drop is on the “data mining” domain. Given that we train our models on the combination of examples from all other domains, it is likely that the probable cause is the domain mismatch. Yet, the reduction on the test sets are smaller than what we observe in our Kfold experiments and validation sets.
In addition, we show in Table 3 the performance advantage we obtained over the next best participant based on the ranking released by the organizers.
One can see that the greater performance advantage is from the structured resource setting. This suggests that the “Concept view frequency” and the Wikidata embedding features are effective. 5
During the creation our systems we perform several experiments over the possible features to use. We did 10-fold cross validation for the in-domain experiments except with the Italian-BERT9, for which we used a stratified split of 30% for validation set. Table 4 shows the experimental results over the training (validation) set for both, in-domain and cross-domain scenarios. The “Resources” column serves to identify the type of resources used for the current feature.
We observe that the “data mining” domain appears to be difficult in the cross-domain scenario, models based on the non-contextual embedding features obtain results of zero. We suspect that this difficulty is due to the domain mismatch.
Based on these results, we select the ItalianBERT for the raw-text setting, and the “complexity +page view” and the addition of Wikidata embeddings (“wd embedding”) for the structured resource setting for our submissions.
9Due to its high computational requirements
We tackle the task of prerequisite relation learning using a variety of systems that explore three set of features: handcrafted features based on complexity intuitions, embedding models from Wikipedia and Wikidata, and contextual embedding from Italian-BERT model. We examine the capabilities of our models in in-domain and cross-domain scenarios. Our models ranked first in all the subtask of the PRELEARN competition at EVALITA 2020. We found that although our Italian-BERT model outperformed the others, the simpler models show competitive results.
We plan to further examine the impact of using a combination of all possible domains as training set on the performance of our models.
The authors thank CONACYT for the computer resources provided through the INAOE Supercomputing Laboratory’s Deep Learning Platform for Language Technologies.