The present paper describes the approach proposed by the UNIGE SE team to tackle the EVALITA 2020 shared task on Prerequisite Relation Learning (PRELEARN). We developed a neural network classifier that exploits features extracted both from raw text and the structure of the Wikipedia pages provided by task organisers as training sets. We participated in all four subtasks proposed by task organizers: the neural network was trained on different sets of features for each of the two training settings (i.e., raw and structured features) and evaluated in all proposed scenarios (i.e. in- and cross- domain). When evaluated on the official test sets, the system was able to get improvements compared to the provided baselines, even though it ranked third (out of three participants). This contribution also describes the interface we developed to compare multiple runs of our models. 1
Prerequisite relations constitute an essential relation between educational items since they express the order in which concepts should be learned by a student in order to allow a full understanding of a topic. Therefore, automatic prerequisite learning is a relevant task for the development of many educational applications.
Prerequisite Relation Learning (PRELEARN)
The remainder of the paper is organised as follows: we present our approach and system in Section 2, then we discuss the results and evaluation (Section 3). Section 4 describes the interface in detail. We conclude the paper in Section 5. 2
In this Section we present our approach for automatic prerequisite learning between Wikipedia pages. We exploited a deep learning model that can be customised by the user on a dedicated GUI. The model was trained and tested on the official dataset of the PRELEARN task.
ITA-PREREQ
The classifier was built with the aim of testing
the combination of different hand–crafted
features on the automatic prerequisite learning task.
More specifically our classifier, whose
architecture is described in Figure 1, uses a two–dense–
layers Neural Network built using Scikit-Learn
and Keras libraries (wrapped for Tensorflow). The
activation function for the hidden layer is ReLU
while the Adam optimizer
Some structural properties of the classifier can be customised by the user from a dedicated GUI. In particular, for what concerns the structure of the neural network the user can define the size of the hidden layer and the number of epochs, while for the evaluation the user can set the number of cross validation folds. Moreover, training can be performed on a customizable set of features (see Section 2.2 for the complete list) since the input layer is set to dynamically match the size of the feature vector. For the specific purposes of this work, we used in every scenario a model exploiting a 20 neurons hidden layer trained on 15 epochs. A 4fold cross validation was used for the in-domain scenario.
Training The official training set containing concept pairs and their binary labels was formatted as a pair of numpy arrays: one of them has variable length and contains the serialization of the features, which will be the model input, whilst the latter contains the binary labels of the pairs. For the in-domain scenario, the model was trained using stratified random folds of concept pairs that preserve the original proportion of domains’ pairs. For the cross-domain evaluation scenario, a “leave one domain out” approach was used, training the model on all domains but the one used for test. 2.2
We defined a set of features extracted from the Wikipedia page content and structure that are available in the GUI and can be selected by the user to train his model. While the pages content was provided in the official release of the training set, we exploited Wikipedia API 2 to extract the Wikipedia metadata and knowledge structure. Depending on the sub-task requirements, we trained our models with a different combination of features.
Features used for the raw features model:
– titleInText: given a pair (A, B), it
checks if the title of page A/B is
mentioned in the page of the other concept.
– Jaccard similarity: a concept-based
metric that measures the similarity
between two pages by the number of
words shared between them.
– LDA: the Shannon Entropy of the LDA
2https://github.com/martin-majlis/ Wikipedia-API – LDA Cross Entropy: the cross entropy of the LDA vectors AnB.
Features used for the raw and structural
features model. We exploited all the above
features combined with the followings:
– extractCategories: the Wikipedia
category(s) to which each page of the pair
(A, B) belongs.
– extractLinkConnections: for each pair
of concepts (A, B) checks if the
Wikipedia page of B contains a link to
A.
– totalIncoming/OutgoingLinks: it
computes how much a concept is linked
to/from other concepts.
– Reference distance: a link-based
metric that measures the relation between
two pages by the links contained in
each of them using the EQUAL weight
Table 1 reports the results obtained by our models on the runs submitted for all four sub-tasks. On average, the performances of our systems in the different scenarios show that, as expected, training the model in a in–domain scenario allows to achieve better results. Among the different sets of features used, exploiting structural information extracted from Wikipedia pages’ structure is in general more effective than relying only on raw textual data. Thus, our best performing model is the one exploiting both raw and structural features evaluated in-domain scenario which achieves an average accuracy computed across all four domains of 0.700. Interestingly Data Mining constitutes the only case where raw textual features are more effective than a combination of raw and structural features. In fact this domain shows lower accuracies in the structured settings, possibly due to the lower number of entries within the dataset of Data Mining with respect to the other three domains and to the lower coverage of Wikipedia.
If we compare our results obtained for each domain with those obtained by the official baseline, there are only two cases where our models do not outperform the baseline, i.e. Geometry in the raw feats in-domain subtask and physics in both cross-domain subtasks. During error analysis on geometry pairs, we observe that, while pages about geometric figures, e.g. ”Rettangolo” and ”Poligono”, show a prerequisite relation in the gold dataset, our systems always fail to correctly classify them. Concerning Physics, we observe that in both cross-domain settings the classifier did not consider the page ”Fisica” as prerequisite of other pages belonging to the Physics domain, causing the performances to be below baseline.
If we look at the variation of accuracy values for each model with respect to the classifier confidence (see Figure 2), we notice that although the four systems have a similar accuracy when the confidence is low those related to the two indomain settings show a similar increase in accuracy confidence. Comparing cross-domain settings we notice that only the structured one is able to reach higher accuracy but only when it is highly confident. 4
Together with our system we also developed a User Interface aimed at personalizing the network and comparing results obtained with different models. The interface is composed of the following three modules: i) setup module; ii) results module; iii) statistics module.
The setup module, loaded at the start of the program, allows to define:
Raw+Struct in-domain Baseline in-domain Raw Feats cross-domain Raw+Struct cross-domain Baseline cross-domain
The module includes also a table where previously saved Configurations can be selected in order to run them again.
After running the model, the user can reach the results module in which are printed the performance statistics (accuracy, precision, recall, Fscore) achieved by the performed configuration. Besides, the result module is composed of different buttons that allows to:
See the results of the classifier on concept pairs labelling.
Save and download the results as csv file or txt file.
The statistics module plots in four bar charts the values of accuracy, precision, F-score and recall of all configurations saved in the interface. The repository containing the system and its GUI can be consulted on github 3. 5
In the paper we described the approach proposed by the UNIGE SE team for the EVALITA 2020 PRELEARN shared task. The classifier relied on a set of features that was customised to address the specific requests of each sub-task. The results obtained by our models are all above baseline (if considered averaging the accuracies across all domains), although in some cases the results obtained by the baseline are still highly competitive. This suggests that automatic prerequisite learning is a difficult task requiring many different information to train the models. However, the obtained results suggest that, at least in a in-domain setting, 3https://github.com/mnarizzano/ se20-project-16 features extracted from raw texts are sufficient to achieve competitive results. In the cross-domain setting exploiting only this type of features is not enough. Nevertheless, using information extracted from knowledge structures allows to achieve better results in all sub-tasks. Although our obtained results are promising, future work will be focused on analyzing the impact of each feature in training the model and exploring the inclusion of new features to improve the performance of the classifier.