This notebook describes our participation to the ProtestNew Lab, identifying protest events in news articles in English. Systems are challenged to perform unsupervised domain adaptation against three sub-tasks: document classi cation, sentence classi cation, and event extraction. We describe the nal submitted systems for all sub-tasks, as well as a series of negative results. Results indicate pretty robust performances in all tasks (average F1 of 0.705 for the document classi cation sub-task, average F1 of 0.592 for the sentence classi cation sub-task; average F1 0.528 for the event extraction sub-task), ranking in the top 4 systems, although drops in the out-of-domain test sets are not minimal.
The growth of the Web has made more and more data available, and the need for
Natural Language Processing (NLP) systems that are able to generalize across
data distributions has become a urgent topic. In addition to this, portability
of models across data sets, even when assumed to be on the same domain, is
still a big challenge in NLP. Indeed recent studies have shown that systems,
even when using architectures based on Neural Networks and distributed word
representations, are highly dependent on their training sets and can hardly
generalise [
The 2019 CLEF ProtestNews Lab 3 targets models' portability and unsupervised domain adaptation in the area of social protest events to support comparative social studies. The lab is organised along three tasks: a.) document classi cation (Task 1); b.) sentence classi cation (Task 2); and c.) event trigger and argument extraction (Task 3).
Task 1 and 2 are essentially text classi cation tasks. The goal is to distinguish between documents and sentences that report on or contain mentions of protest events. Task 3 is an event extraction task, where systems have to identify the correct event trigger, in this case a protest event, and its associated arguments in every sentence of a document.
As described in [
As already mentioned, the lab's main challenge is unsupervised domain adaptation. The lab organisers made available training and development data for one domain, namely news reporting protest events in India, and asked the participants to test their models both on in-domain data and on out-of-domain ones, namely news about protest events in China. In the remainder of the notebook, we will refer to these two test distributions as India and China.
When analysing the three tasks, it seems evident that the rst two tasks are very similar and can be targeted with a common architecture, and possibly features, modulo the granularity of the text message, i.e. full document vs. sentences. On the other hand, the third task requires a dedicated and radically di erent approach.
In the remainder of this contribution, we illustrate the systems we developed for the nal submissions, and provide some data analysis that may help understand the drops in performance across the two test data. We also describe and re ect on what we tried but did not work as expected. 2
The three tasks have been addressed with two separate systems. In particular, for Task 1 and 2, we opted for a feature based stacked ensemble model based on a set of di erent basic Logistic Regression classi ers, while for Task 3, we used a Bi-LSTM architecture optimized for sequence labelling task. 2.1
The lab organisers made available training and development data for each task. Table 1, summarises the distributions of the labels of the training and development data for Task 1 and 2, i.e. document and sentence classi cation, respectively. As the gures show, the positive class, i.e. the protest documents or sentences, is pretty much unbalanced with respect to the negative one, i.e. non-protest, ranging between 22.41% for Task 1 to 16.78% in Task 2 in training. The distribution of the classes is mirrored in the development data, with minor di erences for Task 2, where the positive class is slightly bigger than the negative one (20.81% vs. 16.78%). For training our systems, we did not use any additional training material. The development data was used to identify which methods to use for the nal systems rather than ne tuning the models, given the fact that at test time the models has to perform optimally for two di erent data distributions, in- and out-of-domain (India vs. China).
Task Data set Task 1 (Document Classi cation) TDreavi.n Task 2 (Sentence Classi cation ) TDreavi.n
Protest Not Protest 769 (22.41%) 2,661 (77.58%) 102 (22.31%) 355 (77.68%) 988 (16.78%) 4,897 (83.21%) 138 (20.81%) 525 (79.18%)
english-events-guidelines-v5.4.3.pdf The average amount of event trigger per sentence is 1.41 in training and 1.35 in development, indicating that multiple event triggers are available in the same sentence. As for the arguments, the average per event trigger is 2.24 in training and 1.85 in development, indicating both that arguments are shared among event triggers in the same sentences and that not all arguments are available in every sentence. Similarly to Task 1 and 2, only the available training data was used to train the system. 2.2
The document and sentence classi cation tasks have been formulated as
standard classi cation tasks. In the perspective of maximizing the system results on
both test distributions, we have developed a stacked ensemble model of
Logistic Regression classi ers, following a previously successful implementation that
showed robust portability [
We extracted three di erent sets of features. Each set of features was used
to train a basic Logistic Regression classi er, as available in the scikit learn
platform [
Word embeddings features We used the pre-trained FastText embedding [
Most important token and character n-grams features These two sets of features have been identi ed as useful features to increase the robusteness and portability of the models across data sets. The features have been extracted by performing two sets of TF-IDF scores over each training data (i.e. Task 1 and Task 2) to select the most important tokens and characters n-gram per class (i.e. protest documents/sentences vs. non-protest documents/sentences). For each extracted token, the maximum and minimum cosine similarity is obtained with respect to all tokens in a document/sentence using the FastText embeddings. Similarly to 5 We used the wiki-news-300d-1M-subword.vec model, available at fasttext.cc/docs/en/english-vectors.html. https:// the word embeddings feature, in case a token is not present in the embeddings, we checked for sub-words embeddings. The character n-grams are represented by means of Boolean features indicating whether they are present or not in the document/sentence, thus capturing and representing di erent information.
Table 3 illustrates the settings used to tune the system for each test
distributions and task. The amounts of token and character n-grams varies per task
as well as per test distribution. Although more experiments are needed, during
the submission phase and quite not surprisingly, we observed that the higher
the number of tokens and character n-grams is extracted, the better the model
performs on the same test data distribution, thus loosing in portability (for
instance, with 4,000 token n-grams and 1,000 characters n-grams, the F1 of Task
2 on China drops to 0.553 to 0.536).
We framed the event mention and argument extraction task as a supervised
sequence labelling classi cation problem, following a well established practice
in NLP [
We have implemented a two-step approach using a common sequence
labelling model based on a publicly available Bi-LSTM network with a CRF
classi er as last layer [
We did not ne-tuned the hyperparameters, but followed the suggestions
in [
Training is stopped after 5 consecutive epochs with no improvements.
Komninos and Manandhar ([
Both for the event trigger and argument detection sub-tasks, we have conducted ve di erent runs to better asses the variability of the deep learning models due to random initialisations. At test time, we selected the model that obtained the best F1 scores on the development set out of the ve runs. 3
In this section we illustrate the results for all three tasks 8. Notice that for Task 3 the scores are cumulative for both event trigger and participant detection. For all tasks, the ranking is based on the average F1 of the systems on the two test distributions (i.e. India and China). Table 5 reports the results for each task and the corresponding rankings. For ease of comparison, we also report the distance from the best ranking system for each task expressed in di erences in F1 scores. 7 We used version 3.9.2 8 Results and ranking were taken from the Codalab page of the ProtestNews Lab available at https://competitions.codalab.org/competitions/22349#results .
Quite disappointingly, the drops against the China test data are not minimal, although with a pretty wide range. The minimal drop is on Task 2, where the system does not perform optimally on the India test data (F1 0.631). On the other hand, the largest drop is in Task 1, where the system looses 0.21 points in F1 when applied to the China data set. As for Task 3, the drop is still relevant (0.144 points). However, we also noticed that the results for Precision and Recall on the China are pretty well balanced (P = 0.492; R = 0.425), indicating some robustness of the trained model. 4
To better understand the results of our systems for the three tasks, we have
conducted an analysis of the data to highlight similarities and di erences.
Following recent work [
The J-S divergence assesses similarity between two probability distributions, q and r and is a smoothed, symmetric variant of the Kullback-Leibler divergence. On the other hand, the OOV rate can be used to assess the di erences between data distributions, as it highlights the percentage of unknown tokens. All measures have been computed between the training data and the two test distributions, i.e. India and China. Results are reported on Table 6.
As the gures in Table 6 show the test distributions, India and China, can be seen as occupying pretty di erent portions of this variety of spaces. Not surprisingly, the China distributions are very di erent from their respective training ones, although with varying degrees. This variation in similarity, however, also a ects the India test distributions, where the highest similarity is observed for Task 1 (0.922) and the lowest for Task 3 (0.703). The J-S scores show that the test distributions for Task 3 and Task 2 are even more di erent than those for Task 1, indicating that the di erences in performances of the models across the three tasks is subject to these variations in similarities. As a further support to this observation, we found that there is a positive signi cant correlation between the J-S similarity scores and the F1 values across the three tasks ( =0.901, p<.05).
The OOV rates support the observations conducted with the J-S divergence. The OOV rates for the India test distributions are much lower then those compared to China, clearly signalling that there are strong lexical di erences among the data sets. The OOV rate for India and China for Task 3 are much closer than those for Task 1 and 2, and still the di erences in overall F1 scores for this task between the two test distributions is pretty large (F1 0.600 for India vs. F1 0.456 for China), suggesting that OOV is actually a less powerful predictor of di erences in performance between data distributions. Indeed, we have found that there is a negative non signi cant correlation between OOV rates and the F1 scores across the tasks ( =-0.804, p>.05).
Finally, a further aspect to account for the behavior of the models concerns the proportion of the predictions. In particular, for Task 1 and 2 the proportions of the predictions in the two classes (protest vs. non-protest) are the same as in the training sets for the India data (i.e. in-domain), while they drop when applied to the China tests. In Task 3, the system predicts the same proportion of event triggers in both test distributions (0.90 event per sentence on India, and 0.95 event per sentence on China, respectively), although slightly lower than that observed in training. On the other hand, the proportions of predicted arguments per event are di erent: on the India data, they are in line with those of the training sets (2.51 vs. 2.24 in training, respectively), while they are lower in the China data (1.94 vs. 2.24 in training, respectively). These observations further indicate that the systems for Tasks 1 and 2 are more dependent on the training set, while the system for Task 3 appears to be more resilient to out-of-domain data. 5
In this section we brie y report on alternative methods that actually resulted to
be detrimental for the performance with respect to the nal settings. We mainly
focused on changing strategies in modelling by using di erent algorithms and
paradigms rather than attempting to extend the training materials.
Task 1 and 2 - Inductive Transfer Learning In an attempt to build a system
that better generalizes across data sets, we tried exploiting recent advancements
in transfer learning. Combining a ne-tuned language model with a classi er has
been shown to be a sound strategy for classifying text [
We obtained promising results on the development set, but, surprisingly, the performances dropped when applied to the test distributions (F1 = 0.466 for ELMo, and F1 = 0.567 for BERT, respectively). The ensemble model using both dense and sparse representations outperformed the simpler model by 0.1 F1 point.
Task 1 and 2 - Convolutional Neural Networks (CNN) It has been shown
that character-level convolutional neural networks perform well in document and
sentence classi cation tasks [31] and, being character based, these models are
in theory not severely harmed by OOV words, thus making portability across
test distributions less prone to errors. We experimented with the architecture
described in [
Task 3 - Multi-task Learning We investigated if a multi-task learning
architecture, still based on the Bi-LSTM network, could be a viable solutions to
improve the system performance and portability. Given the incompatibility of
the Task 3 annotations with other existing corpora for event extraction (e.g.
ACE and POLCON [
Our contribution mainly focused on two aspects: a.) assess the most viable approach for each task at stake and maximize portability with limited e orts; b.) explain the limits of the trained models in terms of similarities and di erences across training and test distributions rather than just limiting to technical aspects of the systems.
Task 1 and Task 2 have shown that a simple system can obtain competitive results in an unsupervised domain adaptation setting. This aspect is actually encouraging and triggers further investigation in this direction by focusing e orts on parameter optimisation. We also believe that the lack of any material for the out-of-domain distributions is a further challenge to take into account, as no ne tuning of the models on the target domain was actually possible. As far as we can put e orts into the development of maximally \generalisable" systems, the dependance of the models on the training materials remains high, thus posing the problem if we are not just modelling data sets rather than linguistic phenomena.
Task 3 has actually highlighted the contribution of both more complex architectures, such as a Bi-LSTM-CRF network, and contextualised embedding representations, such as ELMo. In this speci c case, the trained model is able to predict a comparable amount of event triggers between the two test distributions, although it su ers on the argument sub-task, where less arguments are predicted for the out-of-domain data. Unfortunately, the evaluation format does not allow to quantify the losses per sub-task and per trained models.
Finally, the similarity and diversity measures (i.e. J-S divergence and OOV) resulted in useful tools to better understand the di erent behaviors of the systems on both test distributions. It is worth noticing how J-S similarity scores correlates with F1 scores of the trained models, suggesting that it could be possible to quantify, or predict, a margin loss of systems before applying them to out-of-domain test distributions and, consequently, take actions to minimize the losses. Association for Computational Linguistics (Volume 1: Long Papers). pp. 1192{1202. Association for Computational Linguistics, Berlin, Germany (Aug 2016). https://doi.org/10.18653/v1/P16-1113, https://www.aclweb.org/ anthology/P16-1113 28. Ruder, S., Plank, B.: Learning to select data for transfer learning with bayesian optimization. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 372{382. Association for Computational Linguistics, Copenhagen, Denmark (September 2017), https://www.aclweb.org/anthology/ D17-1038 29. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Dropout: a simple way to prevent neural networks from over tting. Journal of Machine Learning Research 15, 1929{1958 (2014) 30. Weber, N., Shekhar, L., Balasubramanian, N.: The ne line between linguistic generalization and failure in seq2seq-attention models. arXiv preprint arXiv:1805.01445 (2018) 31. Zhang, X., Zhao, J.J., LeCun, Y.: Character-level convolutional networks for text classi cation. In: NIPS (2015)