The CLEF 2019 ProtestNews Lab tasks participants to identify text relating to political protests within larger corpora of news data. Three tasks include article classi cation, sentence detection, and event extraction. I apply multitask neural networks capable of producing predictions for two and three of these tasks simultaneously. The multitask framework allows the model to learn relevant features from the training data of all three tasks. This paper demonstrates performance near or above the reported state-of-the-art for automated political event coding though noted di erences in research design make direct comparisons di cult.
Hurriyetoglu et al. propose a competitive lab in which participants are tasked
with producing models to automatically identify indicators of protest in
crosscountry (but monolingual) text corpora [
This paper proceeds by rst introducing the three challenge tasks and the provided data. Then the two models are described: one model for tasks 1 and 2 and a second model for tasks 1, 2, and 3. Results for each task are discussed and compared to the published state-of-the-art on similar tasks. Finally, directions for future research are highlighted.
The competition comprises three tasks. All three tasks evaluate participants' ability to identify indicators of protest events in English text data. However, the three tasks di er in resolution (document-, sentence-, and word-level data) and in provided data sets. Each task comprises four data sets: train, dev, test, and China. As its name implies, the train data set is used to train models. The dev data set is a validation set provided to participants for model ne-tuning. The test data set is out-of-sample and therefore the labels associated with these data are withheld from participants. Similarly, the China data set is an out-ofsample set used to evaluate cross-country performance of models. Train, dev, and test contain text data from English-language news stories collected from Indian sources. The China data set is composed of English-language news collected from Chinese sources. Lab participants are able to observe X and y, the texts and associated labels, for train and dev. Participants can only observe X, the texts, for test and dev.
The role of the China set is to evaluate the performance of models in a crosscountry setting; more speci cally, the China data appear similar in form to the test data (in that they are English news wire text) but are generated by di erent underlying data generating processes (DGP). The DGP of the train, dev, and test sets represent Indian political and social processes as well as reporting norms, standards, and laws. The China data set represents the same for China.
A small amount of data preprocessing is performed prior to modeling. All non-alphanumeric characters are removed and all whitespace characters (e.g. tabs, newlines, spaces) are replaced with a single space. For tasks 1 and 2, characters are all converted to lowercase.1 All sequences are zero-padded such that every sequence within a given task's corpus is of equal length. The sequence length for each corpus is equivalent to the maximum sequence length observed in that corpus prior to padding (given in the following subsections). This is done to satisfy a software requirement that input sequences are of the same length during model training.
Document Classi cation Task 1 challenges participants to classify documents, in this case news articles, as one of either relating to a protest or not relating to a protest. Documents in the train data set vary in size from 44 words to 1599 words. The mean document length is 312 words. Total data set sizes are given in Table 1.
Sentence Classi cation Task 2 is similar to task 1 performed not on the document level but at the sentence level. Given a sentence, the model is tasked to predict whether the sentence describes a protest event or not. Task 2 train
tively impacted the performance of the named entity recognition preprocessing step, described later.
Task 1 Task 2 Task 3
Train 3,429 5,884 21,873 Dev 456 662 3,224 Test 686 1,106 6,586
China 1,800 1,234 4,387
data set sentences range in length from one word to 150 words and have a mean
of length of 24 words.3
Semantic Role Labeling Task 3 di ers from tasks 1 and 2 in that it is
effectively a multiclass classi cation problem. Given sentences tokenized at the
word level, participants are tasked with identifying sets of words (or phrases)
that represent particular roles in the context of a protest. These roles include
triggers, locations, facilities, organizers, participants, event times, and targets.
Tokens are labeled using IOB, inside, outside, beginning, tags [
The O protesters B-participant blocked B-trigger the O
Rayakottai road B-fname I-fname .
O
A robust research e ort within political science has seen many iterations on
techniques for both manual and automatic coding of event records from unstructured
texts. Most previous work on automated event coding relies on large dictionaries
of terms and phrases organized into known ontological categories. These
dictionaries are provided alongside text data to event-coding software that performs
pattern matching to identify instances of dictionary phrases within the texts.
If those phrases found in the texts match a set of heuristics, the software
produces an event record. Protests, the event category of interest here, represent
3 In fact, two entries appear to have no words { they are empty strings. It is unclear if
this is a problem with the original data, the download process, or the pre-processing
steps.
just one ontological category within the CAMEO ontology, the most common
of event-coding ontologies in use today [
While most dictionaries for coding event data are hand-coded by researchers,
recent e orts have sought to largely automate the dictionary generation process
as well { a step towards a fully-automated event data pipeline.4 One such
effort makes use of distributed word vectors to populate dictionaries given a small
input set of exemplar (\seed") phrases [
Most recently, supervised learning has been applied to the problem of event
identi cation within text with the goal of producing an end-to-end solution. A
neural network technique similar to that presented here was used by Beieler
to label sentences according to the Schrodt's QuadClass ontology [
While the model presented here for tasks 1 and 2 di ers from the model for
task 3, they have several properties in common. Both models are examples of
recurrent neural networks (RNNs). RNNs expect time-ordered inputs and are
able to model time-dependent sequences by persisting information across time
steps. These models di er from traditional autoregressive statistical models in
that the lag structure is variable, not pre-determined. Both models are also
multitask; that is, each model is trained on examples from two or more of the
tasks simultaneously. Finally, the inputs to both models are, at least in part,
sequences of words (or tokens). However, prior to the modeling stage, every
word has been replaced by a its corresponding word vector. The word vectors
are pre-trained on the English Wikipedia corpus using FastText, a neural
network language model that leverages both contextual information and sub-word
information to produce word vectors [
Inputs
The model for tasks 1 and 2 is simple. It takes as input a time-ordered sequence
of tokens (i.e. words) of arbitrary length (and possibly padded with zero
vectors) and outputs a document-level and sentence-level prediction that the given
sequence describes a protest event. The input tokens are length 300 real-valued
distributed word vectors derived from the pre-trained FastText model. The
models rst layer consists of 10 bidirectional long short-term memory (LSTM) RNN
cells with no activation function [
The multitask nature of this model, having separate outputs for document and sentence-level predictions, allows the two tasks to jointly train the single LSTM layer; this e ectively increases the training data size for this layer. Having two outputs allows the model to specialize for the two subtly-di erent tasks. Given a document, it may be the case that only a small portion of the document (a handful of words) refers to a protest. On the other hand, given a single sentence about a protest, it is likely that a relatively larger portion of the words in that sentence refer to the protest in question. Task 1 requires that the model be sensitive to the small proportion of words indicative of a protest event in a larger document; task 2 is not necessarily so constrained. However, in hindsight, separating these tasks may not have been necessary: the document sub-model performs comparably to the sentence sub-model on sentence input and vice versa.
Inputs −→ w1 −→ w2 −e−n→t1 −e−n→t2 −→ wn −−→ entn
biGRU Concatenate Embedding
Hidden Layers
Max Pooling
biGRU
tanh
The model architecture for task 3 di ers from that of tasks 1 and 2.5 The input
is still a time-ordered sequence of word vectors representative of a document
or sentence. The LSTM layer has been replaced by a layer of 20 bidirectional
gated recurrent units (GRU) [
The two models are able to perform all three tasks at levels competitive with the reported results of similar research e orts.
The models were each trained for 100 epochs on consumer-grade hardware
including a 6 core CPU and an NVIDIA 1070Ti GPU.6 Training times were
typically under 30 minutes. Models were written in Keras [
Due to the setup of the Lab, the true outcome values (y) for all test and China data sets are unavailable. Therefore, only the high-level summary metrics (F1 scores) provided as feedback by the ProtestNews Lab online evaluation system are reported for those data sets. More complete results (including precision and recall) are provided with respect to the train and dev data sets because they can be computed without the out-of-sample y values.
While the models are able to generalize out-of-sample, their performance degrades noticeably as the task resolution becomes ner (from documents to sentences to words) and as the data transition from in-sample to validation, out-of-sample, and out-of-DGP. This is not surprising as the training data sets for tasks 1, 2, and 3 contain less overall information for each subsequent task (and, in the case of task 3, ask more of that limited amount of information). The decreasing performance from in-sample to out-of-sample data sets points to over tting, a common problem in models with many parameters and one that can sometimes be remedied with additional training data and data augmentation techniques.
examples.
No Protest l e b a l e u r T l e b a l e u r
T Normalized confusion matrix Normalized confusion matrix Protest The multitask model for tasks 1 and 2 is able to classify documents with 92% accuracy on the dev set with an F1 score of 0.80.7 As can be seen in Figure 4(b), the model correctly predicts 95% of non-protest events and 79% of protest events in the dev set. In Table 2 we can see the true out-of-sample F1 scores associated with the test and China sets are 0.84 and 0.66, respectively. This result on the test set is encouraging because it is actually above the corresponding value on the dev set, 0.80, and suggests that the model has not be over t to the dev set through hyperparameter selection.
Model performance deteriorates somewhat for task 2, sentence classi cation. While the model accuracy is still high at 87%, its precision has dropped markedly. In other words, the model is able to accurately classify non-protest event sentences (96% accuracy) but only classi es protest events correctly 51% of the time. This can be seen in Figure 4(b). In out-of-sample tests, the model achieves an F1 score of 0.66 on the test set and 0.46 on the China set. 7 Note that the dev set was available at training time but was at no point provided to the model before inference was performed. Therefore it is out-of-sample but was available for hyperparameter tuning. Only limited results are available for the true out-of-sample datasets as these sets are held by the lab organizers.
not provided by the contest organizers in the output of test and China data set evaluations and are therefore unavailable here. Unfortunately, the random seed values for the models trained and evaluated on the test and China data sets were lost and so the (train and dev ) and (test and China) results are from two di erent model runs. Train Dev Test China
Task 1 Task 2 Precision Recall F1 Precision Recall F1 0.93 0.84 0.88 0.63 0.82 0.71 0.79 0.81 0.80 0.51 0.79 0.62 { { 0.84 { { 0.66 { { 0.66 { { 0.46 3.2
The model that produced the Lab-submitted results for task 3 is actually capable of performing all three tasks, the results of which are shown in Table 3. However, I focus here on the results for semantic role labeling as this model was only evaluated on the test and China data sets for that particular task. The precision, recall, and F1 scores shown here are multiclass weighted averages computed with a Lab-provided script. The unweighted average accuracy of this model on the dev set is very high, 94%, due largely to class imbalance. The model correctly predicts that most words in each sentence are not one of the selected roles. However, the model appears to generalize poorly: the F1 score for the train data set is 0.82 but drops to 0.50, 0.52, and 0.39 for the dev, test, and China data sets, respectively. This is indicative of a model that is over t to the training data.
An example of a task 3 dev data set sentence with actual and predicted annotations is shown in Figure 5. This example illustrates ve of the seven role categories and includes a target, participants, an organizer, triggers, and a location. For each row of text there are up to two rows of annotations. The top row of annotations represents the true role values provided by the Lab organizers. The bottom row of annotations are those predicted by the model.10
One-versus-all classi cation performance for the various role types is shown in Table 4. These metrics are evaluated on the out-of-sample dev set. The model performs better on common role labels than less common labels; it achieves 9 All values computed using contest-provided code. Due to time constraints imposed by the contest structure, performance was not evaluated for tasks 1 and 2 on the test and China data sets. Unfortunately, the random seed values for the models trained | days.
loc{aztion loc{aztion
} F1 scores greater than 0.5 on triggers, participants, and places. The model fails to label any locations correctly. This is probably due to the model's failure to recognize the prepositions preceding locations as the B tokens in the location phrase. For example, \in front of the state secretariat" should be labeled \Bloc, I-loc, I-loc, I-loc, I-loc, I-loc." Instead, the model predicts \O, I-loc, I-loc, I-loc, I-loc, I-loc." Another example from the dev set reads \near a mosque" and should be labeled \B-loc, I-loc, I-loc." The model instead predicts \B-fname, I-loc, I-loc," where \fname" represents the role \facility name."
The model for task 3 is also able to perform document and sentence classi cation. While test and China set results are unavailable for this model with respect to these two tasks, the model's performance on train and dev improves upon the results presented in Table 2 across the board. Future work should determine whether this is due to the second model's ability to over t to these tasks and data sets or due to the inclusion of task 3 data in the model's rst GRU layer. One approach for exploring this is discussed in the paper's nal section.
Hurriyetoglu et al. present preliminary ndings for tasks 1 and 2 [
Previous studies have evaluated the performance of both human and
machinebased coding for political event data. One of these reports that the ICEWS
Jabari-NLP system achieves an average top-level event category precision of
75.6% (document level). The authors further report that the system achieves
average top-level event category recall values for documents and sentences of
65% and 59%, respectively [
One previous study of undergraduate coders tasked with classifying top-level
event categories found that the three coders achieved precision values of 39%,
and evaluated on the test and China data sets were lost and so the (train and dev )
and (test and China) results are from two di erent model runs.
10 In fact, the model must distinguish between the beginning token of a role phrase
and the \internal" tokens. For example, \Kerala Government Medical O cers
Association" would be annotated, word-for-word, \B-organizer I-organizer I-organizer
I-organizer I-organizer." These are omitted for clarity.
11 The data set used in [
Using convolutional neural networks, Beieler [
A drawback of the multitask RNN models used above is that they do not lend themselves to model interrogation { they are typically viewed as black box models whose parameters resist simple interpretation. However, the addition of an attention layer to the input sequences would allow researchers to identify those input tokens (i.e. words) that contribute the most (or least) to a given prediction. Attention layers do this by masking some input tokens and not others, conditional on the input sequence itself. This would help to answer the question of which words contribute to accurate or inaccurate model predictions and whether those informative words di er from task to task.12
The models presented here make use of sub-word (i.e. character) information
but only in the construction of word vectors from a pre-trained FastText model.
By the time the sequences are input to the recurrent neural network models,
the sub-word information has been aggregated to word-level tokens. Based on
previous research that demonstrates the advantages of character-based models
[
Finally, the CLEF 2019 ProtestNews Lab has provided the research
community with a valuable \ground truth" data set on protest (and non-protest) events.
The lack of hand-annotated and curated event data sets has made di cult the
evaluation of event coding systems. Furthermore, due to copyright concerns that
the ProtestNews organizers have cleverly overcome, previous event data sets have
12 While there does not appear to be a single best citation for attention neural networks,
the earliest use of attention in RNN models may be [
The results presented here indicate that supervised learning can achieve strong results in identifying politically-relevant events within unstructured text. However, the generalization of these models to out-of-sample data is imperfect; the ease with which neural network models like those used here can over t to the training data means that care must be taken to ensure that the models continue to perform well on out-of-sample data. This is especially true if there is reason to believe that the out-of-sample data may represent a di erent data generating process than the in-sample data, as is the case here with the China data set. Nonetheless, when su cient training data are available (perhaps only a few thousand examples), supervised learning can play an important role in generating political event data. 13 The organizers provided a script that allowed participants to download the story data themselves from the original source websites. 25. Schrodt, P.A., Beieler, J., Idris, M.: Three's a charm?: Open event data coding with el:diablo, petrarch, and the open event data alliance. version 1.0 (March 2014), http://eventdata.parusanalytics.com/papers.dir/Schrodt-Beieler-Idris-ISA14.pdf