This paper describes and evaluates a model for event sentence detection in news articles using Attention Models with Bidirectional Gated Recurrent Network (GRU) and Word Embeddings. This model was developed for event sentence detection task in the competition that was organized by ProtestNews lab at CLEF 2019. We also evaluated the generalizability of NLP tools by training our model on data from one country and testing it on data from another country. The model was developed for this task was shown to have the highest score in the organized competition with average F1-score of 0.6547.
This task aims to identifying and labeling sentences that contain protest events in news
articles. It follows the document labeling task which identifies news articles that contain
protest events as identified in the Event Labeling Annotation Manual [
Event sentences, those that are labeled as 1, should contain an explicit reference to any protest event that makes the document eligible for being classified as a protest article. Such reference can be any word or phrase which denotes the said event. They can be direct expressions of the event or the pronouns which stand for the event. The sentence must clearly indicate that the event in question has definitely happened in the past or is an ongoing event.
Non-event sentences, i.e. those that are labeled as 0, are the ones which does not contain any event reference in the past, the present or the future.
The main goal for this task is to set a baseline in evaluating generalizability of the
NLP tools. The setting was proposed facilitates testing and improving state-of-the-art
methods for text classification and information extraction on English news article texts
from India and China. The direction of ProtestNews lab work is towards developing
generalizable information systems that perform comparatively well on texts from
multiple countries [
ProtestNews lab organizing committee have collected online English news articles from
India and China. The annotation process started by labeling articles in a sample of news
articles as containing protest or not which will be used for Task 1. Sentences of these
positively marked documents are then labeled as containing protest information or not.
These sentences should contain either an event trigger or a reference to an event trigger
in order to be labeled as positive [
Our deep learning based model was trained using Training-India set and Validated on Validation-India set, the data retrieved from China was not involved in training at all, so when the model was evaluated on the test sets we could obtain independent and generalized results.
F1-score metric (1), was used in the evaluation process for this task, as it gives more accurate assessment results for this kind of tasks where there is non-equal number of negative and positive samples.
1 = 2 × + × (1) 2.1
Before feeding data into the model, our text samples which are sentences taken from articles were to be cleaned and parsed into lower case words.
Because word embeddings were used in the classification process, a word index had to be created according to the embeddings set in use and once word sequences were obtained, some of irrelevant words had to be dropped and those words were determined by the word index.
Also the sequences had to have fixed length of tokens, so sequence length was limited to 35 tokens. Longer sequences where truncated and shorter sequences were prepadded with 0 indexed token in order to reach 35 tokens. 2.2
To feed those word sequences to our deep learning model every token had to be
represented by some value or vector. In this model word based representations were used,
so every word was replaced by embedding vector. This embeddings vector set was
obtained from Google’s pretrained set [
Every token was replaced by 300 dimensional vector and maximum sequence length was limited to 35 token. So every sentence was represented by matrix of shape (300, 35). 3
In sequence classification tasks, Recurrent Neural Networks (RNN) and its variations
had always been the state-of-art tool. After obtaining an embedding for each sample,
the main approach will be using bidirectional GRU [
The firstly proposed model was meant for Machine Translation purposes, while
using Attention Models mechanism for text classification tasks was proposed in the paper
written jointly by CMU and Microsoft in 2016 [
In this model (see Fig. 3) the Attention layer is added after the last GRU layer. So the Attention Models output is the dot product of Attention Similarity Vector si and GRU cells output ai. 1 http://colah.github.io/posts/2015-08-Understanding-LSTMs/ last accessed on June 2019
The main goal is to create scores (si) for every word in the text, which is the attention similarity score for a word. Here in Fig. 4, we could see how those scores are calculated. 2 https://mlwhiz.com/blog/2019/03/09/deeplearning_architectures_text_classification/ last accessed on June 2019
These final scores are then multiplied by GRU output for words to weight them
according to their importance. After which the outputs are summed and sent through
dense layers and then to the last output function [
Machine learning model is shown in Fig. 4. After Embedding layer two Bidirectional GRU layers are introduced, with 128, 64 cells respectively. On the top of them an Attention with Context layer was added and followed by dense layer of 64 nodes with ReLU as their activation function. Finally an output layer was added with one node containing sigmoid function for binary classification output.
The model was trained on Training-India dataset (see Table 1) for 8 epochs using
Nadam [
While testing the model on test datasets it could be observed (as in Table 2) that performance (F1-score) dropped from 0.70 on Test-India dataset which is the same source that Training data was obtained, to 0.60 on Test-China dataset. (*) Was kept hidden by the commission of the lab. 6
In this task we worked on deep learning model which tries to classify and detect event sentences in news articles. The proposed model uses Bidirectional GRU with Attention Models. The results obtained from this model were the highest in the competition which had been organized by ProtestNews Lab.
With this experiment we could observe the effect of local data on NLP tools, our test results on datasets from the same source of training sets were noticeably higher than those on datasets from other sources.
For further work, we could evaluate POS based features of the words in the sentences by adding one more input layer in parallel with embedding layer.