Extracting Protests from news is very useful because it helps in the early identification and subduing of contentious events and controlling violent public outbreaks. This also helps in the study of social sciences on the difference in protest types and their expression across countries. In this paper, a deep learning approach is presented for classifying input documents developed using data from one country and is exercised on data from others. Long Short-Term Memory(LSTM) a variant of Recurrent Neural Network(RNN) is used to implement our approach for the text classification. Models were created for 2 tasks in specific, the first aimed at identifying if a given news document led to a protest or not while the latter predicted if or not an input sentence contained an event trigger. The data given by CodaLab - CLEF 2019 Lab ProtestNews was used to develop and evaluate the performance of the models. The documents and sentences were assigned labels 'protest'/'non-protest' and 'event-trigger'/'not-event-trigger'. A total of 4 LSTM models were implemented with different attention mechanisms, and the ones implementing Scaled-Luong and Bahdanau obtained higher overall F1 macro scores of 0.3290 and 0.3682 on the final evaluation test set respectively.
Text classification refers to the process of assigning tags or categories to textual data according to its content. It is one of the fundamental tasks of Natural Language Processing (NLP) which involves the application of computer techniques to the analysis and synthesis of natural language. Information Extraction, a field strongly associated with the aforementioned topics is the automated retrieval of specific information from a body of text. Using these in unison, the CLEF 2019
Lab ProtestNews TASK 1 aims at classifying news documents as ‘protest’ or
‘non-protest’ given raw data and TASK 2 aims at classifying a sentence as one
containing an event-trigger or not given the sentence and the news article
containing the sentence and Task 3 the extraction of various information from a
given event sentence such as location, time and participation of the event [
This paper focuses on the use of LSTM, an artificial RNN architecture with different attention mechanisms to perform efficient binary classification of input data and assign class labels for TASK 1 and TASK 2 of CLEF 2019 Lab ProtestNews, functioning along lines similar to that of the novel work published by Peng Zhou et al. [18], which employs Bidirectional LSTM as well as Two-dimensional Max Pooling for the purpose of text classification.
The main challenge is to extract various event information from news articles
across multiple countries around the globe. The events considered mainly focus
on politics and social movements across the globe collected from Indian and
Chinese news articles written in English. Neural Networks can be used for addressing
these kinds of problems, but the ability of RNNs to outperform traditional
models in applications such as Discriminative Keywords Spotting, evidenced by the
works of S Fernandez et al. [
This section serves to analyse the wide array of published literature and helps in getting an insight into the various models adopted to tackle the problem at hand.
Research works [
Sudha Subramani et al. [
Even though many papers [
This model classifies news documents as protest and non-protest news, eventtrigger or not event-trigger and it extracts various information from a given sentence such as location, time and participants of an event. To meet the objectives, the following are done – Gather news articles – Preprocessing of news articles – Grouping and Indexing of news articles – Feature selection from news articles – Labelling the news with the help of the model
Text pre-processing is the initial and important step in news article classification process. pre-processing step makes sure that the learning model receives only the data that are considered useful and the noisy data is removed from the text. The data comes from various sources and ought to be cleaned before it is fed into the model. The noisy data or useless information which includes punctuation marks, non-alpha numeric characters etc. are first removed from the text with the use of regular expressions. A dictionary of unique words is created, and these unique words help in embedding the input sequence to the model to do the classification.
The objects of the training file are separated from one another by using ‘newline’ as the delimiter. A multi-layer RNN with LSTM as a recurrent unit are used to build a deep neural network model for finding the classified labels. Several layers are used to build the deep neural network namely, embedding layer, encoding-decoding layers, projection layer and loss layer. The weight vectors are learnt from the input sentences and label input based on their vocabulary using the embedding layer. 2-layered bi-directional LSTM is used as hidden layers to perform the encoding and decoding using the embeddings of input sentences or paragraphs. This is shown in Fig 1. 4
The current methodology adopted for the TASK1 and TASK2 of CLEF 2019 Lab ProtestNews is implemented using Tensorflow. The data provided, belonging to 4 unique sets namely Development, Training, Testing and Testing_China is presented in Table 1.
A total of 4 models [
A total of 4 models were submitted and were evaluated by the CLEF 2019 Lab ProtestNews Evaluation engine. The test data provided was classified into two sets, the first belonging to news articles from India and the latter containing articles from China, labelled Set 1 for China-Task1, Set 2 for China-Task2, Set 3 for India-Task1 and Set 4 for India-Task2. The Table 2 shows the various F1 macro scores obtained by the models in the final evaluation stage.
Average
Clearly Model 3 for which the 2 Layered, bi-directional LSTM, with Bahdanau attention was adopted produced the best results of the lot with a F1 macro score of 0.3682. 6
From Table 2 given under results, undoubtedly Model3 outperforms its peers for the test data given under Task2. Thus, with the scope of the current study under consideration, adopting the input structure of the training data of Task2 (i.e. sentence wise rather than Document wise) and administering the same in the implementation phase of Task1 may provide better results, thus increasing the overall accuracy. However, there may be two approaches that must be juxtaposed for the post processing of predictions outputted by the model. These are as given below – At-least-one approach: If any of the sentences belonging to a paragraph/news-article is classified as class 1 (I.e. classified under ‘protest’) then the whole paragraph/news-article can be classified as class 1. – Vote-based approach: Each of the sentences can be considered to voting for either class 1 or class 0 when classified. Finally, the whole paragraph/newsarticle can be classified based on which class obtained the maximum votes. 7
We have developed deep learning models for information extraction and text classification using data from one country and tested them on data from different countries. RNN-LSTM models were used to classify documents and sentences to the labels ‘protest’/’non-protest’ and ‘event-trigger’/’not-event-trigger’ respectively. The 4 models implemented differed along the attention mechanism used. The various F1 macro scores obtained were 0.0481, 0.3290, 0.3682 and 0.2064 respectively. The performance may be further improved by the effective splitting of Document data at the pre-processing stage and applying appropriate integration logic of the output at the post-processing stage. 18. Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., Xu, B.: Text classification improved by integrating bidirectional lstm with two-dimensional max pooling. arXiv preprint arXiv:1611.06639 (2016)