This paper presents our contribution to the CLEF 2019 ProtestNews Track, which aims to classify and identify protest events in Englishlanguage news from India and China. We used traditional classi cation models, namely, support vector machines and XGBoost classi ers, combined with various word embedding approaches. Multiple models were tested for experimental purposes, in addition to the two models evaluated within the o cial campaign. Results show promising performance, especially in terms of precision on both document and sentence classi cation tasks.
The CLEF ProtestNews Track was introduced in 2019 aiming to evaluate methods for event classi cation and detection from news articles across multiple countries. This track has two main goals: rstly, development of generalisable methods which can be applied to heterogeneous news article data; and secondly, to support surveys conducted in other scienti c elds such as social and political studies by providing data on political con ict events (e.g. protests, riots). This track includes three tasks: news article classi cation, event sentence detection and event extraction. Our contribution is focused on the rst two tasks. The news article classi cation task consists of identifying news articles associated with political con icts through a binary classi cation scheme ("protest" vs. "non-protest"). The event sentence detection task focuses on identifying and labeling sentences that refer to protest events (e.g. riots, social events).
Both of the tasks attempted here relate to text classi cation and sentence
classi cation. Recent work in natural language processing (NLP) and text
mining shows many applications that leverage text classi cation at di erent levels
of scope. At the document level, many classi cation techniques have been
proposed and have achieved good results in the literature [
At both document and sentence levels, e ective feature extraction is
important to help the accuracy and robustness of classi cation models. Inspired by
recent work in e cient word representation learning [
The rest of this paper is organized as follows. Section 2 presents an overview of the analytical framework. In Section 3, we describe a set of 9 di erent models using di erent kinds of word embedding and classi er, with and without dimension reduction. Then, we present results from experimental testing of these models (Section 4) before giving results for the two models submitted to the o cial CLEF ProtestNews track evaluation (Section 5) . 2
In this section, we present the proposed framework consisting of three parts: data processing, word vector learning and text classi cation. 2.1
For each task respectively documents and sentences were converted to lowercase, all URLs and stop-words were removed. After the tokenization process, all tokens based only on non-alphanumeric characters and all short tokens (with < 3 characters) were also deleted. Then, we perform a morphological analysis of all tokens in order to identify lemmas, replacing each token by its lemma. 2.2
In word vector representations, each word is represented by a vector which is
concatenated or averaged with other word vectors in a context to form a
resulting vector which is used to predict other words in the context [
{ Word2Vec [
Pre-trained word embeddings on large training sets are publicly available, such as
those produced for word2vec [
Despite the popularity of models based on neural networks, linear classi ers
stand as strong baselines for text classi cation problems. Furthermore the
stateof-art about these models has proved their suitability and their robustness when
they are combined with right features [
May 2019.
Nine models were explored for the news article classi cation and event sentence
detection tasks. These models were selected from various combinations within
the framework presented above. The best models were chosen according to their
global performance in terms of precision, recall and F1-score obtained on the
training sets provided for each task. Parameter tuning was performed using
GridSearchCV2 in order to select parameter values that maximize the accuracy
of each model. Top parameters are presented in tables following the description
of each model below. The model architectures described below were used in
similar ways for the two tasks (except for the sum ner model, see below).
- The xgboost fast model uses word vector representations created by
FastText. Vectors were built from the training set provided for each task. Then
the XGBoost classi er was used to identify the class of each input.
Data
Document
Sentence
fraction of Gamma tree max.
columns depth
0.75 0.4 5
0.75 0.4 6
min. sum of alpha fraction of
weights observations
6 0.005 0.8
6 0.001 0.85
- The xgboost fast wiki model uses the same architecture as the xgboost fast
model except for word vector learning, which is performed through the use of
pre-trained word embeddings. The pretrained model3 is composed of 1
million word vectors trained on Wikipedia 2017, UMBC webbase corpus and
statmt.org news dataset. These vectors in dimension 300 were obtained using
the skip-gram model described in [
Data Document Sentence fraction of Gamma tree max. columns depth 0.85 0.1 6 0.8 0.3 5 min. sum of alpha fraction of weights observations 10 0 0.8 12 0.05 0.75 - The svm fast model uses word vectors built from FastText. Vectors were designed from the training sets provided. Then SVM classi ers were used to identify the class of each input.
Data C Gamma Kernel Document 10 1 rbf
Sentence 0.001 0.001 linear - The svm fast wiki model uses classi ers based on SVM. Word vector representations are built from the same pre-trained model that was used for the xgboost fast wiki model. 2 https://scikit-learn.org/stable/modules/grid_search.html Date of access: 16th May 2019. 3 https://dl.fbaipublicfiles.com/fasttext/vectors-english/ wiki-news-300d-1M.vec.zip Date of access: 16th May 2019. - The xgboost glove model uses a pre-trained word vector embedding as initialization for the representation of words. This embedding named GloVe4 is composed of 300-dimensional vectors trained over a larger vocabulary of web data (840B words). Then XGBoost classi ers were used to identify the class of each input.
Data Document Sentence fraction of Gamma tree max. columns depth 0.8 0.0 6 0.75 0.3 6 min. sum of alpha fraction of weights observations 8 0.05 0.75 6 0.05 0.8 - The xgboost w2v model is designed from a word vector representations performed by Word2vec. Vectors were built from the training set provided for each task. Then XGBoost classi ers were used to classify inputs. Data Document Sentence fraction of Gamma tree max. columns depth 0.8 0.0 6 0.85 0.3 6 min. sum of alpha fraction of weights observations 12 0.001 0.8 12 0.001 0.85 - The sum ner model uses slightly di erent text processing according to the application context.
For Task 1, the text was trimmed to capture sentences that are most representative of the source document. In this way, we aimed to gain topical clarity and reduce the vocabulary space. Similar to a text summarization process, each sentence was scored as the sum of the weighted frequencies of its words within the whole document. The highest-scoring sentences were then chosen to give a concise representation of the document. The best performances were observed by keeping the rst 4 sentences with the highest scores.
For both Task 1 (using sentences derived from document level as above) and Task 2 (which begins at sentence level), we then apply text normalization using a named entity recognition tool5. Only entities referring to a person, a location or an organisation are identi ed. Each entity localized is replaced by the name of its class. The aim with this process is to provide harmonized vector patterns which can be bene cial in word representation processes.
For both tasks, the nal step is to perform classi cation using the XGBoost technique. 4 http://nlp.stanford.edu/data/glove.840B.300d.zip Date of access: 17th June 2019. 5 https://nlp.stanford.edu/software/CRF-NER.shtml Date of access: 17th June 2019. Document Sentence fraction of Gamma tree max. columns depth 0.85 0.1 4 0.85 0.4 6 min. sum of alpha fraction of weights observations 12 0.01 0.85 8 0.001 0.8 - The xgboost fast SVD and xgboost fast wiki SVD models were created from the xgboost fast and xgboost fast wiki models above by the addition of dimension reduction alongside feature extraction. Dimension reduction was applied using the Singular Value Decomposition (SVD) method, a commonly applied technique, in order to reduce noise and increase model stability. Brie y, SVD is a matrix decomposition method for reducing a matrix to its constituent parts, to make certain subsequent matrix calculations simpler. 4
In this section, we present experimental results obtained on the test sets provided for intermediate evaluation in the CLEF ProtestNews track. The ProtestNews evaluation process was divided into two phases. The rst phase (intermediate evaluation) was conducted on test sets extracted from Indian news articles. The second phase ( nal evaluation) includes English-language news articles from both India and China (see below).
The training sets used for experimental testing of the di erent models are taken from the nal evaluation phase (3429 documents and 5884 sentences extracted from India news articles). The test sets are from the intermediate phase and are composed of 457 documents and 663 sentences respectively, extracted from India news articles. Evaluation measures used for each task are precision, recall, F1-score and the average of F1-scores obtained in these two tasks (Avg.2).
cial CLEF ProtestNews results In this section, we present results obtained for the nal evaluation phase of CLEF ProtestNews 2019. Proposed models were evaluated on Task 1 (news article classi cation) and Task 2 (event sentence detection) using two di erent testing sets. Task 3 was not attempted.
As in the experimental results above, the training sets provided were composed of 3429 documents (Task 1) and 5884 sentences (Task 2) extracted from English-language news articles from India. Table 2 gives details of the nal evaluation test sets. 6 Due to problems with the ProtestNews submission system, only these two models were entered into the nal evaluation, despite their relative poor performance in experimental testing. suspect that dimension reduction did not o er an advantage here. Usually used on a large set of features, SVD appears not to have helped extraction of suitable discriminant features for these classi cation tasks. With these settings, results are better on the Task 1 than Task 2. This may be explained by the di erence in length of the text records in these two levels of scope. However, it is interesting to observe that the use of a pre-trained model improved results obtained in Task 1.
Comparing performance between experimental testing and the nal ProtestNews evaluation, we see a worse Avg. 2 score for xgboost fast SVD and slightly better Avg.2 score for xgboost fast wiki SVD, for the nal evaluation relative to the experimental test on intermediate evaluation data. We note that in the intermediate phase, models are tested and trained on the same kinds of content (Indian news), whereas in the nal phase models are trained on Indian content and tested on both Indian and China content. It appears that use of a pretrained model is less e ective in the sentence level than in the document level when models are applied on the same kind of content. Conversely models trained on similar content are more suitable. We conclude that with these settings, features extracted are less generalisable, while those extracted from a pre-trained model give a slight decrease in performance but are more robust when confronted with another type of data. 6
In this paper, we presented our contribution to the CLEF 2019 ProtestNews Track. Models evaluated combined word-embedding techniques (Word2Vec, GloVe and FastText) with linear classi ers (SVM and XGBoost), as well as dimension reduction as a pre-processing step (SVD). Models showed worse performance when combined with dimension reduction. Word embedding, which is often sensitive to the domain of application, provided best performance when word vectors were generated from pre-trained models, independent of the level of scope.
In future work, we plan to evaluate all models proposed during the experimental phase on the datasets used in the nal evaluation phase of CLEF ProtestNews. This will help explore the portability of these models to datasets extracted from an another country and estimate their ability to adapt to new domains.