We present in this paper our participation in ProtestNews lab at CLEF 2019 in task 1 and task 2. In task 1, the objective is to predict if an article contains protest news or not. In task 2, we must decide if a sentence contains a protest event or not. For these two tasks, we used a supervised machine learning approach based on the logistic regression model. We combine the supervised learning algorithm with two di erent natural language techniques. The rst relies on text processing with linguistic properties. The second is based on the expansion of the text with related term using word embedding similarity.
This section describes the model that have been used to classify the test data of both task 1 and task 2. The model is applied for both tasks, and use the same set of linguistic settings.
The overall architecture of our proposed framework consists of two main phases: training and testing. In the training phase, the classi er learns from a set of labeled text. Subsequently, the classi er is capable of classifying new unlabeled documents in the testing phase.
Each phase consists of the following steps: documents preparation, features
extraction/selection, and classi cation. We opt for a supervised approach based
on a classi cation algorithm [
We notice that ProtestNews documents present di erent characteristics for each task: documents in task 1 are composed of long sentences, this may cause drift in the classi cation process. In the other side, documents for task 2 are short, so they don't contain enough context for the training step. This can lead to di erent problems: word ambiguity and word mismatch between training data and test data.
To deal with these two tasks, we compared di erent classi cation algorithms, among them logistic regression algorithm, random forest, and naive Bayes algorithm. Preliminary experiments were carried on the development data after a training step. Based on these nding, we decided to use the logistic regression algorithm for the rest of the experiments because it gives the best results. 3
Before extracting the feature vectors it is required to pre-process the data with
stop words removal and text lemmatization. We rely on linguistic processings
since they lead to good results in previous work for sentiment analyis task [
After text processing, we proceed to feature extraction, The aims of the feature selection technique are to nd the most relevant features for the classi cation task. We used unigram because it gives the best results. We didn't rely on any weighting scheme, because using the tf.idf scheme degraded the classi cation performances in preliminaries experiments with the development data. 4
To deal with the term mismatch problem, we decided to expand documents with
the most similar word for each token. Since in previous work using pretrained
word embeddings has proven to have a postif impact on di erent natural
language processing tasks: Word Sense Disambiguation, Relational Similarity,
Semantic Relatedness [
For this approach we trained two word embeddings [
We choose to train two word embeddings with di erent data sets, to see if a specialized dataset have an impact on the classi cation performance, in comparison with text8 dataset which is a 100 megabytes cleaned dataset. For the word embeddings training, we rely on the Gensim python library. 5
We studied the performances of the proposed approach and we performed different experiments using di erent setting and processing: { Run1: consist of applying lemmatization and stop words removal on the training and test set. 3 http://mattmahoney.net/dc/textdata.html 4 English Wikipedia dump on Mar 3, 2006 { Run2: consist of combining lemmatization, stop words and named entities removal on both training and test data sets. { Run3: expanding every word in the sentences with the most similar word from the protest-emb word embeddings. { Run4: expanding every word in the sentences with the most similar word from wiki-emb word embeddings. { Run5: combining run 2 settings with the annotation of all CN.
Table 1 present the o cial submitted runs, there are some runs where there are results only for task 2. The analysis of the results shows that the rst run (lemmatizing and eliminating stop word) allows to obtain 0.7612 for task 1 and it corresponds to our second best run. The best result for task 1 was achieved when we expanded the content of the text with bigram extracted from the same span of text. This run allows to obtain the best overall results, but the best result for task 1 (China and India) and the best result for task 2 (India). We note the degradation of the results for task 2 (China).
The best overall result for task 2 is obtained by a simple approach that consists of lemmatizing the text and eliminating stop words.
In a preliminary study phase with development data, we found that expanding text with the most similar word is only bene cial for task 2 and it degrades results for task1. We decided to study the impact of this approach only for task 2. We notice that the best overall results are obtained when training the word embedding on the training set. The best result for task 2 is obtained with word embedding trained on the same data, but the best result for China data is obtained when we used word embedding from another general corpus. This can be explained by the fact that the text8 dataset is bigger dataset and contain more tokens than task 1 training dataset.
Our o cial nal run was the Run6 and it was ranked sixth among 12 teams. We can note that with this run we achieved the best our results in task 1 and task 2 with India data. Degradation of the performance has been noticed for task 2 with China data, this can be explained by terms mismatch between CN in the training set with india data and the CN with the China data because CN in India data represents other concepts than those extracted in China dataset.
This paper describes our participation in the ProtestNews detection lab at CLEF 2019. The aim of this work is to make a decision if a text contains protest news or not. The objective is to develop text classi cation tools. For this purpose, we used a classi er based on the Logistic regression algorithm. As the rst step, we processed the linguistic data processing as a rst step. Then, we use word embeddings to expand text with the most similar word. Also, we proposed to add CN extracted from the same text. This work is still in progress and needs more investigations. For future work, we plan to use deep neural network since it achieved good results for other NLP tasks.