These working notes summarize the di erent approaches we have explored in order to classify a corpus of tweets related to the 2015 Spanish General Election (COSET 2017 task from IberEval 2017). Two approaches were tested during the COSET 2017 evaluations: Neural Networks with Sentence Embeddings (based on TensorFlow) and N-gram Language Models (based on SRILM). Our results with these approaches were modest: both ranked above the \Most frequent baseline", but below the \Bag-of-words + SVM" baseline. A third approach was tried after the COSET 2017 evaluation phase was over: Advanced Linear Models (based on fastText). Results measured over the COSET 2017 Dev and Test show that this approach is well above the \TF-IDF+RF" baseline.
In this work we have explored several approaches for text classi cation on
Twitter messages, tackling the COSET 2017 task from IberEval 2017 [
The aim of the task was to classify a corpus of tweets in Spanish about the
2015 Spanish General Election into ve categories or topics, namely:
1. Political issues (related to the most abstract electoral confrontation)
2. Policy issues (about sectorial policies)
3. Campaign issues (related with the evolution of the campaign)
4. Personal issues (on the life and activities of the candidates)
5. Other issues
The COSET 2017 organizers provided a corpus split in three sets of tweets, two
of them labelled for training (Train) and development (Dev), and the other one
unlabelled as a Test (which we had to classify, submitting the results for
evaluation). The tweets were provided (password protected) in CSV les containing
the tweet ID, the tweet itself and (only in the Train and Dev les) the label [
We carried out the preprocessing of the Train, Dev and Test sets with our own Python script based on regular expressions (regex library). The aim was to leave the tweets in the best condition possible in order to make it easier later to extract meaningful features for the classi cation models that we used in our approaches.
First of all, as many of the the tweets in the CSV les came divided in several lines (we can detect the end of a tweet because the next line begins with double quotation marks + an 18-digit tweet ID), our script converted each tweet to one string without line breaks (to make it easier to process the les for the rest of the modelling and classi cation steps).
Other modi cations were the conversion of hashtags to lower case (as, for Twitter, the hashtags #COSET2017, #CoSeT2017 and #Coset2017 are the same), removing redundant quotation marks (when a tweet contained quoted text, the API used for tweet retrieval had added redundant quotation marks), removing hyperlinks (URLs are shortened by Twitter and sometimes cut by the API, and this provides zero information), shortening citations (Twitter added a lot of text in order to indicate that a tweet cites another tweet), and basic tokenization (separating punctuation from words).
Apart from the initial preprocessing we just described, in the case of some
of the approaches described below, we also tried converting the full text of the
tweets to lowercase, by using the lowercase.perl script that comes with the Moses
toolkit for statistical machine translation [
We faced the task by exploring di erent techniques and solutions instead of focusing on a single approach, with the aim of achieving better results.
Two approaches were tested during the COSET 2017 evaluation phase: Neural Networks with Sentence Embeddings (based on TensorFlow) and N-gram Language Models (based on SRILM). A third approach was tried after the COSET 2017 evaluation phase: Advanced Linear Models (based on fastText). N-gram Language Models (SRILM). The most basic approach we tried was to train an n-gram language model for each topic. Then, new tweets (the tweets in the Test set) were compared to these 5 topic n-gram models based on perplexity. Each tweet was assigned the topic with which it showed less perplexity.
The well-known n-gram language modelling toolkit SRILM [
SRILM was also used to compute the perplexity of new tweets against each one of the 5 topic n-gram models trained. As explained before, each tweet was classi ed as pertaining to the topic with which it showed less perplexity.
In the end, we sent one submission for evaluation in COSET 2017 based on
this approach. The models chosen were 3-grams with Good-Turing smoothing.
Neural Networks with Sentence Embeddings (TensorFlow). In this case,
the aim was to use a TensorFlow-based neural network [
In order to extract the features of each tweet to train the neural network
(NN) models, we decided to use the Sentence Embeddings approach, by which
each text document (in this case, each tweet) is represented as a feature vector.
To this end, we used the software Doc2vec from the gensim Python library [
After sentence embeddings were computed with Doc2vec for all of the tweets, these feature vectors (one per tweet) were used as input to train with TensorFlow a NN model for tweet classi cation into the task's 5 topics.
The Doc2vec model for converting tweets into vectors of fty features (a compromise between having enough representation power and having too many parameters to train) was built from the train data. The training method of the model was called ten times, each training step taking ten iterations. While each training step had a xed learning rate, between each call, the learning rate was decreased (learning rate annealing, starting at 0.025, decreasing 0.02 per step).
After the Doc2vec model construction, all tweet data was converted into feature vectors using it and saved in new les. Each line contained the tweet id (in the case of test data), the feature vector and the label (in the case of Train or Dev data), all separated by spaces.
Finally, we fed the data into a NN with TensorFlow. A traditional feedforward NN topology was used, with a hidden layer of 1024 nodes (with ReLU as the activation function), an input layer of 50 nodes (feature vector size), and an output layer of 5 nodes (number of classes). The labels were converted from a numerical value to a vector with a component for each class (0,1,2,3,4, following the order from Section 1), with the vector size being determined by the highest label's value plus one (in this case, highest value 4, vector size 5).
The NN was fed with batches of 100 tweets during training. After training, the NN was fed with the test data and the output vector was converted into a class label depending on which position of the vector had the highest value.
We sent one submission with this approach for evaluation in COSET 2017.
Advanced Linear Models (fastText). Linear models with bag of n-grams
representation and hierarchical classi ers have been explored by Facebook
Research as a way to develop text classi ers on par with deep learning classi ers
in terms of accuracy and more e cient for training and evaluation [
In order to predict the labels for the COSET 2017 Test set, a text classi cation model was trained with fastText from the COSET 2017 Train and Dev sets. Then, the Test set was classi ed using fastText with this model.
There are several fastText parameters we can adjust for model training. The best results on Dev were observed with fastText's defaults for word vector size (100), word n-gram order (1), loss function (5-negative sampling), and learning rate (0.05), but increasing the number of epochs (from the default of 5 to 25).
Finally, regarding preprocessing, with this approach we tried converting to lowercase the results of our preprocessing described in Section 3.1. Our measurements showed that using lowercase tweets for system training and for classi cation improved the results on Dev (from F-1 macro 0.5494 with truecase to F-1 macro 0.5982 with lowercase); thus, the result for this approach reported in Section 4 below is based on converting tweets to lowercase during preprocessing. 1 https://github.com/facebookresearch/fastText 4 4.1
The metric designated for evaluating the participating systems in COSET 2017
[
F1 macro = As mentioned before, we only were able to submit results before the COSET 2017 competition deadline using the Language Model approach and the Neural Network approach. After the COSET 2017 submission deadline had passed, we carried out additional experiments using the Linear Model approach.
In Table 3 we show our results measured in F-1 macro over the Dev and Test
sets of COSET 2017. The COSET 2017 baselines and winning system have also
been included for reference (see [
As we can see, our results with the N-gram Language Model and Neural
Networks with Sentence Embeddings approaches were modest: both ranked above
the \Most frequent baseline", but below the \Bag-of-words + SVM" baseline.
However, this does not mean that it would not be possible to obtain better
results with these approaches, if more time had been available for experimentation.
For starters, a di erent preprocessing, such as converting all tweets to lowercase,
could improve the results. Then, in the case of N-gram Language Models, trying
n-gram character models instead of n-gram word models could be e ective. As
to the case of Neural Networks, it could be useful to try Word2vec [
As to the Linear Model approach, results on the COSET 2017 Dev & Test
sets show that this approach ranks well above the \TF-IDF+RF" baseline, being
closer to the winning system [
As explained in Section 2, after the COSET 2017 competition was over, a
second phase test on robustness was proposed by the organizers, in which we ran
our best approach on a new, much larger tweet set. As we can see in Table 4, by
running the Advanced Linear Models system described in Section 3.2 (trained
this time on the original COSET 2017 Train, Dev and Test sets), our results
came in second in the list of participants, close to the winning systems [
As future work, we can recapitulate here some of the possible ways for
improvement: for preprocessing, trying other techniques on the tweets, such as
lemmatization, or smiley and emoji preprocessing [