This paper discusses the IberEval 2017 shared task on Classi cation Of Spanish Election Tweets (COSET) [5]. This task has the goal to analyze tweets that talk about Spanish General Election of 2015 and classify them in one of these 5 categories: political issues, policy issues, personal issues, campaign issues and other issues. The aim of this paper is to perform a study of di erent feature extraction methods and machine learning algorithms for classifying this set of tweets into their categories.
Election dates increase the political conversations in Twitter. We can extract
several useful information from all the conversations if we are able to segment
them in di erent categories. COSET [
Before explaining the feature extraction we need some algorithms that can
classify the tweets in the di erent categories. To solve this problem we tried di erent
machine learning algorithms:
{ Support vector machines (SVM): probably the most used algorithm in tweets
classi cation, the SVM is one of the state-of-the-art algorithm for tasks
related with text.
{ Random forest (RF): this algorithm is not usually employed for a text task
but we decided to use it because we have possibilities of over tting with the
size of the corpus and the random forest prevents that.
{ Neural network (NN): currently the most used machine learning algorithm
due to its exibility. We used both multilayer perceptron and long short term
memory (LSTM) [
We used support vector and random forest from scikit learn [
For this task we have a list of characters for every tweet but we can not feed a machine learning algorithm with a list of characters: before it we have to extract features for every tweet. We try three di erent methods of feature extraction. All of them need a previous preprocessing: { Tokenization: we split every tweet in a list of tokens like words, URL links, punctuation marks, etc. { Cleaning: we remove all URL links, emoticons and some other special characters. 3.1
The n-gram idea is based on grouping the tokens in groups of n. We create a
model based on the idea of bag of words but using the n-gram themselves [
Tf-idf
Another approach is using the Term Frequency Inverse Document Frequency
(tf-idf) to generate vectors for every tweet with the tweets tokenized and clean,
despite of getting the n-gram model. It is similar to the previous one but we
have a probability of a word in every feature related with the importance of a
word in a tweet instead of getting a count in every feature. This method is used
in some experiments like [
The third feature extraction method is the proposed one [
This feature extraction gets a vector for every word, so we got di erent numbers
of vectors for every tweet. To train a neural network we need a recurrent layer to
deal with di erent size of vectors. In our case we used Long short term memory
(LSTM) layers [
We used both feature extraction methods tf-idf and word embeddings in order to improve our past model. We need a neural network which can join both. In Figure 1 we can see our base neural network to work with.
We tried some di erent models, replacing the LSTM by a BLSTM, adding dropout, batch normalization and more fully connected layers. To improve the LSTM learned, we added another output layer just before the LSTM. 4
We are going to show the experiments done. We will divide these sections in two parts: The rst is for the experiments with the rst feature extraction method, which was the n-grams approximation with the tf-idf approach. Next we proceed to do the same with the embeddings method. Finally we join both methods in one. We show the best results with f1 macro score obtained with every model and feature extraction because the task will be evaluate with it due to the unbalanced corpus. In addition we show the average precision for every result. The f1 score considers both the precision and recall. The formula for the f1 score is: To calculate the f1 macro score we only need to calculate the f1 score for every class and nd their unweighted mean. In Table 1 we can see our best result in terms of f1 macro score and its correspondent accuracy obtained in every machine learning algorithm. First of all, we can see the support vector machine column. In the second column we have the experiments with random forest and the third column is for the test with neural networks.
The random forest results are quite far from the others as it is shown in the
table. As we said before the random forest is not so common as svm for task
related to text but we want to make a comparison because we saw some studies
where obtain very good marks [
For the random forest, the best results are obtained with the tf-idf feature extraction and a random forest form by ten decision trees forced to have only one sample in every leaf.
In the case of the support vector machine, we get the best with the same feature extraction and a group of SVM (one for every pair of classes) with linear kernel faced each other to decide the predicted label.
To obtain the best result on this table we use a multilayer perceptron with 3
hidden layers of size 1024, 512, 256 for each layer respectively. Due to the size
of the corpus we need to avoid the over tting. In this case we use dropout [
In this section we trained two models of embeddings. The rst one was trained
with only the training tweets provided for the competition and the second one
with these tweets along with the tweets provided by Javier Palanca Camara [
In Table 2 are shown the best results obtained with LSTM and BLSTM along with the best results obtained in the previous section with svm, rf and nn. As we say before BLSTM have some improvements in LSTM but both have worse results compared to the approach of n-grams and tf-idf. We can conclude that embeddings and recurrent neural networks in this task need much more samples that other approximations to be competitive. 4.3
Best results we obtained are shown in Table 3. As we said before, we got better result with the BLSTM than with the LSTM. We have improved the results of the LSTM and the BLSTM with embeddings but can not improve the results with the neural network and td-idf. One of the biggest problems for this method is the number of parameters we need to estimate. Furthermore, more data to train the model is needed.
We can conclude that traditional models like support vector machines and multilayer perceptron have worked better than fashionable approaches such as LSTM and BLSTM. We saw the same results comparing n-grams and tf-idf against embeddings. These results are due to the size of the training set, so we might obtain better results using data augmentation techniques.
In order to improve our previous models, we can try some new approaches like
fastText [