Obtaining information from tweets has become a eld of interest in recent years due to its power to provide information about the insights of the users when any relevant event occurs. This is useful for companies and political parties that take advantage of this information in order to plan their next actions or to know whether or not their current actions are being received well by their public. In this work we show our approach for addressing COSET shared task (a tweet classi cation problem) using neural networks trained only with the data provided. With this approach we achieve the third position in the competition.
In the last ten years, social media, such as Twitter or Facebook, has experienced
a rapid growth that has changed the perspective of the socialization and public
communication [
In Twitter, political conversation is a common topic and its in uence increases when a General Election comes close since users speak more often about politicians or policies to criticize or praise them. This gives a lot of information related to how politicians or political groups are seen by citizenship. Also, detecting the most discussed topics is a key aspect in order to know people's concerns. As a result, this information can be used strategically during a political campaign to focus on those topics that citizens have doubts or want to address immediately.
In this context, COSET [
The di erent topics in which we have to classify the di erent tweets are: { Political issues. Related to the most abstract electoral confrontation. { Policy issues. Tweets about sectorial policies. { Campaign issues. Related to the evolution of the campaign. { Personal issues. Related to the personal life and activities of the candidates. { Other issues.
The distribution of tweets among the above topics is very imbalanced being the largest class (Policy issues) of a size of approx. 1100 tweets and the smallest (Personal issues) approx. 200 tweets in size. 3
In this section we will cover all the experimentation carried out. First we will focus on feature extraction and text representation. Then we will address the classi cation problem by describing the architectures used. 3.1
In order to transform the text into a continuous representation we apply some
preprocessing to clean as much as possible the data at hand. We have applied the
Tweet tokenizer from the NLTK python library [
Although usernames could seem useful, we decided to remove them because they did not contribute to improve the overall performance of the system and they also increased the size of the vocabulary by as much as 20%. 3.2
We have approached the tweet classi cation problem in a variety of ways, most of them with regards to the text representation issue. Here we will only name the various alternatives we have tested. Results will be presented in next section. As our rst approach we have tried representing the tweets with a xed length representation: { Bag of n-grams. To preserve information of both the vocabulary and the word order in the tweet. { Tf-Idf. To account for frequent and important words. { N-gram of characters. Similar to the previous approach but at character level.
Considering the great performance Recurrent Neural Networks (RNN) are
having while processing sequences[
When dealing with xed-length representation a multilayer perceptron (MLP) is used to tackle the classi cation problem. Due to the size of the training dataset and the size of each sample, the expressiveness of this model is more than enough to learn the representations explained before.
The rst few experiments were carried with a two-layer perceptron with 512 units in each layer. This model su ered from over tting very quickly (99.5% accuracy in training in 5 epochs), that is why we reduced the layer's size to 64 units each. With this con guration the over tting still occurred but it was manageable with noise.
Moreover, the best 4 models (trained with di erent initializations but identical con guration of parameters) were combined forming an ensemble to allow some collaboration among them in order to achieve better classi cation performance.
The combination of models has been implemented in a very straightforward way. The output of the last layer of all models is added and averaged by the number of models. The resulting vector is the one used to select the given class.
The nal architecture (used to submit the best results) is a two-layer perceptron with 64 units in each hidden layer. The activation function used on the hidden layers is ReLU while the one used on the output layer is Softmax. Gaussian noise is used on the hidden layers with 0.5 standard deviation. The model was trained during 13 epochs feeding the model with batches of size 128. No class weighting was used to compensate imbalanced classes.
The model is fed with tweets represented as bag of n-grams with the n-gram size up to 3. This results in a size of vocabulary of 47k. The training of this model takes no more than 30 seconds on cpu. 3.4
In the case of variable-length representation we used recurrent neural networks (RNN). With this model, the tweets are represented as a sequence of vectors of real numbers. These vectors are fed sequentially to the RNN which will obtain a xed length vector containing all the information extracted from the tweet. Then a multilayer perceptron will take that as input and perform the classi cation task as stated in Section 3.3. We use 128 as the embedding size and 64 as the hidden state of the recurrent unit. The multilayer perceptron has 64 units on its hidden layer.
In order to transform the text into a sequence of vectors of real numbers we must project our vocabulary into an embedding layer which will hold the continuous representation of every word (or character, depending on the representation) on the vocabulary. This continuous representation will be learned during the training process (no pretrained word embeddings used).
Moreover, additional tests have been made with a bidirectional RNN [
In Table 1 are compiled all the results obtained by training di erent models with di erent representations of the data and di erent con gurations. By looking at the table we can see that the best results are those obtained with the multilayer perceptron with bag of words representation. We can also see that other representations like tf-idf or other sizes of n-grams obtain similar results although with a slightly lower score. However, with better ne-tuning we were able to get better models that achieved several points above the ones presented in the results with the bag of 3-grams representation. For example, the model that generated the submission le achieved 62.4 f1-measure on the development set.
Ensembles are used to try to obtain better results by combining the same
model with di erent initialization. This has been proven to yield better results
[
Special attention has been paid to select which models were used to be part of the ensemble because the F-measure obtained was greatly in uenced with the performance obtained in one of the classes (usually, the one with fewer samples). If this had not been taken into account, the results obtained would have been much worse.
In the case of RNNs they have performed very poorly compared to the MLP probably due to the short length of the tweets, noisy behavior and low number of samples.
Finally, we only see a slight improvement with the character-wise representation when we use tf-idf bag of n-grams. All the others con gurations performed worse than its word-wise variants. Various ways for classifying tweets have been tested in order to nd which model suits the task the best. From a simple multilayer perceptron to RNN with gated units. Models trained with the rst architecture (MLP) yielded the best results while the ones obtained with RNN were signi cantly lower.
We think that this is due to the small size of the dataset with just a few thousands of samples. This has a major drawback: great over tting (and then, poor generalization). The expressiveness of the MLP is enough to achieve an acceptable performance with the data at hand while being in control of the over tting. However, although RNNs have a greater expressiveness, they also need much more data, especially to learn the word embedding, in order to accomplish good generalization. This makes RNN much more suited to tasks where the number of samples is big enough to learn its underlying characteristics.
The MLP architecture with the bag of 3-grams representation used in our experiments achieved the 3rd best result of all participants in COSET shared task.