This paper describes our participation in the Classi cation Of Spanish Election Tweets (COSET) task at IberEval 2017. During the searching process for the best classi cation system, we developed a comparative study over possible combinations of corpus preprocessing, text representations and classi cation models. After an initial models exploration, we focus our attention over speci c neural models. Interesting insight can be drawn from the comparative study helping future practitioners tackling tweets classi cation problems to create system baseline for their work.
Nowadays the pervasive use of social media as a mean of communication helps researchers to found useful insight over open problems in the eld of Natural Language Processing. In this context, the Twitter social network has a huge role in text classi cation problems, because thanks to its API is possible to retrieve speci c formatted text (i.e., a sentence of maximum 140 characters called tweet) from a huge real-time text database, where di erent users publish their daily statements.
This huge availability of data gives raise to the investigation of new text classi cation problems, with special interest in prediction problems related to temporal events that can in uence statements published by social network users. An example of this problem category is the text classi cation related to general election, where the Classi cation Of Spanish Election Tweets (COSET) task at IberEval 2017 is a concrete example.
In COSET, the aim is to classify a corpus of political tweets in ve categories related to speci c political topics. This task can be analysed as a domain-dependent (i.e., political domain) constrained-text (i.e., tweet sentence) classi cation problem.
To tackle the above problem we built a classi cation system that can be decomposed in three main modules, each representing a speci c approach widely used in the Natural Language Processing literature: text pre-processing, text representation and classi cation model. During the modules design, we explore di erent design combinations leading the system development to a comparative study over the possible modules interactions. Analysing the produced study interesting insight can be drawn to create a system baseline for the tweet classi cation problem.
In the following sections we rstly describe the COSET task (Section 2), then we report the development process of the classi cation system and its module design (Section 3), after that, the evaluation of deployed systems over the provided corpus is analysed (Section 4), nally, conclusion over the whole work are outlined (Section 5). 2
The COSET shared task's [
Participants had access to a labelled corpus composed of training set (2242 tweets) and development set (250 tweets) for system benchmarking. We analysed it and nd the following statistical information presented in table 1. In this section we describe the tweet classi cation systems we built. From a module perspective we can describe our systems as composed of three main blocks: text pre-preprocessing (Section 3.2), text representation (Section 3.3) and classi cation model (Section 3.4). 3.1
To address the tweets classi cation problem we began our investigation analysing some of the
most widely used text representations and classi ers. In the analysing for possible text
representations we began focusing our attention on lexical features based on: Bag Of Words [
Regarding the text pre-preprocessing, it has to be mentioned that the corpus under
observation can not be treated as proper written language, because computer-mediated communication
(CMC) is highly informal, a ecting diamesic3 variation with creation of new items supposed to
pertain lexicon and graphematic domains [
From the previous consideration, we de ne a set of speci c tweet pre-processing approach that take into consideration the following items: (i) mentions (i.e., MT), (ii) smiley (i.e., SM), (iii) emoji (i.e., EM), (iv) hashtags (i.e., HT), (v) numbers (i.e., NUM), (vi) URL (i.e., URL) (vii) and Tweeter reserve-word as RT and FAV (i.e., RW).
For each of these items we left the possibility to be removed or substituted by constant string (e.g. (i) @ierrejon preparados para salir #PodemosRemontada substitutio!n $MENTION preparados para salir $HASHTAG, (ii) @ierrejon preparados para salir #PodemosRemontada removin!g preparados para salir ).
To implement above pre-processing technique we took advantage of the following tools:
(i) NLTK [
The use of neural modelS suggest us to exploit recent trend over text representation, in particular
we decided to use embedding vectors as representation following the approach described by [
Using this approach, after the sentence-matrix embeddings are calculated, following [
In this way four possible combinations of sentence-matrix embeddings can be formulated: (i) the use of a pre-trained embedding function (i.e., FastText from Wikipedia) and (ii) static or non-static weights. From this combination the one composed of static weight without pretrained embeddings won't be take in consideration for obvious reasons, meaning that the cases in consideration will be three: (i) ES static, (ii) ES non-static, (iii) (no pre-trained embeddings) non-static.
N-gram embedding. Choosing n-grams as elements to be mapped by the embedding function, raises more challenges respect simple words, because no pre-trained embeddings are available and 4 Preprocessor is a preprocessing https://github.com/s/preprocessor library for tweet data written in
Python, in this case the corpus has to be signi cantly big, otherwise n-gram frequencies will be really low and the estimation algorithm is not able to learn a valid embedding. Our insight was empirically validated by a very low performance. Nevertheless, as explained in the following, this embedding will be used in a particular model that won't rely its performance just over n-gram embeddings. 3.4
Following, we describe the neural models used for the classi cation module, where for each of
them the input layer uses text representations described in Section 3.3 (i.e., sentence-matrix).
Fast text. This model was introduced in [
Convolutional Neural Network. Convolutional Neural Networks (CNN) are considered state
of the art in many text classi cation problem. Therefore, we decide to use them in a simple
architecture composed by a convolutional layer, followed by a Global Max Pooling layer and two
dense layers. Between the two dense layers we used dropout(0.2) to avoid over tting.
KIM. This model was introduced in [
Bidirectional LSTM. Similar to the previous model, bidirectional LSTM is a variation of LSTM where the two RNN receive di erent inputs, the original and its reverse order, and their results are connected through the recurrent layers. Our architecture follows the previous one with an LSTM layer of 128 units terminating with two dense layers, where all layers used dropout (0.25) and recurrent dropout(0.25). 4
In this section we are going to illustrate results from the comparative study elaborated during the system development. First we illustrate the metric used to evaluate the system (Section 4.1) and then we report results produced by a 10-fold cross validation over the given data set (Section 4.2), nally we report our performance at the shared task (Section 4.3). 4.1
System evaluation metrics were given by the organizers and reported here in the following equations (1) to (4). Their choice was to use an F1 macro measure due to class unbalance in the corpus.
F1 macro = 1
X F1(yl; y^l) L j j l2L Following, we present a comparative study over possible combinations of pre-processing (Table 2) and word embeddings (Table 3), in both cases results are calculated from averaging three runs of a 10-fold cross validation over the complete data set. Notations used in Table 2 refer to the one introduced in Section 3.2, where the listing of a notation means its use for the reported result. Regarding the tweet speci c pre-processing, all the items have been substituted, with the exception for URL and RW that have been removed. We report the contribution of each analysed pre-processing alone. To not overwhelm reader with verbose data, reported results are focused only over the two best performing model (Kim's model and FastText).
From the analysis of Table 2 no absolute conclusion can be drawn, meaning that it wasn't possible to nd a combination of pre-processing that gives the best performance for all the models, meaning that each model is highly sensible to the performed combination. Nevertheless, some relative observation can be made: { SW (i.e., removing Spanish stopwords) and NUM (i.e., substitute numbers with a constant string) lead to performance improvement to all the model respect to no pre-processing at all, { ST (i.e., stemming) and HT (i.e., substitute hashtags with a constant string) decrease the performance of both models respect to non-preprocessing at all.
From above observations, to continue our study and x system's pre-processing pipeline, for each classi cation model we decided to use the combination of pre-processings that singularly applied give improvements to the system.
Analysing results in Table 3, here the used notation refers to the one introduced in Section 3.3, where the listing of a notation means its use as embedded input layer for the reported result. From its analysis the following interpretation can be drawn: For the system's submission, participants were allowed to send more than a model till a maximum of 5 possible runs, table 5 reports our best performing systems at the COSET shared task.
In this paper we have presented our participation in the IberEval2017 Classi cation Of Spanish Election Tweets (COSET) shared task. Five distinct neural models were explored, in combination with di erent types of preprocessing and text representation. From the systems evaluation it wasn't possible to nd a combination of pre-processing that gives the best performance for all the models, meaning that each model is highly sensible to the pipeline combination. Regarding the analysed text representation, the setting of sentence matrix to non-static always leads to good performance as a result of the speci c text under observation (i.e., a CMC corpus) and the use of pre-trained word embedding is always suggested. Moreover, we outline a not so promising performance of the recurrent model, meaning that for this task the word order (a feature well captured by LSTM family model) seems not so prominent as in other tasks.