This paper describes our participation in the Stance and Gender Detection in Tweets on Catalan Independence (StanceCat) task at IberEval 2017. Our approach was focused on neural models, rstly using classical and speci c model from state of the art, then we introduce a new topology of convolutional network for text classi cation. The raising of social networks as worldwide means of communication and expression, is gaining lot of interest from company and academia, due to the huge availability of daily contents published by users. Focusing on academia perspective, especially in the Natural Language Processing eld, the contents available in form of written text are really useful for the study of speci c open problems, where the stance detection related to political events is an example, and the Stance and Gender Detection in Tweets on Catalan Independence (StanceCat) task at IberEval 2017 is a concrete application. In StanceCat, the principal aim is to automatically detect if the text's author is in favor of, against, or neutral towards the Catalan Independence. Moreover, as a secondary aim, participants are asked to infer the author's gender. To tackle the described problem we built a stance&gender detection system mainly decomposed in two modules: text pre-processing and classi cation model. During the system's tuning process, di erent design choices were explored trying to nd the best modules' combination and from their anlysis some interesting insight can be drawn. In the following sections we rstly describe the StanceCat task (Section 2), then we illustrate the module's design of developed stance&gender detection system (Section 3), after that, an evaluation of the tuning process for submitted systems is analysed (Section 4), nally, conclusion over the whole work are outlined (Section 5).
Regarding the text pre-preprocessing, has to be mentioned that the corpus under observation can
not be treated as proper written language, because computer-mediated communication (CMC) is
highly informal, a ecting diamesic3 variation with creation of new items supposed to pertain
lexicon and graphematic domains [
Following, we describe the neural models used for the classi cation module. Before introducing the models we describe the speci c text representation used as input layer Section 3.2 (i.e., sentence-matrix).
Text representation To represent the text we used word embeddings as described by [
Choosing words as elements to be mapped by the embedding function, raise some challenge over the function estimation related to data availability. In our case the available corpus is very small and estimated embeddings could lead to low performance. To solve this problem, we decided 3 The variation in a language across medium of communication (e.g. Spanish over the phone versus
Spanish over email)
4 Preprocessor is a preprocessing library for tweet data written in Python,
https://github.com/s/preprocessor
to used a pre-trained embeddings estimated over Wikipedia using a particular approach called
fastText [
Convolutional Neural Network. Convolutional Neural Networks (CNN) are considered state of the art in many text classi cation problem. Therefore, we decide to use them in a simple architecture composed by a convolutional layer, followed by a Global Max Pooling layer and two dense layers.
Dilated KIM. This model is our new topology of CNN. It can be seen as an extension of Kim's
model [
The original Kim's model is a particular CNN where the convolutional layer has multiple lter widths and feature maps. The complete architecture is illustrated in Figure 1, here the input layer (i.e., sentence-matrix) is processed in a convolutional layer of multiple lters with di erent width, each of these results are fed into Max Pooling layers and nally the concatenation of them (previously atten to be dimensional coherent) is projected into a dense layer. Our extension is to use a dilated lters in combination with normal ones, the intuition is that normal lter capture adjacent words features, while dilated one are able to capture relations between non adjacent words. This behaviour can't be achieved by the original Kim's model, because, even though the lters size can be changed, they will capture only features from adjacent words.
Regarding the architectural references in [
In this section we are going to illustrate the evaluation of developed systems regarding the modules design reported in section 3. First we illustrate the metric proposed by organizers for system's evaluation (Section 4.1), then we outline empirical results produced by a 10-fold cross validation over the given data set (Section 4.2), nally we report our performance at the shared task (Section 4.3). 4.1
System evaluation metrics were given by the organizers and reported here in the following equations (1) to (6). Their choice was to use an F1 macro measure for stance detection, due to class unbalance, while a categorical accuracy for the gender detection.
Gender = accuracy =
P T P + P T N
P sample (1)
Stance =
F1 macro(F avor) + F1 macro(Against) 2 (2) (5) (6) precision recall precision + recall (3) (4) precision = recall = 1
X P r(yl; y^l) L j j l2L 1
X R(yl; y^l) L j j l2L where L is the set of classes, yl is the set of correct label and y^l is the set of predicted labels. 4.2
Following, we describe the ne tuning process of our proposed model over possible combinations of pre-processing (Table 3), then we compare Kim's model against our extension (Table 4) and nally report the improvement over the use of a data augmentation technique (Table 5). For brevity of information only the evaluation of Dilated Kim's model over Spanish stance detection is reported, in details, the results are calculated from averaging three runs of a 10-fold cross validation over the complete data set. Nevertheless, the results obtained after the ne tuning process for all the models are reported in section 4.3, where their development performances are compared against the ones obtained in the StanceCat task.
Notations used in Table 3 refer to the one introduced in Section 3.1, where the listing of a notation means its use for the reported result. Regarding the tweet speci c pre-processing, all the items have been substituted, with the exception for URL and RW that have been removed. We report the contribution of each analysed pre-processing alone. { Most of common used preprocessing decrease model's performance, meaning that their information can be directly exploited by the model { Only stemming and mention removals brought small improvements, therefore they will be used as best tuning for our proposed model. ES
CA
Gender ES
CA Kim 0:624( 0:017) 0:630( 0:022) 0:634( 0:011) 0:655( 0:017)
Dilated Kim 0:658( 0:039) 0:659( 0:028) 0:652( 0:013) 0:715( 0:015)
From the analysis of Table 4 we can outline a signi cant performance's improvement of our proposed model respect the original Kim's model.
Due to the fact that our development data set has few samples, to train our models we decided to apply a data augmentation technique that didn't rely over external data rather exploit the word embedding text representation. In details, we applied Gaussian noise to word embeddings and after every convolutional layers, and, to further improve performances, we take advantage of batch normalization. Best results were archived with additive zero centered noise, with a standard deviation of 0.2. Regarding the batch normalization layers, default keras parameters were used. Results of this technique respect the Dilated Kim's model are reported in table 5. 4.3
For the system's submission, participants were allowed to send more than a model till a maximum of 5 possible runs, therefore in tables 6 and 7 we report our best performing systems (tuned following the process in section 4.2) for the StanceCat shared task.
Unfortunately, due to a submission error caught only after the o cial results were published, we didn't manage to be properly evaluated (in tables 6 and 7 the submitted models with name in parentheses and asterisk), therefore after the closing we asked organizers to evaluate some of our model to see how they would had performed (the models with test results in bold). In this paper we have presented our participation in the Stance and Gender Detection in Tweets on Catalan Independence (StanceCat) task at IberEval2017. Five distinct neural models were explored, in combination with di erent types of preprocessing. From the ne tuning process we System System derived that most of well know pre-processing technique are strongly model dependent, meaning that the preprocessing pipeline has to be optimized depending on the classi er. Finally, our proposal of a dilation technique for NLP task, the Dilated Kim's model, seems to increase performances of CNN base classi ers.