This paper describes the participation of ELiRF-UPV team at the Spanish subtasks of the MultiModal Stance Detection in tweets on Catalan #1Oct Referendum workshop. Our best approach is based on Convolutional Neural Networks using word embeddings and polarity/emotion lexicons. We obtained competitive results on the Spanish subtask using only the text of the tweet, dispensing with contexts and images.
The corpus is composed by tweets labeled with respect to the stance of the Catalan rst October Referendum (2017). There are three classes: AGAINST (AG), NEUTRAL (NE) and FAVOR (FA). These tweets are provided in Spanish and Catalan, however, we worked only with the Spanish subtask. Moreover, although context of the tweet and images are also provided by the organizers, we only used the text of the tweet.
From the o cial training corpus, we randomly selected 80% in order to train
our models. The remaining 20% was used as development set. Table 1 shows the
sample distribution per class in the Spanish corpus.
In this Section, we describe the two models used in the competition. Both
models share the same preprocessing of the tweets by means of the TweetMotif [
As rst model for the experimentation, we used a Support Vector Machine (SVM) classi er with di erent representations of the tweets. Concretely, we used bag-of-word-ngrams and bag-of-char-ngrams with several values of n (including combination of ngrams e.g. bag-of-word-1-4grams means the concatenation of n = [1; 4]grams).
As second model for the experimentation, we used a Convolutional Neural
Network (CNN) architecture inspired by the work presented in [
We used ELHPolar [
We represent each tweet x as a matrix S 2 Rn (d+v), where n is the maximum number of words per tweet, d is the dimensionality of word embeddings and v is the dimensionality of the polarity/emotion features, that is, the number of polarity/emotion lexicons. In order to obtain this representation, we use an embedding model h(w) 2 Rd and a set of lexicons h0 (w) = [h01(w); h02(w); :::; h0l(w)] 2 Rv, where h0k(w) is the polarity value of the word w in the lexicon k.
Therefore, given a tweet x with n tokens, x = w1; w2; :::; wn, we represent it as a matrix S in which, each row i is the concatenation of the embedding of wi (h(wi)) and a vector with the polarity values of wi in each lexicon (h0 (wi)),
S = [h(w1)jh0 (w1); h(w2)jh0 (w2); :::; h(wn)jh0 (wn)]. In the case where a word wi is out of vocabulary for the embedding models, we replace its embedding by the embedding of the word \unknown", h(wi) = h(\unknown"). Similarly, if wi is not included in any lexicon, h0 (wi) = [0; 0; :::; 0] 2 Rv.
Due to the variable length of the tweets, we used zero padding at the start of a tweet if it does not reach the maximum speci ed length. Otherwise, if the length of a tweet is greater than the maximum, we only consider the rst n words of the tweet. In this task, the average number of words per tweet is navg = 18:5, and the maximum length is nmax = 34. We decided to set the length n = 26 which is the mean of navg and nmax.
Regarding the CNN architecture, we applied one-dimensional convolutions with variable height lters in order to extract the temporal structure of the tweet over several region sizes. Figure 1 summarizes the model architecture and its hyperparameters.
Batch Normalization + ReLU Convolution 1D Global Max Pooling
Concat + Batch Normalization Softmax fully-connected layer
4 region sizes ([
Concatenated salient features ℝ1024 do you think humans have sense
Batch Normalization Sentence matrix ℝ ×( + ) … … … … … … … ……… ……… ……… ……… ……… ……… … …… … …
⋮ ⋮ … ⋮ 3 classes
As can be seen in Figure 1, we used 4 di erent region sizes (the lter height range from 1 to 4) and 256 lters for each region size. We used this range of region sizes because, in the development phase, the best baseline was SVM using bagof-word-1-4grams. After the lters were applied, we obtained 256 output feature maps for each region size.
In order to extract the most salient features for each region size, we applied 1D Global Max Pooling to the feature maps of each region size. Therefore, we obtained 4 vectors with 256 components, that were concatenated and used as 4 input to a fully-connected layer which performs the classi cation task. We used a softmax activation function to model the posterior distribution of each class at the output layer. 4
In this section, we describe the experimental work conducted by ELiRF team in the MultiStanceCat task. In addition, we present a study of the performance of our best system in the competition.
Table 2 summarizes the results obtained in the development phase. Three di erent classi ers were considered: Linear SVM with bag-of-ngrams of words, Linear SVM with bag-of-ngrams of chars and CNNs.
For the SVM approaches we tested di erent values of n. With respect to the CNN we explored three loss functions: Cross Entropy (CCE), Mean Squared Error (MSE) and di erentiable approximation of the F1 measure (SMF1). It can be observed in Table 2 that generally bag-of-chars performs worse than bag-of-words. Note that CNN models outperform the results achieved by the SVM classi ers. Moreover, CNN classi er with SMF1 loss function outperforms the results of all the other classi ers. However, a deeper study about which factors such as embeddings or lexicons are more relevant in the results would be interesting.
It can be also observed that the value of the F1 measure for the NEUTRAL class (F1(N E)) is generally lower than the F1 measures for AGAINST and FAVOR classes (F1(AG); F1(F A)). We hypothesize this is due to the fact that NEUTRAL class has less samples in the corpus. However, low values of F1(N E) measure do not a ect the o cial evaluation measure that is de ned as the average between F1(AG) and F1(F A).
For the Spanish subtask competition, we selected the best CNN and SVM models according to the results obtained in the development phase. Concretely, our rst run (ELiRF-1) was the CNN model trained using SMF1 loss function. As a second run (ELiRF-2) we selected a Linear SVM with bag-of-word-1-4grams.
Table 3 shows the confusion matrices of the two submitted systems. It can be observed that both systems confuse the NEUTRAL and the AGAINST classes in a similar way. The best performance achieved by ELiRF-1 run is because it predicts better the FAVOR class. We have also performed a study of the samples that ELiRF-1 system misclassi ed with high con dence. Some of these samples are shown in Table 4. We think that in some cases, errors could be avoided by considering hashtags (sample 5, #4gatos) or user mentions (error 2, @CatalunyaPlural). Unfortunately, we have not included this information in our models. 6
Table 5 shows the results on the test set for all the participating teams in the Spanish task. The ELiRF-1 run obtained competitive results without using the text of previous and next tweets or the images in the user timeline. Moreover, we can observe that the context seems to be useful for this task because all the best participating teams used this information. Finally, we would like to highlight the great di erence observed in the results obtained on the development and the test sets. We have no explanation for this, but we think that a study about this aspect should be done when the test set will be available. In this paper, we have presented the participation of the ELiRF team at MultiStanceCat track of the IberEval workshop. Our team participated in the Spanish subtask of this track and competitive results were achieved using only the text of the tweets. Our best approach is based on CNN with sequential representation of the tweets using word embedding, and polarity/emotion lexicons.
As future work, we plan to include the context of the tweet in our deep
learning system in a similar way as Hierarchical Attention Networks [
We have observed that hashtags and user mentions contains relevant information for this task. For this reason, as future work, we want to explore the inclusion of this information in the tweet representation. 6
This work has been partially supported by the Spanish MINECO and FEDER founds under project AMIC (TIN2017-85854-C4-2-R). Work of Jose-Angel Gonzalez is also nanced by Universitat Politecnica de Valencia under grant PAID-01-17.