This paper describes the participation of ELiRF-UPV team at the two Spanish subtasks of the Stance and Gender Detection in Tweets on Catalan Independence track of the IberEval workshop. We tested several approaches based on di erent models and tweet representations. Our best approaches are based on neural networks with one-hot vector representation and Support Vector Machines using bag-of-ngrams of chars. We achieved the rst place on the gender detection subtask and the fourth place on the stance detection subtask.
The corpus is composed by tweets labeled with respect to the independence of Catalonia (three classes: AGAINST, NEUTRAL, FAVOR) and with respect to the gender of the author of each tweet (two classes: MALE and FEMALE).
These tweets are provided in Spanish and Catalan, however, we have only worked with the Spanish version of the proposed corpus. On the other hand, it is necessary to take into account that the corpus is unbalanced in terms of stance detection, being a clear bias between classes AGAINST and NEUTRAL with respect to class FAVOR. This unbalance does not occur in the gender detection subtask as can be seen in Table 1. In this section we describe the main characteristics of the system developed to the Stance and Gender Detection in Tweets on Catalan Independence track of the IberEval workshop. This description includes the preprocessing used, the di erent tweets representations used and, the di erent models that were taken into account during the tuning phase. The preprocessing process of the tweets was a bit di erent depending on the subtasks. In both cases, we removed the accents and converted all the text to lowercase. The web links (URL), and the numbers were substituted by a speci c label.
We assumed that the hashtags, the emoticons and the mentions to other users would be informative to determine the opinions of a user but not his/her gender. Accordingly to this assumption, we substituted the hashtags, the emoticons and user's mentions by a speci c label for the Gender subtask, but we kept their values for the Stance subtask.
Since stance detection may be related, in some way, to sentiment analysis, we
tested the use of polarity lexicons for the Stance subtask. Speci cally, we tried
to include NRC lexicon [
We explored di erent models depending on the representation of the tweets.
This way, Long short-term memory (LSTM) neural networks [
The NRC polarity lexicon was used only with the embeddings representation.
The topology used was similar to the one described in [
In order to select the representations and the models (including their parameters) more appropriated to each subtask, a tuning process was performed. The corpus provided by the organizers of the task was split into two sets, a set with the 80% of the tweet for learning the model and the remaining 20% of the corpus was used as tuning. The partitions were the same for all the tuning process. For tuning the models of each subtask, the o cial evaluation measure of each subtask was taken into account as optimization criterion.
Faced with the impossibility of testing all combinations of models and representations, only those combinations we thought that made more sense were considered. Table 2 shows the most relevant combinations of features and models as well as the results obtained during the tuning phase.
Regarding the stance detection subtask, as can be seen in the rst row of Table 2, with sequential representation at word level (Wikipedia embeddings) the result obtained was 51.84 for F1 measure. Unfortunately, worse results were obtained when the polarity sequence according to the NRC lexicon was added to the embeddings representation (Emb+NRC).
We also tried the use of a di erent sequential representation formed by onehot vectors at character level and processed with the same network used in the previous experiments. This representation together with CNN + LSTM obtained the best results on tuning, 55.10 of F1 measure as can be seen in the third row of Table 2.
Regarding the gender detection subtask, although the sequential representation at word level (embeddings from Wikipedia) processed using CNN + LSTM obtained good results, 64.47% in Accuracy, the best results in the tuning phase were achieved by the representations based on bag-of-ngrams of chars. The representation based on bag-of-ngrams of words achieved signi cantly worst results.
The models that obtained better results were the Support Vector Machines with linear kernel. Speci cally, the SVM model using bag-of-unigram of chars as representation of the tweets achieved 66.92% of Accuracy; while adding bagof-bigrams of chars to the previous model slightly increases the Accuracy to 66.99%. These results correspond to the last two rows in Table 2. 4
In view of the results obtained during the tuning phase and due the limitation of the track, we decided to send the following two runs to the competition. { run1 stance detection: CNN + LSTM + char-one-hot gender detection: SVM + bag-of-2grams of chars { run2 stance detection: CNN + LSTM + char-one-hot (the same as in run1) gender detection: SVM + bag-of-1grams of chars + bag-of-2grams of chars (the best accuracy at tunning)
Once we have analyzed the results, both in the tuning phase and in the o cial competition, we want to point out some interesting things.
In both subtasks, methods based on deep-learning have shown to o er competitive results. However, in the case of gender detection subtask, the best results have been obtained with a priori simple model, SVM and bag-of-chars. We hypothesize that the good results achieved by SVM models in this subtask is due to the greater robustness of these models (compared with deep-learning based models) to deal with the bias problem. If the imbalance is very large, it can cause that the network assigns all the samples only to the majority classes. The solution used for the stance detection subclass was to perform a scaling of the loss function during the training phase. This has prevented the network from classifying all tweets in the AGAINST and NEUTRAL classes (the majority classes with much di erence in the Spanish version of the stance detection corpus).
Regarding the stance detection subtask, a sequential character-level
representation has been chosen due to the increasing interest this kind of representations
are having in the deep-learning area and the good results they are achieving
[
We have presented the participation of the ELiRF-UPV team at the Stance and Gender Detection in Tweets on Catalan Independence track of the IberEval workshop. Our team has participated in the two Spanish subtasks of the track and has achieved competitive results. Our best approaches were based on neural networks with sequential representation of the tweets and Support Vector Machines with bag-of-ngrams of chars.
As future work, we plan to use representations based on one-hot vectors at character level and CNN + LSTM on other tweet classi cation problems (TASS, SemEval, ...) in order to study their behavior in tasks other than stance detection.
This work has been partially supported by the Spanish MINECO and FEDER founds under project ASLP-MULAN: Audio, Speech and Language Processing for Multimedia Analytics, TIN2014-54288-C4-3-R.