We describe the system submitted to IBEREVAL-2017 for stance and gender detection in tweets on Catalan Independence [1]. We developed a supervised system using Support Vector Machines with radial basis function kernel to identify the stance and gender of the tweeter using various character level and word level features. Our system achieves a macro-average of F-score(FAVOR) and F-score(AGAINST) of 0.46 for stance detection in both Spanish and Catalan and an accuracy of 64.85% and 44.59% for Gender detection in Spanish and Catalan respectively.
The task of opinion mining and sentiment analysis on natural language texts in social media has gained a lot of popularity and importance in recent times. Stance detection is related to sentiment analysis but is very di erent from it. In sentiment analysis we check if a tweet has a positive, negative or neutral emotion while in stance detection we check whether the tweet is in favor, neutral or against a given target which in this paper is independence of Catalonia. For example, consider the following sentence: \Recent studies have shown that global warming is in fact real ". We can say that this sentence's author is most likely to be in favor of the concept `global warming'.
There have been several experiments [2],[10] in the eld of sentiment analysis and opinion mining on social media text. Opinion mining can provide a lot of information about the texts that are present in social media and bene ts a lot of other tasks such as information retrieval, text summarization, etc.
On the other hand gender detection is the task of inferring the gender of author from the content of the tweet. Gender detection has many applications in the eld of marketing and advertising and thus there have been a lot of studies [3],[4],[6],[11] on gender detection in social media text. Twitter pro les don't provide a eld for person's gender which makes the task of identifying author's gender from the tweet much more important.
In this paper we present a system for stance and gender detection in tweets. Our system uses character and word level features and Support Vector Machines with radial basis function kernel for classi cation.
The organizers provided training and test dataset which consisted of 4319 tweets and 1081 tweets respectively, for both Spanish and Catalan. All the tweets in the training dataset are annotated with stance (`FAVOR' or `AGAINST' or `NONE') and gender (`FEMALE' or `MALE').
Stance detection systems are evaluated using macro-average of F-score (FAVOR) and F-score (AGAINST) i.e.
(F scoreF AV OR + F scoreAGAINST )=2
On the other hand gender detection systems are evaluated using accuracy i.e. number of tweets for which the gender is predicted correctly per hundred tweets. 3
3.1
Initially tweets are tokenized in a way such that hashtags, URLs and mentions are preserved. Then URLs, mentions and stopwords are removed from the tweets.
It can be observed from the tweets present in the training and test datasets that almost all the hashtags are written in camel case format. Therefore, `#' is removed from the hashtags and all the words are extracted from the hashtag. And then each word is considered as a separate token.
All the tokens in Spanish are then stemmed using Snowballstemmer implemented in NLTK. 3.2
We extracted various features from the given tweets to train our machine learning model. We list and describe these features below.
Character N-grams Character n-grams feature refers to presence or absence of contiguous sequence of n characters. It can be seen from previous work [2],[3],[4] that character level features have a signi cant e ect on stance and gender detection.
We extract character n-grams for all values of n between 1 and 3. Including all the n-grams increases the size of feature vector enormously. Therefore, we consider only those n-grams in our feature vector which occur at least 10 times in the training dataset. This reduces the size of feature vector signi cantly and also removes noisy n-grams. 3 Word N-grams Word n-grams feature refer to presence or absence of contiguous sequence of n words or tokens. Word n-grams have proven to be important features for stance and gender detection in previous studies [5],[6]. We extract word n-grams for all values of n between 1 and 5. We include only those n-grams in our feature vector which occur at least 10 times in the training dataset. Stance and Gender Indicative Tokens This feature refers to presence or absence of stance and gender indicative tokens. We use a variation of the approach to nd stance indicative hashtags [2] and extract stance and gender indicative tokens. We calculate a score for each token for both stance and gender where score is de ned as :
Scorestance(token) = maxstance label2Stance Set Scoregender(token) = maxgender label2Gender Set f req(token; stance label)
f req(token) f req(token; gender label) f req(token) where Stance-Set = fFAVOR, AGAINST, NEUTRALg, Gender-Set = fMALE, FEMALEg.
We consider only those tokens as features for stance indication which have a score 0.6 and occur at least ve times in the training dataset. For gender indication we consider only those tokens which have a score 0.7 and occur at least twice in the training dataset. The threshold value for scores and number of occurrences has been decided after empirical ne tuning. 3.3
Previous studies [4],[7] have shown that feature selection algorithms improve e ciency and accuracy of classi cation systems. We use chi square feature selection algorithm which uses chi-squared statistic to evaluate individual feature with respect to each class. This algorithm was run for both stance and gender detection in order to extract the best features and reduce the feature vector size. 3.4
Support Vector Machines have been used many times previously [2],[8],[9] for stance and gender detection and has proven to be a very e ective classi cation technique for the same.
After pre-processing the dataset and extracting all the desired features, we use scikit-learn Support Vector Machine implementation with a radial basis function kernel for classi cation. We also perform 10-fold cross validation on the provided training dataset to develop the system. 10-fold cross validation is run for each of the individual features separately to observe the e ect of each feature on classi cation. 4 3.5
Our system achieves a macro-average of F-score(FAVOR) and F-score(AGAINST) of 0.46 for stance detection in both Spanish and Catalan and an accuracy of 64.85% and 44.59% for gender detection in Spanish and Catalan respectively for the given test dataset.
Table 1. shows the accuracy in percentage achieved for stance and gender detection for Spanish tweets while Table 2. shows the accuracy in percentage achieved for stance and gender detection in Catalan tweets considering one feature at a time and also considering all the features together. These are the results achieved in 10-fold cross validation on training dataset. In this paper, we presented our approach for stance and gender detection for tweets in both Spanish and Catalan using character and word level features and Support Vector Machine technique for classi cation. It can also be observed from the results of 10-fold cross validation on training dataset that character n-grams have a signi cant e ect on classi cation.
Our system has a lot of room for improvement and future work will include extracting more features such as POS-tags and word embeddings and using several other supervised and unsupervised machine learning algorithms for classi cation.