Deep Stance and Gender Detection in Tweets on Catalan Independence@Ibereval 2017

Abstract

This paper discusses deepyCybErNet submission methodology to the task on Stance and Gender Detection in Tweets on Catalan Independence@Ibereval 2017. The goal of the task is to detect the stance and gender of the user in tweets on the subject "independence of Catalonia". Tweets are available in two languages: Spanish and Catalan. In task 1 and 2, the system has to determine whether the tweet is in favor of, against or neutral to the tweets on the subject pertaining to the task in Spanish and Catalan languages respectively. In task 3 and 4, the system has to decide whether the person who tweets is a male or female. We submitted three systems for this task a Bag-of-Words (BOW) representation for tweets with logistic regression classi er, Recurrent Neural Network (RNN) based approach, Long Short Term Memory (LSTM) based approach and gated recurrent based approach. These methods are highly language independent and can be used for the declarations of stance of tweets and identifying the gender of twitter user in any language. These methods have performed better in detecting stance and gender in tweets of Catalan language than in those of Spanish.

Keywords

• Sentimental analysis
• Bag-of-words embedding
• Deep learning
• Recurrent neural network (RNN)
• Long short-term memory (LSTM)
• Gated recurrent unit

-

Stance and gender detection in tweets is the task of automatically determining the polarity of the tweets and the gender of the twitter user who posted this particular message. In stance detection, the system has to detect whether this tweet is in favor of, against or neutral towards a proposition such as "independence of Catalonia".

The internet has given people a plenty of platforms to express their views on di erent subjects like Twitter, Facebook, WhatsApp etc. So people use these media to share their perspective on various topics in the society. So analyzing these twitter information is very much helpful in understanding the opinion of people and it also helps the respective o cials to take up necessary action. However, determining the stance and gender of texts which are phrased in gurative languages like tweets are very di cult for machines to unfold. Human can easily understand the underlying meaning of such expressions but, for a machine to unravel the meaning of rhetorical expressions such as sarcasm, irony, metaphor, analogy [ 14 ], it requires much additional information.

Many methods have been devised for automatically determining the stance and gender of microblog posts such as tweets. G. Zarella and A. Marsh [ 14 ] employed a Recurrent Neural Network (RNN) based method for classifying stance of tweets where word2vec skipgram method was used to represent features. In [ 3 ], I. Augenstein et.al used a bag-of-words autoencoder for extracting features and classi cation was performed using logistic regression. I. Augenstein et.al [ 2 ] used Long Short Term Memory (LSTM) with bidirectional embedding for stance detection. W. Wei et. al [ 13 ] used a Convolutional Neural Network (CNN) for the e ective detection of stance in tweets. Mohammad et.al [ 11 ] uses Support Vector Machine (SVM) and n-gram based method to detect the stance in tweets. A. Mislove et. al [ 10 ], C. Fink [ 6 ] introduced various measures for detecting the gender of the user who tweets.

Our method uses both Bag-of-Words (BoW) and a BoW-based recurrent embedding system for analyzing the stance in tweets. In rst case, BOW is used to obtain the feature representation for the tweets and classi cation is done using logistic regression. We also employed an RNN based method and LSTM based method for mining the stance of tweets. These methods are language independent. So irrespective of the language, we can use these approaches for nding the stance of micro blogging posts. 2

Task description

The main objective of Stance and Gender Detection in Tweets on Catalan Independence@Ibereval 2017 is to detect the stance and gender of people who tweets based on the topic "independence of Catalonia" [ 12 ].

The aim of this task can be divided into two. 1. Predict the stance of a given message. i.e., given message, the system has to predict whether the message is in favor of, against or neutral to the subject "independence of Catalonia". 2. Predict the gender of the user who tweets on the subject "independence of

Catalonia".

Tweets are given in two languages Spanish and Catalan. The system should be able to analyze tweets in these two languages and to detect the stance and gender of the tweet.

Data for this task are tweets on the topic "independence of Catalonia" during the regional election in September 2015. Shared task Task 1 and Task 2-Spanish Task 1 and Task 2- Catalan 3

Methodology

We set embedding size to 128 and word length to 40. Each word in tweet is mapped in to 128 dimensional vectors. Task 1 and task 2 has 4319 training samples. Thus, we formed a matrix of shape 4319X40. Each word is replaced in the resultant matrix of shape 4319X40 with their word embedding. This forms an input tensor of shape 4319X40X128. Finally, using the max-pooling approach, we converted an input tensor in to matrix of shape 4319X128 by xing 40 as maximum value for word length. This matrix is passed to logistic regression classi er and using argmax the prime stance and gender is selected. Recurrent neural network is an appropriate deep learning architecture for sequence data modeling. This has achieved intriguing results in various tasks in the eld of natural language processing [ 8 ]. It typically looks same as feed forward networks (FFN), additionally contains self-recurrent connection in units [ 5 ]. This cyclic loop carries out information from one time-step to another. As a result, RNN are able to learn the temporal patterns, value at current time-step is estimated based on the past and present states. Generally, RNN takes input as xt 2 Rq and hit 1 2 Rp of arbitrary length to compute succeeding hidden state vector hit by using the following formulae recursively.

ht = f (WxhXt + Whhht 1 + b) ot = sf (Wohht + bot) (1) (2) Where f is the nonlinear activation function, particularly logistic sigmoid function ( ) applied on element wise, hi0 is usually initialized to 0 at time-step t0 and Wxh 2 Rpxq, Wh 2 Rpx and b 2 Rm are arguments of a ne transformation. Here ot is the output at time step t.

We implemented a system based on RNN for stance and gender detection and run all experiments of them in GPU enabled Tensor ow [ 1 ]. Using the previously discussed mechanism such as bag-of-words embedding, we formed an input tensor of shape 4319X40X128. Each tweet embedding 40X128 is reduced to 128 dimension embedding vectors. This embedding vector are given to RNN layer to obtain optimal feature representations and followed by logistic regression and argmax function for classi cation. 3.3

Long short-term memory (LSTM) based system for Analysis of opinion and the gurative language on Twitter tweets RNN has vanishing and exploding gradient issue. To alleviate and to learn the long-term dependencies [ 7 ] introduced long short-term memory (LSTM). LSTM has a memory block instead of a simple RNN unit. A memory block contains one or more memory cell with a pair of adaptive multiplicative gates such as input and output gate. A memory block stores an information and updates them across time-steps based on the input and output gates. Input and output gate controls the input and output ow of information to a memory cell. Additionally, it is has a built-in value as 1 for Constant Error Carousel (CEC). This value will be activated when in the absence of value from the outside the signal. The newly proposed architecture has performed well in learning long-range temporal dependencies in various arti cial intelligence (AI) tasks [ 9 ]. Generally, at each time step an LSTM network considers the following 3 inputs; xt, ht 1, ct 1 and outputs ht, ct through the following below equations.

it =

(WiXt + Uiht 1 + Vimt 1 + bi) ft = ot = (Wf Xt + Uf ht 1 + Vf mt 1 + bf ) (WoXt + Uoht 1 + Vomt 1 + bo) mt = tanh (WmXt + Umht 1 + bm) mt = fti mt 1 + it

m ht = ot tanh (mt) (3) (4) (5) (6) (7) (8) where Xt is the input at time step t, is sigmoid non-linear activation function, tanh is hyperbolic tangent non-linear activation function, denotes elementwise multiplication. Concretely, at t = 0 hidden and memory cell state vectors such as h0 and c0 are initialized to 0.

We developed LSTM based system for stance and gender detection by replacing RNN layer with LSTM. 3.4

Gated recurrent unit (GRU) based system for Analysis of opinion and the gurative language on Twitter tweets As from the above formulae, we can say that LSTM has complex set of processing units. As a result, this needs more training time. Further the research on LSTM, [ 4 ] introduced Gated recurrent unit (GRU). GRU has less number of units in compared to LSTM, computationally e cient. The mathematical formulae of GRU is given below, i ft =

Wxi f + Whi f ht 1 + bi f ft =

(Wxf Xt + Whf ht 1 + bf ) mt = tanh (WxmXt + Whm (f r

ht 1) + bm) ht = f ht 1 + (1 f ) m (9) (10) (11) (12) Where 9 represents Update gate, 10 is for Forget or reset gate, 11 shows the equation for Current memory and 12 gives the equation for Updated memory.

Formulae shows, unlike LSTM memory cell with a list of gates (input, output and forget), GRU only consist of gates (update and forget) that are collectively involve in balancing the interior ow of information of the unit. In GRU, input gate (i) and forget gate (f ) are combined and formed a new gating units called update gate (i f ) that mainly focus on to balance the state between the previous activation (m) and the candidate activation (f ) without peephole connections and output activations. The forget gate resets the previous state (m). GRU networks looks simpler than LSTM with required only less computations. 4

Experiment and Results

We trained all experiments of various deep learning architectures using Tensorow [ 1 ].

Cross-validation performance To select the optimal parameters for tweet length and embedding size, 5-fold cross-validation is done on the given training samples of Catalan and Spanish for stance and gender detection. 10-fold crossvalidation accuracy across varied tweet length and embedding size for stance detection in Catalan language is displayed in Figure 1. 10-fold cross-validation accuracy across varied tweet length and embedding size for gender detection in Spanish language is displayed in Figure 2. 4.1

Evaluation results We have submitted 3 runs for each task; run1 is based on RNN mechanism, run2 is based on LSTM mechanism and run3 is based on GRU mechanism. The detailed evaluation results has been given by the Independence@Ibereval 2017 organizing committee are displayed in Table 2. The scores we obtained for task 1 are 0.285, 0.304 and 0.307 in detecting stance and 0.477, 0.490 and 0.501 in detecting gender for Spanish language. The scores we obtained for task 1 are 0.360, 0.379 and 0.326 in detecting stance and 0.465, 0.483 and 0.486 in detecting gender for Spanish language. This working note has presented a language independent method for the Independence@Ibereval 2017 shared tasks such as stance and gender in Twitter messages written in Catalan and Spanish using BoWs and embedding of RNN, LSTM and GRU. The presented supervised learning method has not relied on any resources; semantic resources such as dictionaries and ontologies or computational linguistics or feature engineering mechanisms for stance and gender detection in twitter tweets. Due to the less training corpus, the e cacy of RNN in stance and gender detection trails the classical BoWs approach. Though the e cacy of embedding of RNN, LSTM and GRU is acceptable and paves the manner in future to use for the analysis of stance and gender detection on Twitter tweets. Evaluating the performance of RNN, LSTM and GRU embedding with more training corpus for justi cation will be remained as one direction towards future work.

References

1. Abadi , M. , Barham , P. , Chen , J. , Chen , Z. , Davis , A. , Dean , J. , Devin , M. , Ghemawat , S. , Irving , G. , Isard , M. , et al.: Tensor ow: A system for large-scale machine learning . In: Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI). Savannah, Georgia, USA ( 2016 )

2. Augenstein , I. , Rocktaschel, T., Vlachos , A. , Bontcheva , K. : Stance detection with bidirectional conditional encoding . arXiv preprint arXiv:1606.05464 ( 2016 )

3. Augenstein , I. , Vlachos , A. , Bontcheva , K. : Usfd at semeval -2016 task 6: Anytarget stance detection on twitter with autoencoders . Proceedings of SemEval pp. 389 { 393 ( 2016 )

4. Cho , K. , Van Merrienboer, B. , Gulcehre , C. , Bahdanau , D. , Bougares , F. , Schwenk , H. , Bengio , Y. : Learning phrase representations using rnn encoder-decoder for statistical machine translation . arXiv preprint arXiv:1406.1078 ( 2014 )

5. Elman , J.L. : Finding structure in time . Cognitive science 14 ( 2 ), 179 { 211 ( 1990 )

6. Fink , C. , Kopecky , J. , Morawski , M. : Inferring gender from the content of tweets: A region speci c example . In: ICWSM ( 2012 )

7. Hochreiter , S. , Schmidhuber , J.: Long short-term memory . Neural computation 9(8) , 1735 { 1780 ( 1997 )

8. LeCun , Y., Bengio , Y. , Hinton , G.: Deep learning . Nature 521 ( 7553 ), 436 { 444 ( 2015 )

9. Lipton , Z.C. , Berkowitz , J. , Elkan , C. : A critical review of recurrent neural networks for sequence learning . arXiv preprint arXiv:1506.00019 ( 2015 )

10. Mislove , A. , Lehmann , S. , Ahn , Y.Y. , Onnela , J.P. , Rosenquist , J.N. : Understanding the demographics of twitter users . ICWSM 11 , 5th ( 2011 )

11. Mohammad , S.M. , Kiritchenko , S. , Sobhani , P. , Zhu , X. , Cherry , C. : Semeval-2016 task 6: Detecting stance in tweets . Proceedings of SemEval 16 ( 2016 )

12. Taule M , Mart M.A , R.F.R.P.B.C.P.V. : Overview of the task of stance and gender detection in tweets on catalan independence at ibereval 2017 . Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017 ), CEUR Workshop Proceedings. CEUR-WS.org ( 2017 )

13. Wei , W. , Zhang , X. , Liu , X. , Chen , W. , Wang , T.: pkudblab at semeval-2016 task 6: A speci c convolutional neural network system for e ective stance detection . Proceedings of SemEval pp. 384 { 388 ( 2016 )

14. Zarrella , G. , Marsh , A. : Mitre at semeval-2016 task 6: Transfer learning for stance detection . arXiv preprint arXiv:1606.03784 ( 2016 )