=Paper=
{{Paper
|id=Vol-1881/StanceCat2017_paper_8
|storemode=property
|title=Deep Stance and Gender Detection in Tweets on Catalan Independence@Ibereval 2017
|pdfUrl=https://ceur-ws.org/Vol-1881/StanceCat2017_paper_8.pdf
|volume=Vol-1881
|authors=R. Vinayakumar,S. Sachin Kumar,B. Premjith,Poornachandran Prabaharan,Kotti Padannayil Soman
|dblpUrl=https://dblp.org/rec/conf/sepln/VinayakumarKPPP17
}}
==Deep Stance and Gender Detection in Tweets on Catalan Independence@Ibereval 2017==
https://ceur-ws.org/Vol-1881/StanceCat2017_paper_8.pdf
Deep Stance and Gender Detection in Tweets on
Catalan Independence@Ibereval 2017
Vinayakumar R1? , Sachin Kumar S1 , Premjith B1 , Prabaharan P2 and Soman
K P1
1
Center for Computational Engineering and Networking,Amrita School of
Engineering, Amrita Vishwa Vidyapeetham, Amrita University, India
{vinayakumarr77}@gmail.com
2
Center for Cyber Security Systems and Networks, Amrita School of Engineering,
Amrita Vishwa Vidyapeetham, Amrita University, India
Abstract. This paper discusses deepyCybErNet submission methodol-
ogy to the task on Stance and Gender Detection in Tweets on Catalan
Independence@Ibereval 2017. The goal of the task is to detect the stance
and gender of the user in tweets on the subject ”independence of Catalo-
nia”. Tweets are available in two languages: Spanish and Catalan. In task
1 and 2, the system has to determine whether the tweet is in favor of,
against or neutral to the tweets on the subject pertaining to the task in
Spanish and Catalan languages respectively. In task 3 and 4, the system
has to decide whether the person who tweets is a male or female. We sub-
mitted three systems for this task a Bag-of-Words (BOW) representation
for tweets with logistic regression classifier, Recurrent Neural Network
(RNN) based approach, Long Short Term Memory (LSTM) based ap-
proach and gated recurrent based approach. These methods are highly
language independent and can be used for the declarations of stance of
tweets and identifying the gender of twitter user in any language. These
methods have performed better in detecting stance and gender in tweets
of Catalan language than in those of Spanish.
Keywords: Sentimental analysis, Bag-of-words embedding, Deep learning: Re-
current neural network (RNN), Long short-term memory (LSTM), Gated recur-
rent unit
1 Introduction
Stance and gender detection in tweets is the task of automatically determining
the polarity of the tweets and the gender of the twitter user who posted this par-
ticular message. In stance detection, the system has to detect whether this tweet
is in favor of, against or neutral towards a proposition such as ”independence of
Catalonia”.
The internet has given people a plenty of platforms to express their views
on different subjects like Twitter, Facebook, WhatsApp etc. So people use these
?
vinayakumarr77@gmail.com
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
media to share their perspective on various topics in the society. So analyzing
these twitter information is very much helpful in understanding the opinion of
people and it also helps the respective officials to take up necessary action. How-
ever, determining the stance and gender of texts which are phrased in figurative
languages like tweets are very difficult for machines to unfold. Human can easily
understand the underlying meaning of such expressions but, for a machine to
unravel the meaning of rhetorical expressions such as sarcasm, irony, metaphor,
analogy [14], it requires much additional information.
Many methods have been devised for automatically determining the stance
and gender of microblog posts such as tweets. G. Zarella and A. Marsh [14] em-
ployed a Recurrent Neural Network (RNN) based method for classifying stance
of tweets where word2vec skipgram method was used to represent features. In
[3], I. Augenstein et.al used a bag-of-words autoencoder for extracting features
and classification was performed using logistic regression. I. Augenstein et.al [2]
used Long Short Term Memory (LSTM) with bidirectional embedding for stance
detection. W. Wei et. al [13] used a Convolutional Neural Network (CNN) for
the effective detection of stance in tweets. Mohammad et.al [11] uses Support
Vector Machine (SVM) and n-gram based method to detect the stance in tweets.
A. Mislove et. al [10], C. Fink [6] introduced various measures for detecting the
gender of the user who tweets.
Our method uses both Bag-of-Words (BoW) and a BoW-based recurrent
embedding system for analyzing the stance in tweets. In first case, BOW is used
to obtain the feature representation for the tweets and classification is done
using logistic regression. We also employed an RNN based method and LSTM
based method for mining the stance of tweets. These methods are language
independent. So irrespective of the language, we can use these approaches for
finding the stance of micro blogging posts.
2 Task description
The main objective of Stance and Gender Detection in Tweets on Catalan Inde-
pendence@Ibereval 2017 is to detect the stance and gender of people who tweets
based on the topic ”independence of Catalonia” [12].
The aim of this task can be divided into two.
1. Predict the stance of a given message. i.e., given message, the system has to
predict whether the message is in favor of, against or neutral to the subject
”independence of Catalonia”.
2. Predict the gender of the user who tweets on the subject ”independence of
Catalonia”.
Tweets are given in two languages Spanish and Catalan. The system should
be able to analyze tweets in these two languages and to detect the stance and
gender of the tweet.
Data for this task are tweets on the topic ”independence of Catalonia” during
the regional election in September 2015.
223
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
Table 1. Statistics of tweets
Shared task Number of tweets in Training Number of tweets in Testing
Task 1 and Task 2-Spanish 4319 1081
Task 1 and Task 2- Catalan 4319 1081
3 Methodology
This section discusses the mechanism adopted for Stance and Gender Detection
in Tweets on Catalan and Spanish Independence@Ibereval 2017. We have used
two methods for stance and gender in Twitter messages; (1) Bag-of-words (BoW)
embedding (2) recurrent neural network based word embedding.
3.1 Bag-of-words based system for Analysis of Stance and Gender
Detection in Tweets
We set embedding size to 128 and word length to 40. Each word in tweet is
mapped in to 128 dimensional vectors. Task 1 and task 2 has 4319 training
samples. Thus, we formed a matrix of shape 4319X40. Each word is replaced in
the resultant matrix of shape 4319X40 with their word embedding. This forms an
input tensor of shape 4319X40X128. Finally, using the max-pooling approach,
we converted an input tensor in to matrix of shape 4319X128 by fixing 40 as
maximum value for word length. This matrix is passed to logistic regression
classifier and using argmax the prime stance and gender is selected.
3.2 Recurrent neural network (RNN) based system for Analysis of
Stance and Gender Detection in Tweets
Recurrent neural network is an appropriate deep learning architecture for se-
quence data modeling. This has achieved intriguing results in various tasks in
the field of natural language processing [8]. It typically looks same as feed for-
ward networks (FFN), additionally contains self-recurrent connection in units
[5]. This cyclic loop carries out information from one time-step to another. As a
result, RNN are able to learn the temporal patterns, value at current time-step
is estimated based on the past and present states. Generally, RNN takes input
as xt ∈ Rq and hit−1 ∈ Rp of arbitrary length to compute succeeding hidden
state vector hti by using the following formulae recursively.
ht = f (Wxh Xt + Whh ht−1 + b) (1)
ot = sf (Woh ht + bot ) (2)
Where f is the nonlinear activation function, particularly logistic sigmoid func-
tion (σσ) applied on element wise, hi0 is usually initialized to 0 at time-step t0
and Wx h ∈ Rpxq , Wh ∈ Rpx and b ∈ Rm are arguments of affine transformation.
Here ot is the output at time step t.
224
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
We implemented a system based on RNN for stance and gender detection
and run all experiments of them in GPU enabled Tensorflow [1]. Using the
previously discussed mechanism such as bag-of-words embedding, we formed an
input tensor of shape 4319X40X128. Each tweet embedding 40X128 is reduced
to 128 dimension embedding vectors. This embedding vector are given to RNN
layer to obtain optimal feature representations and followed by logistic regression
and argmax function for classification.
3.3 Long short-term memory (LSTM) based system for Analysis of
opinion and the figurative language on Twitter tweets
RNN has vanishing and exploding gradient issue. To alleviate and to learn the
long-term dependencies [7] introduced long short-term memory (LSTM). LSTM
has a memory block instead of a simple RNN unit. A memory block contains one
or more memory cell with a pair of adaptive multiplicative gates such as input
and output gate. A memory block stores an information and updates them across
time-steps based on the input and output gates. Input and output gate controls
the input and output flow of information to a memory cell. Additionally, it is
has a built-in value as 1 for Constant Error Carousel (CEC). This value will
be activated when in the absence of value from the outside the signal. The
newly proposed architecture has performed well in learning long-range temporal
dependencies in various artificial intelligence (AI) tasks [9]. Generally, at each
time step an LSTM network considers the following 3 inputs; xt , ht−1 , ct−1 and
outputs ht , ct through the following below equations.
it = σ (Wi Xt + Ui ht−1 + Vi mt−1 + bi ) (3)
ft = σ (Wf Xt + Uf ht−1 + Vf mt−1 + bf ) (4)
ot = σ (Wo Xt + Uo ht−1 + Vo mt−1 + bo ) (5)
mt = tanh (Wm Xt + Um ht−1 + bm ) (6)
mt = fti mt−1 + it m (7)
ht = ot tanh (mt ) (8)
where Xt is the input at time step t, σ is sigmoid non-linear activation function,
tanh is hyperbolic tangent non-linear activation function, denotes element-
wise multiplication. Concretely, at t = 0 hidden and memory cell state vectors
such as h0 and c0 are initialized to 0.
We developed LSTM based system for stance and gender detection by re-
placing RNN layer with LSTM.
3.4 Gated recurrent unit (GRU) based system for Analysis of
opinion and the figurative language on Twitter tweets
As from the above formulae, we can say that LSTM has complex set of processing
units. As a result, this needs more training time. Further the research on LSTM,
225
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
[4] introduced Gated recurrent unit (GRU). GRU has less number of units in
compared to LSTM, computationally efficient. The mathematical formulae of
GRU is given below,
i ft = σ Wxi−f + Whi−f ht−1 + bi−f (9)
ft = σ (Wxf Xt + Whf ht−1 + bf ) (10)
mt = tanh (Wxm Xt + Whm (f r ht−1 ) + bm ) (11)
ht = f ht−1 + (1 − f ) m (12)
Where 9 represents Update gate, 10 is for Forget or reset gate, 11 shows the
equation for Current memory and 12 gives the equation for Updated memory.
Formulae shows, unlike LSTM memory cell with a list of gates (input, output
and forget), GRU only consist of gates (update and forget) that are collectively
involve in balancing the interior flow of information of the unit. In GRU, input
gate (i) and forget gate (f ) are combined and formed a new gating units called
update gate (i f ) that mainly focus on to balance the state between the previous
activation (m) and the candidate activation (f ) without peephole connections
and output activations. The forget gate resets the previous state (m). GRU
networks looks simpler than LSTM with required only less computations.
4 Experiment and Results
We trained all experiments of various deep learning architectures using Tensor-
flow [1].
Cross-validation performance To select the optimal parameters for tweet
length and embedding size, 5-fold cross-validation is done on the given training
samples of Catalan and Spanish for stance and gender detection. 10-fold cross-
validation accuracy across varied tweet length and embedding size for stance
detection in Catalan language is displayed in Figure 1. 10-fold cross-validation
accuracy across varied tweet length and embedding size for gender detection in
Spanish language is displayed in Figure 2.
4.1 Evaluation results
We have submitted 3 runs for each task; run1 is based on RNN mechanism,
run2 is based on LSTM mechanism and run3 is based on GRU mechanism. The
detailed evaluation results has been given by the Independence@Ibereval 2017
organizing committee are displayed in Table 2. The scores we obtained for task
1 are 0.285, 0.304 and 0.307 in detecting stance and 0.477, 0.490 and 0.501
in detecting gender for Spanish language. The scores we obtained for task 1 are
0.360, 0.379 and 0.326 in detecting stance and 0.465, 0.483 and 0.486 in detecting
gender for Spanish language.
226
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
Fig. 1. 10-fold cross validation with Tweet length [10-70] and embedding size [64,128]
for stance detection in Spanish language
Fig. 2. 10-fold cross validation with Tweet length [10-70] and embedding size [64,128]
for gender detection in Catalan language
227
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
Table 2. Macro F-score for all tasks
Accuracy
Shared task
RNN LSTM GRU
Spanish - Stance 0.2849 0.3042 0.3066
Spanish - Gender 0.4764 0.4903 0.5014
Catalan - Stance 0.3603 0.379 0.3257
Catalan - Gender 0.4653 0.4829 0.4857
5 Conclusion
This working note has presented a language independent method for the In-
dependence@Ibereval 2017 shared tasks such as stance and gender in Twitter
messages written in Catalan and Spanish using BoWs and embedding of RNN,
LSTM and GRU. The presented supervised learning method has not relied on
any resources; semantic resources such as dictionaries and ontologies or com-
putational linguistics or feature engineering mechanisms for stance and gender
detection in twitter tweets. Due to the less training corpus, the efficacy of RNN
in stance and gender detection trails the classical BoWs approach. Though the
efficacy of embedding of RNN, LSTM and GRU is acceptable and paves the man-
ner in future to use for the analysis of stance and gender detection on Twitter
tweets. Evaluating the performance of RNN, LSTM and GRU embedding with
more training corpus for justification will be remained as one direction towards
future work.
References
1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghe-
mawat, S., Irving, G., Isard, M., et al.: Tensorflow: A system for large-scale machine
learning. In: Proceedings of the 12th USENIX Symposium on Operating Systems
Design and Implementation (OSDI). Savannah, Georgia, USA (2016)
2. Augenstein, I., Rocktäschel, T., Vlachos, A., Bontcheva, K.: Stance detection with
bidirectional conditional encoding. arXiv preprint arXiv:1606.05464 (2016)
3. Augenstein, I., Vlachos, A., Bontcheva, K.: Usfd at semeval-2016 task 6: Any-
target stance detection on twitter with autoencoders. Proceedings of SemEval pp.
389–393 (2016)
4. Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk,
H., Bengio, Y.: Learning phrase representations using rnn encoder-decoder for
statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
5. Elman, J.L.: Finding structure in time. Cognitive science 14(2), 179–211 (1990)
6. Fink, C., Kopecky, J., Morawski, M.: Inferring gender from the content of tweets:
A region specific example. In: ICWSM (2012)
7. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation
9(8), 1735–1780 (1997)
8. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444
(2015)
228
Proceedings of the Second Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2017)
9. Lipton, Z.C., Berkowitz, J., Elkan, C.: A critical review of recurrent neural networks
for sequence learning. arXiv preprint arXiv:1506.00019 (2015)
10. Mislove, A., Lehmann, S., Ahn, Y.Y., Onnela, J.P., Rosenquist, J.N.: Understand-
ing the demographics of twitter users. ICWSM 11, 5th (2011)
11. Mohammad, S.M., Kiritchenko, S., Sobhani, P., Zhu, X., Cherry, C.: Semeval-2016
task 6: Detecting stance in tweets. Proceedings of SemEval 16 (2016)
12. Taule M, Mart M.A, R.F.R.P.B.C.P.V.: Overview of the task of stance and gen-
der detection in tweets on catalan independence at ibereval 2017. Proceedings of
the Second Workshop on Evaluation of Human Language Technologies for Iberian
Languages (IberEval 2017), CEUR Workshop Proceedings. CEUR-WS.org (2017)
13. Wei, W., Zhang, X., Liu, X., Chen, W., Wang, T.: pkudblab at semeval-2016 task
6: A specific convolutional neural network system for effective stance detection.
Proceedings of SemEval pp. 384–388 (2016)
14. Zarrella, G., Marsh, A.: Mitre at semeval-2016 task 6: Transfer learning for stance
detection. arXiv preprint arXiv:1606.03784 (2016)
229