Natural language systems have to deal with many problems related with texts comprehension, but these problems become very hard when creativity and figurative devices are used in verbal and written communication. Human can easily understand the underlying meaning of such texts but, for a computer to disentangle the meaning of creative expressions such as irony and humor, it requires much additional knowledge, and complex methods of reasoning. model. The system works with an attention layer which is applied at the top of a BiGRU to generate a context vector for each word embedding which is then fed to another BIGRU network. Finally, the learned representation is fed to a Feed Forward Network (FNN) to classify whether the tweet is humorous or not. Motivated by the results shown in [17], we explore to incorporate the linguistic information to the model through initial hidden state in the first BiGRU layer.
The paper is organized as follows. Section 2 presents a brief description of the HAHA task. Section 3 introduces our system for humor detection. Experimental results are subsequently discussed in Section 4. Finally, in Section 5 we present our conclusions and attractive directions for future work.
HAHA 2019 is the second edition of the first shared task that addresses the problem of recognizing humor in Spanish tweets. Similar to the first edition, in the HAHA 2019 task, two subtasks were proposed. The first one, "Humor Detection", aims at predicting whether a tweet is a joke or not (intended humor by the author or not) and the second one "Funniness Score Prediction", is for predicting a score value into 5-star ranking, supposing it is a joke.
Participants were provided with a human-annotated corpus of 30000 Spanish tweets [6], divided in 24000 and 6000 for training and test respectively. The training subset contains 9253 tweets with funny content and 14747 tweets considered as non humorous. As could be observed, the classes distribution are slightly unbalanced, hence a difficulty is added to learn automatically the models.
System evaluation metrics were used and reported by the organizers. They use F1 measure on humor class for the subtask of "Humor Detection", moreover, precision, recall and accuracy were also reported.
The motivation behind of our approach are firstly to investigate the capability of Recurrent Neural Network, specifically, the Gated Recurrent Unit (GRU) [8] to capture long-term dependencies. They showed to be able to learn the dependencies in lengths of considerably large sequences. GRU networks simplified the complexity of the LSTM networks [11], being computationally more efficient. Moreover, attention mechanisms have endowed these networks with a powerful strategy to increase their effectiveness achieving better results [27,29,24,12]. Recently, the initial hidden state of the recurrent neural network has been a successful explored way to inform the networks with contextual information [25]. Secondly, humor recognition based on features engine and supervised learning have been well studied in previous research papers. These features have proved to be good indicators and markers of humor in text. For these reasons, in this approach we propose a method that enrich the Attention-based GRUs model with linguistic knowledge which is passed to the network using the initial hidden state. In Section 3.1 we describe the tweets preprocessing phase. Following, in Section 3.2 we present the linguistic features used for encoding humorous content. Finally, in Section 3.3 we introduce the neural network model and the way in which linguistic features are introduced. The Figure 1 shows the overall architecture of our system.
In the preprocessing step, the tweets are cleaned. Firstly, the emoticons, urls, hashtags, mentions, twitter-reserve words as RT (for retweet) and FAV (for favorite) are recognized and replaced by a corresponding wildcard which encodes the meaning of these special words. Afterwards, tweets are morphologically analyzed by FreeLing [18]. In this way, for each resulting token, its lemma is assigned. Then, the tweets are represented as vectors with a word embedding model. This embedding was generated by using the FastText algorithm [2] from the Spanish Billion Words Corpus [3] and an in-house background corpus of 9 millions of Spanish tweets.
In our work, we explored several linguistic features useful for humor recognition in texts [16,15,22,20,1,5] which can be grouped in three main categories: Stylistic, Structural and Content, and Affective. Particularly, we considered stylistic features such as: length, dialog markers, quotation, punctuation marks, emphasized words, url, emoticons, hashtag, etc. Features for capturing lexical and semantic ambiguity, sexual, obscene, animal and human-related terms, etc., were considered as Content and Structural. Finally, due to the relation of humor with expressions of sentiment and emotions we used features for capturing affects, attitudes, sentiments and emotions. For more details about the features see [17].
Notice that, our proposal did not consider the positional features used in [17]. Moreover taking into account the close relation between irony and humor and motivated by the results presented in [14] we include psycho-linguistic features extracted from the LIWC [19]. This resource contains about 4,500 entries distributed in 65 categories. Specifically, for this work we decided to use all categories as independent features.
Taking into account the previous features, we represent each message by one vector V Ti with dimensionality equal to 165. Also, in order to reduce and improve this representation we applied a feature selection method. Specifically we use the Wilcoxon Rank-sum test for paired samples. By using this test all features were ranked considering their p − value.
We propose a model that consists in a BiGRU neural network at the word level. Each time step t the BiGRU gets as input a word vector w t . Afterward, an attention layer is applied over each hidden state h t . The attention weights are learned using the concatenation of the current hidden state h t of the BiGRU and the past hidden state s t−1 in the second BiGRU layer. Finally, the target humor of the tweet is predicted by an FFN with one hidden layer, and an output layer with two neurons. Our overall architecture is described in the following sections.
In NLP problems, standard GRU receives sequentially (left to right order) at each time step a word embedding w t and produces a hidden state h t . Each hidden state h t is calculated as follow:
(reset gate) ĥt = tanh(W ( ĥ) x t + U ( ĥ) h t−1 + b ( ĥ) ) (memory cell)
Where all W ( * ) , U ( * ) and b ( * ) are parameters to be learned during training. Function σ is the sigmoid function and ⊕ stands for element-wise multiplication.
Bidirectional GRU, on the other hand, makes the same operations as standard GRU, but processing the incoming text in a left-to-right and a right-to-left order in parallel. Thus, it outputs two hidden state at each time step − → h t and ← − h t . The proposed method uses a BiGRU network which considers each new hidden state as the concatenation of these two ĥt = [
The idea behind this BiGRU layer is to capture long-range and backwards dependencies simultaneously. In this layer is where the linguistic information is passed throughout the model. We initialized both initial hidden state [
where g(.) receives a tweet and returns a vector which encodes contextual and linguistic knowledge g(T i ) = V Ti .
With an attention mechanism we allow the BiGRU to decide which segment of the sentence should "attend". Importantly, we let the model learn what to attend on the basic of the input sentence and what it has produced so far. Let H ∈ R 2×N h ×T the matrix of hidden states [ ĥ1 , ĥ2 , . . . , ĥT ] produced by the first BiGRU layer, where N h is the size of the hidden state and T is the length of the given sequence. The goal is then to derive a context vector c t that captures relevant information and feeds it as input to the next BiGRU layer. Each c t is calculate as follow:
Where W a and V a are the trainable attention parameters, s t−1 is the past hidden state of the second BiGRU layer and ĥt is the current hidden state. The idea of the concatenation layer is to take into account not only the input sentence but also the past hidden state to produce the attention weights.
The goal of this layer is to obtain a deep dense representation of the message with the intention to determine whether the tweet is humorous or not. This network at each time step receives the context vector c t which is propagated until the final hidden state s T . This vector is a high level representation of the tweet. Afterwards, it is passed to a feed forward network (FFN) with 3 hidden layers, and we use a softmax layer at the end as follow:
Where W j and b j (j = 0, ...3) denote the weight matrices and bias vectors for the last three layers with a softmax at the end. Finally, cross entropy is used as the loss function, which is defined as:
Where y i is the ground true classification of the tweet (humor vs. not humor) and ŷi is the predicted value by the model.
For tuning some parameters of the proposed model we used a stratified k-fold cross validation with 5 partitions on the training dataset. At this time, a features selection process was not performed, therefore we consider all linguistic features. During the training phase, we fixed some hyper-parameters, concretely: the batch size =256, epochs=10, units of the GRU cell was defined as 256, op-timizer="Adam" and dropout in the GRU cells=0.3. After that, we evaluated different subsets of linguistic features, particularly, five setting of features were explored. The considered subsets were: N o F ea (linguistic information is not considered), 64 F ea (the 64 best ranked features according to p−value), 128 F ea (the 128 best ranked features according to p − value), 141 F ea (all features with p − value ≥ 0.05) and All F ea (all linguistic features). As can be observed in the Table 1, a slight improvement is obtained when linguistic features were passed to the model. Particularly, the subset of F ea 128 achieved the best F1 score (F1 h=0.785) in the humor class. Also, when linguistic information was missing a gradual drop of 3.5%, in term of F1-score was observed in the humor class.
Regarding official evaluation and results, for the system's submission, participants were allowed to send more than one model till a maximum of 10 possible runs. Taking into account the results showed in the Table 1 we submitted three runs. The difference among them is the number of linguistic features considered for informing the U O U P V 2 model. In Run1 we use the subset of features F ea 64, for Run2 we used the subset of features F ea 128 while in Run3 the subset of features F ea 141 was considered. We achieved 0.765, 0.773, and 0.765, in terms of F1 score in the humor class, for Run1, Run2 and Run3 respectively. These values are consistent with the results obtained in the training phase. Regarding the official ranking, a first glance at Table 2 allows to observe that our best submissions (U O U P V 2 ) was ranked as 7th from a total of 18 of teams.
In this paper we presented our modification of the UO UPV system (U O U P V 2 ) for the task of humor recognition (HAHA) at IberLEF 2019. We only participated in the "Humor Detection" subtask and ranked 7th out of 18 team. Our proposal combines linguistic features with an Attention-based BiGRU Neural Network. The model consists of a Bidirectional GRU neural network with an attention mechanism that allows to estimate the importance of each word and then, this context vector is used with another BiGRU model to estimate whether the tweet is humorous or not. Regarding the feature selection, the best result was achieved when the 128 best ranked (according to p−value) features were considered. The results, also shown that adding linguistic information through initial hidden state caused an improvement in the effectiveness based on F1-measure. munication in social media: FAKE news and HATE speech (PGC2018-096212-B-C31).
The work of the second author was partially funded by the Spanish MICINN under the research project MISMIS-FAKEnHATE on Misinformation and Miscom-