Natural Language Understanding becomes very hard when creativity and gurative language are used in social communication. Humor constitutes an illustrative example of how humans use creative language to produce funny content. Therefore, create new methods and resources for analyzing properly humorous texts is an important issue in Natural Language Processing (NLP) and even more in Human Computer Interaction (HCI). In this sense, this paper introduces our UO UPV system developed for the Humor Analysis based on Human Annotation (HAHA) track proposed in IberEval 2018 Workshop. The task focuses on classifying tweets in Spanish as humorous or not, and predicting how funny they are. To solve this task, our proposal combines both linguistic features and an Attention-based Recurrent Neural Network, where the attention layer helps to calculate the contribution of each term towards targeted humorous classes. Experimental results show that our model achieves encourage results.
Social Media have provided to modern societies easy and attractive ways for sharing their point of views on the most diverse subjects. For this reason, new challenges in information processing and management, decision support systems, and human computer interaction has been opened. Dealing with multilingualism, with multiples genres and written styles, as well as other subjectives language devices (sentiments, emotions, attitudes and opinions) has captured the focus of several research. However, these faced problems become very hard when creativity and gurative devices are used in verbal and written communication. Human can easily understand the underlying meaning of such expressions but, for a computer to disentangle the meaning of creative expressions such as irony and humor, it requires much additional knowledge. 2
Humor is an illustrative example of how humans use creative language devices in social communication. Humor not only serves to interchange information or share implicit meaning, but also engages a relationship between those exposed to the funny message. It can help people see the amusing side of problems and can help them distance themselves from stressors. In the same way, it helps to regulate ours emotions. Moreover, the manners in which people produce funny content also reveal insight about their genre and personal traits.
Twitter has become as a popular information source for gathering, in
transparent way, spontaneous user's generated content. It enables accessing humorous
messages, it is useful for recognizing and identifying how humor arises through
language. From a computational linguistics point of view many methods have
been proposed to tackle the fascinating task of recognizing humor from texts
[
Considering the advantages of linguistic features for capturing deep linguistics aspects of the language also the capability of Recurrent Neural Network for learning features and long term dependencies from sequential data, in this paper, we present a new method that combines the most relevant linguistic features used for humor recognition and an Attention based LSTM. The system works with an attention layer which is applied on the top of a Bidirectional LSTM to generate a context vector for each word embedding which is then fed to another LSTM network to detect whether the tweet is humorous or not. To the best of our knowledge, there has not been any other work exploring the use of attention-based architectures for humor recognition in Spanish tweets.
The paper is organized as follows. Section 2 presents a brief description of the HAHA task. Section 3 introduces our system for humor detection. Experimental results are subsequently discussed in Section 4. Finally, in Section 5 we present our conclusions and attractive directions for future work. 2
Humor recognition and generation on social media content have become interesting research areas from the computational point of view. Most studies in the eld of humor recognition from textual sources have focused on English more than on other popular languages such as Spanish. In this context, HAHA can be considered as the rst shared task that facing the fascinating problem of recognizing humor in Spanish tweets. In the HAHA task, two subtasks were proposed by the organizers. The rst one, \Humor Detection": aim to predict if a tweet is a joke or not (intended humor by the author or not) and the second one \Funniness Score Prediction": for predicting a score value (average stars) for a tweet into 5-star ranking, supposing it is a joke.
Participants were provided with a human-annotated corpus of 20000 Spanish
tweets [
System evaluation metrics were used and reported by the organizers. Their choice was to use F1 measure on humor class for the subtask of \Humor Detection", moreover, precision, recall and accuracy were also reported. The Root Mean Squared Error (RMSE) was used to assess the systems e ectiveness in the subtask of \Funniness Score Prediction". 3
The motivation of our approach is twofold: rstly, the capability of Recurrent
Neural Network, speci cally, the LSTMs [
In the preprocessing step, the tweets are cleaned. Firstly, the emoticons, urls,
hashtags, mentions, twitter-reserve words as RT (for retweet) and FAV (for
favorite) are recognized and replaced by a corresponding wildcard which encodes
the meaning of these special words. Afterwards, tweets are morphologically
analyzed by FreeLing [
R. Ortega-Bueno, C. Mun~iz-Cuza, P. Rosso, J. Medina-Pagola representation of words in context and also taking advantage of the peculiar writing style used by Twitter's users. 3.2
In our work, we explored some linguistic features useful for humor recognition in
texts [
{ Negation: The amount of negation words in the tweet is counted.
{ Sentiments : These feature count the number of positive and negative words
according to a sentiment resource. Notice that, for each resource two
features are built. In this work we explore four distinct dictionaries: Spanish
Sentiment Lexicon [
We also try to analyze the occurrence of some of the previous features into speci c positions in the tweets. For instance, the feature occurring at the beginning or at the ending of a tweet. Based on all the features mentioned above a vector (V Ft) is built for each tweet in the training and test datasets. 3.3
We propose a model that consists in a Bidirectional LSTM neural network (BiLSTM) at the word level. Each time step t the Bi-LSTM gets as input a word
6
R. Ortega-Bueno, C. Mun~iz-Cuza, P. Rosso, J. Medina-Pagola
vector wt with syntactic and semantic information, known as word embedding
[
In NLP problems, standard LSTM receives sequentially (left to right order) at each time step a word embedding wt and produces a hidden state ht. Each hidden state ht is calculated as follow: ft = ot = ut = ct = it ht = ot it = (W (i)xt + U (i)ht 1 + b(i)) (W (f)xt + U (f)ht 1 + b(f)) (W (o)xt + U (o)ht 1 + b(o)) (W (u)xt + U (u)ht 1 + b(u)) +ft ct 1 tanh(ct) (input gate) (forget gate) (output gate) (new memory cell) ( nal memory cell)
Where all W ( ), U ( ) and b( ) are parameters to be learned during training. Function is the sigmoid function and stands for element-wise multiplication.
Bidirectional LSTM, on the other hand, makes the same operations as standard LSTM but, processes the incoming text in a left-to-right and a right-to-left order in parallel. Thus, it outputs two hidden state at each time step !ht and ht . The proposed method uses a Bi-LSTM network which considers each new ! hidden state as the concatenation of these two h^t = [ht ; ht ]. The idea of this Bi-LSTM is to capture long-range and backwards dependencies. 3.5
With an attention mechanism we allow the Bi-LSTM to decide which part of the sentence should \attend". Importantly, we let the model learn what to attend on the basic of the input sentence and what it has produced so far.
Let H 2 R2 Nh Tx the matrix of hidden states [h^1; h^2; : : : ; h^Tx ] produced by the Bi-LSTM model, where Nh is the size of the hidden state and Tx is the length of the given sentence. The goal is then to derive a context vector ct that captures relevant information and feed it as input to the next level (Pos-AttLSTM). Each ct is calculate as follow:
Tx ct = X t0=1 t;t0 h^t0 t;i =
Tx X j=1 t;i t;j = tanh(Wa
[ht; ^st 1] + ba)
Where Wa and ba are the trainable attention parameters, st 1 is the past hidden state of the Pos-Att-LSTM and h^t is the current hidden state. The idea of the concatenation layer is to take into account not only the input sentence but also the past hidden state to produce the attention weights. 3.6
The goal of the Post-Att-LSTM is to predict whether the tweet is humorous or not. This network at each time step receives the context vector ct which is propagated until the nal hidden state sTx . This vector is a high level representation of the tweet and is used in the nal softmax layer combined with the linguistic feature vector as follow: y^ = sof tmax(Wg
[STx ; LFd] + bg) LFd = relu(Wd
LFt + bd)
Where Wg and bg denote the weight matrix and bias vector for the last layer with a softmax at the end. LFd is the result of passing the V Ft vector associated to each tweet through a dense layer before the softmax layer. Finally, cross entropy is used as the loss function, which is de ned as:
L =
X yi log(y^i)
i
Where yi is the ground true classi cation of the tweet (humor vs. not humor). For predicting the funniness score we use an architecture similar to the one described above. For predicting the funniness score we use an architecture similar to the one described above. The most salience change is at the last hidden layer and the loss function. Speci cally, the last layer was changed to a dense one with just one neuron as output and the mean square error (MSE) was used as loss function for optimizing the model. 4
In this section we show the results of the proposed method in the shared task of \Humor Detection" and discuss them. For the system's submission, participants were allowed to send more than one model till a maximum of 4 possible runs. In Tables 1 we report our three best performing systems ( run1, run2 and run3 sent by UO UPV) for humor detection task on two classes (humor, not humor). The rst run is based on the Attention based model mixed with linguistic features which are fed to the model in the last Dense Layer. Run2 is similar to run1, the major di erence consists in a dimension reduction of linguistic features by using Random Forest7 as strategy to rank the importance of each feature. Finally, in
8
R. Ortega-Bueno, C. Mun~iz-Cuza, P. Rosso, J. Medina-Pagola the run 3 we evaluate our proposal without linguistic features. As can be shown in Table 1, run1 and run2 achieved F1=0.7851 and F1=0.7785 scores, respectively while run3 obtains a F1=0.7702. Experiments showed that introducing linguistic information to the Attention LSTM model (run1, run2) improves the performance of the model. Contrary, to our expectations, the reduction of the dimensionality of the linguistic feature vector applied in run2 obtained a drop in term of F1 measure. Our 4th run (UO UPV run4) addressed the task of \Funiness Score Prediction". For that, we consider a setting similar to run1 where all linguistic features and recurrent neural network model were combined, but considering the modi cation of the model explained in the Section 3.6 to deal with the regression task. Our run4 obtained a value of 1.5919 in terms of RSME and positioned at last place out of two runs. Also our result do not surpass the baseline established by the task.
Regarding the o cial results, at a rst glance on Table 1 it is possible to observe that our submissions (run1, run2 and run3) were ranked on 2nd, 3rd and 6th respectively from a total of 7 runs of 3 teams. Notice that, our best result (run1) achieves slightly better accuracy that the rst score obtained by the INGEOTEC Team. It is important to remark that in term of precision on humor class our run1 and run2 outperform the best ranked submissions (INGEOTEC run2). 5
In this paper we presented the UO UPV system for the task of humor recognition (HAHA) at IberEval 2018. We participated in the \Humor Detection" subtask and ranked 2nd out of 7 submissions. Our proposal combines linguistic features with an Attention-based Long Short-Term Memory Network. The model consists of a Bidirectional LSTM neural network with an attention mechanism that allows to estimate the importance of each word and then, this context vector is used with another LSTM model to estimate whether the tweet is humorous or not. The results shown that the consideration of linguistic features in combination with the deep representation learned by the neural network model obtains better e ectiveness based on F1-measure the in humor class. Due to encouraging results of our approach, we think that including the linguistic features of humor into the embedding layer could be a way to increase the e ectiveness. We would like to explore this approach in the future work.
The work of the fourth author was partially supported by the SomEMBED TIN2015-71147-C2-1-P MINECO research project.
R. Ortega-Bueno, C. Mun~iz-Cuza, P. Rosso, J. Medina-Pagola