Automatic humor detection is a hard but challenging task. For the competition HAHA at IberLEF 2019 we built a neural networks classi er that uses di erent types of neural networks for speci c sets of features. After being trained separately, the layers are concatenated to give the general output. The performance of our system on the binary detection of humorous tweets reaches F-score of 0.76 which is comparably higher than results of baseline machine learning classi ers and earns us the ninth place in the ranking table. As for task 2, where the system has to guess how funny the tweet was based on the number of stars that it got, our result is similarly good: RM SE = 0:945. However, much needs to be done to evaluate contribution of each of the feature sets and our choice of the type of neural network.
Humor detection is a non-trivial task that was considered by (
Our approach at HAHA@IberLEF2019 is not an exception from this trend. 2
Dataset and Preprocessing
\HUMOR: A Crowd-Annotated Spanish Corpus for Humor Analysis" by (
We rst preprocess tweets with the help of our own software that includes the following steps: 1. Convert some markers of emotions into words: :( to tristeza and :) :D xD
XD to reir. 2. Convert repetitive sequences of jaja... and JAJA... to a simple ja. 3. Pre-tokenize: add a space before and after punctuation symbols except # and . 4. Convert repetitions of the same letter of length 3 into a single letter and add a lemma EMPHASIS to the tweet so that it lexically denotes sentiment implied by letter repetitions: sooooomos to somos EMPHASIS. 5. Tokenize hashtags # and mentions : (a) De-capitalize capitalized sequences of more than one letter leaving the rst letter capitalized: NUET to Nuet. (b) Add a space before and after every non-letter character in the sequence:
PP#CiU to PP # CiU. (c) Add a space before every capitalized character in the sequence: DaviniaBono to Davinia Bono. 6. Convert emoticons into their word representations using unicode tables in Spanish1.
Our Python script for tweet preprocessing (except emoticons) and the system we used at the Competition will soon be available at https://github.com/ evrog/Spanish_Humor. As concerns the choice of steps, it is basically a tradeo between not ruining words with traditional orthography and extracting as many
lemmas from Internet-speci c speech as possible. The nal stage of preprocessing is lemmatization with the help of SpaCy2: the output is a list of lemmas and special characters in a tweet. 3
As mentioned above, our model is based on a neural network. It processes several
types of features in parallel, then concatenates the layers' outputs and passes
them to the Dense layer. (
We used a subset of 1,200 tweets from the Training dataset for validation and accuracy, as a validation measure. To avoid over tting, we used the early stopping strategy (the value of patience is 5) and the dropout regularization for the output layer (the fraction of the input units to drop is 0.8).
The model learns on four sets of features in parallel: 1. Tweets represented as sequences. The length of word emebeddings is 300. The weight matrix is built from pretrained word embeddings for Spanish3. In our experiment, the Convolutional neural network learned better from these features than the Recurrent one, so these features are fed to CNN and a further combination of MaxPooling and a attening layer. The Convolutional layer contains 64 lters with a kernel size of 5. The size of the max pooling windows is 20. 2. Tweets represented as a Bag-of-Words and smoothed with TFIDF. These features are restricted to the 5,000 most frequent words, due to computation complexity of a larger vocabulary, and passed to a Dense layer. 3. Features of sentiment and topic modelling, extracted from tweets.
To represent sentiment in every tweet, we used \a ective norms" calculated
by (
xmin
xmax
xmin
(
3 https://www.kaggle.com/rtatman/pretrained-word-vectors-for-spanish/ 4 The dictionary of norms can be downloaded here: http://crr.ugent.be/archives/ 1844. (b) normalized with MinMax quantitative features: number of words, lines, minimum and maximum distance between embeddings, minimum Levenstein distance between a pair of words (to detect puns) applied to all possible pairs of words in a tweet.
The features are also passed to a Dense layer.
Scheme 1 demonstrates the general outline of our best performing neural network used for the task of binary classi cation. The scheme includes main parameters, e.g. the size of windows is 20 in 20: MaxPooling. The optimizer is adam (adaptive moment estimation); the loss function is binary crossentropy; the activation functions at hidden layers are ReLU, and for the output layer it is softmax. The last layer includes Dropout regularization with probability = 0:8. For the second task the architecture is similar to that of the rst task, but for the input values that are separated into classes according to the average number of stars that the tweets earned. Also, in the second task, we use the mean standard error for validation. 4
Table 1 demonstrates the result of our system compared to the winners of two tasks. The rst four measures are for Task 1, and the last column presents Task 2. As concerns Task 1, the performance of our system is average compared to other teams' results. However, it is high above the chance value and comparably higher than results of baseline machine learning classi ers. Application of computer methods, in particular word embeddings and neural networks, in analysis of gurative speech and its varieties, such as humor, has recently proved to be very e ective in annotation of large corpora. But also, it gives a new perspective on the analysis of language features that are important in humor production and appreciation. Our approach included testing sets of di erent features in a growing combination: at each step we added a feature set and a subset of a neural network into the architecture and checked if they improved our result. For example, including the sentiment dictionary (see above: \a ective norms") improved our F-score by 0.015. The features we chose are usual in the systems we mentioned in 1: vocabulary represented as word embeddings, TF-IDF weighted Bag-of-Words, sentiment dictionary, special characters that represent emotions in Twitter (e.g. :)).
As for the system architecture, we tested the so-called hard parameter sharing. Our system uses di erent neural networks that we empirically found to be more capable of dealing with each speci c set of features. In general, we use CNN for embeddings and Dense layers for other feature sets. The result of our system is much higher than that of the baseline and is medium compared to other participants. However, the value of each of the sets features and the model of neural network have yet to be evaluated more closely. We plan to combine our features with other types of neural networks. Also, the model might have given a higher result in Task 2 if we used a regression model instead of multi-class classi cation. However, this is yet to be tested.
The reported study was funded by RFBR according to the research project No. 18-37-00272.