A characterization of humor based upon machine learning that allows its automatic detection is not yet speci ed. This report describes the system used by the Aspie96 team in the HAHA shared task (part of IberLEF 2019) for humor recognition in tweets in Spanish: a neural network using exclusively character-level features. Humor Detection Telling if a tweet was intended to be humorous by the author or not. The results of the subtask were measured using F1-score for the positive (humorous) class. Funniness Score Prediction Predicting the level of funniness of humorous tweets. The results of the subtask were measured using RMSE (root-meansquared error). The tweets had been crowd-annotated in [1] according to a voting scheme: annotators could, for each tweet, mark it as not humorous or mark it as humorous and give it a number of stars (1 to 5) according to its funniness (5 being the best), for a total of six options. Combining the votes of the annotators, each tweet was labelled as humorous or not humorous and each humorous tweet was given a funniness score, from 1 to 5, equal to its average number of star. In the training dataset, for each tweet, the following information was provided:
HAHA (Humor Analysis based on Human Annotation), a shared task organized
within IberLEF 2019 (Iberian Languages Evaluation Forum) and described in
[
Text The text of the tweet.
Is humorous Whether the tweet is humorous.
Votes (Not humor, 1 star, 2 stars, 3 stars, 4 stars, 5 stars) The number of votes given to the tweet, for each possibility.
Funniness score The average funniness score of the tweet, only provided for humorous tweets.
In the testing dataset, just the text of each tweet was provided.
The competition was run using the CodaLab platform1. Each team was allowed up to 10 submissions per day and a total maximum of 20 submissions. Each team could decide, at any moment, which one submission to include in the leaderboards, which were always visible to all participants and updated in real time. A separate leaderboard was used for each subtask.
Each submission could be meant for the humor detection subtask only or for both subtasks and annotated each tweet in the testing dataset with a binary label, indicating whether it had been detected as humorous and, if it was meant for the funniness score prediction dataset also, the predicted funniness score (even for tweets not detected as humorous). The predicted funniness score was only considered for tweets classi ed as humorous in the gold testing dataset.
The Aspie96 team took part in both subtasks, using a neural network with character-level features.
The structure of the model and its results are described in the following sections. 2
The system used by the Aspie96 team strictly works at character level, without using any word-level features (such as word embeddings) or any data other than what is provided for the speci c task at hand.
It is a neural network adapted from [
The input of the system is a xed-size list of arrays. The neural network begins with a series of unidimensional convolutional layers: each lter has a small width (of 3) and convolves trough the input. The output of each convolutional layer is again a list of arrays: the length of each array is constant for each layer and is set trough a hyperparameter (it is 8 for all convolutional layers), while the length of the list is slightly smaller than that of the input one, because the convolutional layers don't use padding. The series of convolutional layers is followed by a bidirectional recurrent layer. The output of the bidirectional layer, which is an individual dense vector representing information about the whole tweet, is the input of a simple fully connected layer, with one output, whose 1 https://competitions.codalab.org/ activation function is the logistic function (the logistic function has an output between 0 and 1).
The purpose of the unidirectional convolutional layers is to create a higherlevel dense representation of each input vector, together with its surrounding ones, providing context. The purpose of the bidirectional recurrent layer is to produce an individual vector which can be considered as a representation of the tweet. Additional layers meant for regularization are used (a gaussian noise layer applied to the input of the neural network and several dropout layers).
The input tweet is represented as a list with xed length (leading to padding on the left or truncation to the right, where needed) of sparse vectors. Each vector of the list represents an individual character of the tweet and contains ags whose values are either 0 or 1.
Most of the ags are mutually exclusive and are used to identify a character among a list of known ones. Additional ags are used to represent properties of the character.
The full list of known characters is the following:
Space ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; = ? @ [ ] _ a b c d e f g h i j k l m n o p q r s t u v w x y z | ~
Emojis are represented similarly to their Unicode name (in English), with additional ags.
The full list of additional ags is: Uppercase letter Indicates whether the character is an uppercase letter. Accent Indicates whether the character is an accented vowel, regardless of the accent being acute or grave.
Emoji Indicates whether the character is part of the Unicode name of an emoji. Emoji start Indicates whether the character is the rst in the Unicode name of an emoji.
Letter Indicates whether the character is a letter.
Number Indicates whether the character is a numerical digit.
Inverted Indicates if the character is an inverted question mark or exclamation mark ( or ).
Tilde Whether the character is an N with virgulilla (~n or N~).
Multiple spaces are represented as one and unknown characters are ignored.
A more in-depth description of the role of each layer is given in [
The main di erence between the model used for the HAHA task and the one
presented in [
The same structure was used for both HAHA subtasks: for the humor detection subtask, the output of the network was rounded to 0 or 1 to get a binary value and for the funniness score prediction subtask it was multiplied by ve to get a value between 0 and 5 (in principle this could have allowed the model to output funniness scores below 1, but that was not the case for any of the labels in the submission). 3
A total of 18 teams took part in the humor detection subtask.
In the humor detection subtask, the Aspie96 team got an F1-score, for the positive class, of 0.711, ranking in the 13th position, below LadyHeidy, with an F1-score of 0.725 and above dodinh, with an F1-score of 0.660.
As a comparison, the best ranking team (adilism) got an F1-score of 0.821.
The values for precision, recall and accuracy for the Aspie96 team were, respectively, 0.678 (14th position), 0.749 (11th position) and 0.763 (13th position).
The baseline system was one marking tweets as humorous with a probability of 0.5. It got an F1-score, precision, recall and accuracy of, respectively, 0.440, 0.394, 0.497 and 0.505.
The results of the Aspie96 team are summarized in Table 1 and compared with the best system and the baseline system.
A total of 13 out of 18 teams took part in the funniness score prediction subtask.
In the funniness score prediction subtask, the Aspie96 team got a RMSE of 1.673, ranking in the 12th position, below garain, with a RMSE of 1.653 (a lower RMSE is better) and above dodinh, with a RMSE of 1.810. Similar models had been presented before.
A roughly similar model based on convolutions for text classi cation was
presented in [
In [
The results, for both HAHA subtasks, were signi cantly better than the baseline, however there is clearly much room for improvement.
As the system is a mere adaptation of the one presented in [
In the binary classi cation subtask presented in [
However, in the HAHA binary classi cation subtask (humor detection) the
results in the leaderboard were much more spread out and the results obtained
by the Aspie96 team were quite di erent from those of the best systems. Despite
the scores of the best ranking teams being better than those shown in [
The structure of the neural network used in the HAHA task by the Aspie96
team, although having been originally presented in [
This does not mean a character-level approach cannot be used e ectively for humor detection, but the structure of the neural network must be improved in order to be more general, resulting in better results and consistency among di erent tasks.
For the simple case of binary tweet classi cation, the ideal result would be that of obtaining a structure capable of working across di erent languages (at least Spanish, English and Italian), regardless of the high-level task (whether humor detection, irony detection or any other).
Results outside the scope of this paper, regarding the performance of the model, in di erent tasks, in di erent languages, also con rm the current inconsistency of the results obtained by the structure, but show its applicability and its ability to get good results for at least some di erent tasks, suggesting it is indeed needed to make it more general.