In this paper, we present our work for the Automatic Humour Analysis (JOKER) Lab at CLEF 2024. The objective of the JOKER Lab is to research the automated processing of humour that includes tasks such as retrieval, classification, and interpretation of various forms of humorous texts. Our task involved the classification of humorous texts into diferent genres for which we undertook two diferent approaches. These approaches involved the usage of BERT (a transformer architecture) and a traditional machine learning model such as a Random Forest classifier. Out of the two models, BERT had a higher accuracy score of 0.6731. From this, we concluded BERT is better for most Natural Language Processes. We showcase our experiments on the training data and the results on the provided test dataset are presented in the forthcoming pages.
Humor plays a crucial role in human communication and social interaction. However, it is multifaceted and elicits diferent types of responses from various types of audiences. Accurate classification of humor not only enhances our understanding of its various forms but also has practical application in fields such as sentiment analysis, human-computer interaction and social media content moderation.
Traditional humor classification techniques can be labor and time consuming. Automating this process through NLP and ML techniques can improve the eficiency and accuracy of humor classification, benefiting academic research. With the proliferation of digital media, humor is more pervasive and varied than ever, presenting a challenge to even state of the art models to discern the diferences between various genres of humor.
The CLEF 2024 JOKER [
By leveraging some advanced natural language processing techniques and fine-tuning some of the well-known pre-trained models, this study for the chosen task - 2 aims to develop a system capable of accurately classifying text into the following humor categories humorous twist or reversal.
proportions. • IR - Irony relies on a gap between the literal meaning and the intended meaning, creating a • SC - Sarcasm involves using irony to mock, criticize, or convey contempt.
• EX - Exaggeration involves magnifying or overstating something beyond its normal or realistic • AID - Incongruity refers to the unexpected or contradictory elements that are combined in a humorous way and Absurdity involves presenting situations, events, or ideas that are inherently illogical, irrational, or nonsensical. • SD - Self-deprecating humor involves making fun of oneself or highlighting one’s own flaws, weaknesses, or embarrassing situations in a lighthearted manner. • WS - Wit refers to clever, quick, and intelligent humor and Surprise in humor involves introducing unexpected elements, twists, or punchlines that catch the audience of guard.
This automated approach significantly benefits various fields by providing deeper insights into the mechanics of humor and enhancing the way machines understand and respond to human emotions.
We took up 2 approaches for the humor classification task: multiclass classification using BERT base uncased and classification using Random Forest classifier. Preprocessing of the data was done diferently for both methods.
The provided dataset [
Basic text preprocessing was done to the provided dataset. Firstly, the class identifiers for each humorous text were mapped with respective numerical values. All texts were stripped of punctuation, stop words, and other special characters. These texts were then lemmatized. This preprocessed dataset was directly used for BERT (see figure 1)
For the approach involving the use of the Random Forest classifier, the preprocessed text data were further prepared by combining Sentence Transformer, a pre-trained model, and Tfidf Vectorizer, a scikit-learn tool, to generate sentence embeddings and TF-IDF feature vectors, respectively (see figure 2 )
SentenceTransformer: This pre-trained model (multi-qa-mpnet-base-dot-v1) from the sentencetransformers library is utilized to generate sentence embeddings. This model captures the semantic meaning of text at the sentence level, efectively embedding the contextual nuances and relationships between words within sentences.
Tfidf Vectorizer : This is a tool from scikit-learn that converts textual data into TF-IDF feature vectors. TF-IDF vectors highlight the importance of words within a document relative to the entire corpus, thus providing a measure of the significance of terms.
To generate TF-IDF vectors for the test and training data, the TF-IDF vectorizer is first fitted to the text data within the training set. This fitting process involves learning the vocabulary and the inverse document frequency (IDF) values from the training corpus. After fitting, the text data is transformed into TF-IDF vectors, resulting in a sparse matrix representation of the documents where each entry reflects the importance of a term within a document. The SentenceTransformer model encodes the training text data into sentence embeddings, which capture the semantic content of the text. The concatenation of TF-IDF vectors and sentence embeddings in each document creates a comprehensive feature set that considers both local word importance and sentence semantic meaning.
The target labels (classes) are extracted from the data frame to prepare the target variable for model training and evaluation. This extraction isolates the dependent variable, which the machine learning model will learn to predict based on the input feature set which is a combination of the TF-IDF vectors and sentence embeddings.
2.2.1. BERT
BERT [
BERT [
At every layer, BERT calculates the attention scores for each token in the input sequence, indicating the importance of other tokens about it. This allows BERT to understand contextual information by trying to understand all tokens in the input sequence simultaneously, in both directions. After selfattention, the output is passed through a feedforward neural network, typically with a ReLU activation function. This network helps find complex patterns in the data and further improves the representations learned by the self-attention mechanism (see figure 3).
Before inputting text into BERT, it undergoes tokenization into subword units using WordPiece tokenization. This allows BERT to handle out-of-vocabulary words efectively. Each input sequence is then represented as a combination of three types of embeddings namely token, segment, and positional embedding. Token Embedding represents the identity of each token in the input sequence. These embeddings are learned during the pre-training stage and understand the semantic meaning of individual words. Segment Embedding indicates whether a token belongs to the first sentence or the second sentence in a pair of sentences. This helps BERT understand the relationship between sentences, especially in tasks like question answering and natural language processing. Positional Embedding encodes the position of each token in the input sequence allowing BERT to capture sequential information and understand the order of words in a sentence.
After pre-training, BERT can be fine-tuned on specific tasks using task-specific labeled data[ 6]. During ifne-tuning, the pre-trained parameters are adjusted to optimize performance on the task. Fine-tuning BERT on specific tasks enables it to achieve state-of-the-art results across various natural language processing tasks. 2.2.2. Random Forest
Random Forest [7] is an ensemble classifier that contains several decision trees. Instead of using a single decision tree, this ensemble method leverages the decision-making ability of multiple decision trees and based on the majority number of predictions, the final output is predicted. The prepared input feature set is passed to the Random Forest classifier comprising 1500 decision trees. (see figure 5) The use of out-of-bag samples is also enabled to estimate the generalization accuracy of the model. This provides an internal cross-validation measure of the model performance. Decision trees make up the most fundamental component of the Random Forest classifier. Each decision tree works to find the best split to divide the data into multiple subsets and is trained through the Classification and Regression Tree (CART) algorithm. Gini impurity, information gain, or mean square error are some of the commonly used metrics to evaluate the quality of the split. A single decision tree can be prone to bias and over-fitting, hence an ensemble classifier consisting of multiple decision trees is used to improve the accuracy of the predictions. Random Forest algorithm (see figure 4 ) makes use of bagging and feature randomness to create an uncorrelated forest of decision trees. Each tree in the ensemble comprises of data sample drawn from the provided training data set with replacement. One-third of it is set as the out-of-bag sample. The diversity of the dataset is increased and correlation among the decision trees is reduced through feature bagging. For a classification task, such as the one performed, the most frequent categorical variable will yield the predicted class. Finally, the out-of-bag sample is used for cross-validation.
The metrics of precision, recall, accuracy, and f1-score are reported for the two models that were used to complete the given task. Precision is calculated mathematically as the ratio of true positives and the sum of true and false positives. Accuracy is the ratio of the number of correct predictions to the total number of data points. Recall is calculated as the ratio of true positive and the sum of true positive and false negative. The F1 score is calculated from the values of precision and recall. It mathematically, is equal to twice the ratio of the product of precision and recall to the sum of precision and recall.
As mentioned before two diferent approaches were used to solve the given task. The first approach involved using a transformer architecture such as BERT. The second approach involved using a traditional machine learning model such as a Random Forest classifier. Higher accuracy (0.6731) of BERT suggests that using transformer architecture like BERT for classification proves to be more accurate than traditional and feature-dependent machine learning models that are commonly used for classification. Overall, it can be concluded that BERT’s deep contextual and language understanding with its ability to leverage transfer learning, makes it better suited for the nuanced task of humor classification according to genre. [6] S. Prabhu, M. Mohamed, H. Misra, Multi-class text classification using bert-based active learning, arXiv preprint arXiv:2104.14289 (2021). [7] G. Biau, E. Scornet, A random forest guided tour, Test 25 (2016) 197–227.