Humor classification is a complex task in natural language processing (NLP) that involves identifying and categorizing humor based on its various forms and techniques. This study explores the use of two pre-trained transformer models, RoBERTa and BERT-uncased, for humor classification according to genre and technique. The models are fine-tuned on a dataset annotated with diferent humor genres (such as sarcasm, irony, exaggerations and witty jokes). The performance of each model is evaluated based on accuracy, precision, recall, and F1-score. Our results demonstrate that both RoBERTa and BERT-uncased are efective in capturing the nuances of humor, with RoBERTa showing a slight edge in overall performance. This research highlights the potential of transformer models in advancing the field of humor classification and provides insights into their applicability for more nuanced and context-aware NLP tasks.
Humor is an intricate aspect of human communication that poses significant challenges for natural language processing (NLP). The ability to recognize and classify humor is crucial for enhancing humancomputer interactions, particularly in applications like virtual assistants, chatbots, and social media analysis. Despite its importance, automatic humor classification remains a dificult task due to the subjective and context-dependent nature of humor.
The motivation for this research stems from the need to improve the accuracy and sophistication of humor classification systems. Previous approaches have often struggled to capture the subtle nuances and diverse techniques of humor, which can vary significantly across diferent genres. By leveraging advanced transformer models such as RoBERTa and BERT-uncased, we aim to develop more robust and nuanced humor classification methods. These models, pre-trained on vast amounts of text data, have shown remarkable performance in various NLP tasks and ofer the potential to enhance humor recognition.
This paper focuses on humor classification according to genre and technique, as outlined in the CLEF
2024 JOKER task 2 [
JOKER lab, introduced in several key publications, provides a comprehensive dataset for this task, allowing for rigorous evaluation of our models.
Recent advancements in NLP have been dominated by transformer-based models, which have achieved state-of-the-art results in numerous tasks. RoBERTa, a robustly optimized BERT approach, has demonstrated superior performance due to its extensive pre-training and fine-tuning capabilities. BERTuncased, known for its bidirectional training of transformers, excels in understanding context and nuance in text. Prior studies have explored these models for various classification tasks, but their application to humor classification remains under-explored. This study seeks to address this gap by evaluating the performance of RoBERTa and BERT-uncased models in humor classification tasks.
This structured approach aims to provide a comprehensive understanding of the application of transformer models in humor classification, highlighting their strengths and limitations in this nuanced task.
The dataset for this task is provided by the CLEF 2024 JOKER lab, which focuses on automatic humor
analysis. The dataset consists of humorous texts categorized according to genre (such as sarcasm, irony,
exaggerations and witty jokes). This diverse dataset enables the development and evaluation of models
capable of distinguishing between diferent types and techniques of humor. The two datasets train-input
and train-qrels are combined to have the format as in Table 1 .
BERT [
RoBERTa [
• Data Split: The dataset is split into training, validation, and test sets with an 80-20 ratio. • Batch Size: 8 • Learning Rate: 5e-5 • Epochs: 3 • Optimizer: Adam • Loss Function: Sparse Categorical Cross-Entropy Loss
The models are fine-tuned using the training set, with hyperparameters optimized based on performance on the validation set. Fine-tuning involves adjusting the weights of the pre-trained model to minimize the classification error.
The performance of the models is evaluated based on their accuracy in distinguishing between diferent genres and techniques of humor. By evaluating the models based on accuracy, we aim to assess their ability to capture the nuances of humor and classify texts correctly. This metric provides a straightforward measure of the models’ performance and their efectiveness in humor classification tasks.
This methodological approach ensures a rigorous and reproducible framework for humor classification, leveraging state-of-the-art NLP models to capture the complexities of humor in text.
Our study delved into the realm of humor classification using state-of-the-art transformer models, specifically RoBERTa and BERT-uncased. The results of our experiment showcase the efectiveness of these models in deciphering the intricacies of humor, albeit with some limitations.
Overall, our models demonstrated competitive performance, with RoBERTa slightly surpassing BERTuncased by achieving an accuracy of 70% compared to 67%. This indicates that both models are adept at grasping the nuances of humor, as evidenced by their ability to classify jokes based on genre and technique with reasonable accuracy.
model roberta bert uncased accuracy
SD_precision
SD_recall SD_f1-score
SD_support 0.70 0.67 0.37 0.69 0.66 0.84 0.47 0.76 38.00 91.00
Despite their overall efectiveness, both RoBERTa and BERT-uncased exhibited challenges when confronted with certain types of humor. Exaggeration-style jokes and those heavily reliant on cultural references proved to be particularly problematic for the models. These types of humor often require a deep understanding of context and cultural nuances, which may have posed dificulties for the models’ classification abilities.
To address these challenges and further enhance the performance of transformer models in humor classification, several strategies could be explored. One approach could involve incorporating additional contextual information, such as cultural context or background knowledge, into the models. This could potentially help the models better understand and classify jokes that rely heavily on cultural references.
Furthermore, fine-tuning strategies tailored to these specific types of humor could also prove beneficial. By focusing on optimizing the models’ performance on exaggeration-style jokes and culturally nuanced humor, we may be able to improve their overall classification accuracy.
As a result, while transformer models like RoBERTa and BERT-uncased show promise for humor classification, there is still room for improvement, especially when dealing with more nuanced forms of humor. Future research in this area could focus on developing more sophisticated models and ifne-tuning strategies to better capture the complexities of humor in text.
The experiments conducted demonstrate the potential of transformer-based models in humor classification. Both BERT-uncased and RoBERTa showed considerable efectiveness, but several challenges remain. These models struggled with jokes heavily dependent on cultural references or requiring an understanding of specific exaggerations. This indicates a need for incorporating more diverse training data and possibly integrating external knowledge sources to handle such cases better. Additionally, the importance of model interpretability should not be overlooked, as understanding why a model classifies a text as humorous is crucial for further development and trust in AI systems.
Future work could involve exploring more sophisticated ensemble methods, incorporating additional features such as sentiment and semantic similarity, and applying these models to larger and more diverse datasets. Further investigation into interpretability and the handling of ambiguous or context-dependent humor could also provide valuable insights for enhancing humor classification systems. Enhancing the dataset with more varied and culturally rich humorous texts could also improve the models’ robustness and applicability across diferent contexts and populations.
In this paper, we explored the use of advanced transformer-based models, specifically BERT-uncased and RoBERTa, for humor classification according to genre and technique. By leveraging these pretrained models and fine-tuning them on the provided dataset, we aimed to improve the accuracy and robustness of humor classification systems. Our experiments demonstrated that both models are efective in capturing the nuances of humor, with RoBERTa slightly outperforming BERT-uncased in overall performance metrics.
Key observations from our experiments indicate that transformer models, with their deep contextual understanding and extensive pre-training, can significantly enhance the ability to classify humorous texts. The results also highlight the importance of fine-tuning and hyperparameter optimization in achieving optimal performance.