PunDerstand @ CLEF JOKER 2024: Who's Laughing Now? Humor Classification by Genre & Technique Ryan Rony Dsilva 0 Nidhi Bhardwaj 0 Purdue University , West Lafayette, IN , USA 2024

Humor is subject to individual interpretation, with each person perceiving it diferently. Given that humor itself is subjective, this work explores classification of humor by genre and technique through three approaches: manual guided annotation, multi-class classification using BERT-based models with and without sampling, and prompting with large language models. Our experiments revealed insights into the performance of diferent models and approaches on the humor classification task and opens up further discussions on using guidelines from the annotation to aid large language models.

 eol>humor classification BERT large language models humor theory guided annotation 1. Introduction 2. Methodology

For this task, the dataset contained manually annotated examples from the JOKER 2023 corpus [ 5 ] as well as new data.

2.1. Guided Annotation

To classify the type of humor present in sentences, we implemented a guided annotation process. This method involved developing a comprehensive codebook that provided explicit guidelines for categorizing sentences into predefined humor types. To minimize bias arising from preconceived notions of humor, we assigned pseudo names to the categories. This anonymization aimed to ensure that annotators based their classification decisions solely on the structural and contextual cues detailed in the codebook, rather than on any prior subjective understanding of the humor types. Two annotators were tasked with categorizing humor based on guidelines outlined in the provided codebook. Sentences where both annotators agreed on the humor category were considered final and included for submission, while those with disagreement were excluded. Upon reaching consensus, a total of 350 sentences were submitted as the final annotated dataset. The codebook outlined specific characteristics and markers for each humor type. The instructions were derived from the definitions of each category and the patterns observed in the training dataset. The detailed codebook used in this process is included in the appendix.

Construction of the Codebook

To classify wit in sentences, we adopted definitions from [ 6 ], which describe wit as involving an unexpected twist or element that generates humor. In the training dataset, sentences containing wit often exhibited patterns like the use of words with multiple meanings or homophones. These linguistic features were incorporated to facilitate easier identification of wit.

For categorizing sentences as incongruous or absurd, the training dataset revealed a common bipartite structure: the first part typically posed a question, followed by an unexpected or unrealistic answer. Absurd humor was detected by identifying nonsensical situations that elicited humor [ 7 ]. Any humorous characteristic that appeared illogical or unrealistic was classified under this category.

Self-deprecation was identified by structural cues indicating that a sentence negatively addressed oneself [ 8 ]. The humor in self-deprecating jokes arises from highlighting one’s weaknesses and flaws in an embarrassing yet unexpected manner, as it is uncommon for individuals to discuss their shortcomings openly [ 9 ].

To identify exaggeration, we focused on detecting hyperbolic terms [ 10 ] within sentences that dramatically described situations as better or worse than they actually were [ 11 ]. For sarcasm, annotators identified elements of contempt, often indicated by negative polarity words used to criticize someone [ 12 ]. Sarcasm, a form of irony, employs implied meanings to mock or deride [ 13 ]. We sought sentences with implied meanings that aimed to ridicule weaknesses or events negatively.

Irony was characterized by having two elements: a literal meaning and an implied meaning, with the two needing to difer to produce a humorous efect [ 14, 15 ].

Annotators followed the sequence defined in the codebook to ensure that categories such as sarcasm [ 16 ] and exaggeration [ 17 ], which are specific types of irony, were correctly classified only when their unique elements were present. Irony served as an overarching category encompassing these specific humor types, providing a structured framework for accurate annotation.

2.2. Multi-Class Classification with DeBERTa

In our study, we employ the DeBERTa [ 18 ] model as the base model for our experiments. DeBERTa has been recognized as one of the leading choices for encoder models due to its superior performance in various natural language processing tasks. We fine-tune the DeBERTa model using our training dataset and conduct two separate experimental runs. The first run involves using the dataset in its original form, without any modifications to address class imbalances. In this approach, we aim to evaluate the model’s performance on the raw, imbalanced data. In the second experimental run, we address the class imbalance by implementing an under-sampling strategy. This method ensures that the representation of each class is balanced, preventing any single class from having a disproportionately high number of samples. Specifically, we cap the number of samples for the majority classes at = 250. For the ifne-tuning process, we utilize the deberta-v3-large model. The fine-tuning parameters are meticulously chosen to optimize performance. The learning rate is set to 2 × 10− 5, with a training batch size of 8 and an evaluation batch size of 16. The model is trained for 5 epochs, and we apply a weight decay of 0.01 to regularize the training process.

2.3. Prompting with LLMs

In our methodology involving large language models (LLMs), we utilized GPT-4o [ 19 ], the most recent model developed by OpenAI. Our approach incorporated the few-shot prompting technique, which involves providing the model with a limited number of examples to guide its responses. Specifically, we included one example for each class, which served as a template to demonstrate the desired output format and content. Detailed descriptions of these prompts can be found in the appendix. The methodology was inspired from [ 20 ] where humor theories were embedded into the prompts. For reproducibility of our results, we set the random seed to 2024 and configured the temperature parameter to 0 as outlined in the OpenAI documentation. By setting the temperature to 0, we aimed to reduce the model’s output variability, thereby enhancing consistency and repeatability in the generated responses.

3. Results

The metrics of precision, recall, accuracy, and F-score are reported and metrics in the tables below are computed for both the training and test dataset.

Table 1 and Table 2 present the weighted average performance metrics for the four approaches on both the training and test sets. On the training set, Guided Annotation shows moderate performance with an accuracy of 0.7126 and a balanced F-score of 0.7148, evaluated on a smaller subset (support of 87). DeBERTa exhibits the highest performance across all metrics, with an accuracy of 0.7983 and an F-score of 0.7939, indicating strong overall performance. DeBERTa has slightly lower accuracy (0.7854) and F-score (0.7906) than DeBERTa but maintains high precision (0.8124). GPT-4o lags significantly with the lowest accuracy (0.4496) and F-score (0.4563), suggesting a need for change with the methodolgy with LLMs. On the test set, DeBERTa’s performance decreases compared to the training set with an accuracy of 0.6870 and an F-score of 0.6731, indicating some loss in generalization. DeBERTa also shows a decrease in performance with an accuracy of 0.6787 and an F-score of 0.6768, but it maintains higher precision than recall, suggesting it still identifies relevant instances well. GPT-4o again shows the lowest performance with an accuracy of 0.4668 and an F-score of 0.4733, consistent with its training performance.

Table 3 and Table 4 provide class-wise performance metrics for the approaches on both the training and test sets. Guided Annotation shows varying performance across classes, with high F-scores in SC (0.8718 on training, 0.9000 on test) and AID (0.8000 on training, 0.8333 on test), but very low performance in EX (0.1538 on training, 0.2222 on test) and WS (0.5000 on training, 0.4000 on test). DeBERTa performs consistently well across most classes, especially in AID (0.9671 on training, 0.8889 on test) and SD (0.8996 on training, 0.7736 on test), but struggles in EX (0.3988 on training, 0.2614 on test) and WS (0.5201 on training, 0.4762 on test). DeBERTa also shows strong performance, particularly in AID (0.8974 on training, 0.8795 on test) and SD (0.8583 on training, 0.7773 on test), with improved performance in EX (0.5745 on training, 0.4240 on test) and WS (0.6021 on training, 0.5000 on test) compared to its non-sampled counterpart, indicating better generalization.

 4. Conclusions

The task of humor classification, particularly identifying specific types of humor, remains a complex challenge due to the subjective nature of humor perception. Our study presented a comprehensive approach involving guided annotation, fine-tuning the DeBERTa model, and using prompting with GPT-4o. The results highlight the efectiveness of DeBERTa in both original and sampled forms, showcasing its strong performance across various humor types. However, GPT-4o demonstrated significant limitations, suggesting that current LLMs may require further refinement or alternative methodologies to handle the nuances of humor classification efectively. Future research should focus on integrating the guided annotation approach directly into the prompting process for large language models (LLMs). By embedding detailed codebook guidelines and structural cues within the prompts, we can provide LLMs with more context and specificity, potentially improving their performance in humor classification tasks.

 A. Prompts

### Instruction ### You are an expert in linguistics and humor. Classify the text into one of the appropriate types of humor from the following: Irony (IR), Sarcasm (SC), Exaggeration (EX), Absurdity & Incongruity (AID), Self-Deprecating (SD), Wit (WS). You must respond with valid JSON with only one key ‘output‘, containing the correct classification of the sentence. ### Text ### I tried to learn how to make puns, but no pun in ten did. ### Humor Type ### { }

"output": "WS" ### Text ### Did you hear about the pasta that got locked out of the house? Gnocci. ### Humor Type ### { }

"output": "AID" ### Text ### Amazing how fast this team can go winning from 13 straight to losing three in a row. Lol . Horrible managing tonight. I really hope this Boone experiment is over soon. ### Text ### Ohio news station reminds viewers what day it is during coronavirus lockdown. ### Humor Type ### { ### Text ### Good day, this is your trashcan speaking. ### Humor Type ### {

 B. Guided Annotation Codebook

Please follow the following instructions to annotate the given sentences into the category most suitable according to the instructions given. Follow the order in which each category is described and move forward to the next category only if the previous category is eliminated.

1. Identify words with multiple meanings or homophones (similar sounding words), supported by contextual clues within the sentence. 2. Note any unexpected elements or sudden changes in the sentence. 1. Look for sentences structured as *?*. Or, 2. There exist words in the second part of the text which you might not expect on the basis of the ifrst part of the sentence. 3. Identify phonetic incongruities in the second part of the text. 4. Detect illogical situations or events that are unrealistic or nonsensical. 1. Look for events which might be embarrassing. Or, 2. Look for human flaws or weakness described and, 3. Look for specific sentence structures indicating self-reference, such as: a) Interjection followed by ’I’, ’we’, or ’you’. b) Conjunction followed by ’I’, ’we’, or ’you’. c) Question followed by ’I’ or ’we’. d) ’I’ or ’we’ followed by a verb. e) ’I’ or ’we’ followed by a negative model verb. f) Frequency of ’my’, ’me’, and ’I’.

g) Presence of negative polarity.

Category 5

1. Identify situations or events described in a manner better or worse than normal. And, 2. Assess the description of events and their impacts for overly dramatic elements.

 Category 6 [1] W. Ruch , Psychology of humor, in: V. Raskin (Ed.), The Primer of Humor Research, number 8 in Humor Research , Mouton de Gruyter, Berlin, 2008 , pp. 17 - 100 . doi: 10 .1515/9783110198492. 17. [2] L. Ermakova , T. Miller , A.-G. Bosser , V. M. Palma Preciado , G. Sidorov , A. Jatowt , Overview of JOKER - CLEF -2024 track on Automatic Humor Analysis , in: L. Goeuriot , P. Mulhem , G. Quénot , D. Schwab , L. Soulier , G. M. Di Nunzio , P. Galuščáková , A. G. Seco de Herrera , G. Faggioli, N. Ferro (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2024 ), 2024 . [3] L. Ermakova , A.-G. Bosser, T. Miller , T. Thomas-Young , V. M. Palma Preciado , G. Sidorov , A. Jatowt , CLEF 2024 JOKER lab: Automatic Humour Analysis , in: N. Goharian , N. Tonellotto , Y. He , A. Lipani , G. McDonald , C. Macdonald , I. Ounis (Eds.), Advances in Information Retrieval: 46th European Conference on Information Retrieval , ECIR 2024 , Glasgow, UK, March 24 -28, Proceedings, Part VI , volume 14613 of Lecture Notes in Computer Science, Springer, Cham, 2024 , pp. 36 - 43 . doi: 10 . 1007/978-3- 031 -56072- 9 _ 5 . [4] V. M. Palma Preciado , et al., Overview of the clef 2024 joker task 2: Humour classification according to genre and technique , in: G. Faggioli , et al. (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024 ), CEUR Workshop Proceedings, CEUR-WS.org, 2024 . URL: http://ceur-ws. org. [5] L. Ermakova , A.-G. Bosser , A. Jatowt , T. Miller , The joker corpus: English-french parallel data for multilingual wordplay recognition , in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23) , Association for Computing Machinery, New York, NY, USA, 2023 , pp. 2796 - 2806 . doi: 10 .1145/3539618. 3591885. [6] D. L. Long , A. C. Graesser , Wit and humor in discourse processing , Discourse processes 11 ( 1988 ) 35 - 60 . [7] O. Couder , Problem solved? absurdist humour and incongruity-resolution , Journal of Literary Semantics 48 ( 2019 ) 1 - 21 . [8] A. Kamal , M. Abulaish , Self-deprecating humor detection: A machine learning approach , in: Computational Linguistics: 16th International Conference of the Pacific Association for Computational Linguistics , PACLING 2019 , Hanoi, Vietnam, October 11-13 , 2019 , Revised Selected Papers 16 , Springer, 2020 , pp. 483 - 494 . [9] R. A. Martin , P. Puhlik-Doris , G. Larsen , J. Gray , K. Weir, Individual diferences in uses of humor and their relation to psychological well-being: Development of the humor styles questionnaire , Journal of research in personality 37 ( 2003 ) 48 - 75 . [10] E. Troiano , C. Strapparava , G. Özbal, S. S. Tekiroğlu , A computational exploration of exaggeration , in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , 2018 , pp. 3296 - 3304 . [11] J. Patro , S. Baruah , A simple three-step approach for the automatic detection of exaggerated statements in health science news , in: P. Merlo, J. Tiedemann , R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Association for Computational Linguistics , Online, 2021 , pp. 3293 - 3305 . URL: https://aclanthology.org/ 2021 .eacl-main. 289 . doi: 10 .18653/v1/ 2021 .eacl-main. 289 . [12] S. K. Bharti , K. S. Babu , S. K. Jena , Parsing-based sarcasm sentiment recognition in twitter data , in: Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015 , 2015 , pp. 1373 - 1380 . [13] J. D. Campbell , A. N. Katz , Are there necessary conditions for inducing a sense of sarcastic irony? , Discourse Processes 49 ( 2012 ) 459 - 480 . [14] H. P. Grice , Logic and conversation , in: Speech acts, Brill, 1975 , pp. 41 - 58 . [15] D. Sperber , D. Wilson, Irony and the use-mention distinction , Philosophy 3 ( 1981 ) 143 - 184 . [16] H. L. Colston , Irony and sarcasm, in: The Routledge handbook of language and humor , Routledge, 2017 , pp. 234 - 249 . [17] D. Wilson , D. Sperber , On verbal irony, Lingua 87 ( 1992 ) 53 - 76 . [18] P. He , J. Gao , W. Chen, DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing , in: The Eleventh International Conference on Learning Representations , 2023 . URL: https://openreview.net/forum?id= sE7 - XhLxHA . [19] J. Achiam , S. Adler , S. Agarwal , L. Ahmad , I. Akkaya , F. L. Aleman , D. Almeida , J. Altenschmidt , S. Altman , S. Anadkat , et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 ( 2023 ). [20] R. R. Dsilva , Augmenting Large Language Models with Humor Theory To Understand Puns ( 2024 ). URL: https://hammer.purdue.edu/articles/thesis/Augmenting_Large_Language_Models_ with_Humor_Theory_To_Understand_Puns/25674792. doi: 10 .25394/PGS.25674792. v1 . 1. Determine if the sentence conveys negative polarity, showing contempt or criticism . And, 2. Assess whether the sentence criticizes something or mocks a phenomenon or event . And, 3. Verify if the sentence's meaning difers from its literal interpretation . 1. Identify the literal meaning of the sentence . And, 2. Discern any implied meanings, ensuring they difer from the literal interpretation .