In this paper, we present the submission of our team NaiveNeuron to the JOKER 2024 task competition on Automatic Humour Analysis. Our first approach involves utilizing a fastText classifier for Task 2. Our subsequent approaches then incorporate Retrieval-Augmented Generation (RAG) with Large Language Models (LLMs) and adapt it for Humour Classification. By utilizing fastText, known for its eficiency, along with the depth and contextual understanding provided by RAG-enhanced LLMs, we demonstrate significant potential for improvements in humour detection accuracy. Although such systems may not be able to obtain the best possible accuracy as of yet, due to their simplicity we view them as highly potent baselines and believe they may be a very solid starting point for future research in this area.
For humour detection tasks, researchers have mainly explored using transformer models, such as
BERT. For instance, the study [
When it comes to the use of Large Language Models (LLMs) for humour detection, the work closes
to ours would be [
• id: a unique identifier, • text: humorous text.
The JOKER 2024 datasets [
The train labels were provided in the format of JSON qrels file ( joker-2024-task2-classification-trainqrels.json) with the following fields: • id: a unique identifier from the input file, • class: class identifier for each humorous fenomena.
There were 6 possible classes: IR (irony), SC (sarcasm), EX (exaggeration), AID (incongruity-absurdity), SD (self-deprecating) and WS (wit-surprise). The train data consisted of 1742 samples and the test data consisted of 6642 samples.
Task 2 is a multiclass classification task with the aim to automatically classify text according to the following classes: irony, sarcasm, exaggeration, incongruity-absurdity, self-deprecating and wit-surprise. For Task 2, we employed the fastText library.
Our approach began with partitioning the original training dataset, joker-2024-task2-classificationtrain-input.json, and its corresponding labels, joker-2024-task2-classification-train-qrels.json , into training (70%), validation (15%), and test (15%) sets.
Prior to training the fastText classifier [
We tested all possible combinations of hyperparameter values as shown in Table 1. Additionally, we set the following parameters to fixed values: epoch=75 (number of training epochs) and dim=100 (size of word vectors). We evaluated the model’s precision, recall, accuracy, and F1-score for each hyperparameter combination using the validation set. Based on these metrics, the optimal hyperparameter configuration was determined to be minCount=5, wordNgrams=1, minn=1, maxn=4, and lr=1.
We also explored the impact of various preprocessing methods on the input data. Table 2 provides an overview of the preprocessing methods tested and their corresponding validation set accuracies. Our analysis concluded that using the original data without any preprocessing yielded the best results.
Our aim with Large Language Models (LLMs) is twofold. First we aim to study how capable are they of classifying the input text into one of the six categories in a zero-shot setup – that is, without being provided any specific example. To evaluate this sort of performance we utilize the following prompt: # Task You a r e a t e x t c l a s s i f i e r t h a t c l a s s i f i e s E n g l i s h t e x t t o t h e f o l l o w i n g c l a s s e s : − IR ( i r o n y ) : I r o n y r e l i e s on a gap between t h e l i t e r a l meaning and t h e i n t e n d e d meaning , c r e a t i n g a humorous t w i s t or r e v e r s a l . − SC ( sarcasm ) − Sarcasm i n v o l v e s u s i n g i r o n y t o mock , c r i t i c i z e , or convey contempt . − EX ( e x a g g e r a t i o n ) − E x a g g e r a t i o n i n v o l v e s m a g n i f y i n g or o v e r s t a t i n g something beyond i t s normal or r e a l i s t i c p r o p o r t i o n s . − AID ( i n c o n g r u i t y − a b s u r d i t y ) − I n c o n g r u i t y r e f e r s t o t h e u n e x p e c t e d or c o n t r a d i c t o r y e l e m e n t s t h a t a r e combined i n a humorous way and A b s u r d i t y i n v o l v e s p r e s e n t i n g s i t u a t i o n s , e v e n t s , or i d e a s t h a t a r e i n h e r e n t l y i l l o g i c a l , i r r a t i o n a l , or n o n s e n s i c a l . − SD ( s e l f − d e p r e c a t i n g ) − S e l f − d e p r e c a t i n g humour i n v o l v e s making fun o f o n e s e l f or h i g h l i g h t i n g one ’ s own f l a w s , weaknesses , or e m b a r r a s s i n g s i t u a t i o n s i n a l i g h t h e a r t e d manner . − WS ( wit − s u r p r i s e ) − Wit r e f e r s t o c l e v e r , quick , and i n t e l l i g e n t humour and S u r p r i s e i n humour i n v o l v e s i n t r o d u c i n g u n e x p e c t e d e le m en t s , t w i s t s , or p u n c h l i n e s t h a t c a t c h t h e a u d i e n c e o f f guard .
For any i n p u t , o u t p u t ONLY t h e u p p e r c a s e d c l a s s from t h e l i s t above .
In the second case we aim to see to what extent the performance of the model improves when its prompt includes a list of examples of both the input as well as its correct classification. To implement this approach the examples, which are obtained from the the training set based input text, are added towards the end of the aforementioned prompt in the format
Input: {text}\nOutput: {class}\n\n, where class is the correct output class.
All of the LLM-based experiments were done with the temperature being set to 0 to ensure their reproducibility.
The final fastText classifier with parameters minCount=5, wordNgrams=1, minn=1, maxn=4, and lr=1, dim=100, and epoch=200, was trained on a combined set of training and validation data (85% of the original dataset joker-2024-task2-classification-train-input.json and joker-2024-task2-classification-trainqrels.json) and achieved a test accuracy of 0.5725 on the remaining 15% of the original data. This trained model was then used to predict labels for the test dataset, joker-2024-task2-classification-test.json . The predictions were saved in naiveneuron_task2_fasttext.json and submitted.
While the fastText classifier demonstrated moderate performance with a test accuracy of 0.5725, there is significant potential for improvement by employing more advanced models such as large language models (LLMs). On one hand, the fastText classifier had the advantage of being straightforward and easy to implement. On the other hand, transformer-based models like BERT (Bidirectional Encoder Representations from Transformers) could potentially ofer enhanced capabilities in understanding context and semantic nuances.
Accuracy
Precision Model GPT-3.5-Turbo GPT-4o GPT-4 GPT-3.5-Turbo + RAG GPT-4o + RAG GPT-4 + RAG LLama3 70b + RAG GPT-4 + RAG UAE LLama3 70b + RAG UAE
The results of the LLM-based classification, which was done across multiple models including GPT-3.5Turbo, GPT-4, GPT-4o as well as LLama-3, can be seen in Table 3. All of the models were evaluated on the same test split as the fastText evaluation used.
As the results suggest, the zero-shot performance of LLMs on this task (the first three rows of Table 3) leaves quite a bit to be desired, which is a phenomenon we can observe across multiple models.
Involving Retrieval Augmented Generation (RAG) does indeed help, with the best obtained performance tying that of the fastText model. All of the RAG models used the mixedbread-ai/mxbai-embed-large-v11 embedding model which features 335M parameters. We also experiment with using WhereIsAI/UAE-Large-V12 which is of similar size but has yielded better
2https://huggingface.co/WhereIsAI/UAE-Large-V1 results in independent benchmarks. As the final two rows of Table 3 suggests, however, its inclusion in the overall pipeline did not lead to an increase in performance.
The Large Language Models (LLMs) were also able to uncover various issues with the actual evaluation (test set) data. In various cases all three of the models considered (that is GPT-3.5-Turbo, GPT-4 as well as the LLama3 model) refused to respond with a direct classification, despite being instructed to do so.
One such example was the sample with the ID 6471 with the content Where PR = Public Reactions.. To illustrate how dificult it was for the models to respond to this input, we provide their output below verbatim: GPT-3.5-Turbo The input is not a valid example of any of the given classes. Please provide a text that can be classified as either IR, SC, EX, AID, SD, or WS.
GPT-4 Sorry, but I’m not sure what you mean by "PR = Public Reactions." Could you please provide more context or clarify your question? Llama3 I’m ready when you are! Please provide the input text, and I’ll classify it into one of the following classes: - IR (irony) - SC (sarcasm) - EX (exaggeration) - AID (incongruity-absurdity) - SD (self-deprecating) - WS (wit-surprise)
Go ahead and provide the input!
This highlights the inherent limitations of LLMs and their combination with RAG: as their output space is not limited, they might end up responding in free text which won’t match one of the desired classes, especially when the input is indeed dificult to classify. In this particular case, however, we believe it would be best to remove this particular item from the test set due to its significant ambiguity, as it is not clear whether it does indeed fall into one of the aforementioned classes.
In this study, we explored the use of the fastText and LLM-based classifiers for fine-grained sentiment analysis of short humorous texts as part of Task 2 in JOKER 2024. Despite the fastText classifier achieving a moderate test accuracy of 0.5725, which was only tied by the evaluated LLM-based systems, our findings indicate potential for improvement. The simplicity and ease of use of the fastText model were advantageous; however, its limitations in capturing the complex nuances of humour were evident. Similarly, we only used LLMs in zero-shot and/or few-shot manner, without changing their parameters at all, which was only able to yield a classification of the same accuracy as the fastText model. This suggests that using classification models that utilize pre-trained Language Models as well as actually updating the parameters of Large Language Models that are being used might be necessary for obtaining higher performance.
Despite that, we believe that thanks to their simplicity we believe that the proposed models have the potential to serve as potent yet easy to setup baseline for any future exploration in humour classification.