The CLEF 2024 JOKER track focuses on the automatic processing of wordplay through three tasks: humour-aware information retrieval, humour classification by genre and technique, and translation of puns from English to French. Recent advancements in Large Language Models (LLMs) have enhanced their conversational abilities, yet they struggle with humour detection and generation. Addressing this gap can significantly improve humancomputer interactions. For Task 1, we implemented a TF-IDF vectorizer and logistic regression model to identify and rank humorous sentences. The model achieved an F1 score of 0.93 for non-puns and 0.73 for puns, indicating robust performance in humour detection. In Task 2, we classified jokes into five categories using logistic regression, Naive Bayes, and support vector machines. The SVM model performed best, with F1 scores ranging from 0.14 to 0.61, showing particular eficacy in classifying wit. Task 3 involved translating puns using the MarianMT model from the Hugging Face library. Although successful, the process was time-intensive, suggesting the need for more eficient methods. Overall, our approaches demonstrated efective humour identification and translation capabilities but faced challenges in genre-specific classification. This research underscores the importance of improving LLMs' humour processing abilities for better human-computer interaction.
In recent years, Large Language Models (LLMs) have increased in their capabilities exponentially.
Since the release of ChatGPT-4 in 2023, the world has quickly become accustomed to interacting with
LLMs via a chatbot API that made users feel like they were truly chatting with the machine. A 2024
study applied the Turing Test to participants using ChatGPT-4, finding that humans incorrectly judged
ChatGPT-4 to be human 54 percent of the time, showing just how convincingly LLMs can now emulate
human conversation. [
As impressive as this is, LLMs are still found to be lacking in several areas, humour being one of
them. Large language models struggle to reliably detect and explain humour[
Puns are a form of humour based on wordplay. Usually puns exploit double meanings of words or similarity of sounds between diferent words to create a humorous or witty efect, with frequent use of double entendré, homophones, or similar-sounding words. Words which could be used to form puns are words like “profit” and “prophet”, for example, or “check” and “Czech”.
Perhaps what makes humour analysis such a challenge for LLMs is the many diferent subtle forms of it that exist. Humour can be “on-the-nose”, physical, awkward, subtle, obvious, visual, childish, or intelligent; it’s especially hard to define, which presents problems for LLMs.
Wordplay, by its very nature, exploits the intrinsic structure of the source language used and certain
characteristics used that may be impossible or dificult to find replacements or analogues of in the
target language.[
The JOKER track of CLEF 2024 aims to develop interdisciplinary approaches to the automatic
processing of wordplay. [
• Task 3: Translation of puns from EN to FR.
Can LLMs succeed in identifying, classifying, and translating humour? In this paper we will explore this question, ahead we detail our workflow and results for tackling the three tasks of the JOKER track.
The data provided consisted of four JSON files. There was a “corpus” file containing a list of 61,268 pun and non-pun sentences, “queries test” and “queries train” files that list the corresponding keyword linked to the sentences, and a “qrels train” file that if the sentence was a pun (“1”) or not (“0”).
Our first step was to merge the data, creating a table with five columns: “qid”, “docid”, “qrel”, “text, and “query”. We then used a TF IDF vectorizer to train the model on all our text and the corresponding “qrel” values.
# q u e r y t e x t and j o k e t e x t i n t o a s i n g l e column − TF− IDF V e c t o r i z e r d a t a merged [ ’ t e x t a l l ’ ] = d a t a merged [ ’ q u e r y ’ ] + " " + d a t a merged [ ’ t e x t ’ ] # F i t and t r a n s f o r m t h e c o m b i n e d t e x t t f i d f m a t r i x = t f i d f v e c t o r i z e r . f i t t r a n s f o r m ( d a t a merged [ ’ t e x t a l l ’ ] )
Then we created a logistic regression model based of of our training data and used the model to make predictions based of of our “queries test” data. from s k l e a r n . l i n e a r model i m p o r t L o g i s t i c R e g r e s s i o n # L o g i s t i c R e g r e s s i o n model model = L o g i s t i c R e g r e s s i o n ( ) # T r a i n e d model t r a i n e d model = model . f i t ( X t r a i n , y t r a i n )
The model returned relevance scores for each joke in the corpus for each query. (Appendix A) Based on these results we produced a JSON file listing the “best” or rather, most relevant, jokes. (Appendix B)
The results obtained from our model were very positive. Calculation of an F1 score for our model produced results of 0.93 for the “0” (no pun) class, and 0.73 for the “1” class. This indicates that it performed extremely well, with high precision and recall, for identifying sentences without puns and moderately well for identifying those with puns.
The data provided consisted of 3 JSON files. “classification test data” and “classification train input data” each contained “text” column containing thousands of jokes. The classification data was contained in the file “classification train qrels data”, which provided the labels for our training data, categorising each joke into one of five classes, IR (irony), SC (sarcasm), EX (exaggeration), AID (incongruity), SD (self-deprecating), WS (wit).
We merged the training data JSONs to create a dataframe with the text and classes side by side. We preprocessed our text by removing contractions, making all letters lowercase, removing special characters and URLs, then replaced the original text with our cleaned text in our data frame containing the training labels. (Appendix C) prompt t e r m s = " " "
You a r e a r o b o t t h a t ONLY o u t p u t s JSON .
You r e p l y i n JSON f o r m a t w i t h t h e f i e l d ’ t e r m s ’ .
You p r o v i d e ONLY s e m i c o l o n − s e p a r a t e d l i s t o f MAXIMUM 3
s c i e n t i f i c t e r m s o f a s o u r c e s e n t e n c e ONLY .
You DO NOT add ’ Sure , Here a r e t h e s c i e n t i f i c t e r m s o f your s e n t e n c e : ’ .
Example s o u r c e s e n t e n c e : I n t h e modern e r a o f a u t o m a t i o n and r o b o t i c s , \ autonomous v e h i c l e s a r e c u r r e n t l y t h e f o c u s o f a c a d e m i c and i n d u s t r i a l r e s e a r c h . ? \ Example answer : { ’ t e r m s ’ : ’ r o b o t i c s ; autonomous v e h i c l e s ’ } Now h e r e i s my s e n t e n c e : " " "
We encoded our classifications to numbers and used TF-IDF to vectorise the text. The data was then used to train a logistic regression model, a naive bayes model and a support vector machines model. # T r a i n L o g i s t i c R e g r e s s i o n model l o g i s t i c r e g r e s s i o n model = L o g i s t i c R e g r e s s i o n ( max i t e r = 1 0 0 0 ) l o g i s t i c r e g r e s s i o n model . f i t ( X t r a i n t f i d f , y t r a i n ) # T r a i n N a i v e B a y e s model n a i v e b a y e s model = M u l t i n o m i a l N B ( ) n a i v e b a y e s model . f i t ( X t r a i n t f i d f , y t r a i n ) # T r a i n SVM model svm model = SVM ( k e r n e l = ’ l i n e a r ’ ) # You c a n s p e c i f y d i f f e r e n t k e r n e l s l i k e ’ l i n e a r ’ , ’ p o l y ’ , ’ r b f ’ , e t c . svm model . f i t ( X t r a i n t f i d f , y t r a i n )
With our models trained, we were able to make predictions based of of our test data, which we preprocessed in the same way as we did our training data. We then had to convert our classes back to their original names, from numbers. (Appendix D)
The results of our F1 testing reveals big diferences in the performance of each model. The logistic regression model achieved F1 results of between 0.05-0.54 for all the 5 categories. The SVM model achieved better results, with F1 results between 0.14 and 0.61. Interestingly, for both models, the “WS” (wit) class achieved the highest F1 score, indicating this was the easiest to classify. In both models, “IR” (irony) and “EX” (exaggeration) achieved lowest scores, indicating dificulty in classifying those types of jokes.
The data provided consisted of 3 JSON files: “translation EN FR train input”, consisting of 1406 jokes in English, “translation EN FR train qrels”, containing 5839 jokes in french, and “task3 2024 test”, consisting of 4502 rows of jokes in English. There were also two .tsv files, “joker translation EN-FR train input.tsv, and “joker translation EN-FR train qrels.tsv”. In the parent folder, there was also a JSON, “joker translation test”, and “joker translation test.tsv”.
First we merged our data to create a unified data frame around the English and French jokes, joined using the “id en” variable. We loaded the hugging face “transformer” library and used the MarianMTModel and MarianTokenizer then used a pre-trained model, “Helsinki-NLP Opus”, after trying first with EasyNMT and finding better results from the Helsinki model. (Appendix E)
We iterated through the whole list of English jokes, translating to French, then decoded the vectors back to letters, before saving in a new column alongside the originals. (Appendix F)
The translations seemed to be successful however, one drawback of using our method was that the process was highly time intensive, suggesting there may be more practical methods available.
We can conclude that our techniques are efective at identifying the presence of a joke, but rather less efective at classifying them into one of our five classes. Thanks to the existence of pre-trained models which are tailored to the task, we can conclude that the translation of our jokes from English to French was successful. We would like to extend our gratitude to the University of Brest for organising the Blended Intensive Programme (BIP) AI For Humanities. We would also like to thank Liana Ermakova for her teaching of the course and Caroline L’haridon for her support during our stay in Brest. .1. Appendix A r e s u l t s = [ ] # I t e r a t e o v e r e a c h t e s t q u e r y f o r i n d e x , t e s t q u e r y i n d a t a t e s t q u e r i e s . i t e r r o w s ( ) : q u e r y i d = t e s t q u e r y [ ’ q i d ’ ] q u e r y t e x t = t e s t q u e r y [ ’ query ’ ] # C a l c u l a t e r e l e v a n c e f o r e a c h j o k e i n t h e c o r p u s w i t h t h i s q u e r y s c o r e s = [ ] f o r , j o k e i n d a t a c o r p u s . i t e r r o w s ( ) : i f j o k e [ ’ t e x t ’ ] i s None :
c o n t i n u e
e l s e :
t e x t a l l = q u e r y t e x t + " " + j o k e [ ’ t e x t ’ ]
v e c t o r i z e d t e x t = t f i d f v e c t o r i z e r . t r a n s f o r m ( [ t e x t a l l ] )
r e l e v a n c e s c o r e = model . p r e d i c t p r o b a ( v e c t o r i z e d t e x t ) [
# P r e p a r e o u t p u t JSON f o r m a t f o r rank , s c o r e i n f o i n enumerate ( s c o r e s , s t a r t = 1 ) : r e s u l t s . append ( { ’ run id ’ : " T omi sl av&Rowan t a s k 1 TFIDF " , ’ manual ’ : 0 , ’ rank ’ : rank , ’ s c o r e ’ : s c o r e i n f o [ ’ s c o r e ’ ] , ’ docid ’ : s c o r e i n f o [ ’ docid ’ ] , ’ qid ’ : query i d } ) with open ( ’ r e s u l t j o k e r t a s k 1 . j s o n ’ , ’w’ ) a s o u t f i l e :
j s o n . dump ( r e s u l t s , o u t f i l e , i n d e n t = 4 ) .3. Appendix C # P r e p r o c e s s i n g f u n c t i o n from n l t k . stem i m p o r t WordNetLemmatizer i m p o r t c o n t r a c t i o n s i m p o r t r e i m p o r t n l t k n l t k . download ( ’ stopwords ’ ) n l t k . download ( ’ wordnet ’ ) from n l t k . c o r p u s i m p o r t s t o p w o r d s lem = WordNetLemmatizer ( ) d e f p r e p r o c e s s t e x t ( t e x t ) : sms = c o n t r a c t i o n s . f i x ( s t r ( t e x t ) ) # c o n v e r t i n g s h o r t e n e d words t o o r i g i n a l ( Eg : " I ’m" t o " I am " ) sms = sms . lower ( ) # lower c a s i n g t h e message sms = r e . sub ( r ’ h t t p s ? : / / S + |www. S + ’ , " " , sms ) . s t r i p ( ) # removing u r l sms = r e . sub ( " [ ^ a −z ] " , " " , sms ) # removing symbols and
numbers ( k e e p i n g o n l y c h a r a c h t e r s from a −z ) sms = sms . s p l i t ( ) # s p l i t t i n g # l e m m a t i z a t i o n and stopword removal sms = [ lem . l e m m a t i z e ( word ) f o r word i n sms i f not word i n s e t ( s t o p w o r d s . words ( " e n g l i s h " ) ) ] sms = " " . j o i n ( sms ) r e t u r n sms X = d f t r a i n [ " t e x t " ] . a p p l y ( p r e p r o c e s s t e x t ) .4. Appendix D d f b a y e s t e s t = pd . DataFrame ( t e s t d a t a ) # Apply t e x t p r e p r o c e s s i n g d f b a y e s t e s t [ ’ c l e a n t e x t ’ ] = d f b a y e s t e s t [ ’ t e x t ’ ] . a p p l y ( p r e p r o c e s s t e x t ) # TF−IDF V e c t o r i z a t i o n f o r t e s t d a t a X t e s t t f i d f = t f i d f v e c t o r i z e r . t r a n s f o r m ( d f b a y e s t e s t [ ’ c l e a n t e x t ’ ] ) # P r e d i c t b a y e s p r e d i c t i o n s = n a i v e b a y e s model . p r e d i c t ( X t e s t t f i d f ) # Convert back t o o r i g i n a l names b a y e s p r e d i c t e d c l a s s e s = l a b e l e n c o d e r . i n v e r s e t r a n s f o r m ( b a y e s p r e d i c t i o n s ) .5. Appendix E from t r a n s f o r m e r s i m p o r t MarianMTModel , M a r i a n T o k e n i z e r # Load pre − t r a i n e d MarianMT model and t o k e n i z e r f o r E n g l i s h t o
French t r a n s l a t i o n model name = " H e l s i n k i −NLP / opus −mt−en − f r " model = MarianMTModel . from p r e t r a i n e d ( model name ) t o k e n i z e r = M a r i a n T o k e n i z e r . from p r e t r a i n e d ( model name ) # D e f i n e i n p u t t e x t i n p u t t e x t = " T r a n s l a t e t h i s t e x t t o French . " # T o k e n i z e i n p u t t e x t i n p u t s = t o k e n i z e r ( i n p u t t e x t , r e t u r n t e n s o r s =" p t " ) # Perform t r a n s l a t i o n o u t p u t s = model . g e n e r a t e ( ∗ ∗ i n p u t s ) # Decode t r a n s l a t e d o u t p u t t r a n s l a t e d t e x t = t o k e n i z e r . decode ( o u t p u t s [ 0 ] , s k i p s p e c i a l t o k e n s =
True ) # P r i n t t r a n s l a t e d t e x t p r i n t ( " T r a n s l a t e d t e x t : " , t r a n s l a t e d t e x t ) # Assuming you have a l r e a d y l o a d e d t h e t e s t d a t a i n t o a DataFrame d f t e s t d a t a # T r a n s l a t e j o k e s f o r , row i n d f t e s t d a t a . i t e r r o w s ( ) : # T r a n s l a t e each row ’ s E n g l i s h t e x t t o French t r a n s l a t i o n = model . g e n e r a t e ( ∗ ∗ t o k e n i z e r ( row [ ’ t e x t en ’ ] , r e t u r n t e n s o r s =" p t " , padding = True ) ) t r a n s l a t e d t e x t = t o k e n i z e r . decode ( t r a n s l a t i o n [ 0 ] , s k i p s p e c i a l t o k e n s = True ) # Append t h e t r a n s l a t i o n r e s u l t t o t h e r e s u l t s l i s t r e s u l t s . append ( { ’ run id ’ : " T omi sl av&Rowan t a s k 3 MarianMTModel " , ’ manual ’ : 0 , ’ i d en ’ : row [ ’ i d en ’ ] , ’ t e x t f r ’ : t r a n s l a t e d t e x t } ) # Convert r e s u l t s l i s t t o DataFrame t r a n s l a t e d d f = pd . DataFrame ( r e s u l t s ) # P r i n t or use t h e t r a n s l a t e d DataFrame a s needed p r i n t ( t r a n s l a t e d d f )