The problem of accurately translating wordplay in automatic humour analysis remains challenging, as highlighted by the CLEF 2024 JOKER Track. This problem is particularly interesting because humour is deeply cultural and context-dependent, making it difficult for language models to handle. Our study compares last year's and this year's results to determine if there have been improvements in the automatic translation of wordplay using BLOOM and Googletrans. The findings indicate that while numerical metrics show high precision, recall, and F1 scores, the actual translation and conveyance of jokes' meanings still rely heavily on coincidence, underscoring the need for further training and enhancement of language models.
According to the Overview of the CLEF 2024 JOKER Track (Automatic Humour Analysis), this Lab was established in 2022 at the Conference and Labs of the Evaluation Forum (CLEF) [1]. The author of this article is participating in the tasks of this laboratory for the second consecutive year [2]. In this way, this article will attempt to compare last year's results with this year's results to see if the automatic translation of wordplay using language models and automatic translators has improved.
Before sharing the methodology and the results of this year's runs in comparison with last year's, we will describe the JOKER tasks for this year, which include three different tasks:
• Task 1: Humour-aware information retrieval • Task 2: Humour classification according to genre and technique • Task 3: Translation of puns from English to French This year, task 3 includes translation only from English to French, whereas in CLEF 2023, it also included Spanish. However, despite the exclusion of Spanish this year, the organizers of JOKER Track at CLEF 2024 allowed the author of this article to carry out the task using the Spanish language. Therefore, these working notes will focus solely on describing task 3 of JOKER Track.
The overall objective of this paper is to translate puns from English to Spanish using BLOOM and Google Translate. BLOOM is the only language model used, as the previous study found that it produced better results than all other models, including GPT, Simple T5, EasyNMT-Opus, and EasyNMT-mbart [2].
These working notes are organized as follows: after the introduction is the Experimental Setup section dedicated to the approach (data and method description); the third section is Experimental Results that includes some tables with metrics and scores, comparing the results of the task 3 of CLEF 2023 and CLEF 2024; and the fourth section is the discussion of the results and conclusions.
In this section we will give a brief description of the data provided to perform task 3, the methods used to solve it and the evaluation methods.
The task will provide an updated test set of English punning jokes, for French language the training data is also available. However, considering that the training data for Spanish is not needed for this experiment, we only use the test data. The test data is provided in JSON format. As we can see, we were provided with 5,727 jokes in English to translate. The data itself looked as follows: She dumped him because of all their lousy dates. After all, whining and dining does get tiresome after a while.
Each entry in the table contains an identifying number for each joke, and a one-or two-sentence joke that includes a pun.
As mentioned in the introduction, to carry out task 3, we used BLOOM with two different prompts and Google Translate.
BigScience Large Open-science Open-access Multilingual Language Model (BLOOM) is a transformerbased language model [3]. Because the number of tokens is limited in BLOOM, each run was performed with only 100 puns.
The prompts used are provided below.
• Prompt 1:
"Original: Diabetics should not be allowed to have sweet dreams.\n\ Translation: Los diabéticos no deberían tener dulces sueños.\n\ \n\ Original: I'm going to the guillotine at dawn and my wife has already collected my severance pay.\n\ Translation: Al amanecer me van a pasar por la guillotina y mi mujer ya ha firmado la separación.\n\ \n\ Original: After 5 years with the same chiropractor, I moved and had to change doctors. It was quite an adjustment.\n\ Translation: Me mudé y tuve que buscar otros médicos después de estar cinco años con el mismo quiropráctico. Fue un mero ajuste.\n\ \n\ Original: A scientist doing a large experiment with liquid chemicals was trying to solve a problem when he fell in and became part of the solution.\n\ Translation: Un científico que hacía un gran experimento con productos químicos líquidos estaba intentando solucionar un problema cuando cayó en que él se convertiría en parte de la solución.\n\ \n\ Original: Old electricians never die, they just keep plugging away.\n\ Translation:"
• Prompt 2:
"Request: Translate from English to Spanish: Diabetics should not be allowed to have sweet dreams.\n\ Answer: Los diabéticos no deberían permitirse soñar dulce (expresión figurativa para decir que no deberían permitirse deseos excesivos o indulgencias).\n\ \n\ Request: I'm going to the guillotine at dawn and my wife has already collected my severance pay.\n\ Answer: Me voy a la guillotina al amanecer y mi esposa ya ha recogido mi indemnización ( expresión figurativa para decir que me estoy preparando para un evento desafortunado y mi esposa ya ha preparado los asuntos financieros para el futuro sin mí).\n\ \n\ Request: After 5 years with the same chiropractor, I moved and had to change doctors. It was quite an adjustment.\n\ Answer:"
It is curious that when we created the second prompt, BLOOM itself, along with the Spanish translation in brackets, provided an explanation of the wordplay:
• 'expresión figurativa para decir que no deberían permitirse deseos excesivos o indulgencias' = 'figurative expression to say that excessive desires or indulgences should not be allowed' • 'expresión figurativa para decir que me estoy preparando para un evento desafortunado y mi esposa ya ha preparado los asuntos financieros para el futuro sin mí' = 'figurative expression to say that I am preparing for an unfortunate event and my wife has already prepared the financial affairs for the future without me'
Thus, it appears that BLOOM has attempted to apply a sort of translator's explanatory note.
Googletrans is a free and unlimited python library that implements Google Translate API [4]. Since we use Google Colab for code execution, it was absolutely impossible to process all of them at once with over 5000 examples. Therefore, we were compelled to divide the test data into four parts.
The task organizers used two methods for evaluating translations: BLEU and BERT Score [1]. BLEU (BiLingual Evaluation Understudy) measures the lexical similarity between a candidate translation and a reference translation [5]. The organizers utilized the sacreBLEU implementation9 with the default tokenizer 13a, which emulates the mteval-v13a script from Moses [6]. Their report includes the BLEU score (harmonic mean) and BLEU precisions for n-grams.
BERT Score, obtained from the python bert-score package10 [7], presents mean values of precision, recall, and F1 scores.
We divided the results section into two parts: numerical results and linguistic results. In the first subsection, we examine the evaluation metrics provided by the organizers, while in the second, we discuss some examples of translations produced using each method.
Above all, we retrieved the table of results from CLEF 2023 for task 3 with BLOOM and Googletrans. The results for BLOOM using prompt 1 remain the same for both BLEU and BERT score in CLEF 2023 and CLEF 2024. Googletrans shows superior metrics, but this advantage should be viewed with caution due to the larger number of evaluations. BLOOM_2 performs slightly better than BLOOM_1 in the few evaluations conducted, but both methods need more evaluations to provide a more reliable analysis. For a fairer and more accurate comparison, the number of evaluated examples should be equalized for all methods. With a larger amount of data, we might see a reduction in the performance gap between Googletrans and the BLOOM methods. Similarly, we can observe that there has been a slight change in the metrics performed by BLEU for Googletrans when comparing CLEF 2023 and CLEF 2024, this change should be analyzed qualitatively by comparing the translations.
Next, we will analyze some examples of the translations produced by BLOOM and Googletrans in 2023 and 2024. Los pianistas saben en qué bar se encuentran.
Los pianistas saben en qué bar se encuentran.
If we compare the translations produced by Googletrans in 2023 and 2024, we can observe that the vast majority of jokes are translated identically. In Example 1 of Table 7, the only difference is in the use of capitalization. We can infer that the differences in BLEU metrics are due to the fact that this year's test data does not include all the same jokes as last year's. OLD POLICEMEN never die they just cop out.
Los viejos policías nunca mueren, sólo se retiran.
Los viejos policías nunca mueren, sólo se retiran.
Los viejos policías nunca mueren, solo se jubilan. The boy swallowed a pillow, the hospital described his condition as comfortable.
El niño se tragó un cojín, el hospital describió su estado como cómodo.
El niño se tragó un cojín, el hospital describió su estado como cómodo.
El niño se tragó un cojín, el hospital describió su condición como cómoda.
I've never taken an elevator to the basement floor, that's just beneath me.
Nunca he cogido un ascensor hasta el sótano, eso está por debajo de mí.
Nunca he cogido un ascensor hasta el sótano, eso está por debajo de mí.
Nunca he tomado un ascensor al sótano, eso está justo debajo de mí.
Regarding the translations done by BLOOM, the outputs from 2023 and this year using prompt 1 are identical. In some cases, the joke is translated understandably, while in others, it is not. Concerning the translations with prompt 2, we observe that the target texts are more literal than those from prompt 1. While BLOOM_1 attempts to adapt the vocabulary to make it more natural, BLOOM_2 performs a highly literal translation.
The numerical results, which are quite high in terms of precision, recall, and F1, suggest that the translations of the jokes are well-executed. However, when we analyze the results from a translational and linguistic perspective, we observe that the successful translation and conveyance of a joke's meaning are more coincidental. Although jokes are a cultural matter and usually require adaptation in translation, some jokes can be understood when translated literally, which explains some of the good translations in this task. By comparing the numerical and linguistic results, we can conclude that the more literal the translation, the higher the metrics. Therefore, we can conclude that language models still require significant training and improvement to accurately translate jokes in a "conscious" manner.
Future work could involve analyzing other language models, such as GPT, as it continues to improve and might offer different results. Additionally, exploring the translation of other types of jokes and wordplay, and attempting to train models with high-quality translations of jokes could be beneficial.
I would like to thank the organizers of CLEF 2024 in general and the organizers of JOKER in particular for providing us with the opportunity to continually improve our research, learn, and develop. Above all, I would like to express my gratitude to Professor Liana Ermakova for her invaluable advice and great support throughout the execution of this task.
This project has received a government grant managed by the National Research Agency under the program "Investissements d'avenir" integrated into France 2030, with the Reference ANR-19-GURE-0001.