The problem of accurately translating wordplay in automatic humour analysis remains challenging, as highlighted by the CLEF 2024 JOKER Track. This problem is particularly interesting because humour is deeply cultural and context-dependent, making it dificult for language models to handle. Our study compares last year's and this year's results to determine if there have been improvements in the automatic translation of wordplay using BLOOM and Googletrans. The findings indicate that while numerical metrics show high precision, recall, and F1 scores, the actual translation and conveyance of jokes' meanings still rely heavily on coincidence, underscoring the need for further training and enhancement of language models.
In this section we will give a brief description of the data provided to perform task 3, the methods used to solve it and the evaluation methods.
The task will provide an updated test set of English punning jokes, for French language the training data is also available. However, considering that the training data for Spanish is not needed for this experiment, we only use the test data. The test data is provided in JSON format.
test data (jokes) in English entries 5727
As we can see, we were provided with 5,727 jokes in English to translate. The data itself looked as follows: text_en
Each entry in the table contains an identifying number for each joke, and a one- or two-sentence joke that includes a pun.
As mentioned in the introduction, to carry out task 3, we used BLOOM with two diferent prompts and Google Translate. 2.2.1. BLOOM
• Prompt 1:
BigScience Large Open-science Open-access Multilingual Language Model (BLOOM) is a
transformerbased language model [
The prompts used are provided below.
"Original: Diabetics should not be allowed to have sweet dreams.\n\ Translation: Los diabéticos no deberían tener dulces sueños.\n\ \n\ Original: I’m going to the guillotine at dawn and my wife has already collected my severance pay.\n\ Translation: Al amanecer me van a pasar por la guillotina y mi mujer ya ha firmado la separación.\n\ \n\ Original: After 5 years with the same chiropractor, I moved and had to change doctors. It was quite an adjustment.\n\ Translation: Me mudé y tuve que buscar otros médicos después de estar cinco años con el mismo quiropráctico. Fue un mero ajuste.\n\ \n\ Original: A scientist doing a large experiment with liquid chemicals was trying to solve a problem when he fell in and became part of the solution.\n\ Translation: Un científico que hacía un gran experimento con productos químicos líquidos estaba intentando solucionar un problema cuando cayó en que él se convertiría en parte de la solución.\n\ \n\ Original: Old electricians never die, they just keep plugging away.\n\
Translation:" • Prompt 2: "Request: Translate from English to Spanish: Diabetics should not be allowed to have sweet dreams.\n\ Answer: Los diabéticos no deberían permitirse soñar dulce (expresión figurativa para decir que no deberían permitirse deseos excesivos o indulgencias).\n\ \n\ Request: I’m going to the guillotine at dawn and my wife has already collected my severance pay.\n\ Answer: Me voy a la guillotina al amanecer y mi esposa ya ha recogido mi indemnización ( expresión figurativa para decir que me estoy preparando para un evento desafortunado y mi esposa ya ha preparado los asuntos financieros para el futuro sin mí).\n\ \n\ Request: After 5 years with the same chiropractor, I moved and had to change doctors. It was quite an adjustment.\n\
Answer:"
It is curious that when we created the second prompt, BLOOM itself, along with the Spanish translation in brackets, provided an explanation of the wordplay: • ’expresión figurativa para decir que no deberían permitirse deseos excesivos o indulgencias’ = ’figurative expression to say that excessive desires or indulgences should not be allowed’ • ’expresión figurativa para decir que me estoy preparando para un evento desafortunado y mi esposa ya ha preparado los asuntos financieros para el futuro sin mí’ = ’figurative expression to say that I am preparing for an unfortunate event and my wife has already prepared the financial afairs for the future without me’
Thus, it appears that BLOOM has attempted to apply a sort of translator’s explanatory note.
2.2.2. Googletrans
Googletrans is a free and unlimited python library that implements Google Translate API [
The task organizers used two methods for evaluating translations: BLEU and BERT Score [
BLEU (BiLingual Evaluation Understudy) measures the lexical similarity between a candidate
translation and a reference translation [
BERT Score, obtained from the python bert-score package10 [
We divided the results section into two parts: numerical results and linguistic results. In the first subsection, we examine the evaluation metrics provided by the organizers, while in the second, we discuss some examples of translations produced using each method.
Above all, we retrieved the table of results from CLEF 2023 for task 3 with BLOOM and Googletrans.
The results for BLOOM using prompt 1 remain the same for both BLEU and BERT score in CLEF 2023 and CLEF 2024. Googletrans shows superior metrics, but this advantage should be viewed with caution due to the larger number of evaluations. BLOOM_2 performs slightly better than BLOOM_1 in the few evaluations conducted, but both methods need more evaluations to provide a more reliable analysis. For a fairer and more accurate comparison, the number of evaluated examples should be equalized for all methods. With a larger amount of data, we might see a reduction in the performance gap between Googletrans and the BLOOM methods. Similarly, we can observe that there has been a slight change in the metrics performed by BLEU for Googletrans when comparing CLEF 2023 and CLEF 2024, this change should be analyzed qualitatively by comparing the translations.
Next, we will analyze some examples of the translations produced by BLOOM and Googletrans in 2023 and 2024.
If we compare the translations produced by Googletrans in 2023 and 2024, we can observe that the vast majority of jokes are translated identically. In Example 1 of Table 7, the only diference is in the use of capitalization. We can infer that the diferences in BLEU metrics are due to the fact that this year’s test data does not include all the same jokes as last year’s.
Regarding the translations done by BLOOM, the outputs from 2023 and this year using prompt 1 are identical. In some cases, the joke is translated understandably, while in others, it is not. Concerning the translations with prompt 2, we observe that the target texts are more literal than those from prompt 1. While BLOOM_1 attempts to adapt the vocabulary to make it more natural, BLOOM_2 performs a highly literal translation.
The numerical results, which are quite high in terms of precision, recall, and F1, suggest that the translations of the jokes are well-executed. However, when we analyze the results from a translational and linguistic perspective, we observe that the successful translation and conveyance of a joke’s meaning are more coincidental. Although jokes are a cultural matter and usually require adaptation in translation, some jokes can be understood when translated literally, which explains some of the good translations in this task. By comparing the numerical and linguistic results, we can conclude that the more literal the translation, the higher the metrics. Therefore, we can conclude that language models still require significant training and improvement to accurately translate jokes in a "conscious" manner.
Future work could involve analyzing other language models, such as GPT, as it continues to improve and might ofer diferent results. Additionally, exploring the translation of other types of jokes and wordplay, and attempting to train models with high-quality translations of jokes could be beneficial.
I would like to thank the organizers of CLEF 2024 in general and the organizers of JOKER in particular for providing us with the opportunity to continually improve our research, learn, and develop. Above all, I would like to express my gratitude to Professor Liana Ermakova for her invaluable advice and great support throughout the execution of this task.
This project has received a government grant managed by the National Research Agency under the program “Investissements d’avenir” integrated into France 2030, with the Reference ANR-19-GURE-0001.