This paper provides a comprehensive overview of Task 3 of the CLEF 2024 JOKER track on automatic houmous analysis. The overarching objective of the JOKER track series is to facilitate collaboration among linguists, translators, and computer scientists to advance the development of automatic interpretation, generation, and translation of wordplay. Task 3 specifically concentrates on the automatic translation of puns from English into French. This overview outlines the overall structure of the shared task we organised as part of the CLEF 2024 evaluation campaign. We discuss the approaches employed by the participants and present and analyse the results they achieved. We also describe the work of participants who used our data to translate puns from English to Spanish as part of the open task of the track.
This paper describes Task 3 of the CLEF 2024 JOKER1 challenge, where the goal is to accurately translate
puns between diferent languages. This is the final task of JOKER-2024 [
A pun is a form of wordplay that exploits multiple meanings of a word or words with similar sounds
but diferent meanings. Puns pose challenges in translation, as they often rely on language-specific
nuances that may not have direct equivalents in other languages. Nonetheless, it can be important
to preserve wordplay in the target text, even if the exact type of wordplay or the specific meaning
is changed. In Task 3, the goal is to translate English punning jokes into French. The translations
should aim to preserve, to the extent possible, both the form and meaning of the original wordplay –
that is, to implement the pun→pun strategy described in Delabastita’s typology of pun translation
strategies [
In the previous editions of the JOKER track, we observed that the success rate of wordplay translation
is extremely low even in the case of LLMs, for both language pairs: English-French and English-Spanish
[
Spanish achieved 18% [
This year eleven teams submitted 23 runs for Task 3 showing stable interest of the community in the pun translation task. Note that six submissions were made by translators rather than computer scientists (teams Olga and UBO).
In the following sections, we describe the data preparation process (Section 2) and participants’
approaches (Section 3), and then present an analysis of their results (Section 4). In addition to the
traditional machine translation evaluation measures, such as BLEU [9] and BERT Score [10], we
examined the participants’ performances using the dataset we created to identify words or phrases that
have multiple meanings (pun locations) for the CLEF 2023 JOKER Task 2 [
The data is an extension of the JOKER parallel wordplay corpus [12]. The training data for Task 3 consists of 1,405 English wordplay instances, with a total of 5,838 professional human French translations. The maximum number of reference translations per English pun is 29. 72% of English puns have multiple reference translations. The histogram of the test references per English pun is shown in Figure 1.
The test files shared with participants consist of 4,501 English wordplay instances, in JSON format. Over these English puns, we used new 376 distinct source texts with 832 corresponding reference translations created by professional French native speaker translators. The maximum number of reference translations per English pun is eight. However, the majority of source texts have a single reference translation. The histogram of the test references per English pun is given in Figure 2. An example of the source data is as follows: "id_en": "en_1007", "text_en": "Save the whales, spouted Tom." The corresponding human reference translations are as follows: {
"text_fr": "\"Il faut sauver les baleines\", jeta Tom avant de se tasser." { } }, { }, { }, { }, { }, { } "text_fr":"\"Sauvez les baleines, cracha Toto, Cétacé!\"" "text_fr": "\"Il faut sauver les baleines\", interjeta Tom." "text_fr": "Moi je sauve les baleines, Tom s’en venta." "text_fr":"Louis évent-a le projet de sauvetage des baleines." "text_fr":"\"Sauvez les baleines\", proclama Tom à tout évent."
Ten teams submitted 20 oficial runs for this task. In addition, the team Olga [13] used the data from this task to explore translation from English to Spanish as part of the open task. She submitted three additional runs. Statistics on the runs are summarised in Table 1. The approaches used were as follows:
The UAms team [14] submitted two runs. MarianMT – a sequence-to-sequence (Seq2Seq) model based on the Marian framework was used. The second run was based on the T5 (t5-base) model with the same standard preprocessing as for the first run.
The Arampatzis team submitted six runs for this task employing (among others) MarianMT, Google Translate, Helsinki-NLP Opus, and mBART.
The Petra&Regina team [15] submitted a single run. The authors relied on the EasyNMT library, which they used with the Helsinki-NLP Opus-MT model.
The Tomislav&Rowan team [16] preprocessed the data and used it to build prompts to translate the jokes with the translation pre-trained model (Helsinki-NLP Opus) through the MarianMT framework. The authors judged that EasyNMT was less efective for this task. Two runs were submitted.
The Farhan [17] team provided two runs. They used single-shot prompting techniques with GPT-4 and GPT-4o.
The Frane team submitted one run. They used neural machine translation models like MarianMT. The translations were refined with a custom module to preserve the pun elements. This module used bilingual dictionaries and contextual embeddings. A similar approach was taken by the Dajana&Kathy team.
The AB&DPV team [18] used simple prompts with Llama-2-7b and reported that in a number of instances the translations were found to be incomplete or mixing two languages. They submitted a single run.
The Jokester [19] team submitted one run. They also used the MarianMT framework.
The UBO team submitted three semi-manual runs using commercial models such as DeepL, Google Translate, and ChatGPT.
The Olga team [13] submitted three runs as part of the open task. The team explored the topic of
translating humour from English to Spanish, comparing the BLOOM model with Google Translate. For
the BLOOM translations, two diferent prompts were employed. We provide a comparison of her runs
with the CLEF JOKER 2023 participants [
BLEU (BiLingual Evaluation Understudy), which measures the vocabulary overlap between the candidate translation and a reference translation [9]. We used the sacreBLEU implementation2 with the default tokeniser 13a which mimics the mteval-v13a script from Moses [20]. We report the BLEU score (harmonic mean) and the BLEU precisions for n-grams on 376 distinct English texts with corresponding 832 reference translations to French.
BERT Score from the Python bert-score package3 [10]. We report mean values of BERT score precision, recall, and F1 over all 832 references.
We make the following observations. First, the best results were obtained by participants who used the commercially produced machine translation engines (such as Google Translate and DeepL) integrated into SDL studio. The MarianMT models, which are similar to BART, showed very similar results. Second, the same models fine-tuned by diferent teams achieved diferent scores. Third, the
BLEU scores of the UBO submission are very low, while the BERT scores are very high. More analysis is needed to investigate this diference.
For a fine-grained analysis of the generated translations, we decided to evaluate the translations based on the presence of words or phrases carrying multiple meanings (pun location) from the reference texts. This approach allows us to focus on specific elements of translation quality that standard evaluation metrics might overlook. Thus, for this analysis we used the data created within CLEF 2023 JOKER Task
Preserving wordplay is often crucial to maintain the sense of the text as in the pun from Alice’s Adventures in Wonderland by Lewis Carroll, : “ ‘That’s the reason they’re called lessons,’ the Gryphon remarked: ‘because they lessen from day to day.’ ”. The oficial translation preserves wordplay by using the pair cours/courts: “C’est pour cette raison qu’on les appelle des cours: parce qu’ils deviennent chaque jour un peu plus courts.’’. While machine translation destroys the wordplay, resulting in the sentence becoming nonsensical: “C’est la raison pour laquelle on les appelle leçons, remarqua le Grifon: parce qu’elles diminuent de jour en jour.’’
The train data contained 5,838 French translations of 1,405 distinct English puns. These translations
had 4,355 distinct locations, i.e. words or phrases with multiple meanings. These locations were
manually annotated by master’s students specializing in translation who are native speakers of French.
The location annotation was not shared with participants this year. However, in 2023 we released 2,000
annotations as train data [
The overlap between the JOKER 2023 Task 2 pun location data and the test data of Task 3 this year reveals 13 distinct locations for 8 unique English puns. 400 y c n euq300 e r F 200 100 0
For each English pun, we computed a boolean value that is True if the corresponding French translation
contains at least one word from the set of locations corresponding to that English pun. We considered
only exact string matches. The participants’ results in terms of translations containing locations are
presented in Table 6. As the overlap between the JOKER 2023 Task 2 pun location data and the test data
of Task 3 of this year is small, the results are close to zero and are not entirely interpretable. However,
on the training data, we observed similar results between the non-fine-tuned MarianMT and Google
Translate. The T5 and MarianMT models fine-tuned by UAms [14] on our data show improved overlap
with the punning words in the reference translations. The fine-tuned MarianMT [ 14] includes nearly
twice as many translations containing location terms compared to T5. The fine-tuned MarianMT [ 14]
achieves a higher BLEU score than T5, whereas the BERT Score shows the opposite trend. The BLEU
score of both GPT-based runs submitted by the Farhan team [17] is notably low, as are the BERT
Scores. However, one of the GPT-based runs has a higher number of locations than Google Translate,
while another is close to the encoder-decoder model M2M100. In general, only a small percentage of
translations contain at least one word identified as carrying multiple meanings in references. Models,
ifne-tuned on our training data achieve a maximum of 23% of translations containing at least one
pun location word from reference translations. In contrast, non-fine-tuned models use pun location
words in only 11% of cases. These results closely mirror those obtained last year [
The team Olga [13] submitted three additional runs providing translations of English puns into Spanish. The translations into Spanish were generated by the commercial translation engine Google Translate and the LLM BLOOM with few-shot prompting. Both BLOOM runs were partial.
We provide the evaluation of her submissions in comparison with last year’s participants’ results on the train data as we have not constructed new references in Spanish this year. However, as the models used by Olga have not been trained on our data, last year’s train references are not problematic. For this evaluation, we used 215 puns in English with 644 corresponding references.
Tables 7 and 8 display the results obtained on the CLEF 2023 JOKER training data for the participants
of both 2023 and 2024. For the details of the previous year’s runs refer to the corresponding JOKER
overview papers [
Among the runs submitted in 2023, those using mBART, OpusMT, and Google Translate produced
the best results for Spanish according to our manual evaluation in terms of the number of successful
translations – i.e., translations preserving, to the extent possible, both the form and sense of the original
wordplay [
In this paper, we have described Task 3 of the JOKER track at CLEF 2024. The task aims to advance
the automation of wordplay translation. The EN→FR parallel JOKER corpus we used in the previous
edition [
This year, eleven teams submitted 23 runs for Task 3, demonstrating a stable interest from the community in the pun translation task. Participants mainly used LLMs, commercial machine translation engines, and out-of-the-box translation models.
We evaluated the participants’ results using automated measures, specifically BLEU and BERT scores. According to these automatic measures on the test data, the best results were achieved by commercial machine translation models, with Google Translate producing the top results. While on the training data, the fine-tuned models largely outperform Google Translate. The BLEU scores showed a lot of variation across the runs, while the BERT scores produced results that could be grouped into two distinct levels.
To conduct a fine-grained analysis of the generated translations, we evaluated them based on the
presence of words or phrases with multiple meanings (pun locations) from the reference texts. To
this end, we combined the reference translations in French with pun location annotations from the
data created for CLEF 2023 JOKER Task 2 – Pun Location and Interpretation [
We observe that the success rate of wordplay translation remains extremely low, even in the case of LLMs. However, one of the major obstacles in the development of wordplay machine translation is its evaluation. Destroying the wordplay may result in the text becoming nonsensical. The existing metrics do not take into account punning words which can reward translations with completely lost sense. In future work, we will explore new perspectives on evaluating wordplay in machine translation based on the data constructed within the JOKER track.
Additional information on the track is available on the JOKER website: https://www.joker-project. com/
This project has received a government grant managed by the National Research Agency under the program “Investissements d’avenir” integrated into France 2030, with the Reference ANR-19-GURE-0001. We thank all other colleagues and students who participated in data construction, the translation contests, and the CLEF JOKER track. [9] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: A method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002, pp. 311–318. URL: https://www.aclweb.org/anthology/P02-1040. doi:10.3115/ 1073083.1073135. [10] T. Zhang*, V. Kishore*, F. Wu*, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, in: International Conference on Learning Representations, 2020. URL: https://openreview. net/forum?id=SkeHuCVFDr. [11] L. Ermakova, T. Miller, A.-G. Bosser, V. M. Palma Preciado, G. Sidorov, A. Jatowt, Overview of JOKER 2023 Automatic Wordplay Analysis Task 2 – pun location and interpretation, in: M. Aliannejadi, G. Faggioli, N. Ferro, M. Vlachos (Eds.), Working Notes of CLEF 2023 – Conference and Labs of the Evaluation Forum, volume 3497 of CEUR Workshop Proceedings, 2023, pp. 1804–1817. [12] L. Ermakova, A.-G. Bosser, A. Jatowt, T. Miller, The JOKER Corpus: English–French parallel data for multilingual wordplay recognition, in: SIGIR ’23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, Association for Computing Machinery, New York, NY, 2023, pp. 2796–2806. doi:10.1145/3539618.3591885. [13] O. Popova, Comparative Evaluation of Humour Translation from English to Spanish: A Study with BLOOM and Googletrans, in: G. Faggioli, N. Ferro, P. Galuscakova, A. G. Seco de Herrera (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), CEUR Workshop Proceedings, CEUR-WS.org, 2024. [14] L. Buijs, M. Cazemier, E. Schuurman, J. Kamps, University of Amsterdam at the CLEF 2024 Joker Track, in: G. Faggioli, N. Ferro, P. Galuscakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [15] R. Elagina, P. Vučić, Convergential approach in machine learning for efective humour analysis and translation, in: G. Faggioli, N. Ferro, P. Galuscakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [16] R. Mann, T. Mikulandric, CLEF 2024 JOKER Tasks 1–3: Humour identification and classification, in: G. Faggioli, N. Ferro, P. Galuscakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [17] F. Dhanani, R. Abbas, Translating English puns to French, in: G. Faggioli, N. Ferro, P. Galuscakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [18] D. P. Varadi, A. Bartulović, JOKER 2024 by AB&DPV: From ‘LOL’ to ‘MDR’ using AI models to retrieve and translate puns, in: G. Faggioli, N. Ferro, P. Galuscakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [19] H. Baguian, H. N. Ashley, JOKER Track @ CLEF 2024: The Jokesters’ approaches for retrieving, classifying, and translating wordplay, in: G. Faggioli, N. Ferro, P. Galuscakova, A. García Seco de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum, 2024. [20] M. Post, A call for clarity in reporting BLEU scores, in: Proceedings of the Third Conference on Machine Translation: Research Papers, Association for Computational Linguistics, Belgium, Brussels, 2018, pp. 186–191. URL: https://www.aclweb.org/anthology/W18-6319.