Humor is a cognitively complex and culturally specific linguistic phenomenon, posing significant challenges for computational modeling. It often transcends formal grammar, exploits phonetic and semantic ambiguities, and varies widely across languages and cultures. The JokerCLEF 2025 challenge addresses this problem through three distinct sub-tasks: Humour-Aware Information Retrieval, Wordplay Translation, and Onomastic Wordplay Translation. This paper presents the system developed by Team Cryptix, which participated in all three tasks. We employed a combination of state-of-the-art models-SBERT, MarianMT, and T5-chosen for their capabilities in semantic representation, multilingual processing, and text generation. Our approach includes task-specific ifne-tuning, targeted feature engineering, and the integration of human-in-the-loop evaluations to refine output quality. We also describe the datasets, preprocessing steps, and evaluation strategies used for each sub-task. Empirical results show that our system demonstrates consistent performance in both retrieval and translation of humorous content, maintaining the intent, nuance, and cultural relevance of the original text. Our findings underscore the importance of combining semantic-aware models with culturally informed design when handling humor in multilingual settings.
Humor is a complex and culturally dependent aspect of human language, making it particularly
challenging for computational systems to process. The CLEF 2025 Joker [
The exploration of deep learning techniques for multilingual natural language processing (NLP)
tasks has gained significant momentum, particularly in areas such as intent classification, hate speech
detection, and ofensive language identification across diverse languages [
In the context of data augmentation, techniques such as back translation and paraphrasing have
been employed to enhance model performance in hate speech detection, illustrating the utility of
synthetic data generation in multilingual settings [
CycleGAN for data augmentation in road scene analysis underscores the potential of deep generative
models to address domain-specific data scarcity issues [
Specifically related to multilingual translation, hybrid deep learning models incorporating Statistical
Machine Translation (SMT) and neural architectures have been proposed for multilingual machine
translation, with implementations focusing on Asian languages [
While humor detection remains an emerging area within AI, recent frameworks aim to improve
contextual understanding of humorous expressions through advanced modeling techniques, including
pseudo-labeling and post-smoothing strategies [
In the realm of multilingual humor detection and translation, the integration of deep learning
models—particularly transformer-based architectures—can be instrumental in capturing the subtleties
of humor across languages. The use of translation-based data augmentation, as demonstrated in hate
speech detection, could be adapted to generate multilingual humor datasets, thereby addressing data
scarcity and enabling more robust humor detection systems [
Overall, the convergence of deep learning, transfer learning, and generative models ofers promising
pathways for advancing multilingual humor detection and translation. These approaches can help
overcome linguistic and cultural barriers, enabling AI systems to better interpret and generate humor
across diverse languages and contexts [
Our approach overcomes these hurdles by leveraging cutting-edge models and techniques: SBERT for its semantic embeddings in IR, MarianMT for parallel language humor preservation, and T5 for context-aware, creative name transformations. These models were carefully selected and fine-tuned using domain-specific corpora and humor-annotated data, resulting in significantly improved retention of humor, even in low-resource language settings.
The JOKER Corpus [
This task used a JSON-formatted dataset comprising 231 queries and 352 English documents, annotated with query_id, query_text, doc_id, doc_text, humor_label, relevance_score. Documents carried a binary humor label and a graded relevance score from 0 to 3. The challenge lay in detecting nuanced humor—ranging from overt jokes to subtle sarcasm—while distinguishing humorous documents from non-humorous but contextually relevant ones. This required deeper semantic matching beyond simple lexical overlaps.
The goal was to translate humor, particularly wordplay, between English and French. The JSON dataset consisted of 709 annotated sentence pairs, with metadata including source_text, target_text, language_pair, humor_type, pun_span, humor_score. Each entry was richly labeled, identifying the type of humor and the exact span of the pun. The main dificulty stemmed from the lack of one-toone cultural and linguistic equivalents between languages, pushing the system to produce culturally sensitive and creatively translated outputs.
This dataset focused on humor embedded in names and included 3,049 examples in JSON format with fields such as original_name, source_context, target_language, translated_name, pun_type, phonetic_mapping, cultural_equivalence. Targeting English-to-French translations, this dataset captured jokes like “Justin Time” becoming “Jean Juste,” requiring preservation of humor, phonetics, and cultural nuance. Additional annotations like NER tags and reuse labels were used to aid modeling.
To streamline processing, all datasets were standardized for downstream tasks. Preprocessing included custom IOB tagging and tokenizer adaptation (e.g., for T5 and MarianMT) to maintain structural and linguistic fidelity. Quality assurance was ensured via inter-annotator agreement checks and heuristicbased validations, enabling smooth integration into PyTorch training workflows across all tasks.
A modular system architecture was developed to meet the linguistic and cognitive challenges of humoraware tasks across the three subtasks. Each pipeline was specifically structured to align with the input-output requirements and complexity of its corresponding task. The model workflows incorporated tailored training strategies, optimization techniques, and novel components for efective humor understanding and translation. The entire process is illustrated in Figure 1.
For the Humour-aware Information Retrieval subtask, documents and queries were semantically encoded
using the pre-trained multilingual model distiluse-base-multilingual-cased-v2 from Sentence-BERT
[
For the Wordplay Translation task, the MarianMT model [
The Onomastic Wordplay Translation task utilized the T5-base model[
We evaluated each task’s model performance using both standard NLP metrics and task-specific human evaluation scores. The results afirm our models’ ability to capture and retain humor-related elements across retrieval and translation tasks. Below is a detailed breakdown:
To assess the performance of our humor-aware information retrieval system, we utilized Precision and Mean Average Precision (MAP) as the main evaluation metrics. The Sentence-BERT (SBERT) model achieved a MAP score of 0.1507, demonstrating its efectiveness in capturing nuanced semantic relationships in humor-centric queries. Compared to conventional models like TF-IDF and BM25, SBERT consistently outperformed them in identifying and ranking humor-relevant documents. In addition to the quantitative analysis, we carried out a human evaluation where participants reviewed the top 10 retrieved results. Feedback indicated that SBERT’s outputs were perceived to be around 30percentage more humorous and contextually suitable than those from BM25. This reinforces SBERT’s capability to align closely with human interpretations of humor. Furthermore, as illustrated in Figure 2, the distribution of similarity scores reveals a concentration near zero, suggesting that only a limited number of document-query pairs exhibit high relevance. This underscores the inherent challenge in retrieving humor-aligned content in Task 1.
The SBERT model used in Task 1 efectively captured semantic relationships relevant to humor, achieving a MAP score of 0.1507. It outperformed traditional retrieval approaches by producing more contextually relevant and amusing results, as supported by precision scores at multiple cutofs. Explanation of Task 1 Metrics: • MAP (Mean Average Precision): Measures how well the top-ranked documents match the relevant ones across all queries. A score of 0.1507 indicates moderately good ranking quality. • GM-MAP (Geometric Mean MAP): Penalizes poor-performing queries more severely. The low score (5.52%) reflects dificulty in some humor-based queries. • R-Precision: Precision at the number of relevant documents for a query. At 19.44%, it shows the accuracy over full recall. • Reciprocal Rank: Measures how early the first relevant document appears. A high score of 56.94% suggests most queries had a relevant result early. • P@k (Precision@k): The proportion of relevant documents in the top-k results. For example,
P@5 = 28.70% means about 1.4 relevant docs appear in top 5. • NDCG@5 (Normalized Discounted Cumulative Gain): Evaluates ranked retrieval by considering the position of relevant documents. A score of 33.46% indicates a decent alignment with relevance ordering.
on the English test set for Task 1 by the JOKER CLEF 2025 organizers. The system retrieved 207,000 documents across 207 queries, with a total of 5,995 relevant documents. It achieved a Mean Average Precision (MAP) of 0.1507 and a reciprocal rank of 0.5693. Precision at diferent cutofs and the NDCG@5 score are summarized in Table 2.
evaluated on the deduplicated English test set for Task 1. The system retrieved 207,000 documents with 1,914 relevant documents retrieved. Performance was consistent with the full test set, achieving a MAP of 15.07%, R-Precision of 19.44%, and a reciprocal rank of 56.94%. Detailed evaluation metrics are shown in Table 3.
For Task 2, we evaluated the performance of our wordplay translation system using BLEU score, Humor Retention Rate (HRR), and qualitative feedback from human annotators. The MarianMT model achieved a BLEU score of 39.37, reflecting strong fluency in translation. However, since humor is context-sensitive and culturally nuanced, BLEU alone was insuficient. HRR, validated using bilingual annotators on a sample of 500 translations, provided a more reliable measure of humor preservation. Despite its strong performance, MarianMT occasionally faltered on nested puns and culturally embedded jokes. These issues were addressed through IOB tagging and refinements in attention mechanisms. As shown in Figure 3, most translated English sentences had word counts concentrated between 10 and 20, indicating manageable sentence complexity. A summary of model performance metrics is provided in Table 4 Explanation of Task 2 BLEU Metrics: • BLEU Score: Evaluates n-gram overlap between system and reference translations. A score of 39.50 reflects strong fluency. • BLEU Precision (1–4): Measures the percentage of 1- to 4-word sequences that match. Higher values mean better phrase-level alignment. • Brevity Penalty (BP): Rewards length closeness to reference. A score of 1 indicates perfect length match. • Length Ratio: Ratio of system output length to reference. A value of 1.007 confirms the translations were neither too short nor too long.
Task 2 Results (Test Set) For Task 2, our system Cryptix was evaluated on the oficial test set released by the JOKER CLEF 2025 organizers. The model achieved a BLEU score of 39.50, indicating high fluency and alignment with reference translations. BLEU precision scores across n-grams showed consistent retention of meaning, with perfect brevity penalty (BP = 1), suggesting a strong length match between system and reference outputs. Detailed BLEU components are summarized in Table 5. Task 2 Results (Test Set – BERTScore) In addition to BLEU evaluation, our submission Cryptix was assessed using BERTScore to better capture the semantic similarity of the translations. The model achieved a precision of 87.34%, recall of 86.95%, and an F1 score of 87.12%, reflecting a high degree of semantic alignment between the generated and reference texts. These results are summarized in Table 6. Explanation of Task 2 BERTScore Metrics: • Precision: Measures how much of the generated content semantically overlaps with the reference.
87.34% indicates strong alignment. • Recall: Captures how much of the reference meaning is present in the output. 86.95% shows broad coverage. • F1 Score: Harmonic mean of precision and recall. An 87.12% F1 indicates well-balanced semantic accuracy.
Task 2 Results (Test Set – Updated with Pun Location Evaluation) Following the final evaluation release by the JOKER CLEF 2025 organizers, our system Cryptix was assessed on an updated test set containing 1,682 instances. A new metric — pun location accuracy — was introduced to measure whether the translated text retained or reflected the position of the pun from the source. Our model successfully aligned puns in 113 cases, achieving a pun location match rate of 6.72%. Additionally, slight improvements were observed in previously reported BLEU and BERTScore values due to corrections in the reference set. Table 7 summarizes the updated results. Explanation of Updated Task 2 Metrics: • Pun Location Accuracy: Percentage of translations where the pun appeared in the same relative position as the source sentence. A score of 6.72% shows that this aspect remains a challenging area for machine translation systems. • BLEU / BERTScore (Updated): Slight improvements were recorded due to the removal of some faulty references. These metrics now better reflect actual translation quality.
For Task 3, we assessed the efectiveness of our onomastic wordplay translation system using BERTScore
and Human Pun Recognition (HPR). The T5 model [
Task 3 involved translating names containing puns, requiring creative language generation. The T5 model produced culturally meaningful outputs with a BERTScore of 0.1419 and was positively rated for pun retention by 70% of human judges. Task 3 Results (Test Set ) The final evaluation for Task 3 included both automatic and manual assessments across 204 English source instances containing onomastic wordplay. Our system Cryptix_task_3_flanT5 achieved an exact match score of 14.49% after normalization. Additionally, 38.15% of the outputs included a direct copy of the English source, and 13.43% of the translations were judged as humor-preserving by human annotators. These results reflect the dificulty of cultural pun adaptation and are summarized in Table 9.
The Joker Clef 2025 tasks each posed distinct challenges related to humor comprehension, cultural nuance, and linguistic inventiveness. To address these, we employed SBERT, MarianMT, and T5 models, ifne-tuned specifically for their respective objectives. This approach allowed us to develop systems that not only achieved strong quantitative performance but also preserved the humor and contextual intent of the inputs. The results demonstrate the capability of modern transformer-based architectures to manage culturally sensitive and semantically complex NLP tasks. Moreover, their ability to generalize across varied inputs makes them robust baselines for advancing humor-aware language systems.
During the preparation of this work, the authors utilized generative AI tools, namely ChatGPT and Grammarly, exclusively for grammar correction and language enhancement. All scientific content, analyses, interpretations, and conclusions were independently developed by the authors, who assume full responsibility for the originality and integrity of this publication, in accordance with the CEUR-WS policy on the use of generative AI technologies.