Humor processing remains a challenging problem for NLP due to linguistic ambiguity, language-specific nuances, and intricate wordplay. The CLEF JOKER 2025 lab tackles this with two tasks we participated in: humour-aware information retrieval in Portuguese and English (Task 1), and pun translation from English to French (Task 2). For Task 1 we developed a hybrid pipeline combining BM25, dense retrieval with multilingual-e5-small, and a cross-encoder reranker, achieving MAP 0.050 and NDCG@100 0.172 in English, and MAP 0.074 and NDCG@100 0.184 in Portuguese. For Task 2 we fine-tuned Lucie-7B-Instruct and CroissantLLMChat-v0.1 using supervised ifne-tuning (SFT) and Adaptive Rejection Preference Optimization (ARPO), obtaining a best BLEU of 42.40 (Lucie + SFT) and demonstrating a modest overlap trade-of (41.32 BLEU) when integrating ARPO, while CroissantLLM variants scored 35.17 and 35.28 BLEU. Our experiments show the baseline IR setup underperforms compared to more advanced systems, while the LLMs-based pun translation achieves best results conrfiming the promise of their cross-lingual wordplay transfer.
While humour plays an essential role in human interaction, it remains a complex challenge for advanced natural-language-processing (NLP) systems—even for the latest large language models (LLMs). Cultural diferences, implicit meanings, intricate wordplay, and the inherently subjective nature of humour all blur the clear indicators models rely on, making computational detection, translation and transfer of humour far from trivial.
The JOKER Lab at CLEF 2025 [
• A systematic evaluation of hybrid retrieval with reranking for humour-aware IR in English and
• An LLM-based pun translation pipeline that contrasts Supervised Fine-Tuning (SFT) with Adaptive
We follow the standard pipeline of (i) dense retrieval, (ii) lexical retrieval, and (iii) cross-encoder
reranking.
2.1.1. Data
The oficial English and Portuguese corpora and query sets provided by the organizers [
You a r e an a s s i s t a n t t h a t c l a s s i f i e s s h o r t d o c u m e n t s a s w o r d p l a y ( j o k e s ) o r no t , and − i f i t i s w o r d p l a y − g e n e r a t e s a c o n c i s e s e a r c h − s t y l e q u e r y t h a t would r e t r i e v e t h i s j o k e .
F o r e x a m p l e : − T e x t : "Why d i d t h e s c a r e c r o w win an award ? B e c a u s e he was o u t s t a n d i n g i n h i s f i e l d . " i s _ w o r d p l a y : T r u e g e n e r a t e d _ q u e r y : " s c a r e c r o w award " − T e x t : " The m i t o c h o n d r i a i s t h e p o w e r h o u s e o f t h e c e l l . " i s _ w o r d p l a y : F a l s e g e n e r a t e d _ q u e r y : " "
We combined the original (query, reference) pairs with these synthetic pairs to form our positive training set. Next, we performed hard-negative mining using a pretrained SentenceTransformer (all-MiniLM-L12-v2) with the following configuration: • Score range: retrieve candidates ranked 8–100 by cosine similarity • Maximum similarity: 0.8 (to avoid too-easy negatives). • Relative margin: 0.05 (filter out near-duplicates).
• Negatives per positive: 5, sampled at random.
2.1.2. Models
Next we fine -tune intfloat/multilingual-e5-small [
Humour often hinges on very subtle semantic shifts (puns, wordplays) where positives and hard
negatives lie close in embedding space; MNRL is given in Equation (1), it’s forces the model to pull
true humour examples away from all negatives, while Adaptive Margin loss introduces a temperature
and bias (Equation (2)), this dynamic penalty preserves learning signal for near-tie cases, helping
the retriever tease apart genuinely funny hits from near misses. Preliminary experiments on all-nli
dataset [
ℒMultipleNegativeRanking = 1 ∑︁[︁log∑︁ exp(︀ ⟨, ⟩︀) − =1
=1 ⏟ negative si⏞milarity ℒAdaptiveMargin = 1 ∑︁[︁log∑︁ exp(︀ ⟨, ⟩ + )︀ − =1
=1 where = ′ , ∈ R. ⟨ , ⟩ posi⏟tive similarity ⏞
]︁. ︀( ⟨, ⟩ + )︀ ]︁, (1) (2)
Our translation system follows a two-stage strategy: supervised fine -tuning (SFT) and ARPO preference
optimization.
2.2.1. Data
We merge the JOKER Task 2 corpus [
E n g l i s h : { s o u r c e } F r e n c h : { t a r g e t }
2.2.2. Models
We experiment with croissantllm/CroissantLLMChat-v0.1 [
2.2.3. Supervised Fine-Tuning
The supervised fine-tuning (SFT) script uses a completions-only data collator that masks out prompt
tokens and computes loss solely over the generated responses, which—given our instruction-style
data—forces the model to focus on learning to produce high-quality completions rather than memorizing
the prompts. Fine-tuning is done with the trl library [
ℒARPO = − E(,,)∼
To enhance humour retention, we apply ARPO2 [
too far from its original distribution, and an adaptive preference term that rejects low-quality candidates.
[︁
log (︀ log (|) − (, ) log (|))︀ + log (|)]︁. (3) Here, the first term inside the expectation is the preference loss (with a temperature ), and the second term is the BC regularization.
The adaptive penalty weight (, ) ∈ [
(, ) = ⃒⃒⃒ |1| log (|) − |1| log (|)⃒⃒⃒ . (5)
bit version of X-ALMA model 3. We used the library’s default (specified by relax_coefficient_2) and the scale of (, ), (specified by relax_coefficient_1). The default values are = 0.4 and = 0.9.
A família (do latim: familia) é um agrupamento humano formado por duas ou mais pessoas com ligações biológicas, ancestrais, legais ou afetivas que, geralmente, vivem ou viveram na mesma casa.
Vision is the ability to think about or plan the future with imagination and wisdom.
edge over its ARPO-enhanced counterparts: the SFT baseline scored 42.40 BLEU, compared to 41.32 for the Lucie-7B-Instruct-v1.1 + ARPO variant. This suggests that, although ARPO’s adaptive preference loss can improve qualitative aspects—such as preserving humour or other nuanced translation properties—it may do so at the cost of n-gram overlap as measured by BLEU.
Table 4 provides a concrete example (source en_83) where ARPO better preserves the pun: the SFT-only translation “poule en vitesse” is a literal but awkward translation, whereas the ARPO output “poule pressée” more naturally mirrors the play on “pullet” and “pressé.” In practical terms, if maximizing standard BLEU is the primary objective, the pure SFT approach remains the stronger choice. However, if downstream qualities that BLEU cannot fully capture—such as humour preservation—are important, integrating ARPO may still be worthwhile despite the slight BLEU trade-of.
We updated the SFT configuration by merging the JOKER and X-ALMA translation pairs and compared two training regimes:
of translation examples.
First, we split the JOKER parallel dataset into training and validation sets (train size = 0.97, validation size = 0.03) and mixed the SFT training split with the X-ALMA parallel dataset. A grid search was performed over a range of hyperparameters (see Appendix A).
For ARPO, negative samples were generated for the JOKER Task 2 dataset using the X-ALMA model 3 with the same train/validation split. We again compared two setups:
dataset4.
The ARPO hyperparameter grid is also detailed in Appendix A. Finally, we selected the best ARPO models in terms of COMET-22 for each dataset combination and each SFT model (see Appendix B).
As shown in Table 5, the SFT-only models achieve higher BLEU scores, consistent with the findings in Section 3. The ARPO-enhanced variants, however, fail to produce any significant gains on this metric. Notably, the configurations in Appendix B show that optimal performance required diferent values: = 0.4 for the JOKER-only dataset versus = 1.0 for the combined dataset. This suggests that smaller,
less diverse datasets benefit from weaker penalties that preserve more translation variants, while larger, more diverse datasets require stronger adaptive penalties to efectively filter translation quality while maintaining optimization stability.
In this paper, we presented two baseline systems for the CLEF 2025 JOKER Lab: a hybrid retrieval pipeline combining BM25, dense retrieval with multilingual-e5-small, and cross-encoder reranking for Task 1; and an LLM-based pun translation framework that combines SFT with ARPO preference optimization for Task 2. Our retrieval system achieved mid-tier performance (MAP 0.050/0.074, NDCG@100 0.172/0.184), revealing that purely lexical or semantic matches often miss true wordplay. In our translation experiments, SFT maximizes BLEU (42.40) but tends to produce overly literal translations, whereas
By using the activity taxonomy in https:// ceur-ws.org/ genai-tax.html:
• Formatting assistance: ensuring adherence to the formatting guidelines required by journals or institutions. • Peer review simulation: simulating peer review by providing feedback on the strengths and weaknesses of the manuscript.
• Coherence enhancement: improving the overall clarity and logical flow of the text. After using this tool/service, the author(s) reviewed and edited the generated content as needed and take(s) full responsibility for the publication’s content. SFT Hyperparameter Grid ARPO Hyperparameter Grid Parameter Learning rate
Values
× 10− 7 sigmoid 5 1 8
SFT configurations: dataset, learning rate, and epochs. v1 JOKER + X-ALMA preference