Introduction

This paper presents details of Task 1 of the JOKER-2024 Track 1 , which was held as part of the 15th Conference and Labs of the Evaluation Forum (CLEF 2024) 2 . The overall objective of the JOKER track series [1,2,3], which began in 2022, is to facilitate collaboration among linguists, translators, and computer scientists to advance the development of automatic humour analysis. In each edition of the JOKER track, we construct and publish reusable, quality-controlled datasets to serve as training and test data for various humor processing tasks. In Task 1, participants build systems aiming to retrieve short humorous texts from a document collection based on a given query. For details on JOKER-2024's other two tasks, we refer the reader to their respective overviews [4,5]. Further information and insights are also presented in the Track's overview paper [1].

Search engines generally do not account for humour, ambiguity, or subversion of linguistic rules as features for selecting relevant documents to be returned. However, humour-aware retrieval, such as retrieval of wordplay-containing passages, can be useful for certain use cases or for user groups who appreciate or are interested in humorous qualities of text [6,7].

To foster research in humour-aware information retrieval, in JOKER 2024 we have introduced a novel task that consists of retrieving short humorous texts from a document collection. The intended use case is to search for a humorous text on a particular topic. Besides users who especially like humour, this could be useful for writers, for humour researchers, for second-language learners as a learning aid, for advertisement copywriters, for professional comedians as a writing aid, or even for translators who might need to adapt certain jokes to other cultures. Formally, for Task 1, the objective is to retrieve short humorous texts from a document collection based on a given query. The retrieved texts should fulfill two criteria: to be relevant to the query, and to be humorous, which in our task means to be 1 Tomislav&Rowan [16] 1 UAms [17] 8

Total 26

instances of wordplay. The intended use case is to search for a joke on a specific topic. For example, a search query of "math" would mean that the goal is to find math jokes, while the query "Tom" would mean that the goal is to find jokes about some person or entity named Tom. The test collection was built based on the English corpora constructed within the previous edition of the CLEF JOKER track:

• JOKER 2023 Task 1 -pun detection [2,8,9]; • JOKER 2023 Task 2 -pun location and interpretation [2,10,9]; • JOKER 2023 Task 3 -pun translation [2,11,9]. This year, ten teams, out of the total 22 active JOKER participants, submitted 26 runs for Task 1 out of the 103 runs submitted to the track (see run statistics in Table 1).

This paper presents an overview of our data preparation process in Section 2. In Section 3, we describe the participants' runs, and we present the analysis of their results in Section 4. We provide some concluding remarks in Section 5.

Dataset

The data for this task extends that which was originally used for JOKER-2023's tasks on wordplay detection in English [8,2,9]. Those texts were annotated according to whether they are humorous; we supplement this data with texts from Task 3 of JOKER-2023 [11,2,9], used for humour translation, and with some new wordplay instances. We further extended the data with text passages collected from non-humourous sources, as well as data that was automatically generated in relation to the queries. Specifically, the non-humorous data related to queries was obtained from the following sources:

• negative examples from the JOKER corpus.

• Wikipedia extracts returned for the queries. We used the Wikipedia Python package 3 for this and then collected sentences to form non-humorous text instances. • Descriptions of queries generated by Meta's Llama 2 with 7B parameters [18].

In total, we provided our participants with a collection consisting of 61,268 documents, where 4,492 texts are humorous. The latter encompasses 3,507 texts from JOKER 2023 and 985 new wordplay instances. The remaining 56,776 texts are non-humorous. These consist of 4,954 negative examples taken from the JOKER 2023 wordplay detection corpus, 12,523 texts generated using Llama 2, and 39,299 sentences from Wikipedia extracts. All the texts were typically one or two sentences long and were released in the form of JSON files. For creating the set of queries, we harnessed data from CLEF 2023 JOKER Task 2 -Pun Location and Interpretation [10,2,9], and in particular, the locations of wordplay in texts, i.e. words or phrases carrying multiple meanings. In CLEF 2023 JOKER Task 2, puns were either homographic (identical spelling as in I used to be a banker but I lost interest) or heterographic (i.e. exploiting paronymy as propane/prophane in When the church bought gas for their annual barbecue, proceeds went from the sacred to the propane.) To expand the queries, we used the semantic annotations of pun locations (pun interpretation), i.e. pairs of lemmatized word sets, containing the synonyms (or, if absent, hypernyms) of the two words involved in the pun, excluding any that share the same spelling as the pun. The lists of query expansions were manually checked. The document was deemed humorous and relevant to the query if it came from the positive examples of the JOKER corpus and included the query term or its expansions.

Twelve queries with their judgments (qrels) were created for training or validating participants' systems. Then, another 45 queries were created as a test set. 4 For all 57 queries (combined test and training), 11,831 documents were deemed topically relevant. We considered a document to be topically relevant to a given query if it contained the term from this query, or its synonyms, or its hyperonyms. Among the topically relevant documents, 1,730 were considered to be humorous. The descriptive statistics of relevant humourous texts is given in Table 2 while Figure 1 presents the histogram of the number of relevant humourous texts per query. The average number of relevant humorous texts per query is 30, while the median is 18 texts.

Evaluation Measures

When it comes to evaluation measures, a set of standard information retrieval metrics were used: map mean average precision -i.e., the mean of the average precision scores for each query ndcg normalised discounted cumulative gain, the gain of each document based on its relevance, discounted logarithmically by its position in the ranking normalised over the ideal ranking P1, P5, P10 precision -i.e. the ability of a system to present only relevant items, at different numbers of top ranked results R5, R10, R100, R1000 recall -measuring the ability of systems to find all or many relevant items, at different top numbers of results bpref binary preference, a sum-based metric showing how many relevant documents are ranked before irrelevant documents MRR mean reciprocal rank, the average of the multiplicative inverse of the ranks of the first correct answer of results for a sample of queries We used the pyterrier platform [19,20] implementation of these metrics. "docid": "3", "text": "The organic compound primarily responsible for the characteristic odor of musk is muscone." }, { "docid": "51135", "text": "I've inherited a fortune, said Tom, willfully" }, { "docid": "591", "text": "My name is Will, I'm a lawyer." } ]

Input format

Queries

The train and test queries are also JSON files, this time with the following fields: qid a unique query identifier from the input file query the search query Input example:

[ {"qid":"qid_train_1","query":"steps"}, {"qid":"qid_train_3","query":"math"}, {"qid":"qid_train_4","query":"Tom"} ]

Qrels

Finally, we provide training/validation data in the format of JSON qrels files with the following fields: qid a unique query identifier from the query input file docid a unique document identifier from the corpus qrel indication the document docid is relevant to the query qid and is a wordplay instance Example of a qrel file:

[ { "qid": "qid_train_0", "docid": "27260", "qrel": 0 }, { "qid": "qid_train_0", "docid": "591", "qrel": 1 }, { "qid": "qid_train_0", "docid": "51135", "qrel":1 } ]

Output format

We required results to be provided in a JSON format with the following fields: run_id run ID starting with <team_id>_<task_id>_<method_used>, e.g. UBO_task_1_TFIDF manual flag indicating if the run is manual 0,1 qid a unique identifier from the input file docid an identifier of the document retrieved from the corpus to the qid query rank retrieved document rank score normalised document relevance score (in the [0-1] scale)

For each query, the maximum allowed number of distinct documents (docid field) is 1000. A sample output file is as follows:

[ { "run_id":"team1_task_1_TFIDF", "manual":0, "qid":"qid_train_0", "docid":"27260", "rank":1, "score":0.97 }, { "run_id":"team1_task_1_TFIDF", "manual":0, "qid":"qid_train_0", "docid":"591", "rank":2, "score":0.8 }, { "run_id":"team1_task_1_TFIDF", "manual":0, "qid":"qid_train_1", "docid":"27261", "rank":1, "score":0.7 } ]

Participants' Approaches

In total, ten teams submitted 26 runs (see run statistics in Table 1). The approaches used by the participating teams are as follows:

• The jokester team [12] provided a single run based on an approach that uses TF-IDF for feature weighting and a Logistic Regression classifier. • The Arampatzis team 5 provided ten runs, testing a range of diverse models such as TF-IDF, LSTM, Random Forest, XGBoost, LightGBM (Light Gradient-Boosting Machine), SVM, Decision Tree, Gaussian Naive Bayes, KNN, and neural nets. • A run submitted by LIS team [13] was based on T5 transformer model, query processing, expanding terms with their synonyms collected from WordNet, choosing the optimal tokenisation method for queries and documents, and then selecting the best threshold for the similarity score. Finally, a pre-trained model was applied to filter texts with puns. • The Frane uses fine-tuned BERT models for estimating humorousness together with the wellknown retrieval model such as BM25. The team submitted one run. • The Dajana&Kathy team processed text using stemming, lemmatisation, and stop word removal, and employed TF-IDF and BM25, together with fine-tuning BERT for submitting their run • The AB&DPV team [14] used TF-IDF for ranking humourous text within the collection for constructing thier run. • The RubyAiYoungTeam team's run was submitted without any description of the employed method. • The Petra&Regina team [15] submitted a single run employing logistic regression with TF-IDF vectorised documents and queries and iterative relevance scoring. • The Tomislav&Rowan team [16] employed logistic regression with TF-IDF vectorised documents to create a single run. • The UAms team [17] provided two runs based on BM25 and BM25+RM3 using default settings.

Two other runs employ neural cross-encoder rerankings of the latter runs based on zero-shot application of an MSMARCO-trained ranker. The last four runs are based on two trained versions of the SimpleT5 model, one with a batch size of 6 and the other with a batch size of 8 and a trained BERT model using LoRa.

Note that we do not detail the zero-scored runs, nor the runs with problems that we could not resolve.

Results

Evaluation on Test Data

The majority of submitted runs had some issues -for example, some runs were submitted on only part of the data, and there were runs for the training data only. We tried to solve these problems whenever possible.

In Table 3 we show the main results for participants' runs on test data. We make the following observations based on the results:

• First, in general, both precision and recall are extremely low. Low precision is due to the presence of the query terms in the non-humorous texts which is considered as topical relevance by the retrieval systems. The low recall is probably related to the length of the text and the fact that in many texts, both humorous and topically relevant, the query terms do not appear. • The runs based on pseudo-relevance feedback RM3 query expansion outperform the BM25 baselines. • Cross-encoder rerankers do not exhibit better performance than the baseline models.

• Filtering trained on the wordplay detection task improved systems' results quite a lot.

• Simple solutions such as ones with TF-IDF and Logistic Regression remain quite competitive.

• Using T5 and BERT language models with RM3 is one of best approaches both in terms of precision and recall.

To evaluate the errors produced by the rankers, we compared the results with those obtained using topic relevance alone, disregarding the humorousness of the texts. Topical relevance results on the test data are given in Table 4. Traditional models without filtering, such as RM3, TF-IDF, BM25, showed high performance on topical relevance only with 𝑀 𝐴𝑃 > 0.35 and 𝑁 𝐷𝐶𝐺 > 0.55 but the official results, which take into account humorousness of the texts, go down with 𝑀 𝐴𝑃 < 0.1 and 𝑁 𝐷𝐶𝐺 < 0.3. Post-filtering applied with different ranking models improved MAP up to 50% (cf. UAms_rm3_T5_Filter2 and UAms_Anserini_rm3) according to the official results but dropped topical relevance. UAms_bm25_BERT_Filter demonstrated high scores according to both the official results and topic relevance alone.

Evaluation on Training Data

Here we also report the results on the training data in order to provide additional insights as to the performance and characteristics of different approaches. Table 5 shows the performance based on the submitted runs.

Looking at the results we can make the following observations: • While precision is quite high, recall still poses many challenges, even on training data. This may also support the hypothesis that low recall may arise due to the absence of query terms in texts which are relatively short. • The approaches using decision trees, and relatively standard approaches like SVM and kNN, achieve the best results. However, these results are from a team (Arampatzis) that have not submitted a system description paper, so they should be treated with caution. • Considering the remaining results, the ordering of the best runs is similar to that of the test data.

• Considering topical relevance alone on the training data (see Table 6), in general, we observe similar trends as seen in the test set where unfiltered runs tend to have higher topical relevance alone but a significant drop according to the official ranking. • The filtered runs exhibited identical scores in both the official ranking and for topical relevance alone, indicating that they retrieved (almost) exclusively humorous documents. However, the relative ranking between filtered and unfiltered runs differs, except for the Arampatzis runs, which are at the top of both tables. • The topical relevance scores between the test and training data are similar, but the ranking that considers both topical relevance and humor is nearly twice as low on the test data, indicating potential overfitting in humor classification (cf. the UAms runs).

Conclusions

This paper has given an overview and discussed the results of Task 1 of the JOKER-2024 challenge on the retrieval of humorous texts. Based on the data for wordplay detection and interpretation previously constructed within the CLEF JOKER track [8, 10, 2, 9], we constructed a unique reusable test collection for wordplay retrieval in English.

Ten participating teams submitted 26 runs in total for Task 1. The teams applied diverse methods, ranging from traditional approaches rankers such as TF-IDF, BM25, and RM3 to cross-encoders with and without post-filtering based on classical machine learning methods (logistic regression, and SVMs) to more modern ones, including SimpleT5 and BERT.

The participants' results confirm that humour-oriented information retrieval remains a rather challenging task with both precision and recall being extremely low. Filtering trained on the wordplay detection task significantly improved the systems' results. However, while topical relevance scores between the test and training data are similar, the ranking that considers both topical relevance and humor is nearly twice as low on the test data, suggesting potential overfitting in humor classification.

In general, our results confirm that retrieval models are humour-agnostic and humour detection is still a challenge for machine learning models and LLMs. Developing new test collections, including those for non-English languages, could help address this issue.

Additional information on the track is available on the JOKER website: https://www.joker-project. com/