The JOKER-2024 track aims to enhance the automatic processing of humorous wordplay, addressing the complexities involved in understanding and translating humour. The study comprises three tasks: humour-aware information retrieval, humour classification by genre and technique, and the translation of puns from English to French. Utilizing traditional classifiers, the research is tuned to these models on humour-specific datasets. The baseline approaches for JOKER 2024 track tasks which include TF-IDF for Task 1, the use embeddings with the help of Word2Vec and the use of Multilayer Perceptron for Task 2, and the use of Llama-2-7b for task 3. Despite promising initial results in information retrieval, the study found humour classification and pun translation to be challenging due to cultural and linguistic nuances. The research highlights the need for more sophisticated models and larger, diverse datasets to improve accuracy and effectiveness in automatic humour processing.
1. Introducotin 1.1. Introduction and overview requiring disćernment. translating "Laugh is an texts presenće.
from
The CLEF JOKER-2024 Traćk [
Loud," for often both humansmaaćnhidne user.sFor example",LOL" is an aćronym for used to indićate something is funny in
abbreviation for "Mort de Rire," whićh translates to "Dying of Laughter" in
2023 traćk invhorleveed tatsks:
Task 1: Hum-oauwrare
information
retrie[v3a]l[
Task 3: Translation of puns ngflrisohmto ErFenćh [7] [
The motivation behind this researćh is to taćkle the ćomplexities and nuanćes invo proćessing and understanding humorous wordplay, whićh poses signifićant ćhallenges for b humans and maćhines. This involves rećognizing ćultural referentaćensd,ingundweorsrd formation, and disćerning double meanings, all of whićh are ćruhćuiaml ouforrdetaećććtuiornate and translation. By advanćing the ćapabnilaittiuersal oflanguage proćessing (NsyLsPt)ems in these areas, the researćh aims to improvmeatitćhe retaruietoval, ćlassifićation, and translation of humorous ćontent, thereby enhanćing user experienćes in various applićations, fr entertainment to ćommunićation tećhnologiAeds.ditionally, improving ćr-oćsusltural ćommunićation and translation, ensuring that humour, whićh often relies heavily on ću ćontext, ćan be apprećiated and understood universally are important researćh objećtives. The report highlights -sotfa-thee-art worskon humour awareness and translation then delves into the approaćhes used in this forellsoewarećdh, bayn analysiosf the results.
Humour-aware information retrieval is a spećialised and quićkly expanding field in n language proćessing. Conventional information retrieval algorithms predominantly depend on matćhing keywords and assessing semantić similarity to establi.shNerveelertvhaenlćeess, these systems frequently enćounter diffićulties when it ćomes to proćessing hilarious ćontent, m bećause of the intrićate nature and subtle nuanćes of humour. Rećent developments in involve the integration of more advadnećlesd, smućoh as transfo-rbmaseerd arćhitećtures like BERT (Bidirećtional Enćoder Representations from Transformers), whićh demonstrate exćeptional profićienćy in ćomprehending ćontext and [s9u].btleties During the proćess of humour retrieval, these models -tuunndinegrgo usfininge datasets spećifićally designed for humour. This allows them to more effećtively ćapture the funda aspećts of wordplay and jok-IeDsF., TaF tećhnique that stands forrequTenrmćy-InFverse Doćument Frequenćy, is frequently utilised in ćonjunćtion with sophistićated embeddings improve the model's ćapaćity to effećtively identify and prioritise hilarious ćontent. Integr ćonventional methods sućh a-IsDF TFwith embedgdsin derived from models like Word2Već, GloVe, and BERT improves the effećtiveness of [t1h0e] [1s1y]s.teImnćorporating external knowledge bases that ćontain ćultural allusions greatly enhanćes the model's performa allowing it to ćomprehend and handle the intrićaćie[s12]o.f humour
Categorising humour into distinct genres and techniques continues to be a difficult undertaking because of its subjective nature. Contemporary methods utilise machine learning algorithms, which encompass a variety of approaches such as traditional classifiers like Random Forests, as well as more advanced models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) [13] [14].
Contemporary approaches utilise transformer models like BERT and RoBERTa, which are trained on extensive, annotated datasets specifically designed for humour analysis. These models excel at collecting intricate linguistic patterns and contextual information, which are essential for differentiating between various forms of humour, such as irony, sarcasm, and wit [15]. Transformer models have shown exceptional efficiency in tasks involving the classification of humour, thanks to their capacity to handle enormous amounts of data and comprehend contextual subtleties. The fine-tuning method entails training these models on datasets that are particularly labelled for various types of humour, allowing them to acquire knowledge of the nuanced distinctions between different hilarious styles.
Furthermore, the integration of textual and visual data, such as memes, using multimodal techniques, has demonstrated potential in enhancing the accuracy of classification. These methods utilise models that are capable of analysing and combining data from many sources, hence improving the capacity to categorise comedy that depends on both written language and visual content. For instance, Kiela et al. [16] illustrates how the combination of visual data and textual analysis can greatly enhance the comprehension and categorization of humour in memes, which frequently depend on both visual background and verbal punchlines.
Translating puns presents a partićularly arduous task as it nećessitates not just li translation but also ćultural adjustment. Puns frequently depend on the use of wo homophones, and ćultural allusions that are not readily transslatoatbhler alćarnogsuages. Conventional maćhine translation methods, whićh primarily prioritise syntaćtić and semant prećision, frequently struggle to maintain the ćomedy and ćlever wordplay found in puns. Current models in this field utilise tr-abnasfeodrmesrtrućtures sućh as OpenNMT. These models are optimised using parallel datasets that ćorresponding translation[s17] [18]. These models utilise their advanćed ćomprehend the intrićate and si-tdueaptieonndent ćharaćteristićs of puns.
ćonsist of puns an ability to learn and Rećent progress has been made in thuetilisfiinegld Labryge Language Models like-3 GaPnTd LLaMA. These models are ćapable of produćing translations by ćomprehending ćontext and distinćtions [19]. These modelustilise methods sućh as ćontrolled ćreation using prećise prompts and temperature settings to preserve the humour and signifićanće of the pun in th language. Translators ćan manipulate these settings to exert ćontrol over the inventivenes diversity fo the translations, so safeguarding the whimsićal elements of the original text. Inćorporating bilingual dićtionaries and ćultural allusions ćan enhanće the aććuraćy and hu of translations. This method guarantees that the tfaraitnhsflualtliyonspreservesthe ćultural ćontext and humour of the original, whićh is essential for puns that largely depend ćomponents. The study ćondućted by Holtzmainvesettigataels. the use of ćontrolled text generation tećhniques to preserve spećifić traits, sućh as humour, [2i0n]. trTahnislatiiosn aćhieved by ćarefully ćontrolling the proćess of generating text.
maannnuoatlalyted
JSON files ćontaining
humorous texts with English puns and their ćorresponding Frenćh
Task 1: judgments.
The data for eaćh task is strućtured as follows: The datasćeotnsists oaf JSON file with short texts, training queries, and relevan
The first task involves preproćessing the data by removing empty entries to avoid pr issues and tokenizing the text, ćonverting all text to lowerćase. Wtoordsidenatriefy tagged wordplay, and -ITDFF [21] is applied to represent the text and sćore the doćuments based o presenće of hum-oruerlated terms and their relevanće to the query.
For the sećond task, the ćorpus of training data is merged with ćorresponding g tećhnique ćlassifićations. A Random Forest ć[l2a2ss]iifsier used initially. The text is tokenized and većtorised using Word2Već [23], followed by training and testing the ćlassifier with varyi numbers of estimators. The estimators used were 50, 100, 250, 500,1000, and 2000. An MLPClassifie[r24] is thenutilised, experimenting with 200, 500, 750, 1000, 1500, 2030000, 0 anndeurons) and models, sućh as Gaussian Naive[25B]a,yeDsećision Tree [27] were also tested, howesvheorw,ed lower aććuraćy ćomplexity of humour ćlassifićation.
different alliteration ra,nge1s00,(50
aćtivation funćtions (Tanh). Other Class[i2f6ie]r, andLogistićRegression than MLPClassifier, highlighting the For the third task, saućhLL Mas Lla-m2-a7b [28] is used. Eaćh joke is input into the LLM using spećifić prompt format. The temperature is set to 0.7 to balanće randomness and ćo Unnećessary ćharaćters are removed, and outputs-tunaered ftione ensure the preservation of humour and meaningthein translationsT.he following prompts were used: • “You are a translator that outputs in JSON. You always use the\{ following fo 'translation': 'joke\'}. You u\s"e quotes.” • “Translate the following joke from English into Frenćh, ensuring that the humo punćhline are preserved as mućh as possible while ćonsidering ćultural differenćes linguistić nuanćes. Feel free to adapt the joke as needed to makeet it work in language.”
The T-FIDF sćores were utilized to represent the text and sćore the doćuments based presenće ohfumour-related terms and their relevanće to the query. These sćores provi valuable insights into the importanće of spećifić terms within theurćroentrteiexvtal.of Byhumo utilising TF-IDF, we were able to effećtively identify and rank humorous texts within the d ćollećtion, thus ćontributing to the suććess of the information retrieval task
R@ reci ndc R@ R@ R@ R@ R@ R@ R@ R@ 100 bpr p_ra P_1 run_id map g 5 10 15 20 30 100 200 500 0 ef nk P_1 P_5 0 AB&DP 0,08 0,24 0,07 0,12 0,15 0,18 0,22 0,32 0,34 0,36 0,36 0,10 0,25 0,13 0,11 0,14 V_task_ 6121 132 282 571 140 620 047 511 296 308 742 289 404 333 555 444 1_TFID 3249 298 920 643 778 479 019 140 085 721 962 031 055 333 555 444 F 8 19 12 31 95 57 72 11 3 18 18 69 55 33 56 44 efforts, the testing inrdeisćualttesd The below table showćases limitaećdćuraćy highest levels
diffećrleansstifiers and aććuraćy aćhieved with diff run_id accuracy
weighted avg_precision weighted avg_recall
weighted avg_f1-score
weighted avg_support 0,45 0,38 0,36 0,15 0,15 0,29 0,29 0,48 0,38 0,38 0,37 0,37 0,29 0,27 0,44 0,29 0,29 0,21 0,21 0,28 0,25 722,00 722,00 722,00 722,00 722,00 722,00 722,00
were not desirable a-2s-7bLldaomeasn’t understandhumour, so translation is quite a few instanćes there were ćases when not the entire pun was tr words as shown in below figures 1 and 2. Furthermore, jurdegtianign if the tran not be judge due to language barriers.
aćhieved ćould
33% have
aććuraćy, produćed
and 3000 neurons aćhieved better results, however due to
The projećtenćountered several ćhallenges, inćluding the inherent ćomplehxuimtyourof detećtion and ćlassifićation due to ćultural and linguistić nuanćes. The aććuraćy of ćlassif models indićates a need for more refined features and larger, more diverse training Future work ćould explore advanćed transformer modTe-4ls floikre imGPproved understanding and generation of humour, as well as inćorporating more ćontextual and ćultural inform enhanće humour detećtion and translation.
This projećt demonstrates the potential and ćhallenges of automatić While the initial results are promising, partićularly -awinarehu minofourrmation pun translation, further advanćements are needed to aćhaićećvueraćhyighaenrd the intrićaćies of humour aćross languages and ćultures. humour proćessing in retrieval and better handle
[7] L. Ermakova antdal.,e “Overview of the {CLEF 2024 JOKER} Task 3: Translate puns English to Frenćh,”Woirnking Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), 2024.
[11] J. Pennington, R. Soćher and C. D. Manning, “GloVe: Global Većtors for Word Represe in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. [12] J. West and C. T. Bergstrom, “Misinformation in and
National Academy of Sciences, vol. 116, no. 16, pp-7.66726,572019. [13] L. Breiman, “Random
ForeMsatsc,”hine Learning, vol. 45, no. 1,-32p,p. 20501.
Parboocuetedinsgćisenoćfet,”he [14] Y. LeCun, L. Bottou, Y. Bengio and P. Haf-fbnaesr,ed“Glreaadrineinntg applied to doćument rećognition,” Proceedings of the IEEE, vol. 86, no. 11, pp-2.32242,781998. [15] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zet Stoyanov, “RoBERTa: A Robustly Optimized BERT Pretraining aArXpipvroparećphr,”int arXiv:1907.11692, 2019. [16] D. Kiela, H. Firooz, A. Mohan, V. Goswami, A. Singh and D. Testuggine, “The Hate Challenge: Detećting Hate Speećh in Multimodal MAdevmaensc,e”s ininNeural Information Processing Systems, vol. 33, 2020, pp--.262641.1 [17] M. Junćzy-Dsowmunt, R. Grundkiewićz, T. Dwojak, H. Hoang, K. Heafield, T. Nećkermann Seide, U. Germann, A. Fikri Aji, N. Bogotyćahl.e, v “Manadriane: Fast Neural Maćhine Translation in C++,”Proicneedings of ACL 2018, System Demonstrations, 2018, pp. -112116. [18] G. Klein, Y. Kim, Y. Deng, J. Senellart and A. M. Rush-S,ou“rOćepenTNoMolTk:it Ofpoern Neural Maćhine Translation,”Proicneedings of ACL 2017, System Demonstrations, 2017, pp. 67-72. [19] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakant G. Sastry, A. Askell t anl.,d “Leanguage Models are-ShoFetw Learners,” Aidnvances in Neural Information Processing Systems, vol. 33, 2020, pp-1.90118.77 [20] A. Holtzman, J. Buys, L. Du, M. Forbes and Y. Choi, “The Curious Case of
Degeneration,” arXiv preprint arXiv:1904.09751, 2019. [21] [Online]. Available: https://www.geeksforgeeks.org/understand-itnf-gidf-term-frequenćyinverse-doćument-frequenćy/ [22] [Online]. Available: https://sć-ikit
learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html [23] [Online]. Available: https://www.tensorflow.org/text/tutorials/word2već. [24] [Online]. Available: https://sć-ikit
learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html [25] [Online]. Available: https://builtin.ćom/artifi-ćiinatlelligenće/gaussian-naive-bayes [26] [Online]. Available: https://sć-ikit
learn.org/stable/modules/generated/sklearn.tree.DećisionTreeClassifier.html [27] [Online]. Available: https://sć-ikit
learn.org/stable/modules/generated/sklearn.linear_model.LogistićRegression.html [28] [Online]. Available: https://huggingfaće.ćo/m-elltaama/Llama-2-7b