The following work has the purpose of describing the participation in the JOKER 2023 track in the classification Task 1 in which it is asked to classify sentences that contain wordplay and the translation Task 3 in which this same type of sentence has to be translated into Spanish trying to maintain the wordplay and the sense in a certain way tropicalizing the sentence to Spanish language from English. Given the constant use of models such as GPT and T5, it was decided to take the direction of language models, making a slight finetuning of BLOOMZ & mT5 models with which to address the problem of translation with a simple prompt and on the other hand the use of BERT as a model proposed for the classification task. The results were mixed since in the manual review of the BLOOMZ & mT5 translated sentences not many were found to contain the Spanish pun, in the case of classification somewhat similar results were obtained given the structure of the dataset. It is believed that given their performance the models can still be optimized and improve the accuracy with which they classify, in the case of the translations perhaps another methodology could be chosen to improve the results obtained in this task, in general, the results are satisfactory for a first approach.
since in other instances of humor, this model has demonstrated a certain level of effectiveness. Therefore, this time we aim to test if it can really be used for different cases and if its use can be generalized.
An important aspect of language-based models, especially BLOOMZ & mT5 [
The use of homophones can complicate the task even more, as there may not be a similar word in the translation that matches or has the same intention, altering in a certain way the interpretation of the joke with the pun, thus changing the original meaning. It could be said that the pun is maintained, but not the meaning. With the understanding that most puns become partial translations of what the original sentence proposes, we can say that there are different phenomena present in the translation of puns whether they are homophones, homographs, or this comes from a regional homophony of the sentence.
The task of classification presents a different challenge because puns can be confused by those sentences that unintentionally exhibit similar characteristics to a pun. Understanding the type of sentences that our model needs to screen can give us a broader idea of what needs to be done. In this case, the classification of puns appears to include certain sentences that emulate puns, but in reality, they are not. This can hinder and confuse the model, as it introduces a certain amount of noise with which we must deal.
The tasks proposed by JOKER in CLEF 2023 primarily focus on the automatic analysis of puns. In
this edition, a corpus consisting not only of English and French text but also of Spanish was available.
Based on previous successful works [
The first task focused on pun detection, which can be solved as a binary classification task since it
has yes and no labels that indicate whether a text is a pun or not. Initially, it was considered to train a
multilingual model such as multilingual BERT [
As a result, two models were trained: one specifically for English and the other for a mixture of
English-Spanish, both utilizing the same BERT architecture [
Due to the fact that transformers do not require extensive preprocessing, the training process was relatively straightforward. However, during the fine-tuning phase, it was necessary to experiment with different parameters to achieve the best results in validation accuracy and loss. In the end, the best models were saved in a Keras H5 format for future reference.
For the training of the BERT-like models, the following parameters were utilized: a batch size of 8, 3 training epochs, and a learning rate of 7e-6 for the BERT model trained on English data. On the other hand, the parameters for BERT-multilingual trained on English-Spanish data were a batch size of 32, 3 training epochs, and a learning rate of 5e-5. Considering the text length from the JOKER corpus, a maximum length of 70 tokens was declared.
To develop Task 3, consisting of the translation of word plays from English to Spanish, BLOOMZ & mT5 was implemented, which is a more robust version of BLOOM, offering an improvement in task performance. This model can follow human instructions through the use of a prompt such as the examples used in the experimentation: “Translate into Spanish:”, “Translate the text into Spanish:”, “Translate into Spanish the next text:”, and “Translate into Spanish the next sentence:”. It is worth mentioning that the result obtained depends on the prompt used. In this case, four passes or runs were made on the text to achieve a better result, each pass corresponding to one of the provided prompts.
These iterations were necessary because in some passes the results were not favorable, either because the model fail to produce a translation (resulting in blank output), generating incomplete translation, or because the model produced more text than necessary (generating new text beyond the necessary translation).
The queries were conducted using the Hugging Face API, specifically the Inference API [
Although the generation of the translation was fully automated, a sample of the results were manually reviewed to identify potential errors. These errors were taken into consideration to determine which texts needed a re-runs. For instance, in the case of the pun “What do you call a doctor who treats retired soldiers? A veteran-arian.” the first two attempts did not produce any results. It was in the third run that the text was successfully translated to “¿Qué es un doctor que trata a soldados retirados? Un veterano-ario.”. Another example is the sentence “Recent survey revealed 6 out of 7 dwarf's aren't happy.” which was generated on the second pass: “Una reciente encuesta reveló que 6 de cada 7 enanos no son felices.”
Among the resources used to train and evaluate the models, Google's Colab environment was employed. This platform enables Python programming and execution, while also providing easier use of GPU. The use of GPU resources allowed for faster execution in the performance of the described tasks, by enabling multiple simultaneous computations. The server used has the following specifications: GPU NVIDIA-SMI 525.85.12, CUDA v12.0, and 25 of RAM.
As part of the resources, it was decided to use a combination of Google Colaboratory and Petals [
On the other hand, for other BERT-like models that are less demanding in the use of resources, it is
necessary to consider the data volume and the desired width of tokenization. In the experimentation
process BETO [
3. And a mixture of English-Spanish data.
As can be seen in the tables below, the probabilities of each of the tasks and a brief explanation of the cases are provided. Some of the sentences demonstrate a clear example of a pun, while others are not so straightforward in terms of their intended meaning. This variability in pun structures contributes to the challenge of accurately identifying puns. It is expected that the model may have some confusion in finding the characteristics between sentences that lack a pun’s connotation but exhibit high similarity to those that do. This type of concern seems to be overlooked by the model to some extent. It is logical to evaluate the data set to get some insights into how the model works, which is exactly what is presented in the following tables.
In this section, the results obtained for the binary classification of English and Spanish puns are presented. As the test set labels were not provided, some inferences about the wordplays are made to explain the model performance when evaluating this type of sentence.
The BERT model trained on the English corpus consisting of 5,292 wordplays, was divided into a training and validation set in a 80:20 distribution. Subsequently, the model was used to evaluate the test set provided by JOKER, which comprised 3,183 sentences. The probability of a text containing a wordplay was obtained, and the top 5 wordplays and non-wordplay were identified.
In Table 1, it can be seen that a positive result indicating a wordplay involves a specific structure, where a sentence contains references in pairs (the words indicating this event are marked in bold letters). The highest results obtained a probability of around 0.99, where three of the puns with the highest probability of being wordplay refer to a pun referring to lawyers, as an example:
The lawyer asked a loaded question about guns.
The pun talks about a loaded question, and it is easily understood that the play on the words lies in the relationship between “loaded” and “guns”. This example provides a small idea of how the model classifies such positive instances.
Pun
Regarding the results with a low probability of being puns, that indicate a non-wordplay sentence. It can be observed that, unlike the previous table, these examples are not as obvious in terms of finding the wordplay. Therefore, if even for humans it is challenging to identify this word matching, it can be even more difficult for the model to understand the nuances that these sentences have with the use of puns. The results in Table 2 show how these sentences do not contain wordplay, indicating that the model correctly classified them as non-wordplay.
Pun
''Let's take a vacation in the south of France,'' said Tom loudly.
The multilingual BERT model used to classify the Spanish test set (2,241 sentences) presents distinct curiosities compared to the problems encounter in the English dataset. This may be attributed to the fact that incorporating a mixture of datasets strengthened the model's ability to infer the type of wordplays, but more so the type of redundant wordplays that was able to correctly classify. The example below shows a sentence that was accurately classified as a wordplay:
¿Sabes cuánto pago de alquiler por la frutería? - No, ¿cuánto? - Pimientos euros.
In this instance, it is observed how the model was able to infer alquiler (rent), and connected in to the word “euros” which corresponds to the word Pimientos (bell peppers). The intended meaning is to convey Pimientos to quinientos (five hundred) in relation to a monetary amount, a very similar word to the word pimientos. This wordplay is easy to understand, unlike other types of puns that are presented in Table 3, like the sentence:
Hey Jesús. - ¿Salió cara la cena? No, salió mala.
To understand this pun, relate to religion, humans typically require prior knowledge about the subject. It can be challenging to identify, as it often involves cultural references. However, it is among the results with the highest probability of being a wordplay, which is correctly classified. As seen in the table, the confidence in the probability is lower than that of the English model, ranging between 0.80 percent in the probability that it will be a wordplay.
Pun
¿Sabes cuánto pago de alquiler por la frutería? - No, ¿cuánto? - Pimientos euros.
In the case of the examples with lower probability, it can be noted that the presence of sentence structures where the word order is altered can make it more difficult to find the wordplay. This is evident in the example provided in Table 4:
Un Señor Ruiz... que un ruiseñor.
The use of Señor (Mr.) and then the surname Ruiz may create a pun that refers to a type of bird, the ruiseñor (nightingale bird), but the model was unable to recognize the relationship between these two statements, where only the order of the pun was altered. The other four sentences do not contain a pun, this shows that most of the non-wordplay text where correctly identify as such.
Translations posed an extremely interesting case due to the fact that translating a term that makes a play of words in English may be difficult to interpret in Spanish. For a translation to fulfill the same objective as its counterpart in Spanish, it is extremely important to pay attention to certain structures. For example, the use of proper names that refer to a particular characteristic can be difficult to understand when they are translated into Spanish.
Understanding these nuances often becomes the pivotal part that modifies the behavior of what one wants to translate but that on other occasions can play in favor of the translation by making sense in the other language. Without forgetting that normally these words are quite related to the internal structure of the word in both languages.
These observations can be seen in Table 5, where different results are presented. The first sentence “If there's one person you don't want to interrupt in the middle of a sentence it's a judge.” although correctly translated to “Si hay una persona que no quieres interrumpir en medio de una oración, es un juez.” the pun is not maintained, resulting in a literal translation.
On the contrary, in the case of the sentence “I tried to learn how to drive a stick shift but couldn't locate the manual.” that was correctly translated to “Intenté aprender a conducir con una caja de cambios manual pero no pude encontrar el manual.”, maintained not only the sense, but also the funniness of the wordplay (as indicated in bold letters).
In conclusion, we can say that the approaches implemented to cope with the classification and translation of text with puns is not misguided, perhaps more work could be done to strengthen and further explore and improve the results in these tasks. Especially in the case of translation, as the aspect of translating while maintaining the original context of the language to another makes this type of task a very interesting one to delve into, and it calls for continued research to find better methods and techniques.
Moreover, the integration of more advanced language models, such as BLOOMZ & mT5, showcased the potential of the use of prompts to enhance the translation of puns, which are instructions carried out by humans but this itself influences the result obtained. Regarding the classification of wordplays, the utilization of wordplays with BERT-like showed good results but they could be improved with better fine-tuning and exploring different variations of BERT-like architectures. Overall, this study obtained good results that demonstrate the potential of transformer-based models and language models in the classification and translation of puns.