We explore the use of data augmentation techniques and their impact on pun detection in English, leveraging the oficial JOKER Task 1 - English Pun Detection dataset. The paper reports on various data augmentation strategies such as synonym replacement, back-translation, paraphrasing, shortening, and extension, and their respective impact on pun detection performance. An important aspect of the study is ensuring that the augmentation techniques retain the humorous efect present in the original text. The results from the experiments suggest however that the augmentation techniques did not substantially enhance pun detection capabilities.
Humor is a crucial aspect of human communication that lends nuance to interactions and enhances our understanding of complex situations. One of the primary forms of humor in English is punning or wordplay, characterized by the use of words in such a way that multiple meanings or efects occur. The ability to detect puns in text is an interesting albeit challenging task in Natural Language Processing (NLP).
This paper presents our empirical study of the impact of various data augmentation techniques
on the performance of pun detection, based on the oficial JOKER Task 1 - English Pun Detection
dataset [
We present a detailed discussion on the various data augmentation techniques used, as well as the measures taken to ensure the quality of augmented data. Despite the multiple techniques employed, our study suggests that the data augmentation methods we apply have not significantly enhanced pun detection capabilities. However, this paper ofers some insights into pun detection and the potential influence of data augmentation techniques, providing a foundation for future research in this area.
The primary source of data for this research was the oficial JOKER Task 1 - English Pun
Detection dataset [
The pun detection task was performed using a classifier model built on the transformer architecture. The choice for the underlying pre-trained model was the BERT model, specifically the ’bert-base-uncased’ variant. This model was selected for its high efectiveness in handling a wide range of NLP tasks, including text classification.
The BERT model was fine-tuned using the Adam optimizer [ 3] with a learning rate of 1e-5 and a weight decay of 0.01. Our training procedure used a batch size of 32 and was conducted over two epochs.
To measure the performance of the model, the CrossEntropyLoss function was used.
To further improve the ability to successfully classify sentences containing puns, without tuning the classifier, data augmentation could be applied.
Although applying diferent data augmentation techniques to humorous data containing wordplay can be challenging without losing the humorous efect, we decided to take this path and perform experiments in order to understand to what extent diferent data augmentation technologies would be applicable for wordplay detection.
In the following, we describe a few simple techniques that were implemented and experimented with.
Data augmentation by substitution involves replacing parts of the original data to create slightly diferent versions, yet keeping the underlying meaning the same. One popular substitution technique is synonym replacement, where certain words are replaced by their synonyms. For example the sentence "The quick brown fox jumps over the fence" could become "The fast brown fox jumps over the fence", by replacing "quick" with one of its synonyms "fast".
Synonym replacement can be implemented in a variety of ways. Two popular approaches are Wordnet-based [4] and Word2vec-based [5] synonym augmentation. To implement synonym replacement, this project uses the TextAugment [6] library, which provides easy-to-use methods for data augmentation.
TextAugment’s Wordnet-based augmentation algorithm chooses ’r’ words from a sentence and replaces them with one of their synonyms ’s’, where ’r’ and ’s’ are randomly selected. [6] Note that the synonyms selected by Wordnet are context insensitive. This might lead to less accurate replacements if a word strongly depends on its context.
Word2vec-based augmentation uses a diferent approach by utilizing a word embedding model. Words that are close to each other in the model’s vector space are likely to have a similar meaning, but must not strictly be synonyms. In contrast to Wordnet-based augmentation, it can handle words based on their context, making it possible to suggest more suitable synonyms in some situations.
As for the embedding model, we are using the pre-trained Google News Word2vec model, which was trained upon roughly 100 billion words from diferent Google News articles. However, we encountered an incompatibility issue between the TextAugment library and the latest gensim version (v4.3.1). Starting with gensim version 4.0.0, the ’wv’ attribute was removed from the ’KeyedVectors’ class. To resolve this breaking change, we had to slightly modify the TextAugment library as a workaround.
Another family of data augmentation techniques is augmentation by paraphrasing. In this case, instead of only substituting certain parts of the text, the entire text is modified by rewording or restructuring it (by paraphrasing). By paraphrasing, the example from before: "The quick brown fox jumps over the fence" could become "The fence is jumped over by the quick brown fox".
Back-translation is a type of paraphrasing, in which text is translated from its original language to some other language and then back-translated to its original language. Often, this results in a reworded or restructured sentence while retaining its original meaning.
For the translation of sentences, we used the MarianMTModel [7], which was trained on a large dataset of texts. First, each sentence is translated from English to German using the pre-trained model ’Helsinki-NLP/opus-mt-en-de’, and then back-translated from German to English using the ’Helsinki-NLP/opus-mt-de-en’ model. After, the same was done for English and French using the ’Helsinki-NLP/opus-mt-en-fr’ and ’Helsinki-NLP/opus-mt-fr-en’ models. This results in 2 new augmented samples for every input sentence.
PEGASUS [8] stands for Pre-training with Extracted Gap-sentences for Abstractive Summarization and is used for tasks like text summarization. However, it is also capable of rephrasing text while retaining its original meaning. This makes PEGASUS not only suitable for summarization but also for data augmentation.
For our experiments, we use the ’tuner007/pegasus_paraphrase’ model, a variant of PEGASUS, which was fine-tuned specifically for the task of paraphrasing.
In this approach, a text or sentence is reduced in length. One way to achieve this is by removing unnecessary details, i.e., the least important parts of the text. However, removing a word from a sentence, in our case, a sentence that might contain pun wordplay, without altering its meaning or humorous efect, turns out to be quite challenging. Every word in a sentence containing wordplay might contribute to its meaning.
To implement augmentation by shortening we applied the Rapid Automatic Keyword Extraction (RAKE) algorithm[9]. RAKE selects the key words and phrases of a sentence by analyzing the frequency of word appearance and its co-occurrence with other words in the text. By removing some of the words that were not identified as key words, a new shortened sentence can be formed.
Augmentation via extension is another widely used technique. Since there are only new words added, but none removed, it could be considered one of the safer techniques in regards to not altering the text’s meaning.
In our implementation, we use GPT-2 [10] to generate a new additional sentence based on the original text. This sentence is then randomly appended either at the beginning or the end of the original text.
When applying any of those techniques to humorous texts containing wordplay, special care must be taken to not lose the humorous efect. In the experiments, it was inevitable that there were samples generated in which the humorous efect was lost. Therefore, after augmentation, we filter out / reject some generated samples that do not meet certain criteria.
The following filters were used: • Duplicate Filter: If a generated text sample replicates the original word for word, it is discarded. Having duplicate data in the dataset can lead to overfitting. Thus, it is important to prevent augmentation techniques from generating duplicate samples. • Last Words Filter: Most of the time the punning word is one of the last two words in the sample [11]. By requiring every augmented sample to end with the same two words, it’s possible to filter out generated samples that are likely to have lost their punning word in the process of augmentation. • Similarity Filter: Filtering out generated samples that are semantically "too far" from the original sample can also ensure a higher quality of the entire augmented dataset. Here it is done by using Sentence-BERT [12] (SBERT), more specifically using the ’allMiniLM-L6-v2’ [13] model, which is able to generate sentence embeddings that capture semantic information. These embeddings are then used to determine the similarity between the original and augmented samples. The similarity is calculated using the cosine distance. If this similarity measure falls below a certain threshold, then the generated sample will not be included in the augmented set.
In addition to the work we submitted to JOKER, which involved making predictions on the test set, we conducted more experiments using only the training set, which we further divided into testing and training subsets via the application of K-Fold Cross Validation with k = 4.
Table 2 provides evaluations of diferent datasets generated using various DA techniques. The technique "None" implies an unaltered dataset and can be used as a standard for other techniques to be compared against. Note that due to the randomness in machine learning processes, we noticed fluctuations of around 1 percent in the results. Therefore, it is challenging to determine if any of those techniques provide a significant advantage for data augmentation in detecting wordplay.
According to the evaluation table, back-translation exhibits a notable increase in size, because every sample is once Back-translated from German and French. Here, adding filters helps to significantly reduce the data size, but it seems to have almost no efect on performance.
Synonym replacement via WordNet and Word2Vec yields similar performance. The use of WordNet with filters seems to produce the highest accuracy of 74%.
Pegasus paraphrase technique’s unfiltered values are missing, because it generated so many samples (including bad quality ones) which were too many to train the classifier on our setup. But after applying filters, and reducing the size this technique also does not show an increased performance.
The application of filters in the extension technique results in the highest recall value (92.61%), contributing to the best F1 score in the table (80.07%), but with only a minor improvement in accuracy.
Overall, synonym replacement (WordNet) with filters seems to be the most balanced technique in terms of performance and accuracy. Yet, if recall and F1 score are of higher importance, extension with filters could be a preferable choice.
In conclusion, we found that the accuracy of detecting puns on the JOKER Task 1 test data was much lower compared to our experiments where we divided the training data into separate training and testing groups. One explanation could be that training and test data have diferent origins. Our attempts to enhance the performance via data augmentation approaches specific to humorous text and pun wordplay did not yield substantial improvements in pun detection capabilities. It is worth mentioning that larger language models, such as GPT-4 might have a superior accuracy on pun detection. Therefore, it may be an area worth exploring for further enhancement of pun detection techniques. [3] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, 2017. arXiv:1412.6980. [4] G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, K. J. Miller, Introduction to wordnet: An on-line lexical database, International journal of lexicography 3 (1990) 235–244. [5] T. Mikolov, K. Chen, G. Corrado, J. Dean, Eficient estimation of word representations in vector space, arXiv preprint arXiv:1301.3781 (2013). [6] V. Marivate, T. Sefara, Improving short text classification through global augmentation methods, in: International Cross-Domain Conference for Machine Learning and Knowledge Extraction, Springer, 2020, pp. 385–399. [7] M. Junczys-Dowmunt, R. Grundkiewicz, T. Dwojak, H. Hoang, K. Heafield, T. Neckermann, F. Seide, U. Germann, A. F. Aji, N. Bogoychev, A. F. T. Martins, A. Birch, Marian: Fast neural machine translation in C++, in: Proceedings of ACL 2018, System Demonstrations, Association for Computational Linguistics, Melbourne, Australia, 2018, pp. 116–121. URL: https://aclanthology.org/P18-4020. doi:10.18653/v1/P18-4020. [8] J. Zhang, Y. Zhao, M. Saleh, P. Liu, PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization, in: H. D. III, A. Singh (Eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, PMLR, 2020, pp. 11328–11339. URL: https://proceedings.mlr.press/v119/ zhang20ae.html. [9] S. Rose, D. Engel, N. Cramer, W. Cowley, Automatic keyword extraction from individual documents, Text mining: applications and theory (2010) 1–20. [10] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9. [11] L. Ermakova, A.-G. Bosser, A. Jatowt, T. Miller, The joker corpus: English–french parallel data for multilingual wordplay recognition, in: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR, ACM Press, 2023. [12] N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, 2019. arXiv:1908.10084. [13] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, M. Zhou, Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020. arXiv:2002.10957.