The big challenge in NLP for local Indian languages, mainly Tamil and Malayalam, is to develop a strong sarcasm detection system. The linguistic characteristics of these Dravidian languages are quite diferent, and there are very few annotated datasets. This makes sarcasm detection challenging because it relies on hints that aren't always obvious and depends on a person's understanding of the situation or topic. Hence, a model capable of capturing those aspects is required. To address the above problem, a deep learning-based model is proposed where Convolution layers take advantage of automatic feature extraction, max-pooling for reducing dimensions, and bidirectional GRUs to capture long-range dependencies and contextual information in sentences. The model performs very well in non-sarcastic and sarcastic detection, scoring an F1 of 0.94 for non-sarcastic detection 0.71 for sarcastic detection in Malayalam, and 0.94 for non-sarcastic and 0.81 for sarcastic detection in Tamil. This indicates it was able to handle the complexities of sarcasm detection in these low-resource languages successfully. Our rank for sarcasm detection in the Tamil language was 10, followed by rank 12 in the Malayalam language for the same.
Rajnish Pandey and Jyoti Prakash Singh [
Aggarwal, and Akshita [
Rosid, and Mochamad Alfan [
Bhaumik, Anik Basu, and Das, Mithun [
Maity, Krishanu et al. [
Tejasvi, Koti, et al. [
Shah, Aditya, and Maurya, Chandresh Kumar [
Ratnavel, Rajalakshmi et al. [
Bansal, Srijan, et al. [11], showcased how code-switching patterns are very impactful on the performance of NLP applications such as humor, sarcasm, and hate speech detection. They explained how these patterns can significantly improve model performance by identifying and utilizing them.
N. Sripriya et al. [12], provided an in-depth exploration of the Dravidian Language, particularly Malayalam and Tamil. The shared Task focuses on developing a dataset of code-mixed texts in Malayalam, English, and Tamil. On the other hand, Chakravarthi, and Bharathi Raja [13], investigated the problem of identifying hope speech detection which utters positive as well as supportive sentiment.
Chakravarthi, Bharathi Raja et al. [14] showed the typology, i.e., identifying hate speeches of the category Homophobia and Transphobia in code-mix. This posed an increased number of ofensive languages towards the LGBTQIA+ community, and fewer methods focusing on its detection inspired us to create approaches — solutions to enhance social life among these communities. In future work, we target to optimize the loss functions and construct a multilingual Homophobia and Transphobia detection system for numerous languages. In this paper, we provide a novel rationale for the detection of sarcasm speech.
Chakravarthi, Bharathi Raja, et al [15] introduced their work to extract humor in the Dravidian language, mainly Tamil and Malayalam text The paper exon preprocessing techniques incorporating pre-trained models, in particular, BERT and distilBERT and conventional method SVM, TF-IDF.
Chakravarthi, et al [16] highlighted the linguistic complexities of these mixed-language texts and discussed methodologies through their work in their shared task at FIRE 2024 conference to improve sarcasm detection for better natural language understanding in Dravidian languages.
The dataset from the DravidianCodeMix@FIRE-2024 shared task has been utilized for sarcasm detection, consisting of Tamil-English and Malayalam-English posts and comments. This dataset is code-mixed, meaning it includes a blend of words or phrases from both Tamil and English or Malayalam and English, within a single sentence or text. For prior context, Vijay, Deepanshu, et al. [17], introduced a dataset that is specifically created in Hindi-English code-mixed social media posts to detect irony. In this paper, they presented an annotated corpus of Hindi-English code-mixed text which consists of tweet IDs and the corresponding annotations. They have also presented a supervised system that is used for detecting irony in code-mixed text.
Suhaimin, Mohd Suhairi Md, et al. [18], focuses on issues within the public safety section in the dataset. Constructing this type of dataset that includes English and Malay code-mixing elements.
Swami, Sahil, et al. [19] created a collection for both English and Hindi to develop code-mixed tweets specifically for sarcasm detection. The paper highlights the diferent aspects of social media, which have numerous challenges in detecting sarcasm from a code-mixed language and the informal way of writing style of the public.
Chakravarthi, Bharathi Raja, et al. [20] showed an important benchmark dataset for sarcasm detection in code mixing in the Dravidian language.
Table 1 and 2 provided a summary of the key features of the datasets, presenting an in-depth look at the statistics for each of the three sets (Training, Development, and Test) in both languages. This breakdown ofers a clear understanding of how the data is distributed and the size of the datasets, giving insight into the structure of the code-mixed texts used for this task.
TAM(Train) TAM(Dev) TAM(Test) Total no. of sentences
Total no. of words
Max. length of sentences Non-sarcastic to sarcastic ratio
Our data was processed and annotated at the sentence level, with each sentence labeled as either sarcastic or non-sarcastic. To support model development and testing, the datasets for both Tamil and Malayalam were divided into three distinct sets: the training set, the development set, and the test set.
This all-inclusive methodology allows for a strong approach in the building, training, and evaluation of a model capable of detecting sarcasm in both Malayalam and Tamil code-mixed and code-switched languages, using a combination of Convolutional and Recurrent Neural Networks and advanced preprocessing and evaluation techniques.
Text pre-processing is critical before feeding the text data to the model in order to make sure that the model can learn from the inputs at hand. The text is tokenized, and it thus transforms into sequences of integers, with each integer indicating one word from Malayalam or Tamil languages. Such sequences have been padded to equal uniform input sizes. Such uniformity is required by the model for consistent processing of data. This preprocess will make the model strong at handling variable lengths of sentences, as all input data would be in the expected format, thus helping detect sarcasm and allow generalization using multilingual text data.
The design is tailor-made for the specific nature of multilingual text data, including an embedding layer that transforms integer sequences into dense vectors and a Conv1D that captures local patterns in the data. For capturing both sides of dependencies that ultimately help in context understanding and nuance detection in complicated sarcasm, there is a bidirectional GRU layer. These representations are further refined by additional LSTM layers, and Dense layers having ReLU activation and Dropout have been added to prevent overfitting; the final Dense layer uses Sigmoid activation for binary classification. From Figure 1, it is observed that the process starts by loading the Malayalam and Tamil text data mixed with both languages. Next, the text is broken down into numbers (tokenized) and padded to make sure all input has the same length. The numbers are turned into dense vectors in building the model, followed by layers that detect patterns and connections. Extra layers are added to prevent overfitting, and the final layer helps classify the text into two categories. After this, the models are compared, and a report is created to measure how well they perform using metrics like precision and F1 scores.
An analysis of the model shows that quite a lot is known about its performance for both Tamil and Malayalam. From the confusion matrix, it can be seen that the Malayalam model is successful in correctly classifying 2236 non-sarcastic instances and 323 sarcastic ones but fails to detect 69 nonsarcastic instances, misclassifying them as sarcastic, and 198 sarcastic instances as non-sarcastic, as shown in Figure 2.
On the other hand, the Tamil model correctly classified 4528 non-sarcastic and 1225 sarcastic instances, while labeling 102 non-sarcastic examples as sarcastic and missing 481 sarcastic examples, as indicated in Figure 3. These results are further depicted in the confusion matrices, where it can be visually observed how well the model performed in each category.
Comparing the results, one can easily see that the results from both models are high in terms of accuracy for detecting non-sarcastic content, with an F1 of 0.94. It is quite evident that the Tamil model shows better detection of sarcastic material than the Malayalam model, with an F1 score of 0.81 as compared to 0.71. This shows that although both models perform similarly in correctly classifying non-sarcasm, the Tamil model tends to perform better in detecting sarcasm. The diference between the sarcasm detection systems calls for further development areas, especially the model of Malayalam because it will still need more improvement to increase its performance.
The sarcasm detection model shows a very good performance for both Tamil and Malayalam code-mixed texts, where both the languages showed a very good performance achieving a strong F1 score of 0.94 for detecting non-sarcastic content. However, there is a clear distinction in sarcasm detection, as the Tamil model outperforms the Malayalam model, with an F1 score of 0.81 compared to 0.71. While the model handles non-sarcastic classification eficiently, the gap in sarcastic content detection suggests room for improvement, particularly in the Malayalam model.
Looking ahead, future work could focus on developing a more robust fusion model that combines the strengths of Transformers, RNNs, and GRUs. Such a hybrid approach may better capture the nuances of sarcasm in code-mixed languages and improve overall detection accuracy. Incorporating these advanced architectures could address the current limitations, enhancing performance for both Tamil and Malayalam sarcasm detection tasks.
The author(s) have not employed any Generative AI tools. [11] S. Bansal, V. Garimella, A. Suhane, J. Patro, A. Mukherjee, Code-switching patterns can be an efective route to improve performance of downstream nlp applications: A case study of humour, sarcasm and hate speech detection, arXiv preprint arXiv:2005.02295 (2020). [12] N. Sripriya, T. Durairaj, K. Nandhini, B. Bharathi, K. K. Ponnusamy, C. Rajkumar, P. K. Kumaresan, R. Ponnusamy, C. Subalalitha, B. R. Chakravarthi, Findings of shared task on sarcasm identification in code-mixed dravidian languages, FIRE 2023 16 (2023) 22. [13] B. R. Chakravarthi, Hope speech detection in youtube comments, Social Network Analysis and
Mining 12 (2022) 75. [14] B. R. Chakravarthi, A. Hande, R. Ponnusamy, P. K. Kumaresan, R. Priyadharshini, How can we detect homophobia and transphobia? experiments in a multilingual code-mixed setting for social media governance, International Journal of Information Management Data Insights 2 (2022) 100119. [15] B. R. Chakravarthi, N. Sripriya, B. Bharathi, K. Nandhini, S. Chinnaudayar Navaneethakrishnan, T. Durairaj, R. Ponnusamy, P. K. Kumaresan, K. K. Ponnusamy, C. Rajkumar, Overview of the shared task on sarcasm identification of Dravidian languages (Malayalam and Tamil) in DravidianCodeMix, in: Forum of Information Retrieval and Evaluation FIRE - 2023, 2023. [16] B. R. Chakravarthi, S. N, B. B, N. K, T. Durairaj, R. Ponnusamy, P. K. Kumaresan, K. K. Ponnusamy, C. Rajkumar, Overview of sarcasm identification of dravidian languages in dravidiancodemix@fire2024, in: Forum of Information Retrieval and Evaluation FIRE - 2024, DAIICT , Gandhinagar, 2024. [17] D. Vijay, A. Bohra, V. Singh, S. S. Akhtar, M. Shrivastava, A dataset for detecting irony in hindi-english code-mixed social media text., EMSASW@ ESWC 2111 (2018) 38–46. [18] M. S. M. Suhaimin, M. H. A. Hijazi, E. G. Moung, Annotated dataset for sentiment analysis and sarcasm detection: Bilingual code-mixed english-malay social media data in the public security domain, Data in Brief 55 (2024) 110663. [19] S. Swami, A. Khandelwal, V. Singh, S. S. Akhtar, M. Shrivastava, A corpus of english-hindi code-mixed tweets for sarcasm detection, arXiv preprint arXiv:1805.11869 (2018). [20] B. R. Chakravarthi, N. Sripriya, B. Bharathi, K. Nandhini, S. C. Navaneethakrishnan, T. Durairaj, R. Ponnusamy, P. K. Kumaresan, K. K. Ponnusamy, C. Rajkumar, Overview of the shared task on sarcasm identification of dravidian languages (malayalam and tamil) in dravidiancodemix, in: Forum of Information Retrieval and Evaluation FIRE-2023, 2023.