The exponential growth of social media platforms like Facebook 1 , x (formally Twitter) 2 , and YouTube 3 has provided a global stage for individuals from diverse cultures and social backgrounds to communicate and share their opinions on a myriad of topics. While these platforms uphold the principle of freedom of speech, some users also negatively exploit this freedom and abuse other users on the basis of gender, religion or race. This surge in harmful content has underscored the need for increased research in natural language processing (NLP) to effectively detect instances of hate speech. The consequences faced by victims of targeted hate speech are not limited to physical harm; they also experience a profound sense of dread and rejection within their communities. Recognizing the urgency to create online spaces free of racism and hate speech, researchers emphasize the importance of early detection mechanisms to mitigate the pervasive harm caused by such content [1]. This challenge extends beyond the English language, as millions of users worldwide employ diverse languages as vehicles for spreading f.maqbool@campus.unimib.it (F. Maqbool); blerina.spahiu@unimib.it (B. Spahiu); andrea.maurino@unimib.it (A. Maurino) hate. Despite extensive research in English, there exists a noticeable dearth of datasets and studies on languages like Urdu. Urdu, spoken by over 170 million people globally, faces unique challenges in written communication, with an alphabet that cannot be easily mapped onto an English keyboard. The Urdu language comprises 40 characters, yet English keyboards, accommodating only 26 letters, face limitations in fully supporting Urdu script. Attempting to map the Urdu alphabet directly onto an English keyboard proves impractical due to these constraints. Consequently, the predominant approach among Urdu speakers involves the use of Roman Urdu particularly on social media platforms, Roman Urdu is a transliteration version of Urdu using English letters. The use of Roman Urdu has sharply expanded as a result of social media's rising adoption [2,3]. Users of these tools regularly utilize these platforms to share their opinions about a variety of products, services, politics, and other items. However, despite its widespread use, Roman Urdu (RU) encounters challenges such as a lack of linguistic resources, annotated datasets, and dedicated language models [4]. To address these limitations and enhance model performance, researchers have explored data augmentation techniques [3]. In a notable application, simple data augmentation techniques were employed on a low-resourced language dataset with a limited number of samples. The results demonstrated a significant improvement in model performance, underscoring the potential of augmentation in mitigating the challenges posed by scarce linguistic resources. Additionally, experiments aimed to identify the optimal percentage of augmented data to be integrated with the original dataset, aiming to boost model performance while minimizing training time. This multifaceted approach contributes to ongoing efforts to combat hate speech across various languages, fostering inclusivity and positive discourse in online spaces. In this paper we make the following contributions: (i) enrich RUHSOLD dataset [3] and create a new RUHSOLD++ dataset; (ii) provide a new custom function in Python to dynamically alter the spelling of selected words within a sentence; and (iii) provide an empirical analysis of the different data augmentation methods. The paper is structured as follows: Section 2 discusses approaches to detect hate speech in the Roman Urdu language. In Section 3 we provide the methodology to augment our initial dataset. Section 4 provides the analysis and findings by applying different methods for data augmentation while conclusions and future work end the paper in Section 5.
The issue of abusive speech has been a longstanding focus within the research community. In earlier investigations, attempts to identify abusive users primarily relied on lexical and syntactic features extracted from their posts [5]. However, recent advancements in automated hate speech detection have witnessed substantial growth. The availability of large datasets has prompted a shift in academic research towards more data-intensive and sophisticated models, notably leveraging deep learning techniques [6] and graph embedding methods [7]. Notably, transformer-based language models like BERT [8] have gained immense popularity in various downstream tasks, proving to be particularly effective in surpassing traditional deep learning models such as CNN-GRU [9], LSTM [10], etc., for the detection of abusive language [11], [12]. This evolution highlights the dynamic nature of research in addressing the complexities of abusive speech detection. Detecting hate and abusive speech in low-resourced languages presents a formidable challenge, as exemplified in the context of Roman Urdu. The scarcity of linguistic resources for Roman Urdu spurred the creation of the RUHSOLD dataset by H. Rizwan et al. [3]. Comprising 10,012 tweets, this dataset stands out for its dual approach, offering both coarse-grained and fine-grained labeling of hate speech instances. In their research, Rizwan and colleagues not only curated this valuable dataset but also proposed a deep learningbased architecture specifically tailored for hate speech detection in Roman Urdu. Addressing the broader landscape of multilingual abusive speech, M. Das et al. [13] conducted an indepth investigation into the performance of multilingual models across eight distinct Indic languages. In a noteworthy application, they employed m-BERT [8] and MuRIL [14] models on the RUHSOLD dataset [3] to gauge their efficacy in detecting abusive speech. Through a series of meticulously designed experiments, encompassing various settings, Das and team explored the nuances of multilingual hate speech detection. Their findings underscored the effectiveness of model transfers, revealing that transferring knowledge from one language to another enhances the overall performance of the models. This body of research not only contributes to the evolving field of hate speech detection but also illuminates the specific challenges associated with low-resourced languages like Roman Urdu. By providing a robust dataset and proposing dedicated architectures, these studies lay essential groundwork for future endeavors aimed at combating hate speech across diverse linguistic landscapes. The insights gained from these investigations, especially regarding the transferability of models, offer valuable guidance for the development of more inclusive and effective hate speech detection systems in multilingual contexts. In a meticulous analysis, M. M. Khan. et al. [15] delved into the complexities of hate speech detection in Roman Urdu, manually examining over 90,000 tweets to curate a substantial corpus of 5,000 Roman Urdu tweets. Their significant contribution extended beyond dataset creation, as they systematically employed five supervised learning approaches, including a sophisticated deep learning technique, to rigorously evaluate and compare their effectiveness in hate speech detection. The results of their comprehensive study revealed that, across two levels of categorization, logistic regression outperformed all other techniques, opening up a viable path for robust of hate speech detection in Roman Urdu. Recognizing the challenges posed by the low resources of Roman Urdu, Azam et al. [16] undertook a proactive exploration of data augmentation strategies. Leveraging both easy data augmentation and transformer-based augmentation approaches, they aimed to enhance hate speech detection capabilities in Roman Urdu. The researchers conducted experiments using existing datasets in Roman Urdu and baseline models to meticulously assess the impact of augmentation techniques. Their findings unequivocally demonstrated that the performance of hate speech detection models could indeed be significantly improved by the strategic application of augmentation techniques to the dataset. This research not only contributes to the optimization of hate speech detection in low-resourced languages like Roman Urdu but also highlights the potential of augmentation strategies as valuable tools in mitigating the impact of resource constraints, providing valuable insights for the ongoing evolution of hate speech detection methodologies.
We employed the RUHSOLD dataset, a comprehensive collection of tweets in Roman Urdu created by H. Rizwan et al. [3]. The authors meticulously established a gold standard for two distinct sub-tasks. Our focus centered on the first sub-task, which involves binary labels categorizing content as either Hate-Offensive (labeled as 0) or Normal (labeled as 1), representing inoffensive language. The dataset comprises a total of 10,000 tweets, thoughtfully partitioned into training, testing, and validation sets in a ratio of 70%, 20%, and 10%, respectively.
In our quest to boost our dataset's size and improve the overall performance, we strategically employed Noise-based Data Augmentation techniques on training data. We dove into a detailed exploration, trying out different percentages for augmenting the dataset to strike the right balance. Moreover, we played around with the augmentation process by adjusting the percentage of words in each tweet that underwent these augmentation techniques. This nuanced approach was not just about expanding the dataset; it was about delicately adjusting the augmentation impact and finding the sweet spot between quantity and quality to strengthen our model's resilience. Through methodical experimentation, we aimed to uncover the most effective configurations that could genuinely enhance the overall performance of our model.
Random swapping is an effective technique in the realm of noise-based data augmentation. This technique is based on randomly swapping the words or tokens from a tweet. For example: "hum kisi se km nhi" becomes "km kisi se hum nhi".
This operation adds variances to the dataset without changing the overall sentiment or context of the text. In order to help the model generalize and function well on a variety of linguistic patterns, it is intended to be exposed to various word configurations.The percentage of word augmentation in random swapping directly controls the degree of variability injected into the dataset. Therefore, it is essential to find the optimal percentage of words which should be swapped during augmentation.
Another noise-based data augmentation is Random Deletion. This strategy involves the deliberate and random removal of words or tokens from a given sentence, introducing an element of unpredictability and variability. We designate a specific percentage of words within the sentence for potential removal, aiming to strike a careful balance between introducing noise for robustness and preserving the coherence of the text. By implementing this intentional randomness, we infuse the dataset with a dynamic quality, fortifying the model's adaptability to diverse linguistic nuances. This method serves as a potent instrument, enriching our model's adaptability and efficacy across a wide range of textual inputs.
Spelling Augmentation adds a layer of complexity by altering the spelling of words within a sentence. This process entails replacing one or more characters in a word with randomly chosen alternatives and deliberately introducing a controlled amount of noise into the data. The purpose here is to diversify the linguistic patterns in the dataset, enhancing the model's ability to handle variations in spelling and promoting resilience against potential inconsistencies in user-generated content. This meticulous introduction of noise through character substitution serves as a strategic maneuver, refining our model's capacity to adapt to a wide array of spelling idiosyncrasies. For example: "chal ja tujhy maaf kia" becomes "chal aa tujha maaf kia".
After employing the data augmentation techniques, we created a new RUHSOLD++ Dataset 4which can be accessed publicly to promote future work. The dataset consists of 3 types of data with swap, delete and spelling augmentation applied. The dataset contains the augmented data in train and validation sets distributed uniformly and unaltered test data.
The m-BERT model has garnered significant attention in the realm of abusive speech research. Its efficacy is attributed to being pre-trained on a comprehensive dataset, comprising the extensive content of Wikipedia5 , employing a masked language modeling (MLM) [17] objective across 104 languages. This pre-training involves 12 fully connected transformer encoder layers, incorporating a self-attention mechanism to efficiently process contextual information. It is worth noting that m-BERT, while powerful, has a token limit of 512, necessitating the use of a fine-tuned variant introduced by Das, M. et al. [7]. Das and colleagues enhanced the original m-BERT by incorporating a fully connected layer, aligning its output with the CLS (classification) token in the input. This added layer introduces a level of specificity, with the output reflecting the model's interpretation of the input sentence, often represented by the CLS token output. This nuanced modification allows the model to capture and interpret complex contextual nuances within the given token limit, contributing to its efficacy in understanding and classifying abusive speech patterns.
In this section, we describe the experimentation conducted on the RUHSOLD [3] dataset using fined tuned m-Bert model, a refinement proposed by M. Das et al. [7]. In our implementation, we use the PyTorch library 6 in Python, configuring each model to run for 10 epochs with an Adam optimizer and a batch size of 16.
The experimentation extended to the exploration of data augmentation techniques, aiming to strike an optimal balance between computational efficiency and accuracy. The primary focus was on determining the most suitable percentage of the dataset for applying augmentation. To achieve this, we selected a portion of the original training dataset that underwent Random Swap Augmentation. Leveraging the NLPAug7 library in Python, a random percentage of 30% for words was chosen, indicating that 30% of the total words in a tweet would undergo swapping with each other. Following the generation of augmented data, it was seamlessly integrated with the original dataset and subsequently shuffled to mitigate overfitting concerns. The outcomes of this experiment are succinctly presented in Table 1, showcasing the impact of Random Swap at varying percentages of the dataset. This experimentation highlights the strategic choices made in augmenting the data to achieve an optimal trade-off between efficiency and accuracy. We employed the Macro F1 score (mF1-score) as a performance metric, along with other evaluation measures. The Macro F1 score allows to assess the performance of each class individually while giving equal weight to all classes. Examining the results, we observe a consistent improvement in both accuracy and mF1-score as the model is trained on augmented data. However, a notable finding emerges: the highest accuracy is attained when augmentation is applied to 50% of the data. Beyond this threshold, further increasing the size of augmented data leads to diminishing returns, resulting in a decline in accuracy and mF1-score. This suggests that the model tends to overfit the training data when subjected to an excessive amount of augmented information. While the accuracy of the validation data may show promising signs, the model's performance on unseen data, specifically the test data, begins to decrease.
With the optimal augmented data percentage identified, our exploration extends to varying the percentage of words swapped in each iteration. The results, as depicted in Table 2, indicate that the overall accuracy and mF1-score exhibit minimal fluctuations with changes in the word augmentation percentage. However, precision and recall values do showcase variations corresponding to alterations in the word augmentation percentage. This nuanced observation underscores the importance of fine-tuning not only the quantity of augmented data but also the specific aspects of augmentation. Subsequently, we implemented the Delete Data Augmentation on 50% of the original training dataset. Leveraging the NLPAug library in Python, we conducted experiments to generate new data by selectively removing certain words in each row. This augmented dataset was seamlessly integrated with the original data, effectively amplifying the training set by 50%. Our exploration further extended to varying the percentage of words designated for deletion in each iteration. The outcomes of this experiment are presented in Table 3. The results demonstrate that this approach not only diversifies the training data but also involves fine-tuning the augmentation to achieve improvements in model performance. In implementing spelling augmentation, we crafted a custom function in Python to dynamically alter the spelling of selected words within each sentence. This function provides the flexibility to adjust the percentage of words in each row subject to augmentation. The outcomes of this experiment are presented in Table 4. Notably, the results reveal an improvement in accuracy as we incrementally raise the percentage of augmented words in each row. However, a cautious approach was adopted, refraining from further increasing the percentage to prevent potential distortion of the sentence's meaning. Beyond a certain threshold, excessive alterations could compromise the contextual integrity of the sentence, potentially undermining the model's overall performance.
After conducting a comprehensive array of experiments, our findings show that the most favorable accuracy was achieved through swap data augmentation. The optimal model, exhibiting the highest accuracy, emerged from training with an additional 50% of data, where 30% of words in each row were subject to swapping. This configuration demonstrated the finest balance between data enrichment and model performance enhancement.
To provide a visual representation of the model's performance, Figure 1 presents the confusion matrix for this optimal model, showcasing the details of how well the model navigated and classified instances with the applied swap data augmentation. This synthesis of experimentation outcomes reinforces not only the efficacy of swap data augmentation but also the significance of precise configurations in achieving the model's peak performance.
Recognizing the pressing need to combat hate speech within the constraints of limited resources, our paper lies in the strategic application of data augmentation techniques to linguistic datasets on social media. We assert that applying data augmentation techniques to the dataset helps to increase the dataset size and improves the overall model performance. We experimented with determining the ideal percentage of augmented data to seamlessly integrate with the original dataset. This exploration aimed not only to enhance model training efficiency but also
to circumvent the pitfalls of model overfitting. Our experimentation involved the application of three distinct data augmentation techniques: random swap, random deletion, and spelling augmentation. Notably, the results underscore the prowess of swap data augmentation, exhibiting the highest accuracy at 91.3%. This achievement was realized with a 30% word augmentation rate and a 50% augmented data incorporation. We also published the new RUHSOLD++ dataset containing the augmented data. For future work we envision the exploration of additional augmentation techniques, setting the stage for a comprehensive comparison of model performances. This will improve and add more tools to the ongoing fight against hate speech on social media especially for under-resourced languages such as Roman Urdu.