The Code-mixed text classification is challenging due to the lack of code-mixed labeled datasets and the non-existence of pre-trained models. This paper presents the HASOC-2021 ofensive language identification results and main findings on code-mixed (Hindi-English) Subtask2. In this work, we have proposed a new method of code-mixed data augmentation using synonym replacement of Hindi and English words using WordNet, and phonetics conversion of Hinglish (Hindi-English) words. We used a 5.7k pre-annotated HASOC-2021 code-mixed dataset for training and data augmentation. The proposal's feasibility was tested with a Logistic Regression (LR) used as a baseline, Convolutional Neural Network (CNN), and BERT with and without data augmentation. The research outcomes were promising and yields almost 3% increase of classifier accuracy and F1 scores as compared to baseline. Our oficial submission showed a 66.56% F1 score and ranked 8th position in the competition.
sexism, racism, politics, cyberbullying, and blackmailing. Nockleby [
organizations have been focusing on developing an approach that can identify hate speech or
abusive languages and flag them for human restraints or elimination [
in a sentence, is getting increasingly commonplace in several bilingual communities, which renders the automatic making detection task more challenging [11].
Many works focused on deep learning-based models to identify the aggressive language in social media texts. For instance, Agrawal and Awekar[12] investigated how learning-based models can capture more dispersed features on various platforms and topics. Bu and Cho[13] provided a hybrid deep learning model that combines CNN and Long-term Recurrent Convolutional Networks (LRCN) to detect ofensive in Social Networking Service (SNS) comments. A character-level CNN model with shortcuts was proposed by Lu et al. [14]. In addition, Rosa et al.[15] compared three diferent deep learning approaches, trained from three diferent sources for multiple category textual ofensive detection.
to cope with the scenarios above to stimulate research on hateful speech and ofensive content in social media, such as Semeval-2019[16] and 2020[17], HASOC-2019 [18] and 2020[19]. From previous work, BERT model outperformed most of other state-of-the-art models. The rise of
ofensive code-mixed language identification task in social media-Twitter. In this contest, our team participated to SubTask2 for Hindi-English code-mixed identification of Hate/ofensive Tweets. We have used the HASOC-2021 shared dataset for training and validation. Since the absence of large-scale code-mixed labeled corpus, an intuitive approach is to seek an appropriate data augmentation strategy. It is challenging to obtain universal transformation rules in natural languages that assure the quality of the produced data and easy automated application procedures in various domains. A common approach for a such a transformation is to replace words with their synonyms selected from a handcrafted ontology such as WordNet [22]. Another synonym replacement approach is based on pre-trained word embeddings such as GloVe, FastText, Sent2Vec, etc. The nearest neighbor words in the embedding space are a replacement for some word in the sentence or word similarity calculation [23]. Back-translation and paraphrasing augmentation also shown higher accuracy for supervised learning [24]. A recent trend is contextual data augmentation that stochastically replaces words with other words predicted by a bi-directional language model at the corresponding word positions. However, code-mixed datasets are an amalgamation of multilingual tokens, making them limited to existing augmentation methods. For example, contextual augmentation with the transformer model requires pre-trained models, and to the best of our knowledge, pre-trained models for codemixed are still scarce. Furthermore, many tokes are written native phonetically but in a diferent language; therefore, it is impossible to use synonym replacement without further conversion. We focused on code-mixed data augmentation to overcome these challenges by using synonym replacement with WordNet, phonetics conversion, translation, and back-translation for relevant tweets. The paper posits some main contributions as follow: • A data augmentation scheme has been put forward that has not been experimented for code-mixed dataset. • We developed a new python library for data augmentation, which is the end product of our experiment, and would be released under an open-sourced license for the research community 1. • We constructed a newly extended code-mixed (Hindi-English) dataset and released it publicly.
The paper is structured as follows. Section 2 describes our methodology, consisting of dataset annotation schema, preprocessing, dataset augmentation, and classifier architectures, including the machine learning models and the associated feature engineering. Section 3 details and comments on our experimental result. In Section 4, an error analysis task of our best models is performed. Finally, conclusive statements and potential future work are drawn in the conclusion section.
(vi)The Israel-Palestine conflict in 2021 (vii) Religious controversies in India and (viii) The coronavirus controversy.
The HASOC task organizer already annotated datasets for Subtask2. For hate/ofensive posts, it is labeled as HOF, and for non-hate/ofensive, it is labeled as NONE. The Code-mixed dataset consists of approximately 5740 training data from Twitter tweets, re-tweets, comments, and replies; among 2841(45%) are ofensive, and 2899(55%) are not-ofensive. Table 1 shows the example of annotated code-mixed.
ofensive content.
1https://pypi.org/project/nlp-augment/
The second challenge of Subtask2 is related to its unique annotation criteria. For example, each tweet in the dataset could have a conversational thread that may contain hate and ofensive content, which is not apparent just from a single comment or the reply to a comment but can be identified if given the context of the parent content. Table 2 shows relational annotation; for example, the comment’s reply ’You totally nailed it, can’t stop laughing’ seems not ofensive; however, since it is supporting the original post, which was ofensive, therefore it becomes ofensive as well.
You totally nailed it, can’t stop laughing
Translation They have asked Doctors and Scientists. Not fuckers.
Sit down.
Label HOF HOF 2.1.1. Data Preprocessing
#resignmodi kon chutiya ka interview liye ho Translation of Source posts who make him dog Who has interviewed fucker?
As discussed in Section 2.1, the dataset has three diferent types of tokens that have formed four diferent kinds of posts. Our proposed augmentation methods are followed by three diferent kinds of approaches that have been employed to cover all types of tokens and posts. For example, if posts consist of all 3 types of tokens, it is impossible to translate the sentence and reform it as a meaningful sentence. In that case, each token was targeted individually and identified by its type for further processing. For Hindi/English token, it is replaced by its synonyms. However, if the token was phonetics, it is converted to Hindi word using python library2; finally, synonym replacement is performed. After each token conversion, we have restored the sentence with 2https://pypi.org/project/pyhinavrophonetic/ diferent types of the augmented token. Figure 1 shows the token-based augmentation and reformation of new posts.
were performed (i) English to Hindi translation, (ii) Hindi to a phonetics, written in
again.
performed (i) Hindi to English translation, (ii) Hindi to phonetic, and (iii) Back-translation,
testing and validation, ensuring the same proportion of dataset for all kinds of model learning.
word-level TF-IDF, Convolution Neural Network (CNN) with word-level TF-IDF, and fine-tuned
Where is the mask मुर्ख chuthiye Where is the mask <fool> <pussy> Where is the <cover> मुर्ख chuthiye Where is the mask <idiot> chuthiye Sentence level Aug. Augmentation steps Fugitive son of a cow- Eng. > Hi. > Eng. BT. ard Bhagode Kaayar ka Eng. > Hi. > Phonetics beta Sale bhagode Kaayar Eng. > Hi. > Phonetics ka beta भगोड़े कायर का बेटा Eng. >Hi. translation Hi. > Eng, Phonetic > Hi. > Eng.
Eng. synonym (mask > cover) Hi. > Eng.
We adopted [25] a CNN, architecture, where the input layer is represented by a concatenation of the words forming the post (up to 70 words), except that each word is now a TF-IDF vector representation. Though several experiments showed using word embedding (e.g., fastText) performed better when used with CNN in the embedding layer[20]; however, no available pretrained word embedding was found for code-mixed Hindi-English. A convolution 1D operation with a kernel size 3 was used together with a max-over-time pooling operation over the feature map with a layer dense 50. Dropout on the penultimate layer with a constraint on 12-norms of the weight vector was used for regularization.
datasets and codes3.
BERT – Bidirectional Encoder Representations from Transformers: this seminal transformerbased language model employs an attention mechanism that enables the mode to learn contextual relations between (sub-)words in a text sequence [26]. BERT uses two training strategies:
learns to predict the original tokens, and
corresponding test data for validation. The following models were tested: BERT-base (uncased) and BERT-multilingual. Each model was fine-tuned for 6 epochs with a learning rate of 5e-6, maximum sequence length of 128, and batch size 4. After each epoch, the model was evaluated on the validation set.
3https://github.com/saroarjahan/Hasoc_2021_subtask2 (accessed September 20, 2021) and BERT-multilingual. The results exhibit model accuracy and F1 scores for both augmented and non-augmented dataset for comparison. In both cases, we see that CNN with word-level TF
slightly (.5%) outperform both BERT-base-uncased and BERT-multilingual. It is unusual that CNN showed much better performance compared to BERT models. The possible explanation is that we need a pre-trained model that would reflect our dataset; however, no pre-trained models were found that can be useful for code-mixed. Though BERT multilingual was trained with 104 languages; however, the BERT-multilingual model has not outperformed CNN. This agreed with our intuition since the multilingual pre-trained model was trained on the top 104 languages with the largest Wikipedia using a masked language modeling (MLM) objective; however, it contains only a small percentage of English and Hindi tokens. Therefore, it might have fallen short as the HASOC-2021 code-mixed dataset is related to Hindi, English, and amalgamation of phonetics tokens.
models. After using data augmentation for model training, it resulted in almost 3% increase compared to the non-expanded dataset. This improvement has been shown for baseline, CNN, and BERT models. Since we have kept the experiment test dataset identical for all models, this improvement justifies the proposed augmentation method that has helped the model to train better.
however, the F1 score seems low, which indicates the model exhibits a large portion of false detection. We performed in this section an analysis of the model’s performance and train dataset evaluation to understand this phenomenon better. For this purpose, we randomly prepared 200 subsets of test data then manually inspected each annotation.
From Figure 3, we see that 232 of the error is related to false-negative (FN), and 421 hate samples were correctly identified. In contrast, 477 non-hate samples were correctly detected, and 218 resulted as false positive (FP). This indicates that our model was not performing as good as detecting non-hate samples. Since our train dataset in its majority (55%) contains non-hate samples, it seems our model is better trained or biased towards non-hate classes. Another possible explanation of the overall model’s low performance is due to the fact that errors might be coming from the training samples. For example, our manual inspection of 200 samples showed that 4% of the test sample was annotated wrongly (Table 4). Since this test data was a split of original data, the wrong annotation of test data raises the question of overall training data quality, which might have afected model training. Furthermore, the training data were mostly relational data of tweets, retweets, comments, and replies. Therefore a single post was an amalgamation of hate and not-hate samples, which made the classifier dificult to train.
Subtask2 for hate-ofensive post identification. We performed diferent experiments of twitter preprocessing, classification with LR, CNN, and BERT-finetuned models. Our tweet preprocessing showed removing the mentioned tag and removing special characters and URLs were useful and yield an increased of almost 1% in classification accuracy. However, eliminating hashtags and stemming reduced the overall performance. This provides a basis for optimal preprocessing pipeline. For classification purpose, we proposed a data augmentation strategy for code-mix dataset. Since the code-mixed dataset is a combination of Hindi, English, and HingLish tokens, we applied data augmentation that employs WordNet synonym replacement, conversation phonetics to Hindi, translation, and back-translation of sentences. After training with the augmented dataset, all models showed an approximate 3% improvement in classification accuracy and F1 accuracy. Among BERT and CNN, CNN performed little better than BERT models since there was no available pre-trained model for the code-mix dataset. Our best test CNN models showed 69.6 F1 scores, and the oficial submission result showed 66.65% f1 scores. In the future, it would be interesting to experiment with contextual augmentation using
Picking up any matter, come with casting the cast, now stop, it is not India of 1947, 2021 has come, now leave this thing watch video randeep hooda make dirti joke nation leader mayavati dalit woman voic oppress toler castiest peopl arrest #arresterandeephood chup teri ka bsda watch video randeep hooda make dirti joke nation leader mayavati dalit woman voic oppress toler castiest peopl arrest #arresterandeephood chup kar saal ka baccha joke funni watch video randeep hooda make dirti joke nation leader mayavati dalit woman voic oppress toler castiest peopl arrest #arresterandeephood Supper watch video randeep hooda make dirti joke nation leader mayavati dalit woman voic oppress toler castiest peopl arrest #arresterandeephood The show should be cancelled watch video randeep hooda make dirti joke nation leader mayavati dalit woman voic oppress toler castiest peopl arrest #arresterandeephood God is Ram Rahim for you, now we don’t expect you to use your brain Original La- label should bel be HOF NONE HOF NONE HOF HOF NONE
HOF NONE NONE HOF
[5] Y. J. Foong, M. Oussalah, Cyberbullying system detection and analysis, in: 2017 European
[6] R. Kumar, A. K. Ojha, S. Malmasi, M. Zampieri, Benchmarking aggression identification in social media, in: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018), 2018, pp. 1–11. [7] K. Reynolds, A. Kontostathis, L. Edwards, Using machine learning to detect cyberbullying, in: 2011 10th International Conference on Machine learning and applications and workshops, volume 2, IEEE, 2011, pp. 241–244. [8] K. Dinakar, B. Jones, C. Havasi, H. Lieberman, R. Picard, Common sense reasoning for detection, prevention, and mitigation of cyberbullying, ACM Transactions on Interactive Intelligent Systems (TiiS) 2 (2012) 1–30. [9] Y. Bounab, J. M. Adeegbe, M. Oussalah, Towards storytelling automatic textual summerized, in: Conference of Open Innovations Association, FRUCT, 25, FRUCT Oy, 2019, pp. 434–438. [10] M. S. Jahan, O. Mourad, Team oulu at semeval-2020 task 12: Multilingual identification of ofensive language, type and target of twitter post using translated datasets, in: Proceedings of the Fourteenth Workshop on Semantic Evaluation, 2020, pp. 1628–1637. [11] I. A. Bhat, V. Mujadia, A. Tammewar, R. A. Bhat, M. Shrivastava, Iiit-h system submission for ifre2014 shared task on transliterated search, in: Proceedings of the Forum for Information
[12] S. Agrawal, A. Awekar, Deep learning for detecting cyberbullying across multiple social media platforms, in: European Conference on Information Retrieval, Springer, 2018, pp. 141–153. [13] S.-J. Bu, S.-B. Cho, A hybrid deep learning system of cnn and lrcn to detect cyberbullying from sns comments, in: International Conference on Hybrid Artificial Intelligence Systems, Springer, 2018, pp. 561–572. [14] N. Lu, G. Wu, Z. Zhang, Y. Zheng, Y. Ren, K.-K. R. Choo, Cyberbullying detection in social media text based on character-level convolutional neural network with shortcuts,
[15] H. Rosa, D. Matos, R. Ribeiro, L. Coheur, J. P. Carvalho, A “deeper” look at detecting cyberbullying in social networks, in: 2018 International Joint Conference on Neural Networks (IJCNN), IEEE, 2018, pp. 1–8. [16] M. Zampieri, S. Malmasi, P. Nakov, S. Rosenthal, N. Farra, R. Kumar, Semeval-2019 task 6: Identifying and categorizing ofensive language in social media (ofenseval), arXiv preprint arXiv:1903.08983 (2019). [17] M. Zampieri, P. Nakov, S. Rosenthal, P. Atanasova, G. Karadzhov, H. Mubarak, L. Derczynski, Z. Pitenis, Ç. Çöltekin, Semeval-2020 task 12: Multilingual ofensive language identification in social media (ofenseval 2020), arXiv preprint arXiv:2006.07235 (2020). [18] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, A. Patel, Overview of the hasoc track at fire 2019: Hate speech and ofensive content identification in indo-european languages, in: Proceedings of the 11th forum for information retrieval evaluation, 2019, pp. 14–17. [19] T. Mandl, S. Modha, A. Kumar M, B. R. Chakravarthi, Overview of the hasoc track at ifre 2020: Hate speech and ofensive language identification in tamil, malayalam, hindi, english and german, in: Forum for Information Retrieval Evaluation, 2020, pp. 29–32. [20] M. S. Jahan, M. Oussalah, A systematic review of hate speech automatic detection using natural language processing, arXiv preprint arXiv:2106.00742 (2021). [21] S. Satapara, S. Modha, T. Mandl, H. Madhu, P. Majumder, Overview of the HASOC
in: Working Notes of FIRE 2021 - Forum for Information Retrieval Evaluation, CEUR, 2021. [22] X. Zhang, J. Zhao, Y. LeCun, Character-level convolutional networks for text classification,
Advances in neural information processing systems 28 (2015) 649–657. [23] S. Wang, J. Liu, X. Ouyang, Y. Sun, Galileo at semeval-2020 task 12: Multi-lingual learning for ofensive language identification using pre-trained language models, arXiv preprint arXiv:2010.03542 (2020). [24] D. R. Beddiar, M. S. Jahan, M. Oussalah, Data expansion using back translation and paraphrasing for hate speech detection, Online Social Networks and Media 24 (2021) 100153. [25] Y. Kim, Convolutional neural networks for sentence classification, arXiv preprint arXiv:1408.5882 (2014). [26] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [27] S. Modha, T. Mandl, G. K. Shahi, H. Madhu, S. Satapara, T. Ranasinghe, M. Zampieri,
ifcation in english and indo-aryan languages and conversational hate speech, in: FIRE 2021: Forum for Information Retrieval Evaluation, Virtual Event, 13th-17th December 2021, ACM, 2021.