This report outlines the problem formulation and methodology employed by team Chetona for identifying hate-speech in low resource languages from social media comments. We focus on HASSOC 2023 Task 4, which involves binary classification of Twitter, Facebook, and Youtube comments for hate speech in Bengali, Bodo, and Assamese languages. We propose ensembling IndicBERT and Naive Bayes, along with synthetic data upsampling techniques, and attain macro F1 scores of 0.73, 0.68, and 0.84 for Assamese, Bengali, and Bodo. The scores are significant improvements over existing baselines, placing us within the top 10 of the leaderboard for all languages. The code and method is available on Github.1 Hateful comments are prevalent on social media platforms. Although tools for automatically detecting, flagging, and blocking such false, ofensive, and harmful content online have matured lately, research in low-resource languages such as Assamese, Bengali, and Bodo is still lacking [1, 2, 3]. Most prior research pertains to detecting ofensive text in social media while neglecting the subtler and broader task of identifying hateful comments. This document delineates the problem definition and the approach incorporated to tackle the challenges presented by HASOC 2023 Task 4 [4, 5], which revolves around identifying hate speech in low-resource Indo-Aryan textual content sourced from social media platforms.
This task aims to identify hate speech from social media (Twitter, Facebook, and YouTube) text comments spanning the low resource languages of Bengali, Bodo, and Assamese from Eastern India. The datasets for these languages contain sentences labeled as hate/ofensive (HOF) or not hate (NOT), and the goal is to develop robust machine learning models that can reliably predict the correct binary class for each comment. Models are evaluated and compared using the Macro F1 score.
Our approach incorporates robust data augmentation followed by an ensemble approach for hate-speech detection.
The provided training data comprises 1281 Bengali, 4036 Assamese, and 1679 Bodo samples. We
up-sample the training examples of each language by translating the examples from other two
languages to the given language. For a language, we use the IndicTrans2 [
We implement an ensemble approach using IndicBERT [
Since multi-task learning generally yields better results, we train a single model for all three languages. Furthermore, to distinguish between the original and translated examples, we prepend each sample text with <language> + "original:" or "translated:", e.g. "bengali original:<followed by the original text>", "bodo translated:<followed by the translated text>". We ifne-tuned IndicBert’s ’ai4bharat/IndicBERTv2-MLM-Sam-TLM’ version from the Hugging Face library using Pytorch. The model was optimized using AdamW [12] and a learning rate of 2e-5. We trained the model for 20 epochs on a single A5000 GPU with a batch size of 16. The training took approximately 1 hour with early stopping if the validation loss didn’t decrease for 2 epochs. The Naive Bayes models were trained with the default Sklearn settings.
We report our results in Table 1 and compare the test set Macro F1 score against teams that submitted predictions for all three languages. We highlight in bold the best F1 score for each language and underline our results. We attain the best results for Assamese and competitive results for Bengali and Bodo. Since most Indian languages originate from Devanagari, we experimented with using the IndicBERT-SS IndicBERT variant instead of IndicBERT-MLM+Samanantar. IndicBERT-SS was trained with the MLM objective on an Indic language to Devanagari corpus to encourage better lexical sharing among languages. We did not note any improvements in results on the validation and test sets (Note: We do not report the numbers here).
AI Alchemists Avigail Stekel Chen876 Ours CIT TEAM CNLP-NITS-PP Code Fellas FiRC-NLP InclusiveTechies IRLab@IITBHU Komar99 Michal Stekel MUCS Ravens SATLab Team +1 TeamBD
This report outlines the problem formulation and the methodology employed to address the HASOC 2023 competition Task 4, focused on hate speech detection in low resource text from social media. Our proposed method implements a weighted ensemble of IndicBERT v2 and Multinomal Naive Bayes and incorporates a translation-based data augmentation approach. Our results indicate that our implementations can robustly detect social media hate speech in low resource Indo-Aryan languages, thus promoting a safer and more inclusive online environment. P. Kumar, Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages, 2023. arXiv:2212.05409. [10] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018). [11] G. Ramesh, S. Doddapaneni, A. Bheemaraj, M. Jobanputra, R. AK, A. Sharma, S. Sahoo, H. Diddee, M. J, D. Kakwani, N. Kumar, A. Pradeep, S. Nagaraj, K. Deepak, V. Raghavan, A. Kunchukuttan, P. Kumar, M. S. Khapra, Samanantar: The largest publicly available parallel corpora collection for 11 Indic languages, Transactions of the Association for Computational Linguistics 10 (2022) 145–162. URL: https://aclanthology.org/2022.tacl-1.9. doi:10.1162/tacl_a_00452. [12] I. Loshchilov, F. Hutter, Decoupled weight decay regularization, arXiv preprint arXiv:1711.05101 (2017).