This paper presents the novel solution to HODI (Homotranphobia Detection in Italian; [1] at EVALITA 2023 [2]. The task is structured into two subtasks: Task A, homophobic message detection, and Task B, identification of rationales of homophobic messages. We participated in Task A, a binary class classification problem. The main aim of this task is to identify the homotransphobia in Italian tweets. To determine the homotransphobia, we choose the selective models that are available in the huggingface1 related to our task. We use the zero-shot technique to select the models that are working well for the homotransphobia classification task. With those models, we performed a series of few-shot learning experiments. Our best approach achieves a macro F1-score of 0.673, higher than the baseline, and ranking number 6.
gravely concerned about homophobic bullying against gay, lesbian, bisexual, transgender, and "queer" (LGBTQ+) Hate speech refers to any type of ofensive material, in- minorities [15]. Bullying is a form of aggression that cluding verbal, nonverbal, symbolic, or communicative involves a victim, an aggressor, and bully victims. Hoactions that are intentionally used to demean members mophobia, the underlying attitude influencing bullying of a specific social group based on their membership [ 3]. against LGBTQ+ vulnerable minorities, is defined as the Hate speech on social media is a pervasive phenomenon negative beliefs, attitudes, stereotypes, and behaviors afecting diverse categories of users targeted because directed toward sexual minorities [16, 17]. of their race, ethnicity, gender, religion, sexual orienta- Cross-lingual zero and few-shot learning, in addition tion, political views, or other characteristics [4, 5, 6, 7]. to other detection approaches that work well with limSignificant efort has been expended to combat harmful ited or nonexistent training data sets in the target lanand abusive content in the general domain, primarily via guage, have not been extensively studied in the current reactive measures [8, 9, 10, 11, 12, 13]. literature on homophobia [18, 19]. Neither have other
One such form of hate is hate speech towards vulnera- detection methods that work successfully with limited or ble LGBTQ+ individuals. Homophobia, a term frequently nonexistent training data sets in the target language [20]. used to characterize hostile responses to lesbians and The researchers [21] describe zero-shot learning as an gay men, connotes a one-dimensional conception of atti- extreme form of transfer learning. When this concept is tudes as manifestations of irrational fears [14]. Parents, used in natural language processing (NLP), a model that instructors, school administrators, and government are has been trained on one language or domain can learn to predict samples from an unseen language or domain by making use of the latent structures of a pre-trained language model that is aligned across several languages [22, 23]. In the method of cross-lingual few-shot learning, training on the source language is supplemented with samples of the target language. This helps to strengthen both cross-lingual and task-specific alignment.
In this paper, we describe a novel approach for
detecting homotransphobia in Italian-language tweets shared
task conducted by [
task on Homotransphobia in Italian tweets consists of labeled data categorized into two classes: homotranspho2. Related Work bic and not-homotransphobic. The dataset is divided into three sets: training, development, and testing, and each The modern forms of social media are frequently ex- set is carefully constructed to maintain a comparable disploited in ways that promote the dissemination of violent tribution. The distribution of the dataset is visualized messages and remarks as well as hate speech [25]. By an- in Figure 1, which provides a graphical representation alyzing the people’s interaction on these issues through of the data distribution. Additionally, the class-wise disposts, videos, and comments, a number of works have tribution of the training, development, and test sets is been done with the purpose of determining whether or presented in Table 3, as provided by the HODI shared not aggressiveness [26], misogyny [27], racism [28], ha- task organizers. These distributions ofer valuable inrassment, and violence are present in social media [29]. sights into the composition and balance of the dataset, On the other hand, the amount of study that has been con- enabling researchers to analyze and develop models to ducted to identify homophobic and transphobic speech address homotransphobia in Italian tweets efectively. online has been rather limited to few research [18].
Chakravarthi et al. [30] introduced a newly developed hierarchical taxonomy for online homophobia and transphobia, as well as a dataset that has been classified by subject matter experts, that will make it possible for homophobic and transphobic content to be automatically identified. The annotators are provided with thorough annotation criteria because this is a delicate issue. The dataset includes 15,141 comments written in English, Tamil, and Tamil-English, each of which has been annotated. From this data, Chakravarthi et al. [19] organized a shared task to increase research in homophobia/transphobia identification. It garnered 10 systems for the Tamil language, 13 systems for the English language, and 11 systems for the Tamil-English language combination. The best systems for Tamil, English, and Figure 1: Visualization of splitted dataset Tamil-English each received an average macro F1-score of 0.570, 0.870, and 0.610, respectively. Further, we select 400 random samples from the train
A similar shared task for Dravidian languages ho- ing set, and we have split the dataset for the few-shot mophobia/transphobia identification was conducted by learning into five sets (80, 160, 240, 320, and 400). These 10.1145/3574318.3574347. It is conducted in 4 language sets are divided based on 20, 40, 60, 80, and 100 percent settings (Tamil, English, Malayalam, Tamil-English). It of the data taken from the training data. These datasets obtained 8 systems for the Tamil language, 8 systems are visualized in Figure 2. for the English language, 9 systems for the Malayalam language, and 8 systems for Tamil-English codemixed.
Wenpeng Yin and Roth [31] examined the limitations 4. Methodology of prior research on zero-shot text classification (0SHOTTC), the inadequacy of getting the problem and the sig
rate of 2 × 10− 5 and batch size of 2. We used the same setting for all the models used for fine-tuning.
Training Development
Models
twitter-xlm-roberta-base-sentiment-finetunned [34]
xlm-roberta-large-it-mnli
mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 [35]
dehatebert-mono-italian [
solutions for the Homotransphobia Detection in Italian (HODI) Shared Task at EVALITA 2023. Our approach focused on utilizing pre-trained models for zero-shot learning, aiming to detect homotransphobia in Italian text.
We experimented with multiple pre-trained models and evaluated their performance to identify the most efective one. Additionally, we employed a few-shot learning technique by utilizing a dataset consisting of five sets, as depicted in the dataset section diagram. Our findings evaluation campaign of natural language process- org/2022.dravidianlangtech-1.44. doi:10.18653/ ing and speech tools for italian, in: Proceedings v1/2022.dravidianlangtech-1.44. of the Eighth Evaluation Campaign of Natural Lan- [11] M. R. Karim, S. K. Dey, T. Islam, M. Shajalal, guage Processing and Speech Tools for Italian. Final B. R. Chakravarthi, Multimodal hate speech deWorkshop (EVALITA 2023), CEUR.org, Parma, Italy, tection from Bengali memes and texts, in: Speech 2023. and Language Technologies for Low-Resource Lan[3] O. S,tefănit, ă, D.-M. Buf, Hate speech in social media guages: First International Conference, SPELLL and its efects on the LGBT community: A review 2022, Kalavakkam, India, November 23–25, 2022, of the current research, Romanian Journal of Com- Proceedings, Springer, 2023, pp. 293–308. munication and Public Relations 23 (2021) 47–55. [12] B. R. Chakravarthi, M. B. Jagadeeshan, V. Palaniku[4] W. Warner, J. Hirschberg, Detecting hate speech mar, R. Priyadharshini, Ofensive language on the world wide web, in: Proceedings of the identification in Dravidian languages using MPNet Second Workshop on Language in Social Media, As- and CNN, International Journal of Information sociation for Computational Linguistics, Montréal, Management Data Insights 3 (2023) 100151. URL: Canada, 2012, pp. 19–26. URL: https://aclanthology. https://www.sciencedirect.com/science/article/pii/ org/W12-2103. S2667096822000945. doi:https://doi.org/10. [5] A. Schmidt, M. Wiegand, A survey on hate speech 1016/j.jjimei.2022.100151. detection using natural language processing, in: [13] B. R. Chakravarthi, R. Priyadharshini, S. BanerProceedings of the Fifth International Workshop jee, M. B. Jagadeeshan, P. K. Kumaresan, R. Ponon Natural Language Processing for Social Media, nusamy, S. Benhur, J. P. McCrae, Detecting abusive Association for Computational Linguistics, Valen- comments at a fine-grained level in a low-resource cia, Spain, 2017, pp. 1–10. URL: https://aclanthology. language, Natural Language Processing Journal 3 org/W17-1101. doi:10.18653/v1/W17-1101. (2023) 100006. URL: https://www.sciencedirect.com/ [6] P. Fortuna, S. Nunes, A survey on automatic de- science/article/pii/S2949719123000031. doi:https: tection of hate speech in text, ACM Computing //doi.org/10.1016/j.nlp.2023.100006.
Surveys (CSUR) 51 (2018) 1–30. [14] G. M. Herek, Beyond" homophobia": A social psy[7] N. Chetty, S. Alathur, Hate speech review in the chological perspective on attitudes toward lesbians context of online social networks, Aggression and and gay men, Journal of homosexuality 10 (1984) violent behavior 40 (2018) 108–118. 1–21. [8] A. Founta, L. Specia, A survey of online hate [15] S. S. Horn, J. G. Kosciw, S. T. Russell, Special issue inspeech through the causal lens, in: Proceed- troduction: New research on lesbian, gay, bisexual, ings of the First Workshop on Causal Inference and transgender youth: Studying lives in context, and NLP, Association for Computational Linguis- 2009. tics, Punta Cana, Dominican Republic, 2021, pp. [16] J. S. Hong, J. Garbarino, Risk and protective factors 74–82. URL: https://aclanthology.org/2021.cinlp-1.6. for homophobic bullying in schools: An application doi:10.18653/v1/2021.cinlp-1.6. of the social–ecological framework, Educational [9] P. K. Kumaresan, Premjith, R. Sakuntharaj, S. Thava- Psychology Review 24 (2012) 271–285. reesan, S. Navaneethakrishnan, A. K. Madasamy, [17] B. R. Chakravarthi, Detection of homophobia and B. R. Chakravarthi, J. P. McCrae, Findings of transphobia in youtube comments, International Shared Task on Ofensive Language Identifica- Journal of Data Science and Analytics (2023). URL: tion in Tamil and Malayalam, in: Proceedings https://doi.org/10.1007/s41060-023-00400-0. doi:10. of the 13th Annual Meeting of the Forum for 1007/s41060-023-00400-0.
Information Retrieval Evaluation, FIRE ’21, As- [18] B. R. Chakravarthi, A. Hande, R. Ponnusamy, P. K. sociation for Computing Machinery, New York, Kumaresan, R. Priyadharshini, How can we detect NY, USA, 2022, p. 16–18. URL: https://doi.org/ homophobia and transphobia? experiments in a 10.1145/3503162.3503179. doi:10.1145/3503162. multilingual code-mixed setting for social media 3503179. governance, International Journal of Information [10] R. Priyadharshini, B. R. Chakravarthi, S. Cn, Management Data Insights 2 (2022) 100119. URL: T. Durairaj, M. Subramanian, K. Shanmugavadivel, https://www.sciencedirect.com/science/article/pii/ S. U Hegde, P. Kumaresan, Overview of abusive S2667096822000623. doi:https://doi.org/10. comment detection in Tamil-ACL 2022, in: Proceed- 1016/j.jjimei.2022.100119. ings of the Second Workshop on Speech and Lan- [19] B. R. Chakravarthi, R. Priyadharshini, T. Durairaj, guage Technologies for Dravidian Languages, Asso- J. McCrae, P. Buitelaar, P. Kumaresan, R. Ponciation for Computational Linguistics, Dublin, Ire- nusamy, Overview of the shared task on homoland, 2022, pp. 292–298. URL: https://aclanthology. phobia and transphobia detection in social media comments, in: Proceedings of the Second Work- tional Linguistics, Online, 2021, pp. 3181–3197. URL: shop on Language Technology for Equality, Di- https://aclanthology.org/2021.acl-long.247. doi:10. versity and Inclusion, Association for Computa- 18653/v1/2021.acl-long.247. tional Linguistics, Dublin, Ireland, 2022, pp. 369– [28] A. Field, S. L. Blodgett, Z. Waseem, Y. Tsvetkov, 377. URL: https://aclanthology.org/2022.ltedi-1.57. A survey of race, racism, and anti-racism in doi:10.18653/v1/2022.ltedi-1.57. NLP, in: Proceedings of the 59th Annual Meet[20] Y. Wang, Q. Yao, J. T. Kwok, L. M. Ni, Generaliz- ing of the Association for Computational Lining from a few examples: A survey on few-shot guistics and the 11th International Joint Conferlearning, ACM computing surveys (csur) 53 (2020) ence on Natural Language Processing (Volume 1–34. 1: Long Papers), Association for Computational [21] I. Goodfellow, Y. Bengio, A. Courville, Deep learn- Linguistics, Online, 2021, pp. 1905–1925. URL: ing, MIT press, 2016. https://aclanthology.org/2021.acl-long.149. doi:10. [22] G. Winata, S. Wu, M. Kulkarni, T. Solorio, 18653/v1/2021.acl-long.149.
D. Preotiuc-Pietro, Cross-lingual few-shot learn- [29] K. Glasgow, R. Schouten, Assessing violence risk in ing on unseen languages, in: Proceedings of the threatening communications, in: Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Workshop on Computational Linguistics and CliniAssociation for Computational Linguistics and the cal Psychology: From Linguistic Signal to Clinical 12th International Joint Conference on Natural Lan- Reality, Association for Computational Linguistics, guage Processing (Volume 1: Long Papers), Associ- Baltimore, Maryland, USA, 2014, pp. 38–45. URL: ation for Computational Linguistics, Online only, https://aclanthology.org/W14-3205. doi:10.3115/ 2022, pp. 777–791. URL: https://aclanthology.org/ v1/W14-3205.
2022.aacl-main.59. [30] B. R. Chakravarthi, R. Priyadharshini, R. Pon[23] Y. Gu, X. Han, Z. Liu, M. Huang, PPT: Pre-trained nusamy, P. K. Kumaresan, K. Sampath, D. Thenprompt tuning for few-shot learning, in: Pro- mozhi, S. Thangasamy, R. Nallathambi, J. P. McCrae, ceedings of the 60th Annual Meeting of the As- Dataset for identification of homophobia and transociation for Computational Linguistics (Volume 1: sophobia in multilingual youtube comments, arXiv Long Papers), Association for Computational Lin- preprint arXiv:2109.00227 (2021). guistics, Dublin, Ireland, 2022, pp. 8410–8423. URL: [31] J. H. Wenpeng Yin, D. Roth, Benchmarking zerohttps://aclanthology.org/2022.acl-long.576. doi:10. shot text classification: Datasets, evaluation and 18653/v1/2022.acl-long.576. entailment approach, in: EMNLP, 2019. URL: https: [24] L. Tunstall, N. Reimers, U. E. S. Jo, L. Bates, D. Korat, //arxiv.org/abs/1909.00161.
M. Wasserblat, O. Pereg, Eficient few-shot learning [32] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kawithout prompts, arXiv preprint arXiv:2209.11055 plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas(2022). try, A. Askell, et al., Language models are few-shot [25] Z. Waseem, D. Hovy, Hateful symbols or hateful learners, Advances in neural information processpeople? predictive features for hate speech detec- ing systems 33 (2020) 1877–1901. tion on Twitter, in: Proceedings of the NAACL [33] D. Nozza, Exposing the limits of zero-shot crossStudent Research Workshop, Association for Com- lingual hate speech detection, in: Proceedings of the putational Linguistics, San Diego, California, 2016, 59th Annual Meeting of the Association for Compp. 88–93. URL: https://aclanthology.org/N16-2013. putational Linguistics and the 11th International doi:10.18653/v1/N16-2013. Joint Conference on Natural Language Processing [26] S. T. Aroyehun, A. Gelbukh, Aggression detec- (Volume 2: Short Papers), 2021, pp. 907–914. tion in social media: Using deep neural networks, [34] F. Barbieri, L. Espinosa Anke, J. Camacho-Collados, data augmentation, and pseudo labeling, in: Pro- XLM-T: Multilingual language models in Twitter ceedings of the First Workshop on Trolling, Ag- for sentiment analysis and beyond, in: Proceedings gression and Cyberbullying (TRAC-2018), Asso- of the Thirteenth Language Resources and Evaluciation for Computational Linguistics, Santa Fe, ation Conference, European Language Resources New Mexico, USA, 2018, pp. 90–97. URL: https: Association, Marseille, France, 2022, pp. 258–266. //aclanthology.org/W18-4411. URL: https://aclanthology.org/2022.lrec-1.27. [27] P. Zeinert, N. Inie, L. Derczynski, Annotating [35] M. Laurer, W. v. Atteveldt, A. S. Casas, K. Welonline misogyny, in: Proceedings of the 59th bers, Less Annotating, More Classifying – AddressAnnual Meeting of the Association for Computa- ing the Data Scarcity Issue of Supervised Machine tional Linguistics and the 11th International Joint Learning with Deep Transfer Learning and BERT Conference on Natural Language Processing (Vol- - NLI, Preprint (2022). URL: https://osf.io/74b8k, ume 1: Long Papers), Association for Computa- publisher: Open Science Framework.