Homophobia and Transphobia Detection is the task of identifying homophobia, transphobia, and nonanti-LGBT+ content from the given corpus. Homophobia and transphobia are both toxic languages directed at LGBTQ+(Lesbian, Gay, Bisexual, Transgender, and Queer community ) individuals that are described as hate speech. The shared task we worked on was to try and predict if a given comment was homophobic in nature. We were provided with a corpus of comments in Tamil, Malayalam, TamilEnglish, and English. We used the IndicBERT and LaBSE machine learning models to predict the content. The results were as follows: IndicBERT was used to train Tamil, Malayalam, and Tamil-English and LaBSE was used to predict the content in English. We achieved weighted average F1 scores of 0.46, 0.54, 0.39, and 0.28 for English, Malayalam, Tamil-English, and Tamil respectively.[1]
With the advancing reach of social media in today’s world, an increasing number of people are
getting online to consume content, share messages and consequently express their views and
opinions.[
The shared task focuses on trying to identify comments that are homophobic or transphobic and could potentially cause a lot of harm. Hence, we employ the use of deep learning-based NLP models that can be modeled to detect such homophobic and transphobic content to help the social media platforms better deal with the same and ensure social media remains safe for all.
Homophobia and Transphobia are toxic languages aimed at the LGBTQ+ community. The
goal of shared task (task B) is to develop systems to identify such homophobic, transphobic,
and non-anti-LGBT+ content from the given corpus[
The corpus[
The data was split into three sections; one to be used for training and one other for development. The Tamil, English, and Tamil-English files were provided in CSV format, while the Malayalam file was provided in TSV format.
The third and final section consisted of the test data which was provided in CSV format. The test data consisted of only the comment and the respective ID, no labels were provided. The test data was fed into the fully developed model to output the required results.
The task was approached as a binary-label text classification task with the use of transformers.
The data is preprocessed and then data is made suitable for training by creating a data frame
for all the instances of the data. Model training is done by fine-tuning the parameters of two
transformers that include LaBSE[
Data processing is important for any machine learning problem. The data available is often incoherent and unorganized and hence must be cleaned to bring it to a more organized format that can easily be processed and used to draw results from. The irregularities in the data are removed in the preprocessing stage and made suitable for training. Preprocessing steps for this machine learning problem involved diferent steps as follows: • Removal of special character such as ‘#’ , ‘@’, ’;’ and ’!’ etc - Special characters such as ’#,’:’ and ’@’ are very commonly used in comments in social media platforms such as YouTube. These characters can afect the training of the model as the words that end with special characters are treated diferently by the model. Hence they must be removed before the training so as to not interfere with the training of the model. • Replacement of emojis with the appropriate words : Emoticons convey emotional expression in a text and hence must be preserved. In order to take into account the emotions that they bring to a message, they are replaced with appropriate words that convey the same meaning.
The training and development data were concatenated and put into a data frame. The eightytwenty rule was then used and eighty percent of the data was used as training data and twenty percent of the data was used to test the model’s performance. Label encoding was done to the training dataset and testing dataset to convert them into machine-readable form. The subsequent f1 score, recall, and precision obtained were evaluated.
Sample input data:
Key: 0: Homophobic or Transphobic ; 1: Non-Anti LGBTQ+. Sample input data
The transformer models namely IndicBERT and LaBSE were used whose implementation is
explained below. Both the models were trained for 3 epochs. (the rest of the hyper parameters
were restricted to their default values) The IndicBERT uses an autotokeniser with the accents
set to true.
4.2.1. AI4Bharat/IndicBERT
IndicBERT[
The languages that IndicBERT covers are: Assamese, Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Oriya, Punjabi, Tamil, Telugu.
For our implementation, the AI4Bharat/IndicBERT model was used on the vernacular
languages ( Tamil, Malayalam and Tamil-English)
4.2.2. LaBSE
LaBSE, a multilingual embedding model[
The LaBSE model was used for the English dataset.
After training, the sample test data was generated using twenty percent of the concatenated training and development and fed into the model to evaluate its performance. Once the performance metrics were verified, the model was run on the actual test data provided.
LaBSE model that was used on the English dataset achieved a mean F1-score of 0.4625 and our
team was ranked third for the same. IndicBERT model was used for Tamil-English, Malayalam
and Tamil datasets. It achieved a mean F1-score of 0.393 for the Tamil-English dataset and
we were placed third. It achieved a mean F1-score of 0.542 for the Malayalam dataset and we
were placed seventh. For the tamil dataset, the model achieved a mean F1-score of 0.228 and
was ranked eighth. The following performance metrics[
Precision: Precision refers to the probability that a correct classification has been done. It is the ratio of true positives to total positives predicted.
Both the AI4Bharat/IndicBERT and LaBSE fail to attain perfect scores in identifying homophobic or transphobic comments (0 label). These scores were improved by making changes to the data preprocessing. Initially, when the emojis were completely removed from the text, the obtained scores were unsatisfactory. Thus, emoji replacement was done instead, as emoticons provide meaning and emotion to each sentence.
The IndicBERT model had dificulty always predicting the correct label. This
Malayalam sentence, for instance, was incorrectly identified as a non-anti-LGBTQ+ statement. A
possible explanation for this might be that the IndicBERT model is trained on native Indian
language scripts and not their transliterated counterparts such as Malayalam sentences
transliterated into English[
The paper discusses our approach to trying and identifying comments as homophobic, transphobic, or non anti LGBT. Our implementation included a mixture of IndicBert for the vernacular languages and LaBSE for English. The submitted IndicBert model got an F1 score of 0.228 for Tamil, 0.542 for Malayalam, and 0.393 for Tamil-English. The LaBSE submission received an F1 score of 0.4625.
On the whole, LaBSE seems to have performed better when compared to the IndicBERT model on the respective datasets. In the future, the performance could be improved by performing additional pre-processing steps on the data provided and perhaps, even making use of external datasets to improve the aforementioned accuracy of the transfer model implementations. As a future implementation, an ensemble approach could also be adopted to achieve a more accurate transfer model.