<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>Hate Speech and Ofensive Content Identification in Memes in Bangla, Hindi and Gujarati using Auxiliary Text Supervised Learning</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Kongqiang Wang</string-name>
          <email>wangkongqiang60@gmail.com</email>
          <xref ref-type="aff" rid="aff1">1</xref>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Qingli Tan</string-name>
          <email>tanqingli@stu.ynu.edu.cn</email>
          <xref ref-type="aff" rid="aff0">0</xref>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>College of Ecology and Environment, Yunnan University</institution>
          ,
          <addr-line>Kunming 650500, Yunnan</addr-line>
          ,
          <country country="CN">China</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Forum for Information Retrieval Evaluation</institution>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>School of Information Science and Engineering, Yunnan University</institution>
          ,
          <addr-line>Kunming 650500, Yunnan</addr-line>
          ,
          <country country="CN">China</country>
        </aff>
      </contrib-group>
      <pub-date>
        <year>2026</year>
      </pub-date>
      <abstract>
        <p>Our group name on the HASOC2025-meme (Gujarati, Hindi and Bangla) competition platform on Kaggle is kongqiang wang. We are very interested in the seventh edition of the HASOC shared task, in which HASOC 2025 ofer memes for abuse, sentiment, sarcasm, vulgarity detection. This task involves analyzing multimodal data (image and text) to detect abuse, identify targeted communities, assess vulgarity and sarcasm, and assign sentiment labels. So, the task will be in five parts. We mainly participated in the identification of memes for abuse, sentiment, sarcasm and vulgarity detection. Team rank will be determined based on the Average of all four Macro F1 score. We ranked 10th on HASOC2025-meme(Gujarati) with a score of 0.56253 and 16th on HASOC2025-meme(Hindi) with a score of 0.51985. The above two tracks mainly employ machine learning and deep learning methods. We ranked 13th on HASOC2025-meme(Bangla) with a score of 0.52528. This track mainly employ transformer model. The code sources for the paper are available via GitHub.</p>
      </abstract>
      <kwd-group>
        <kwd>eol&gt;Multimodal Data Classification</kwd>
        <kwd>Bert</kwd>
        <kwd>Machine Learning</kwd>
        <kwd>Deep Learning</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>
        Social networking platforms such as Twitter and Facebook have become widely popular due to their
ease of use and broad accessibility, ofering individuals a powerful space to express their thoughts.
Users from all age groups actively engage on these platforms, frequently documenting and sharing
details of their daily lives, which contributes to an ever-growing volume of user-generated content.
Despite the many advantages of social media, it is not without its drawbacks. A significant amount
of harmful and ofensive content—including hate speech—circulates online, posing serious societal
challenges[
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
      </p>
      <p>
        HASOC[
        <xref ref-type="bibr" rid="ref2">2</xref>
        ] provides a forum for developing and testing text classification systems for various
languages. It organized a shared task for FIRE 2025[
        <xref ref-type="bibr" rid="ref3">3</xref>
        ]. The task is aimed at identifying hateful and
ofensive language in social media posts[4]. The task is organized for four languages, Hindi, Gujarati, Bangla
and Bodo[5], but we only conduct our investigation on the Hindi, Gujarati and Bangla[6] dataset.
      </p>
      <p>The following is a description of our research based on our advanced solutions to the specific
problem. Main research fields track description: This task involves analyzing multimodal data (image and
text) to detect abuse, assess vulgarity and sarcasm, and assign sentiment labels. So, the task will be in
four parts.</p>
      <p>Sentiment Detection:
• Positive - The meme conveys a supportive, humorous, or appreciative tone.
• Neutral - The meme is neither overtly positive nor negative in tone.</p>
      <p>• Negative - The meme expresses hostility, mockery, or criticism.
• Sarcastic - The meme presents statements or visuals that imply the opposite of their literal
meaning, often to mock or ridicule.</p>
      <p>• Non-Sarcastic - The meme directly conveys its message without sarcasm or irony.
Vulgarity Detection:</p>
      <sec id="sec-1-1">
        <title>Abuse Detection:</title>
        <p>• Vulgar - The meme contains explicit or ofensive words, gestures, or depictions.
• Not Vulgar - The meme does not include any such content.
• Abusive - The meme includes ofensive, harmful, or derogatory language, imagery, or
implications targeting an individual or a group.</p>
        <p>• Non-abusive - The meme does not contain any ofensive, harmful, or derogatory content.</p>
        <p>All the above tasks are part of HASOC2025-meme (Hindi, Gujarati and Bangla). We did not
participate in the identify targeted communities task. There has also been no research on the Bodo language.</p>
      </sec>
    </sec>
    <sec id="sec-2">
      <title>2. Related Work</title>
      <sec id="sec-2-1">
        <title>2.1. Dynamics of Online Abuse</title>
        <p>Online hostility is a context-dependent notion intended to express hatred and threaten an individual
or group based on discriminatory views. Despite the argument that hateful statements ought to be
tolerated due to free speech acts, the public expression of hate speech propels the reduction of minority
members, and such frequent and repetitive exposure to abusive speech could increase an individual’s
outgroup prejudice[7]. Real-world violent events could also lead to increased hatred in online space
and vice versa[8]. With the rise of online hate, the research community has a massive responsibility
to develop solutions to mitigate online hostility.</p>
      </sec>
      <sec id="sec-2-2">
        <title>2.2. Research on abusive speech</title>
        <p>The concern of abusive speech has long been studied in the research community. Earlier work on
abusive speech attempted to detect abusive users by using lexical, syntactic features extracted from
their posts[9]. Over the past few years, research around automated hate speech detection has matured
tremendously. Most of the current study consists of diverse but related works. In 2016, Zeerak Waseem
and Dirk Hovy[10] contributed a dataset in which thousands of tweets were labeled with racism and
sexism markers, and Davidson et al.[11] focused on distinguishing ofensive from hate content on
Twitter. Using this dataset, the authors examined multiple linguistic features such as character and
word n-grams, POStags, emotion lexicon, and tf-idf vectors with several classifiers such as LR, SVM,
decision tree, etc. Although multiple datasets were being published, a major problem was the lack of
correlation and reusable datasets across hate speech detection tasks[12]. To address this issue, Founta
et al.[13] studied these inconsistencies and drew upon a robust labeling mechanism that attempts to
circumvent the overlap among various forms of abusive speech.</p>
        <p>With the advent of large datasets, most academic research has moved to data-hungry
complex models to improve classifier performance, including deep learning[ 14] and graph embedding
techniques[15]. Pitsilis et al.[16], used deep learning models such as LSTMs to identify the abusive
tweets in English and noticed that it was pretty efective in this task. Zhang et al.[ 17] fused
convolutional and gated recurrent networks to enhance the classification performance and had remarkable
success on 6 out of 7 datasets used. Recently, transformer based language models such as BERT are
becoming immensely popular in several downstream tasks and have outperformed several deep learning
models such as CNN-GRU, LSTM, etc., for detecting abusive language[18].</p>
      </sec>
      <sec id="sec-2-3">
        <title>2.3. Abusive language detection in Indic languages</title>
        <p>In the last few years, several shared tasks, such as Hate-Speech and Ofensive Content Identification
(HASOC)[19], Dravidian Lang-Tech [20] workshop, TRAC[21], etc., have been organized to develop
resources, datasets, and models for abusive speech detection and multiple datasets in Indic languages
such as Hindi, Marathi, Tamil, Malayalam, etc. have been made public. The HASOC[22] shared task in
Indo-European languages is arguably the most well-known series of competitions. It has been
consistently organized from 2019 at the Forum for Information Retrieval (FIRE). The Dravidian Lang-Tech[20]
workshop focused on determining the ofensive language of the code-mixed dataset in three
Dravidian languages, namely, Tamil–English, Malayalam–English, and Kannada–English crawled from
social media. In addition, researchers have also developed several datasets for Bengali[23], code-mixed
Hindi[24], Urdu[25], etc., for abusive language detection. However, a limited number of studies have
been performed on the efect of zero-shot learning, few-shot learning[ 26], instance transfer, etc. In
our work, we try to fill this critical gap by studying various transfer schemes thus opening up new
avenues for future research for abuse detection in Indic languages[27].</p>
      </sec>
    </sec>
    <sec id="sec-3">
      <title>3. Exploratory Data Analysis</title>
      <p>For these tasks, participants are not allowed to use any external resources and datasets. So we
only used the training set provided by the oficial. The test dataset is also provided by the oficial for
the evaluation of models and the submitted result documents. The training and test sets for Hindi,
Gujarati and Bangla languages provided by HASOC2025-meme, as well as the labels in the training
sets, are respectively presented in Table 1.</p>
    </sec>
    <sec id="sec-4">
      <title>4. Graphical Representation</title>
      <p>The following images are the word-cloud for various types of memes given in the text dataset. It is a
graphical representation of word frequency of diferent words used in each category of the content in
memes on oficial resources and datasets.</p>
      <p>The following is the situation of the word cloud generated by sentiment detection in the training set
for the Hindi language provided by HASOC2025-meme (see Figure 1, 2, and 3).</p>
    </sec>
    <sec id="sec-5">
      <title>5. Methodology</title>
      <sec id="sec-5-1">
        <title>5.1. Introduction to Machine Learning</title>
        <p>Machine Learning (ML) is an important branch of artificial intelligence (AI). Its core idea is to enable
computers to automatically learn patterns through data and experience, rather than completing tasks
through explicit programming. In simple terms, traditional programming is Rules (written by
programmers) + data → output. And machine learning is: Data + Output → Learning Algorithm →
Rules (Model). In this way, the machine can use the learned ”model” to predict new data.
5.1.1. K-nearest Neighbor Model
The definition of the nearest neighbor algorithm: To determine the category of an unknown sample,
all training samples are used as references to calculate the distance between the unknown sample and
The disadvantage of the KNN algorithm:
• Large amount of sorting calculation.</p>
        <p>number of samples in others is small).</p>
        <p>• It requires a large amount of memory.
5.1.2. Naive Bayes Model</p>
        <p>• Sample imbalance issue (that is, the number of samples in some categories is large, while the
all training samples, and the category of the nearest neighbor is used as the sole basis for deciding the
category of the unknown sample.</p>
        <p>The K-nearest neighbor model is based on two important assumptions of machine learning: ①
Manifold Assumption: It refers to the fact that examples within a very small local neighborhood have similar
properties, and therefore, their notations should also be similar. ② Cluster Assumption: Examples in
the same cluster are more likely to have the same label.</p>
        <p>The advantages of the KNN algorithm:
• The idea is simple and the theory is mature. It can be used for both classification and regression.
• It can be used for nonlinear classification.</p>
        <p>• No assumptions about the data and insensitive to outliers.</p>
        <p>Logistic regression achieves classification by fitting curves (or learning hyperplanes). Naive Bayes
takes a unique approach by predicting classifications by considering feature probabilities.</p>
        <p>The theorem derivation for implementing classification using Naive Bayes is mainly as follows:
sample is  .
• Given a training dataset {( 1,  1), ( 2,  2), ( 3,  3), … , (  ,   )}, where  is the number of
samples in the dataset. Each sample contains  features, i.e.,   = ( 1 ,  2 , … ,   ). The label set is
{ 1,  2, … ,   }. Let ( =   | = )</p>
        <p>denote the probability that the output is   when the input
• Suppose we now have a new sample  and want to determine which class it belongs to. We can
solve for the values of ( = 
1|) , ( = 
2|) , ( = 
3|) , …, ( = 
 |) . Whichever value
is the largest determines the class it belongs to. That is, we solve for the maximum posterior
probability arg max ( |)</p>
        <p>.
• According to Bayes’ theorem, we have
(1)
(2)
(3)
• In general, the naive Bayes method assumes that all features are mutually independent, so the
above equation can be rewritten as:
• Since the denominator is the same for each class in the solution, it can be omitted in practice.</p>
        <p>Finally, the decision rule for the Naive Bayes classifier is:
( =   |x) =
(  )( x|  )</p>
        <p>( x)
( =   |x) =
(  )( x|  )
( x)
=</p>
        <p>(  )∏=1 (  |  )</p>
        <p>∏=1 (  )




 = arg max (  )( x|  ) = arg max (  )∏ (  |  )

=1
Ent() = −
∑   log2  
||
=1
The smaller the value of Ent() , the higher the purity of  . Suppose a certain attribute has 
possible values { 1,  2, … ,   }. If we use attribute  to partition the sample set  ,  branch nodes will be
generated, where the  -th branch node contains all samples in  whose attribute  takes the value   ,
denoted as   .</p>
        <p>By calculating the information entropy of   using the previous definition, and considering that
diferent branch nodes contain diferent numbers of samples, we assign weights
nodes, i.e., the more samples a branch node contains, the greater its influence. Thus, we can calculate
the ”information gain” obtained by partitioning the sample set  using attribute  :
| 
|/|| to the branch
Gain(, ) =</p>
        <p>Ent() −

∑
=1 ||
|  | Ent(  )
We can use information gain to perform decision tree attribute selection.</p>
        <p>Continuous Values: The simplest method for handling continuous values is to use bi-partition.
Given a sample set</p>
        <p>and a continuous attribute  , suppose  has  diferent values on  . Sort
these values from small to large to obtain { 1,  2, … ,   }. Obviously, for adjacent values   and  +1 , the
candidate partition points can be selected as:
5.1.3. Decision Tree Model
Decision trees make decisions based on tree structures, which is precisely a very natural processing
mechanism for humans when facing decision-making problems.</p>
        <p>The key question in decision tree construction is how to build the tree, which means how to select
the optimal splitting attribute for partitioning. To address this partitioning problem, we introduce the
concept of information entropy, denoted as:
(4)
(5)
(6)
(7)
(8)
point:
tree structure.
5.1.4. Random Forest Model
Random Forest is an extension of the Bagging algorithm in ensemble learning applied to decision trees.
As the name suggests, it is a forest composed of many decision trees.</p>
        <p>Consider a dataset containing  samples. Each time we randomly sample one sample and record
its information, then put it back into the dataset. After performing  such samplings, the sample
information we record forms a dataset of  samples. Then the probability that a certain sample is not
selected in the  samplings is (1 − 1/)  . Taking the limit:
  = {
  +  +1
2</p>
        <p>∣ 1 ≤  ≤  − 1 }
Similarly, calculate the information gain and select the point with maximum gain as the partition
Gain(, ) =
max Gain(, , ) =
∈ 
∈ 
max Ent() −</p>
        <p>∑
∈{−,+} ||
|  | Ent(  )
A decision tree divides based on the maximum information gain and makes decisions according to the
→∞
lim (1 −

1 
1

) ≈
That is, approximately 63.2% of the samples will appear in the sampled new dataset.</p>
        <p>Based on this, for each sampling we conduct, we train a decision tree using 63.2% of the samples. In
this way, by conducting such sampling training n times in total, we can obtain a total of n diferent
decision trees. Ultimately, by using the voting mechanism, if the majority of decision trees agree that it
is a positive class, then the final classification result of the random forest is a positive class. Conversely,
it is a negative class.
5.1.5. Support Vector Machine Model
Support vector Machine is a common supervised learning algorithm, mainly used for classification
and regression tasks. Its core idea is: to find an optimal hyperplane in the feature space, separate data
points of diferent categories, and maximize the classification margin.</p>
        <p>In simple terms: Imagine you have two types of data, red dots and blue dots. SVM will find a
line (in two dimensions) or a surface (in three dimensions or higher dimensions) and separate them.
Moreover, this line should be as far away as possible from the points on both sides, so that the model
is more ”robust” to new data.</p>
        <p>The advantages of SVM: It remains efective in high-dimensional Spaces (suitable for text
classification and genetic data analysis). The classification boundaries are clear and the generalization ability is
strong. It only relies on support vectors and is not greatly afected by redundant samples.</p>
        <p>The disadvantages of SVM: The training speed for large-scale data is slow (with high computational
complexity). Sensitive to the selection of parameters (such as C, kernel function type,  ). It is not very
suitable for handling large amounts of noisy data and overlapping samples.</p>
      </sec>
      <sec id="sec-5-2">
        <title>5.2. Ensemble Learning</title>
        <p>Ensemble learning is a machine learning method that achieves better performance than a single model
by combining the results of multiple models (base learners). Multiple weak models combined together
can form a strong model.</p>
        <p>There are mainly three common ways of ensemble learning:</p>
        <p>Bagging. Idea is perform random sampling with substitution on the data to obtain multiple diferent
training subsets, Train a base learner for each subset, and finally obtain the result by voting/averaging.
Features are Reduce variance and enhance stability. Representative algorithm is Random Forest.</p>
        <p>Boosting. The idea is that the base learner is trained step by step. The subsequent learner will focus
on learning the samples that were wrongly classified by the previous learner. Finally, multiple weak
classifiers are weighted and combined to form a strong classifier. Features are reduces bias, improves
accuracy, but is prone to overfitting. Representative algorithms are AdaBoost, Gradient Boosting
Decision Tree (GBDT), XGBoost, LightGBM, CatBoost.</p>
        <p>Stacking. Idea is train multiple base learners (which can be diferent types of models), and then
train a ”meta-learner” to integrate their prediction results. Features: It can combine the advantages
of diferent models and usually achieves the best results, but it is relatively complex. Representative
application: It is very common in Kaggle competitions1.</p>
        <p>The advantages and disadvantages of ensemble learning. Advantages: It enhances model accuracy
and has strong generalization ability. It can reduce overfitting (Bagging) or minimize deviation
(Boosting). Suitable for handling high-dimensional and large-scale data. Disadvantages: The computational
cost is high (multiple models need to be trained). The model has poor interpretability and is not as
intuitive as a single decision tree. There are many parameters, and parameter adjustment is rather
complicated.</p>
      </sec>
      <sec id="sec-5-3">
        <title>5.3. TfidfVectorizer</title>
        <p>TfidfVectorizer is a text feature extraction tool provided by scikit-learn, which is often used in natural
language processing (NLP). Its function is to convert a set of text (sentences, documents) into a
TFIDF feature matrix for use by machine learning models. The underlying principle of TfidfVectorizer is
based on TF-IDF, that is, term frequency-inverse document frequency.</p>
        <p>Term Frequency (TF). Represents the frequency of a term appearing in a document.
  (, ) =</p>
        <p>Number of times term  appears in document</p>
        <p>Total number of terms in document 
(9)
(10)
(11)
Inverse Document Frequency (IDF). Represents the degree of discrimination a term has in the
corpus.</p>
        <p>() =
log</p>
        <p>1 +  ()
•  : Total number of documents in the corpus
•  ()</p>
        <p>: Number of documents containing term</p>
        <p>If a term appears in many documents (such as “the”, “is”), its IDF will be low, indicating low
discrimination ability.</p>
        <p>TF-IDF Formula.</p>
        <p>(, ) =   (, ) ×   ()
This way, common words have low weights, while important but infrequent words have high weights.</p>
      </sec>
      <sec id="sec-5-4">
        <title>5.4. FastText</title>
        <p>FastText is an eficient word embedding and text classification tool proposed by the Facebook AI
Research (FAIR) team in 2016[28]. Its goal is to ensure high-quality word vectors while training faster
than Word2Vec and GloVe, and to handle Out-Of-Vocabulary words (OOV) more efectively[ 29]. The
core idea of FastText is diferent from that of Word2Vec. FastText does not merely view a word as a
whole but breaks it down into n-gram characters.</p>
        <p>Subword Representation, for example, for the word ”learning”, suppose n=3 (3-gram), then the
subwords include: The word vectors of lea, ear, arn, rni, nin, and ing. FastText are equal to the sum of
these sub-word vectors. In this way, even when encountering new words (such as ”learningssss”), the
model can generate reasonable vectors through its subwords, solving the OOV problem.</p>
        <p>The main functions of FastText: Word vector training, similar to Word2Vec, supports two training
methods: Skip-gram and CBOW. However, due to the introduction of sub-word information, the vector
representation is richer. FastText is equipped with an eficient text classifier that supports multi-class
and multi-label classification[ 30]. Improve the training speed through Hierarchical Softmax.</p>
        <p>The advantages of FastText: Fast training speed (based on eficient C++ implementation). It can
handle unlogged words (represented by sub-words). It performs well in low-resource languages and
small corpora. It can be used for two major tasks: word vector training and text classification.</p>
        <p>The disadvantages of FastText: Compared with Transformer models (such as BERT), its semantic
understanding ability is limited. It is impossible to capture long-distance dependencies (after all, it is
still based on the bag-of-words idea).</p>
        <p>To improve convergence, we apply Cyclic Learning Rate (CLR) scheduling, as proposed by (Smith,
2017)[31]. Specifically, we use the exp range policy to periodically vary the learning rate between
0.001 and 0.006 using a base-2 exponential decay factor (gamma=0.99994). This prevents premature
con- vergence and encourages the model to escape local minima during training. The model is trained
using the Adam optimizer with default hyperparameters ( 1 = 0.9,  2 = 0.999,  = 1 − 8 ). The loss
function used is categorical_crossentropy, and additional evaluation metrics include accuracy and a
custom-defined F1-score metric implemented using Keras backend operations. In the two language
tasks, i.e. HASOC2025-meme (Hindi) and HASOC2025-meme (Gujarati), the word vectors we use are
respectively cc.hi.300.vec and cc.gu.300.vec. They can be download from the fasttext oficial website
2</p>
      </sec>
      <sec id="sec-5-5">
        <title>5.5. Deep Learning</title>
        <p>5.5.1. Introduction to Deep Learning
Deep Learning (DL) is a branch of machine learning, which is based on Artificial Neural networks
(ANN), especially deep neural networks (DNN) with multi-layer structures. Common understanding:
Machine learning. It requires manual design of features (feature engineering) and then learning with
models. Deep learning. Through multi-layer neural networks, it automatically learns features from
data, reducing manual intervention. Therefore, deep learning performs outstandingly in tasks such as
image recognition, speech recognition, and natural language processing.</p>
        <p>The inspiration for deep learning comes from the way neurons in the human brain work: a neuron
receives input → weighted summation → activation function → output result. Multiple neurons
combine to form layers, and multiple layers combine to form a deep neural network.
5.5.2. Deep Learning Architecture
In this study, a text classification model based on deep learning was constructed. This model is built
using the Keras Sequential API. The overall structure includes the embedding layer, the recurrent neural
network layer and the fully connected classification layer. The specific design is as follows:</p>
        <p>Firstly, the discrete word index is mapped to a low-dimensional continuous dense vector by using
the word embedding layer, thereby capturing the semantic relationships between words. The input
length set in this experiment is 2500, and the embedding dimension is 256.</p>
        <p>Secondly, based on the embedded representation, the Long Short-Term Memory (LSTM) layer is
introduced to model the context dependency and long-distance features of the text. This layer adopts
a dropout rate of 0.2 and a recurrent dropout rate during the training process to alleviate the overfitting
problem and enhance the generalization ability of the model.</p>
        <p>Finally, a fully connected Layer (Dense Layer) is used for classification. This layer contains two
neurons, and the softmax function is adopted to map the network output to a probability distribution,
thereby achieving the binary or multiple classification task.</p>
        <p>During the model training process, the Adam optimizer is adopted for parameter update. The
categorical cross-entropy is selected as the loss function, and the Accuracy is used as the evaluation index.
All three models are based on this overall architecture. The detailed model situation is shown in Figure
6.</p>
      </sec>
      <sec id="sec-5-6">
        <title>5.6. TransformerTrainer</title>
        <p>Trainer is a training manager provided by Hugging Face3 to simplify the training/evaluation/inference
process of Transformer models. It helps you encapsulate many underlying details, such as: Training
loops (forward, loss calculation, backward, optimizer update), validation loops, gradient accumulation,
learning rate scheduling, mixed-precision training (FP16/bf16), distributed training (multi-GPU, TPU,
DeepSpeed), log and model saving.</p>
        <p>Its core components include: TrainingArguments: Configure training-related parameters
(number of epochs, batch size, learning rate, save policy, etc.). Trainer: Encapsulates the training logic,
manages models, datasets, and optimizers. compute_metrics: Allows for custom evaluation metrics
such as precision and F1 score. Callback: You can add custom logic, such as EarlyStopping.</p>
      </sec>
      <sec id="sec-5-7">
        <title>5.7. BERT Model</title>
        <p>BERT (Bidirectional Encoder Representations from Transformers) Proposed by Google AI (2018)[32],
it is a pre-trained language model based on the transformer encoder.
3URL: https://huggingface.co/</p>
        <p>Model 1
Layer(Type) Input Shape Output Shape Basic Parameters
Embedding input_dim = 8859 output_dim = 256 input_length = 2500</p>
        <p>LSTM 256 lstm_out = 64 dropout = 0.2</p>
        <p>Its core idea is to conduct pre-training through large-scale unsupervised corpora and then fine-tune
with a small number of supervised tasks, thereby significantly improving performance on various
natural language processing tasks, such as text classification, named entity recognition, question-answering
systems, etc.</p>
        <p>The model structure mainly includes the following parts:
• Transformer Encoder Stacking: BERT uses multiple layers of Transformer Encoders as its basic
structure, and each layer contains: multi-Head self-attention mechanism, feedforward neural
network, residual connection and layer normalization.
• Bidirectional context modeling: Unlike traditional language models (which Model only from left
to right or from right to left), BERT uses the masked language model (MLM) technique, enabling
the model to obtain context information from both left and right directions simultaneously.
• Special symbols: [CLS] : Added at the beginning of the sentence during classification tasks, and
the hidden vector at the end can be used for classification. [SEP] : Delimiter used to distinguish
sentences for tasks such as natural language reasoning.</p>
        <p>The BERT pre-training task is trained through two unsupervised tasks:
• Masked Language Model (MLM). Randomly covering 15% of the words, the model needs to
predict these words. Let the model learn deep context representations.
• Next Sentence Prediction (NSP). Given two sentences, the model determines whether the second
sentence is a true follow-up to the first one. Let the model learn semantic relations at the sentence
level.</p>
        <p>There are two situations regarding the scale of pre-trained models:
• BERT Base: 12-layer transformer encoder, 768 hidden dimensions, 12 attention heads, and
approximately 110 million total parameters.
• BERT Large: 24-layer transformer encoder, 1024 hidden dimensions, 16 attention heads, total
parameters approximately 340 million.</p>
        <p>Fine-tuning BERT pre-training model: The core advantage of BERT lies in the fact that the same
pretrained model can be adapted to diferent downstream tasks. fine-tuning, simply add a task-specific
output layer (such as a classification layer) to the [CLS] vector and then train on the downstream
dataset. Application scenarios include text classification (sentiment analysis, hate speech detection,
etc.), sequence labeling (named entity recognition, word segmentation), question-answering system
(SQuAD), natural language reasoning (NLI), information retrieval (semantic matching).</p>
        <p>The advantages of the Bert pre-trained model include Bidirectional representation:
Considering the information on both sides of the context simultaneously, its efect is superior to that of the
unidirectional model. Versatility: One model is adapted to multiple downstream tasks. Powerful
performance: When BERT was proposed, it refreshed the SOTA (state of the art) results on 11 NLP
tasks.</p>
        <p>In summary, BERT is a pre-trained language model based on the Transformer Encoder. It uses
two tasks, MLM and NSP, to learn deep bidirectional context representations and is widely applied in
various natural language processing tasks through fine-tuning.</p>
      </sec>
    </sec>
    <sec id="sec-6">
      <title>6. Result</title>
      <sec id="sec-6-1">
        <title>6.1. Machine Learning Model Experimental Results</title>
        <p>We used diferent machine learning methods to train the models for the HASOC2025-meme(Hindi) and
HASOC2025-meme(Gujarati) tasks. We extracted 15% of the data from the training set as the validation
set. The performance of these models in the two languages is shown in Table 2 and Table 3.</p>
        <p>The text is processed by TfidfVectorizer into digital matrices and then fed to the machine learning
model respectively. The final performance on the test set is shown in Table 4 and Table 5. Among
them, the voting mechanism was used to conduct ensemble learning on the machine learning
KNeighborsClassifier, LinearSVC, and RandomForestClassifier, and it achieved good results in the overall
situation.</p>
        <p>KNeighborsClassifier</p>
        <p>LinearSVC
RandomForestClassifier
overall (rfc, knc, svc)</p>
      </sec>
      <sec id="sec-6-2">
        <title>6.2. FastText Model Experimental Results</title>
        <p>This word vector-based model performed averagely in these two language tasks, i.e.
HASOC2025meme (Hindi) and HASOC2025-meme (Gujarati), their final results are shown in Table 6.</p>
      </sec>
      <sec id="sec-6-3">
        <title>6.3. Deep Learning Model Experimental Results</title>
        <p>In our experiment, we used the LSTM model in Model 1 as the baseline model. Due to the
overfitting problem of the model, the performance was not satisfactory. Therefore, we added Dropout and
BatchNormalization parts to Model 1. The performance of the model has been improved to a certain
extent, solving the overfitting efect of the training set. We call this model Model 2. To further enhance
the performance of the model, we have added class_weight to address the issue of label imbalance in
the dataset. The use of EarlyStopping and ReduceLROnPlateau strategies has further enhanced the
model, which we call Model 3. The performance of these models on the HASOC2025-meme(Hindi)
and HASOC2025-meme(Gujarati) tasks is shown in Table 7.</p>
      </sec>
      <sec id="sec-6-4">
        <title>6.4. BERT Model Experimental Results</title>
        <p>In our task, we used the l3cube-pune/bengali-bert4 model combined with TransformerTrainer and
achieved good results on the HASOC2025-meme (Bangla) dataset. See Table 8. BengaliBERT is a
Bengali BERT model trained on publicly available Bengali monolingual datasets. Preliminary details
on the dataset, models, and baseline results can be found in their paper[33].</p>
      </sec>
    </sec>
    <sec id="sec-7">
      <title>7. Conclusion</title>
      <p>In this paper, several machine learning and deep learning approaches have been used to detect hate
speech and ofensive language content, and the models have been compared. Several techniques have
been employed to increase accuracy. Our proposed Bert model achieved good results compared to its
simplicity. We believe with proper feature extraction and data augmentation techniques, these will be
able to improve our proposed model.</p>
    </sec>
    <sec id="sec-8">
      <title>Acknowledgments</title>
      <p>We are very grateful to the organizers of the Shared Task on Hate Speech and Ofensive Content
Identification in Memes in Bangla, Hindi, Gujarati and Bodo @ HASOC-meme (2025) and the School
of Information of Yunnan University for providing the environment and equipment.</p>
    </sec>
    <sec id="sec-9">
      <title>Declaration on Generative AI</title>
      <sec id="sec-9-1">
        <title>The author(s) have not employed any Generative AI tools.</title>
        <p>4HuggingFace: https://huggingface.co/l3cube-pune/bengali-bert
[4] K. Ghosh, N. K. Singh, J. Mahapatra, et al., Safespeech: a three-module pipeline for hate intensity
mitigation of social media texts in indic languages, Social Network Analysis and Mining 14 (2024).</p>
        <p>URL: https://doi.org/10.1007/s13278-024-01393-9. doi:10.1007/s13278- 024- 01393- 9.
[5] K. Ghosh, S. Saha, T. Mandl, S. Modha, Findings from shared tasks on hate speech
detection: Performance patterns for low-resource languages, Pattern Recognition Letters (2025). URL:
https://www.sciencedirect.com/science/article/pii/S0167865525003150. doi:https://doi.org/
10.1016/j.patrec.2025.09.004.
[6] M. Das, A. Mukherjee, Banglaabusememe: A dataset for bengali abusive meme classification, in:
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023,
pp. 15498–15512.
[7] W. Soral, M. Bilewicz, M. Winiewski, Exposure to hate speech increases prejudice through
desensitization, in: Aggressive behavior 44, 2, 2018, pp. 136–146.
[8] A. Olteanu, C. Castillo, J. Boy, K. Varshney, The efect of extremist violence on hateful speech
online, in: Proceedings of the International AAAI Conference on Web and Social Media, volume 12,
2018.
[9] Y. Chen, Y. Zhou, S. Zhu, H. Xu, Detecting ofensive language in social media to protect adolescent
online safety, in: 2012 International Conference on Privacy, Security, Risk and Trust and 2012
International Confernece on Social Computing. IEEE, 2012, pp. 71–80.
[10] Z. Waseem, D. Hovy, Hateful symbols or hateful people? predictive features for hate speech
detection on twitter, in: Proceedings of the NAACL student research workshop, 2016, pp. 88–93.
[11] T. Davidson, D. Warmsley, M. Macy, I. Weber, Automated hate speech detection and the problem
of ofensive language, in: Proceedings of the International AAAI Conference on Web and Social
Media, volume 11, 2017.
[12] R. Kumar, A. K. Ojha, S. Malmasi, M. Zampieri, Benchmarking aggression identification in social
media, in: Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying
(TRAC2018), 2018, pp. 1–11.
[13] A. M. Founta, C. Djouvas, D. Chatzakou, I. Leontiadis, J. Blackburn, G. Stringhini, A. Vakali,
M. Sirivianos, N. Kourtellis, Large scale crowdsourcing and characterization of twitter abusive
behavior, in: Twelfth International AAAI Conference on Web and Social Media, 2018.
[14] P. Badjatiya, S. Gupta, M. Gupta, V. Varma, Deep learning for hate speech detection in tweets,
in: Proceedings of the 26th international conference on World Wide Web companion, 2017, pp.
759–760.
[15] M. Das, P. Saha, R. Dutt, P. Goyal, A. Mukherjee, B. Mathew, You too brutus! trapping hateful users
in social media: Challenges, solutions &amp; insights, in: Proceedings of the 32nd ACM Conference
on Hypertext and Social Media, 2021, pp. 78–89.
[16] G. K. Pitsilis, H. Ramampiaro, H. Langseth, Detecting ofensive language in tweets using deep
learning, in: ArXiv abs/1801.04433, 2018.
[17] Z. Zhang, D. Robinson, J. Tepper, Detecting hate speech on twitter using a convolution-gru based
deep neural network, in: European semantic web conference. Springer, 2018, pp. 745–760.
[18] S. Banerjee, M. Sarkar, N. Agrawal, P. Saha, M. Das, Exploring transformer based models to
identify hate speech and ofensive content in english and indo-aryan languages, in: arXiv preprint
arXiv:2111.13974, 2021.
[19] T. Mandl, S. Modha, G. K. Shahi, H. Madhu, S. Satapara, P. Majumder, J. Schaefer, T. Ranasinghe,
M. Zampieri, D. Nandini, Overview of the hasoc subtrack at fire 2021: Hate speech and ofensive
content identification in english and indo-aryan languages, in: arXiv preprint arXiv:2112.09301,
2021.
[20] B. R. Chakravarthi, R. Priyadharshini, N. Jose, T. Mandl, P. K. Kumaresan, R. Ponnusamy, R.
Hariharan, J. P. McCrae, E. Sherly, Findings of the shared task on ofensive language identification in
tamil, malayalam, and kannada, in: Proceedings of the First Workshop on Speech and Language
Technologies for Dravidian Languages, 2021, pp. 133–145.
[21] R. Kumar, A. K. Ojha, M. Zampieri, S. Malmasi, Proceedings of the first workshop on trolling,
aggression and cyberbullying (trac-2018), in: Proceedings of the First Workshop on Trolling,
Aggression and Cyberbullying (TRAC-2018), 2018.
[22] T. Mandl, S. Modha, P. Majumder, D. Patel, M. Dave, C. Mandlia, A. Patel, Overview of the hasoc
track at fire 2019: Hate speech and ofensive content identification in indo-european languages,
in: Proceedings of the 11th Annual Meeting of the Forum for Information Retrieval Evaluation,
FIRE ’19, Association for Computing Machinery, New York, NY, USA, 2019, p. 14–17. URL: https:
//doi.org/10.1145/3368567.3368584. doi:10.1145/3368567.3368584.
[23] N. Romim, M. Ahmed, H. Talukder, M. S. Islam, Hate speech detection in the bengali language: A
dataset and its baseline evaluation, in: Proceedings of International Joint Conference on Advances
in Computational Intelligence. Springer, 2021, pp. 457–468.
[24] A. Bohra, D. Vijay, V. Singh, S. S. Akhtar, M. Shrivastava, A dataset of hindi-english code-mixed
social media text for hate speech detection, in: Proceedings of the second workshop on
computational modeling of people’s opinions, personality, and emotions in social media, 2018, pp.
36–41.
[25] M. P. Akhter, Z. Jiangbin, I. R. Naqvi, M. Abdelmajeed, M. T. Sadiq, Automatic detection of
ofensive language for urdu and roman urdu, in: IEEE Access 8 (2020), 2020, pp. 91213–91226.
[26] T. Ranasinghe, M. Zampieri, Multilingual ofensive language identification with cross-lingual
embeddings, in: arXiv preprint arXiv:2010.05324, 2020.
[27] M. Das, S. Banerjee, A. Mukherjee, Data bootstrapping approaches to improve low resource
abusive language detection for indic languages, in: Proceedings of the 33rd ACM conference on
hypertext and social media, 2022, pp. 32–42.
[28] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, M. Douze, H. Jégou, Fasttext.zip: Compressing
text classification models, arXiv preprint arXiv:1612.03651 (2016).
[29] P. Bojanowski, E. Grave, A. Joulin, T. Mikolov, Enriching word vectors with subword information,</p>
        <p>Transactions of the Association for Computational Linguistics 5 (2017) 135–146.
[30] A. Joulin, E. Grave, P. Bojanowski, T. Mikolov, Bag of tricks for eficient text classification, in:
Proceedings of the 15th Conference of the European Chapter of the Association for Computational
Linguistics: Volume 2, Short Papers, 2017, pp. 427–431.
[31] L. N. Smith, Cyclical learning rates for training neural networks, in: Computer Vision and
Pattern Recognition, arxiv.org, 2017. URL: https://arxiv.org/abs/1506.01186. doi:10.48550/arXiv.
1506.01186.
[32] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional
transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
[33] R. Joshi, L3cube-hindbert and devbert: Pre-trained bert transformer models for devanagari based
hindi and marathi languages, arXiv preprint arXiv:2211.11418 (2022).</p>
      </sec>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>A.</given-names>
            <surname>Bhandari</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S. B.</given-names>
            <surname>Shah</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Thapa</surname>
          </string-name>
          ,
          <string-name>
            <given-names>U.</given-names>
            <surname>Naseem</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.</given-names>
            <surname>Nasim</surname>
          </string-name>
          , Crisishatemm:
          <article-title>Multimodal analysis of directed and undirected hate speech in text-embedded images from russia-ukraine conflict</article-title>
          ,
          <source>in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</source>
          ,
          <year>2023</year>
          , pp.
          <fpage>1994</fpage>
          -
          <lpage>2003</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>Koyel</given-names>
            <surname>Ghosh and Mithun Das</surname>
          </string-name>
          and
          <article-title>Mwnthai Narzary and Saptarshi Saha and Shubhankar Barman and Animesh Mukherjee and Sandip Modha and Debasis Ganguly and Utpal Garain and Sylvia Jaki and Thomas Mandl, Overview of the HASOC Track at FIRE 2025: Abusive Meme Identification - Shadows Behind the Laughter</article-title>
          , in: K. Ghosh,
          <string-name>
            <given-names>T.</given-names>
            <surname>Mandl</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Pal</surname>
          </string-name>
          ,
          <string-name>
            <given-names>S.</given-names>
            <surname>Majumdar</surname>
          </string-name>
          ,
          <string-name>
            <surname>A</surname>
          </string-name>
          . Chakraborty (Eds.),
          <source>Forum for Information Retrieval Evaluation (Working Notes) (FIRE 2025) December</source>
          <volume>17</volume>
          -20, Varanasi , India, CEUR-WS.org,
          <year>2025</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>Koyel</given-names>
            <surname>Ghosh and Mithun Das</surname>
          </string-name>
          and
          <article-title>Sumukh Patel and Nilotpal Bhandary and Alloy Das and Animesh Mukherjee and Sandip Modha and Debasis Ganguly and Utpal Garain and Sylvia Jaki and Thomas Mandl, Overview of the HASOC Track at FIRE 2025: Abusive Meme Identification - Shadows Behind the Laughter</article-title>
          ,
          <source>in: FIRE '25: Proceedings of the 17th Annual Meeting of the Forum for Information Retrieval Evaluation. December</source>
          <volume>17</volume>
          -20, Varanasi , India, Association for Computing Machinery (ACM), New York, NY, USA,
          <year>2025</year>
          .
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>