This paper presents our system developed by our research team at FiRC-NLP for the HASOC-meme 2025 shared task, which focuses on detecting hate speech and ofensive content in multilingual memes. We explore three distinct approaches: prompt-based inference using Gemini, cross-modality encoders combining image and text, and a text-only modality leveraging Optical Character Recognition (OCR), OCR English translation and image descriptions. Our final ensemble system fuses these modalities, achieving macro 1-scores up to 67.50 on test sets and top-1 ranking in 3 out of 4 tracks, demonstrating the efectiveness of multimodal fusion and robust training strategies.
Memes pose unique challenges for hate speech detection due to their multimodal nature [
Recent advances in multimodal machine learning have motivated research into systems capable of
integrating textual and visual signals for classification tasks. Prior work has explored multimodal fusion
for hate detection [
In this paper, we present our system developed for the shared task, which combines multiple modalities and fusion strategies to maximize semantic coverage. Specifically, we explore three complementary approaches: (i) prompt-based inference with large multimodal models, (ii) cross-modality encoders
CEUR Workshop
ISSN1613-0073 jointly processing images and text, and (iii) text-only pipelines enhanced with optical character recognition (OCR), translation into English, and image description generation. We further investigate ensemble strategies that integrate these modalities, demonstrating that their combination yields significant performance improvements over single-modality baselines.
Our contributions are threefold: • We design and evaluate distinct modality-specific pipelines for meme classification. • We propose a fusion strategy that leverages the complementary strengths of prompt-based, cross-modal, and text-enriched approaches. • We provide an empirical analysis of system performance on the HASOC-meme dataset, achieving top-1 ranking in three out of four tracks of the competition.
These findings underline the importance of multimodal integration for robust hate speech detection in memes and suggest promising directions for future research, such as adaptive fusion strategies and meme-specific captioning models.
The HASOC shared task series (Hate Speech and Ofensive Content Identification) has reached its seventh
edition in 2025. The HASOC-meme 2025 [
The 2025 edition introduces a five-part classification challenge, requiring participants to analyze memes across multiple dimensions of ofensive communication: • Positive: The meme conveys support, humor, or appreciation. • Neutral: The meme is neither overtly positive nor negative.
• Negative: The meme expresses hostility, mockery, or criticism.
Abuse Detection • Abusive: The meme includes harmful, derogatory, or ofensive elements targeting individuals or groups. • Non-Abusive: The meme lacks abusive or derogatory content. • Sarcastic: The meme implies the opposite of its literal meaning, often mocking or ridiculing. • Non-Sarcastic: The meme conveys its message directly, without irony. • Vulgar: The meme contains explicit or ofensive words, gestures, or depictions. • Not Vulgar: The meme does not include such content.
Target Community Identification • Gender: Mentions gender identities (male, female, non-binary, transgender). • Religion: References to religious beliefs, practices, or symbols. • Individual: Mentions or depicts a specific person. • Political: Refers to political parties, ideologies, politicians, or policies. • National Origin: Targets groups by country or ethnicity. • Social Sub-groups: Refers to socio-economic, occupational, or cultural groups. • Others: Any community not covered above.
• None: No explicit target community.
Through this multi-dimensional design, HASOC-meme 2025 emphasizes the complexity of abusive content detection, encouraging participants to go beyond binary classification and address fine-grained aspects such as sarcasm, sentiment, and vulgarity.
We used the oficial HASOC-meme dataset. To evaluate and compare diferent strategies, we employed 5-fold cross-validation, partitioning the training data into five subsets and iteratively using four folds for training and one for validation.
While the organizers provided baseline OCR text, we hypothesized that a more detailed extraction of
visual and textual elements would improve performance [
This process yielded a rich set of features for each meme: the original image, a comprehensive OCR text (in its native language), an English translation of the OCR, and a structured description of the visual elements.
One of the key challenges in this work was building a strong classifier from limited training data in low-resource languages. To address this, we unified all datasets into a single multilingual corpus and explored two complementary strategies: (1) prompt-based inference using large multimodal foundation models, and (2) fine-tuning compact multi- and single-modal encoders. Our system integrates multiple components, detailed in the following sections:
Our core methodology revolved around prompt-based inference using a powerful multimodal foundation model. We explored both zero-shot and few-shot paradigms to leverage the model’s capabilities.
Our initial step was to select the most suitable foundation model. We conducted a preliminary evaluation on a small subset of 20 training samples, testing OpenAI’s GPT-4, Anthropic’s Claude 3 Opus, and Google’s Gemini 2.5 Flash. Each model was prompted to classify the meme images across the four target labels. In this comparison, Gemini 2.5 Flash demonstrated consistently superior performance, leading us to select it for all subsequent experiments.
Our first strategy was a zero-shot approach, designed to test the model’s intrinsic understanding of the task without any prior examples. The model was provided with the raw meme image and a carefully crafted prompt instructing it to classify the image according to the four labels (sentiment, abuse, vulgar, sarcasm). The prompt 1 also required the model to provide a brief explanation for its decision, which ofered valuable qualitative insights into its reasoning process. This approach used no samples from our prepared training set.
Our second, more advanced strategy was a few-shot approach that provided the model with relevant examples from the training set to guide its prediction. This method, often referred to as RetrievalAugmented Generation (RAG), relied on our extracted features rather than the raw images. The process was as follows: 1. Vector Database Creation: We used a Gemini text embedding model to generate vector embeddings for the English-translated OCR and the image descriptions of all samples in the training set. These embeddings were stored in a Chroma vector database. 2. Dynamic Example Retrieval: For each new sample, we performed two similarity searches against the vector database to retrieve the top 5 training examples with the most similar translated OCR and the top 5 examples with the most similar image description. 3. Prompt Engineering: The retrieved 10 examples, along with their ground-truth labels, were concatenated and formatted into the prompt. The model was then tasked with classifying the target sample based on this contextual information. Prompt 2
We fused the vision encoder from google/siglip-base-patch16-224 (SigLIP) [
We used the OCR of the images and applied Gemini (using Prompt 3) to generate detailed image
descriptions and translate the content into English. The outputs from Gemini and the OCR process
were then fed into two language models: XLM-R and l3cube-pune/bengali-bert (l3cube-bangla) [
XLM-R l3Cube-Bangla
AbuseJoint ClassifiVcuatligoanr Head
JCoiLnSt PColoalsinsigfication Head
For the final submission, we adopted a hybrid selection strategy to maximize the overall macro 1score. For each of the 16 tasks (4 languages × 4 labels), we compared the performance of all our methods, including zero-shot LLM, few-shot LLM, and our encoder-based models in the internal validation set.
The approach that yields the highest validation 1 score for a specific task was selected to generate the final prediction for the test set. For encoder-based models, we further optimized the 1 score by setting a prediction threshold for each label based on its positive class distribution in the training data. This ensured that the predicted label ratios were aligned with the observed data. This strategic combination allowed us to take advantage of the distinct strengths of each modeling technique.
For the fine-tuned models, we employed 5-fold cross-validation, where four splits were used for training and one split for validation (Valid) in each fold. At the end of training, predictions from the five folds were combined into an ensemble to score the test set. All models were implemented using the HuggingFace Transformers framework and optimized with early stopping to prevent overfitting. A constant learning rate schedule with warmup was applied to enhance training stability and overall performance.
We trained the models on four out of the five classes (sentiment, sarcasm, vulgarity, and abusive), excluding the target community class since it was not requested in the final submission and, in preliminary experiments, its inclusion degraded performance due to potentially low-quality annotations.
We evaluate the performance of multiple approaches across Bangla, Bodo, Gujarati, and Hindi, reporting macro 1-scores on both validation and test sets. Table 2 presents the performance comparison of diferent approaches.
Language → Approach ↓ Backbone Prompt-Based (Zero-Shot) Gemini Prompt-Based (Few-Shot) Gemini Cross-Modality Encoder XLM-R + SigLIP Text Modality (OCR only) XLM-R + l3cube-Bangla Text Modality (OCR + Image Description + OCR Translation) XLM-R + l3cube-Bangla Combined Ensemble
Bangla Training set Valid Test - 58.10 59.45 all 52.34 54.04 all 55.95 56.48 all 58.56 59.48 all 61.02 59.91 Bodo - all - 62.75
Zero-shot prompt-based approaches using Gemini perform surprisingly well for the test set of the Bangla language, outperforming the fine-tuned models. However, when applied to other languages, its performance tends to be more modest. On the other hand, few-shot prompting occasionally outperforms zero-shot prompting but not consistently. This variability makes it challenging to draw a clear conclusion about its overall efectiveness.
Approaches leveraging text-only encoders (XLM-R + l3cube-Bangla) generally outperform prompt-based
strategies, particularly for Gujarati and Hindi. However, their performance varies across languages,
with weaker results observed for Bodo. The performance on Bodo was notably low when the model
was trained jointly with other languages. We hypothesize that this may be due to inconsistencies
in the annotation of Bodo data, warranting further investigation of the annotation process. Similar
low-resource challenges were also discussed in prior shared task analyses [
Models combining multimodal cues (OCR, image descriptions, and OCR translations) consistently outperform text-only methods. This fusion strategy leads to the strongest results, indicating that meme-based hate speech detection benefits substantially from integrating multimodal signals.
The ensemble approach achieves the best overall macro 1-score. These results confirm that ensemblebased strategies, which leverage diverse models and representations, provide robust improvements across all languages.
Our system demonstrates that combining prompt-based reasoning, cross-modality encoding, and enriched text-only pipelines yields strong performance in meme-based hate speech detection.
Future work will focus on exploring dynamic fusion strategies and fine-tuning captioning models for meme-specific semantics. In addition, we plan to investigate encoder fusion using cross-attention mechanisms, as the current implementation relies solely on late aggregation (where inputs are independently processed through encoders and fused only at the end). Cross attention has the potential to capture richer cross-encoder interactions and improve overall system efectiveness.
We would like to thank the HASOC team and all participants for their eforts in organizing and contributing to the shared task.
During the preparation of this work, the author(s) used GPT-4-based tools in order to: Grammar and spelling check. After using this tool, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.
Listing 1: Prompt used for the Zero-Shot approach prompt: >
You are a helpful content moderator. You need to analyze meme images in Bangla, Hindi,
Gujarati, or Bodo.
You should detect: 1) sentiment (one from 3 values): - positive - The meme conveys a supportive, humorous, or appreciative tone. - neutral - The meme is neither overtly positive nor negative in tone.
- negative - The meme expresses hostility, mockery, or criticism. 2) sarcasm (True or False):
- True - The meme presents statements or visuals that imply the opposite of their literal meaning, often to mock or ridicule.
- False - The meme directly conveys its message without sarcasm or irony. 3) vulgar (True or False): - True - The meme contains explicit or offensive words, gestures, or depictions. - False - The meme does not include any such content. 4) abuse (True or False):
- True - The meme includes offensive, harmful, or derogatory language, imagery, or implications targeting an individual or a group.
- False - The meme does not contain any offensive, harmful, or derogatory content. 5) target - (one from 8 values): - "gender" - Any reference to male, female, non-binary, or transgender identities. - "religion" - Mentions or imagery related to any religious belief, deity, or practice. - "individual" - Specifically mentions or portrays a particular person. - "political" - Targets political ideologies, parties, politicians, or policies. - "national" - Targets people based on their country or ethnicity.
- "social subgroups" - Groups based on socio-economic status, occupation, cultural identity, or other affiliations.
- "other" - Any target that does not fall into the above categories.
- "non-targeted" - If the meme does not target any specific community, no target label is assigned. 6) description - finally provide a short explanation of why you make these predictions.
Listing 2: Prompt used for the Few-Shot approach prompt_few_shots: >
You are a helpful content moderator. You need to analyze meme images in Bangla, Hindi,
Gujarati or Bodo.
I will provide you description of every part of the meme (images and texts) You should detect: 1) sentiment (one from 3 values): - positive - The meme conveys a supportive, humorous, or appreciative tone. - neutral - The meme is neither overtly positive nor negative in tone.
- negative - The meme expresses hostility, mockery, or criticism. 2) sarcasm (True or False):
- True - The meme presents statements or visuals that imply the opposite of their literal meaning, often to mock or ridicule.
- False - The meme directly conveys its message without sarcasm or irony. 3) vulgar (True or False): - True - The meme contains explicit or offensive words, gestures, or depictions. - False - The meme does not include any such content. 4) abuse (True or False):
- True - The meme includes offensive, harmful, or derogatory language, imagery, or implications targeting an individual or a group.
- False - The meme does not contain any offensive, harmful, or derogatory content. Use the examples below as a guide: {examples} Meme description to analyze: {meme_description}
Text description: {text_description} description_ocr_template: >
OCR from a meme: {ocr} example_template: > -- {description} Ground Truth: {ground_truth}
Listing 3: Prompt used for Image OCR + Image Description + OCR Translation extraction ocr_prompt: >
You are an expert at analyzing and describing internet meme images in Bangla, Hindi, Gujarati, or Bodo. Your task is to extract the text and provide a concise description of the image content for each distinct part of a meme.
You need to return a list of descriptions of every part of the meme. For every part (subimage) , return the location of the part, image_description and text in the original language it exists on this part and translated_text of this text to the English language (if the original text is English, then return the same text). Your example output: [{"location": "Top", "image_description": None, "text": "work in IT", "translated_text": " work in IT"}, {"location": "right bottom image", "image_description": "picture of software developer working at night", "text": "I will work only when I want", "translated_text": "I will work only when I want"}, {"location": "left bottom image", "image_description": " picture of tired and angry software developer", "text": "And also when my boss will ask me", "translated_text": "And also when my boss will ask me"}]