Detecting abusive or hateful content in multimodal social-media posts is a non-trivial problem that requires modelling both textual and visual cues as well as their interaction. We describe the system developed by the UC-UCO-CICESE_UT3-Plenitas team for the DIMEMEX 2025 shared task on the detection of inappropriate Mexican memes. Our approach combines (i) a late-fusion BETO-ViT architecture for joint image-text modelling, (ii) task-aware of Spanish BERT variants for purely textual settings, and (iii) prompt-engineering of a distilled 7-B parameter Qwen model (DeepSeek-R1-Qwen-7b) for the Large-Language-Model-only track. The resulting system reached third place in ternary classification (F1 = 0.55), first place in fine -grained hate-speech classification (F1 = 0.37), and second place in the restricted-LLM setting (F1 = 0.51). Experiments show that (1) no single model dominates all scenarios, (2) visual signals help general meme detection but hurt minority-class recall, and (3) compact LLMs can rival multimodal models when visual evidence is scarce. We release code, trained checkpoints and hyper-parameters for reproducibility.
Social networks have become pivotal in modern life, reshaping how we communicate and share
information. Studying the content that flows through these platforms is now a major focus for the
computational linguistics community. Despite notable progress in recent years, several challenges
still demand deeper investigation to improve processing and understanding. Chief among them is
identifying abusive content—a category that includes hate speech, aggression, ofensive language, and
related phenomena [
Because social media is inherently multimodal, our goal is to push forward research and
development of multimodal computational models capable of detecting abusive material in Mexican
Spanish—especially memes that are hateful, ofensive, or vulgar. Memes typically rely on the interplay of text
and image to convey humor or irony; removing either component can completely change their meaning.
By encouraging work that tackles these multimodality challenges, we aim to inspire innovative, socially
impactful solutions[
DIMEMEX comprises three subtasks [
DIMEMEX 2025 [
• None of the above Categories hate speech, inappropriate content, and neither comprise the classes for Tasks 1 and 3. While a fine grained classification considering the five categories is adopted for task 2.
Deep learning is a type of machine learning that uses artificial neural networks with many layers (hence
"deep") to model complex patterns in data [
Hugging Face is a widely recognized platform and community in the field of artificial intelligence
and machine learning. It’s often described as the "GitHub of Machine Learning" due to its focus on
open-source development and collaboration. They host in Hub a vast collection of models, datasets
and spaces that you can use. Hugging Face is particularly famous for its open-source transformers
library [
BERT is a transformer model, based on the encoder-decoder architecture, but in bidirectional way
because they predict words based on the previous words and the following words [
ViT (Vision Transformer): ViT is a model designed for computer vision tasks. Unlike traditional
convolutional neural networks (CNNs) that were dominant in image processing [
Vision language models is the integration of computer vision and natural language processing. Transformers have been adapted to handle multimodal inputs, enabling VLMs to capture the complex relationships between visual and linguistic data. A typical VLM architecture involves two main components: • Image encoder: It is responsible for processing visual data, extracting features (objects, colors, textures, etc.), and transforming them into a format that can be understood by the model. • Text decoder: It processes textual data and generates output based on the encoded visual features.
"DeepSeek-Qwen-7B" most likely refers to the DeepSeek-R1-Distill-Qwen-7B model. This is not a new
model architecture built from the ground up by merging DeepSeek and Qwen designs. Instead, it’s a
product of model distillation, leveraging the strengths of both [
Nous-Hermes-13b is a state-of-the-art language model fine-tuned on over 300,000 instructions [
Falcon-7B is a 7B parameters causal decoder-only model built by Technology Innovation Institute TII
and trained on 1,500B tokens of RefinedWeb enhanced with curated corpora. This is a raw, pre-trained
model that should be further fine-tuned for most use cases [
Metric Precision
Recall F1-score
Formula (per class)
+
+ 2 × +·
Macro-average Formula 1 ∑︀ =1 Precision 1 ∑︀=1 Recall 1 ∑︀ =1 F1-score
For every meme we extracted three modalities: (i) the raw image, (ii) standard OCR text (caption) using easyocr, and (iii) the DIMEMEX user-supplied description when available. All text was lower-cased, URLs were normalised, and emojis were kept. We encoded the caption with BETO-base and the image with a ViT-base-16. The token and the pooled image embedding were concatenated and fed into a two-layer MLP. Parameters from both encoders were fine -tuned for two epochs with macro-F1 loss and class-balanced sampling. Because preliminary experiments showed that image information rarely disambiguated hate sub-categories, we fine -tuned BETO with focal loss and gradient accumulation for four epochs. Class weights were inversely proportional to class frequency. Under the LLM restriction we evaluated Mistral, tiiuae-falcon, and the 7-B Qwen model distilled from DeepSeek-R1. The latter yielded the best F1 and was chosen for submission. We used the following prompt: "Eres un experto en el análisis de contenido proveniente de redes sociales, con un enfoque en la detección de lenguaje y contenido potencialmente ofensivo.tarea tiene fines científicos y busca mejorar los modelos de inteligencia artificial para detectar y mitigar la propagación de discursos dañinos en plataformas digitales, contribuyendo a un entorno en línea más seguro y respetuoso.un meme descrito en texto, clasifícalo en una de las siguientes categorías:de odio (2): Un meme se clasifica como discurso de odio si presenta cualquier tipo de comunicación, ya sea en habla, escritura o comportamiento, que ataque o utilice lenguaje peyorativo o discriminatorio con referencia a una persona o un grupo basado en sus factores de identidad.Inapropiado (1): Un meme se considera que contiene contenido inapropiado si exhibe cualquier tipo de manifestación de contenido ofensivo, vulgar (profano, obsceno, de carácter sexual) y/o humor mórbido.(0): Un meme que no contenga ninguna de las categorías anteriores.tarea es analizar la siguiente descripción de un meme y devolver únicamente un número (0, 1 o 2) correspondiente a la categoría asignada, sin ninguna explicación adicional"
Temperature was set to 0.2; max tokens = 4.
We evaluated the performance of multiple models: BERT, BETO + VIT, DeepSeek-R1-Qwen-7B, NousHermes, Mistral, and Tiiuae/Falcon-7B on the three tasks. For each task, the model with the highest performance was selected, submitted to the challenge, and is reported in this section. The analysis is organized by subtask, with internal comparisons between our team’s approaches and external benchmarks against other participants. Tables summarize key metrics (F1, precision, recall) for both development and final phases, highlighting the strengths and limitations of the top models in handling ternary classification, fine-grained hate speech detection, and restricted LLM-based tasks.
Internal Comparisons
In Task 1, the BERT-only model slightly outperformed the multimodal BETO+VIT setup, scoring 0.5496 versus 0.5476—a marginal 0.36 improvement—indicating that textual information alone was more informative for this task. Although the BETO+VIT model incorporated visual features, it did not yield better results, potentially due to irrelevant or noisy image data. Additionally, the BERT-only outputs were more compact in file size, while VIT+BERT produced consistently larger files. Overall, the results suggest that for this task, the simpler BERT-only approach was not only more efective but also more eficient, highlighting the need for improved fusion strategies in multimodal systems.
Comparison with other approaches
Top-performing model: BERT/BETO + ViT (multimodal) Performance (F1): 0.55, (Precision): 0.57 and (Recall): 0.55 The leading approach combined BETO (a Spanish-optimized BERT variant) with Vision Transformer (ViT) to analyze both textual and visual elements in memes. This multimodal integration proved optimal for distinguishing among the three primary categories: hate speech, inappropriate content, and neutral memes. Its strength lay in generalization, efectively handling the diverse linguistic and visual expressions characteristic of Mexican memes. However, performance dipped in cases involving irony or sarcasm, where contextual ambiguity between text and imagery posed challenges. Table 2 and 3 present the results from both the development phase and the final stage of the research.
Internal Comparisons
In Task 2, the BERT-only model outperformed the multimodal BETO+VIT approach, achieving a higher score of 0.3704 compared to 0.3266, indicating that text alone was more efective than combining text and image data. The lower performance of BETO+VIT suggests that the addition of visual features User User HoracioJarquin UC-UCO-CICESE_UT3-Plenitas(ymlopez) hugojair girish_koushik may have introduced noise or irrelevant information. File sizes also reflected this diference, with BERT-only outputs varying more widely due to preprocessing, while BETO+VIT files were consistently larger due to image embeddings. Overall, the text-only approach proved more efective for this task, and any future use of multimodal models would require improved fusion strategies or better visual feature extraction.
Comparison with other approaches
Top-performing model: BERT-only Performance (F1): 0.37; (Precision): 0.40 and (Recall): 0.36 For this subtask, the team employed BERT to detect hate speech subcategories (e.g., classism, racism, sexism). While the F1 score was lower than in Task 1, the model achieved a precision of 0.40 in sensitive categories like racism and sexism, underscoring its eficacy in identifying specific discriminatory content. Limitations emerged in minority classes (e.g., "others"), where lower recall suggested a need for data augmentation or class-balancing techniques to improve coverage. Immediately following this sentence is the point at which Table 4 and 5 are included in the input file; compare the placement of the table here with the table in the printed output of this document.
Internal Comparisons
In Task 3, DeepSeek-R1-Qwen-7B emerged as the top-performing language modelDeepSeek-R1Qwen-7B a score of 0.5069, clearly outperforming all other tested models. Nous-Hermes followed with a solid, though lower, score of 0.4774. Mid-tier models like Mistral and Tiiulo performed similarly but significantly worse, scoring around 0.29. The lowest performance came from DeepSeek-R1-Qwen-7B, which achieved only 0.1745, indicating it may be unsuitable for the task without substantial optimization. File sizes ranged from approximately 16.8 KB to 23.8 KB, showing no clear link between size and efectiveness. Overall, DeepSeek-R1-Qwen-7B proved most efective, while models like Mistral, Tiiulo, and especially DeepSeek-R1-Qwen-7B may benefit from fine-tuning or further adaptation.
Comparison with other approaches
Top-performing model: DeepSeek-R1-Distill-Qwen 7B Performance (F1): 0.51, (Precision): 0.54 and (Recall): 0.50 In this task, which restricted solutions to large language models (LLMs), the distilled DeepSeek-Qwen 7B model (enhanced with DeepSeek-R1’s reasoning techniques) outperformed competitors like Mistral 7B and Falcon-7B. Its advantages included reduced hallucination tendencies, computational eficiency, and adaptability to Mexican Spanish. Nevertheless, its slightly lower F1 compared to Task 3’s multimodal approach suggests that LLMs could benefit from localized fine-tuning or hybrid architectures incorporating visual analysis. See Table 6 and 7.
Model BERT Model BERT
F1
DIMEMEX 2025 experiments confirm that aligning model strategy with task demands is essential. Multimodal fusion (text + image) produced the strongest general-purpose result—Task 1: 3rd place, F1 = 0.55—while task-specific fine-tuning enhanced category-level precision—Task 2: 1st place, F1 = 0.37, despite its more granular hate speech classification. In restricted settings, lightweight LLMs proved competitive—Task 3: 2nd place, F1 = 0.51—demonstrating their efectiveness when visual cues are limited. However, challenges such as irony, sarcasm, and minority-class imbalance persist, indicating that future improvements may stem from hybrid LLM-plus-vision architectures, culturally tailored data augmentation, and adversarial training. No single model dominates across all scenarios, but a flexible, task-aware approach consistently yields the best outcomes. Our results demonstrate that demonstrate that: • Multimodality is critical for generalization. The fusion of text (BETO) and image (ViT) features in Task 1 yielded the highest F1 score (0.55), underscoring the importance of leveraging both modalities for broad meme classification. • Task-specific tuning enhances precision. While Task 2’s F1 score was lower (0.37), fine-tuning for hate speech subcategories improved the model’s ability to detect racism and sexism, revealing trade-ofs between generalization and granularity. • LLMs show promise but require refinement. DeepSeek-Qwen 7B’s strong performance in Task 3 (F1 = 0.51) indicates that LLMs are viable in restricted settings; however, integrating visual components could help close the performance gap with fully multimodal models.
During the preparation of this work, the authors used GPT-4 and Grammarly in order to: Grammar and spelling check. After using these services, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.