Visual Question-Answering (VQA) is a complex multimodal task that requires integrating visual recognition and natural language understanding to answer questions about images. While significant progress has been made in English, resources and models for non-English languages, such as Italian, remain scarce. This paper addresses this gap by evaluating MiniCPM-V 2.6, a state-of-the-art multimodal Large Language Model, on GQA-it, the ifrst large-scale Italian VQA dataset. The primary goal of this work is to investigate the performance of such models when applied of-the-shelf to this task and, if unsatisfactory, to explore how much they can improve with ifne-tuning on Italian data. When applied of-the-shelf, MiniCPM-V 2.6 achieves an accuracy of 33.4%. However, after fine-tuning it on the GQA-it dataset, the performance improves significantly, reaching a state-of-the-art accuracy of 59.4%. These findings highlight the importance of language-specific adaptation in multilingual VQA tasks, especially for under-resourced languages like Italian. The trained model is released to the community on a dedicated Huggingface repository: https://huggingface.co/sag-uniroma2/MiniCPM-V-2_6-gqa-it-finetuned.
Visual Question-Answering (VQA) is a challenging and rapidly evolving task in the field of Artificial
Intelligence (AI). It requires a system to provide an accurate answer to a question posed in natural
language based on the visual content of an image. The question typically depends on various details
within the image, such as objects, relationships, actions, or other visual attributes, and demands that
the system integrates both visual recognition and language understanding to generate a correct and
contextually relevant response [
Figure 1 shows an example image, for which a wide range of questions can be asked, each with its respective answer, between brackets, in both English and Italian: • Q(A): Is the remote to the right or to the left of the book? (right). • Q(A): Il telecomando è a destra o a sinistra del libro? (destra) • Q(A): How thick is the book to the left of the remote? (thick). • Q(A): Quanto è spesso il libro a sinistra del telecomando? (spesso) • Q(A): What device is to the left of the calculator made of plastic? (charger). • Q(A): Quale dispositivo si trova a sinistra della calcolatrice di plastica? (caricabatterie) • Q(A): What’s the charger made of ? (plastic). • Q(A): Di cosa è fatto il caricabatterie? (plastica) • Q(A): Are there any phones? (no).
• Q(A): Ci sono dei telefoni? (no).
The task is particularly complex because an image can be rich in information (in this case, a large
number of objects), express actions, colors, and sensations, and be the subject of many diferent questions,
as demonstrated by the small number of examples listed above. Some questions (such as those about
the color of an object or its position) can be answered by simply observing the image, while others
may require prior knowledge (such as understanding the typical materials of a charger). Additionally,
some questions could be dificult to answer because it is unclear what the observer’s focus is, some may
have no answer (for example, if the question concerns an object not present in the image), or certain
questions might require follow-up clarifying questions to provide an adequate response, necessitating a
dialogical interaction [
As a benchmark for evaluating the efectiveness of AI systems, VQA has garnered increasing attention
due to its role in assessing how well these systems can perform tasks that involve deeper semantic
understanding and cross-modal reasoning between language and vision. In recent years, several models
and resources have been developed to address the challenges of VQA, particularly for the English
language. These advancements have contributed to significant breakthroughs in the field [
Among the most successful approaches, Multimodal Large Language Models (MLLMs) have shown
significant promise [
Some of the most promising Multimodal LLMs, such as LLaVA [
In most cases, these models reuse and specialize existing LLMs that have been pre-trained on
largescale document collections and then fine-tuned within the final multimodal architecture [
This paper aims to answer the following research questions:
• How do state-of-the-art models for VQA and other multimodal tasks involving both images and text
perform when faced with a VQA problem in a non-English language?
• Can these models be used of-the-shelf, or would they benefit from—or even require—further
finetuning on annotated Italian data?
To explore these questions, we utilize the GQA-it dataset proposed by [
In this paper, we evaluate how MiniCPM-V [
Our experimental results provide interesting insights into the research questions posed. MiniCPM-V, when evaluated in its base form, i.e. zero-shot, achieves an accuracy of 33.4%, demonstrating that while the model possesses general multimodal capabilities, it struggles to efectively handle VQA tasks in Italian without any additional adaptation. This aligns with our first research question, indicating that state-of-the-art models may not perform optimally in non-English contexts when used of the shelf. However, after fine-tuning the model over the GQA-it dataset, its performance dramatically improves, reaching a state-of-the-art accuracy of 59.4%. This significant improvement addresses our second research question, highlighting that fine-tuning seems crucial for enhancing the model’s ability to handle Italian, as it allows the model to better understand the linguistic and visual nuances of the target language. The results emphasize that even advanced models require further adaptation to achieve competitive performance in multilingual VQA tasks. These experimental findings underscore the importance of language-specific adaptation in the development of multimodal models, particularly for under-resourced languages like Italian. Even state-of-the-art open-source models, such as MiniCPM-V, struggle to generalize efectively to non-English languages without targeted fine-tuning, as evidenced by the marked performance disparity observed in our GQA-it evaluation. This reinforces the need for dedicated resources and model adjustments to enable competitive performance in multilingual settings, highlighting that fine-tuning is not just beneficial but often necessary to overcome the inherent bias toward English in many pre-trained multimodal models.
The rest of the paper is organized as follows: Section 2 describes the GQA-it dataset, while Section 3 discusses the investigated Multimodal LLMs (MLLMs). The experimental evaluation is presented in Section 4, and conclusions are drawn in Section 5.
GQA-it [
GQA-it mirrors this structure, translating the questions and answers into Italian. The GQA dataset
itself provides a rich framework for visual question-answering by incorporating multi-step reasoning,
compositional questions, and balanced answer distributions to prevent models from exploiting dataset
biases [
The GQA-it dataset consists of more than 1 million question-answer pairs and preserves the structure and balance of the original GQA dataset. Each question is designed to assess various aspects of visual understanding, including object recognition, spatial reasoning, and scene comprehension, making GQA-it a comprehensive benchmark for evaluating VQA systems. In Figure 1, an example image is displayed along with five question-answer pairs, showcasing the type of visual reasoning tasks that can be posed based on the image content, both in Italian and English. This highlights the dataset’s ability to test models across multiple linguistic and visual dimensions.
In the creation of GQA-it, Neural Machine Translation (NMT) was employed to automatically translate the English questions and answers into Italian. Although NMT achieved high-quality translations, manual validation was applied to a subset of 3,000 examples to ensure the reliability of the test data. This validation process involved correcting errors related to gender inflection, lexical ambiguity, and inconsistencies in the translation of answers, which were often sensitive to the context provided by the corresponding question and image.
Dataset Split train validation test-dev (silver) test-dev (gold) #Images #Question-Answer Pairs 72,140 943,000 10,234 132,062 398 12,578 398 3,000
As shown in Table 1, GQA-it is divided into training, validation, and test sets, with the test set further split into silver (automatically translated) and gold (manually validated) subsets. The silver test set comprises automatically translated questions and answers, while the gold test set contains manually corrected samples, providing a high-quality evaluation benchmark for VQA models. GQA-it poses significant challenges for models, as it requires a deep understanding of both the Italian language and the visual content. Furthermore, the dataset supports a wide range of visual reasoning tasks, including object detection, relationship identification, and attribute recognition. This makes GQA-it a valuable resource for advancing research in multilingual multimodal AI.
Multimodal input signals enable virtual or physical agents to perceive and interact with their environments in more meaningful ways. One increasingly explored area in this domain is the conjunction of text and images. A seminal contribution to this field is CLIP [ 24], an architecture that takes a text and an image as input and originally produces a similarity score between the two. The architecture is optimized using a Contrastive learning schema, which defines a specific contrastive loss: this is minimized if the text accurately describes the image and maximized if the two inputs represent entirely diferent topics. The strength of CLIP lies in its ability to bring similar images and texts closer in a shared latent space while pushing dissimilar pairs far apart. This is achieved in a supervised manner, meaning that the training dataset must be annotated. Building upon CLIP, BLIP [25] was introduced to bootstrap the architecture from a pre-trained model in an unsupervised fashion, leveraging web image data along with their captions. This approach eliminates the need for large amounts of annotated data. Both CLIP and BLIP, however, depend on a global comparison of image-text similarities across a batch, even if the pairs are not directly related. As a solution, Sigmoid loss for Language-Image Pre-training (SigLIP) [26] was developed. Unlike the contrastive learning employed by CLIP, which uses Softmax normalization and relies on global pairwise similarity comparisons, SigLIP introduces a sigmoid loss that operates only on individual image-text pairs. This approach significantly enhances performance while reducing the dependence on large batch sizes for normalization. A major limitation of contrastive loss in CLIP is its assumption of a single correct image-text pairing for each example. However, in real-world scenarios, images and captions often have multiple plausible associations. Sigmoid loss addresses this by computing binary cross-entropy for each potential image-text pair, framing the problem as multi-label classification. This allows the model to score multiple relevant image-text pairs, rather than forcing a single correct match. As a result, Sigmoid loss enables smoother and more flexible alignment between images and texts, efectively handling cases where an image may correspond to several valid captions, or a caption may describe multiple images.
These models are specifically designed to learn the best representations of the two modalities: vision
and text. More recently, architectures such as LLaVA [
LLaVA [
On the other hand, CogVLM [
Finally, a smaller multimodal model called MiniCPM-V 2.6 [
The availability of the above pre-trained models underscores their potential for tackling Visual Question-Answering tasks in languages other than English. To assess their efectiveness in a nonEnglish context, particularly Italian, we leverage the GQA-it dataset. The upcoming section outlines the experimental setup and results, focusing on how MiniCPM-V 2.6 performs in both zero-shot and ifne-tuned scenarios. This comparison aims to determine whether fine-tuning significantly enhances the model’s ability to handle the linguistic and visual complexities of the Italian language, ultimately addressing the research questions of this work.
In this section, we address the research questions posed: how well state-of-the-art multimodal models perform on VQA tasks in non-English contexts, and whether they benefit from fine-tuning on Italianspecific data. To verify this, we apply MiniCPM-V to the GQA-it dataset, comparing its of-the-shelf performance to its results after fine-tuning. The aim is to evaluate the extent to which fine-tuning on the target language enhances the model’s capabilities, and we conclude by analyzing the most common error types observed in each setup.
The aim of the evaluation is to compare MiniCPM-V 2.6 in two configurations: of-the-shelf (zero-shot)
and fine-tuned. For this purpose, the model was assessed using the 3,000 examples from the manually
validated (gold) set of the GQA-it test data (Section 2). This choice ensures the highest quality evaluation,
as the manually validated set provides more reliable and accurate data than the automatically translated
silver set. Additionally, this setup allows us to directly compare the results with LXMERT [21], a
BERT-based model introduced in [
In GQA-it, each question is presented in natural language, and the expected answer is a short expression, typically consisting of one to four tokens. The models are evaluated based on their accuracy, which is the percentage of questions for which the system’s response exactly matches the expected answer. This metric allows for a clear and direct measurement of the performance of the model in providing correct answers.
To evaluate the performance of the MiniCPM-V 2.6 model, we conducted experiments in two distinct scenarios, each designed to assess diferent aspects of the model’s capabilities.
First, the model was applied of-the-shelf in a zero-shot setting, meaning no additional training or
Model
Baseline [
This formulation was designed to be clear and direct, ensuring that the model understood it needed to provide a concise, single-word (or few words) response without adding any extra information.
In the second scenario, we employed a fine-tuned setting, where the MiniCPM-V 2.6 model was further trained on the GQA-it training set before being evaluated. This fine-tuning process involved adapting the model to the specific linguistic characteristics of Italian and the visual question-answering requirements of the dataset. The training was executed using the DeepSpeed framework3 on a cluster of four A100 GPUs, each equipped with 80 GB of memory. The hyperparameters for the fine-tuning were set as follows: the standard fine-tuning parameters were used 4, with a learning rate of 1e-6, and the model was trained for 1 epoch due to the high number of examples. Both the vision and large language model (LLM) components were fine-tuned. The batch size per device during training was set to 4 to reduce memory usage. During the fine-tuning process, the model was trained using a simplified prompt, where only the original question and the image itself were presented:
<IMAGE_HERE> <QUESTION_HERE> It is important to note that the prompt was kept extremely simple during fine-tuning, as additional instructions were unnecessary, given that the same prompt structure would be repeated for all observed questions during both training and model application. This streamlined approach aimed to help the model focus purely on the content of the question without additional instructions. By fine-tuning with this minimal prompt, the model was adapted to the task of responding directly to questions in Italian.
Results are presented in Table 2. The first row reports the baseline model, which, due to the significant imbalance in the dataset, simply predicts the most frequent response (“yes” or “sì”). This naive approach achieves an accuracy of 17.6%, ofering a minimal benchmark for comparison.
The LXMERT-it model, introduced in [
3https://github.com/microsoft/DeepSpeed 4https://github.com/OpenBMB/MiniCPM-V/blob/main/finetune/finetune_ds.sh
Error Type Example(s) Our Object tavola (‘table’) vs sedia (‘chair’) 39% Syn or hyp persona (‘person’) vs donna (‘woman’) 16% Attributes blu (‘blue’) vs nero (‘black’); chiuso (‘closed’) vs aperto (‘open’) 17% Morph. feat. bella (‘beautiful’) vs bello (‘beautiful’); persona 5%
(‘person’) vs persone (‘people’) Actions sta dormendo (‘sleeping’) vs sta sdraiato (‘is lying down’) 3% Spatial feat. destra (‘right’) vs sinistra (‘left ’) 2% Binary si (‘yes’) vs no (‘no’) 18% predominantly trained and evaluated on English and Chinese data, they do not yet generalize efectively to Italian.
However, after fine-tuning on the GQA-it dataset, MiniCPM-V 2.6 achieves a substantial improvement, reaching an accuracy of 59.4%, which surpasses both the baseline, the LXMERT-it model, and its own zero-shot performance. This result highlights the critical importance of fine-tuning it on target language datasets to fully leverage a model’s potential. Fine-tuning plays a pivotal role in improving a model’s performance, especially when dealing with language-specific data, as it allows the model to adapt to the linguistic nuances and complexities of the target language. Moreover, it helps address the specific visual reasoning challenges posed by the dataset, ensuring more accurate and contextually relevant responses.
It is important to recognize that LXMERT-it has at least an order of magnitude fewer parameters compared to MiniCPM-V 2.6. Despite this, MiniCPM-V’s architecture, specifically its LLM component based on Qwen2 [20], benefits from being trained on significantly larger and more diverse datasets. This contributes to its superior performance post-fine-tuning, as the extensive training data enables the model to better handle complex multimodal reasoning tasks and cross-lingual understanding.
Additionally, it is worth noting that the fine-tuning process involved adapting both the language model and the vision model. Further experimentation, where fine-tuning is applied selectively or where techniques such as LoRA (Low-Rank Adaptation) [29] are employed, remains an open area of exploration for future research.
The error analysis aimed to identify the most frequent types of misclassifications made by the system.
The resulting percentages, summarized in Table 3, also include a comparison with the analysis from [
We will focus on discussing the diferences between the two analyses at the end of the section.
The most evident diference appears in binary-type questions, such as those involving yes/no answers. These findings underscore the dificulty in managing seemingly straightforward yet context-dependent queries, where subtle diferences in the phrasing of the question can result in incorrect binary decisions, exposing the limitations of the model in reasoning. For instance, in example 2, when asked: “C’è del vino in questa foto?”5, the model incorrectly responds with “si” (“yes”), when in fact, the image shows wine glasses, but no actual wine. Nonetheless, MiniCPM-V only produces errors in this category in 18% of the examples, compared to 31% for LXMERT-it. This could be attributed to the vision and language model
components of MiniCPM-V working more efectively together, particularly in this category where more advanced reasoning may be required to respond correctly.
In general, the distribution of the remaining error types follows a similar pattern between MiniCPM-V and LXMERT-it. However, one notable exception is the category of object-related errors. In MiniCPM-V, object-related answers account for 39% of the errors, whereas in LXMERT-it, this percentage is slightly lower at 30%. Despite the lower overall number of errors in MiniCPM-V, this high percentage indicates that the model struggles significantly when distinguishing between similar objects, such as “tavola” (“table”) and “sedia” (“chair”). In Figure 3, an example is shown where the model incorrectly answers the question “Quale tipo di mobile è nero?” 6 by predicting “tavola” (table) instead of “sedia” (“chair”). This type of confusion likely arises from the inherent visual or semantic similarities between objects, suggesting that the model’s object recognition or feature diferentiation requires further refinement.
Errors related to attributes were also prevalent, contributing 17% to the overall misclassifications. Examples like confusing “blue” with “black” demonstrate that the model struggles with precise attribute identification, potentially due to issues with color or texture recognition. This suggests that the model may require improvements in fine-grained feature extraction. Errors stemming from synonyms (syn) or hypernyms (hyp) followed closely, at 16%. These included confusion between terms like “donna” and “ragazza” (“woman” vs. “girl”) or “uomo” and “persona” (“man” vs. “person”), pointing to challenges in linguistic nuances and semantic hierarchical relations. This kind of confusion may indicate that the language understanding capability of the model is limited in distinguishing between related (e.g. hyperonyms or hyponyms) but distinct concepts.
Other errors stem from genuine ambiguity. In Figure 4, for instance, the answer for the question “Cosa c’è davanti alla felpa?”7 was annotated as “tappeto” (“carpet”) by the human annotator, while the model answered “scrivania” (“desk”). This discrepancy (both answers could be deemed correct) underscores the challenge of interpreting spatial relationships without additional context. If the model were able to ask clarifying questions, it might inquire, “Could you clarify what you mean by ‘in front’ in this context?”. The human annotator may have considered the observer’s perspective, interpreting ‘in
front’ as the space between himself and the chair, whereas the model might have interpreted ‘in front’ as the position of the sweatshirt on the chair. The ability to seek such clarification could help resolve these ambiguities and enhance the accuracy of spatial understanding in similar scenarios.
Another type of ambiguity can arise from diferences in attention between the system and the annotator. For instance, in Figure 5, when asked “Su cosa è sdraiato il gatto?”8, the annotator provided the annotation “tappetino” (“mouse pad”), while the model answered “scrivania” (“desk”). Once again, the discrepancy highlights a potential divergence in focus: the annotator might be concentrating on a more specific object, such as the mat directly beneath the cat, whereas the model may be considering a larger, more general object, like the desk that the mat is placed on. Such diferences in attentional focus
can contribute to ambiguities in interpreting spatial relationships and objects in images.
This study underscores the role that fine-tuning plays in enhancing the performance of multimodal models, particularly for under-resourced languages. Our analysis of MiniCPM-V 2.6, a state-of-the-art MLLM, demonstrates significant improvements when fine-tuned on Italian-specific datasets like GQA-it. While the zero-shot performance of the model yields a 33.4% accuracy, fine-tuning nearly doubles its accuracy to a state-of-the-art 59.4%. This emphasizes the importance of adapting models to the linguistic characteristics of the target language to fully unlock their potential. For languages like Italian, where available training data is scarce, the impact of language-specific fine-tuning becomes even more critical.
The challenges posed by multilingual Visual Question-Answering (VQA) tasks further support the necessity of language adaptation. Our findings show that models pre-trained predominantly on English data often exhibit limited generalization capabilities in other languages, reinforcing the presence of an English-centric bias in many multimodal pre-trained models. Addressing this bias through targeted ifne-tuning on non-English datasets is crucial for developing AI systems that can operate efectively across multiple linguistic contexts.
Lastly, the comparative analysis highlights that MiniCPM-V 2.6, once fine-tuned, surpasses the previously best-performing Italian VQA model, LXMERT-it, which reached 51.0% accuracy. With a new benchmark of 59.4%, MiniCPM-V 2.6 demonstrates the potential for advanced multimodal LLMs to set state-of-the-art standards for VQA tasks in non-English languages. This success further underscores the value of fine-tuning with relevant language-specific data to achieve competitive performance in diverse linguistic settings.
Claudiu Daniel Hromei is a Ph.D. student enrolled in the National Ph.D. in Artificial Intelligence, XXXVII cycle, course on Health and life sciences, organized by the Università Campus Bio-Medico di Roma. We acknowledge financial support from the PNRR MUR project PE0000013-FAIR and support from Project ECS 0000024 Rome Technopole, - CUP B83C22002820006, NRP Mission 4 Component 2 Investment 1.5, Funded by the European Union - NextGenerationEU.