We introduce MAIA (Multimodal AI Assessment), a multimodal dataset developed as a core component of a competenceoriented benchmark designed for fine-grained investigation of the reasoning abilities of Visual Language Models (VLMs) on videos. The MAIA benchmark is characterized by several distinctive features. To the best of our knowledge, MAIA is the first Italian-native benchmark addressing video understanding: videos were carefully selected to reflect Italian culture, and the language data (i.e., questions and reference answers) were produced by native-Italian speakers. Second, MAIA explicitly includes twelve reasoning categories that are specifically designed to assess the reasoning abilities of VLMs on videos. Third, we structured the dataset to support two aligned tasks (i.e., a statement verification and an open-ended visual question answering) built on the same datapoints, this way allowing to assess VLM coherence across task formats. Finally MAIA integrates, by design, state-of-the-art LLMs in the development process of the benchmark, taking advantage of their linguistic and reasoning capabilities both for data augmentation and for assessing and improving the overall quality of the data. In the paper we focus on the design principles and the data collection methodology, highlighting how MAIA provides a significant advancement with respect to other available dataset for VLM benchmarking. Data available at GitHub.
VLMs, assessing their performance on standardized tasks and metrics is becoming more and more challenging.
In recent years, mainly following the success of large First of all, evaluating VLMs understanding in real
language models (LLMs), there has been a growing in- world scenarios requires moving beyond single-frame
terest for large pre-trained models able to manage both scenarios. Unlike static images, videos ofer rich
temtexts and images. Such Vision and Language models poral structure: they capture dynamic scenes, evolving
(VLMs) have been investigated both from a theoretical actions, interactions, and causal dependencies that
unperspective (e.g., Baroni [
The paper is organized as follows. Section 2 reviews entities, by trying to challenge VLMs in identifying the rethe most relevant prior work in the research area. In lationships between them. As in the case of image-based Section 3, we detail the design choices behind the cre- evaluation, early surveys have already stressed the need ation of the dataset and, more broadly, the development for careful and systematic assessment Zhong et al. [25]. of the entire MAIA benchmark. Finally, Sections 4 and 5 While task-oriented benchmarks often report strong perdescribe the specific steps followed for dataset construc- formance [26, 27], more fine-grained evaluations have tion: the former focuses on the selection and collection of revealed critical limitations [28], and competence-based video material, while the latter addresses the collection analyses continue to highlight the substantial gap in and validation of all linguistic data that constitute MAIA. the video understanding capabilities of VLMs [29]. In Both sections are complemented by dedicated analyses this context, MAIA contributes as a new video-language of the collected data. dataset aimed at evaluating VLMs not only on videos featuring temporal dynamics and meaningful content but also through a competence-oriented design that explores 2. Related Work the interplay between language and vision, a dimension largely neglected in prior Video QA benchmarks.
Multimodal datasets combining vision and language have played a crucial role in the development and evaluation of VLMs. Early image-based resources such as the VQA [6], Italian Multimodal Datasets. Most multimodal GQA [7], DVD [8], and HL [9] datasets have provided con- datasets are available in English, with only limited multitrolled environments to assess visual reasoning and nat- lingual or other native-language resources, with Italian ural language understanding through several tasks like being consistently underrepresented. In the image doImage captioning or Visual Question Answering, thereby main, GQA-it dataset [30] is a notable attempt to adapt reinforcing the role of vision as a fundamental compo- a visual question answering dataset into Italian. More nent in the evaluation of multimodal models [6]. Over recent benchmarks like XGQA [31] and EXAMS-V [32] time, contributions of this kind have been instrumental in include translated Italian multiple-choice questions, but shaping the foundations of multimodal evaluation, where lack original content and do not target high-level reasoning. MAIA fills this gap as the first Italian-native and Q
A
(1Q : 8A) 10 Q
A1 : A8 TS FS
True- False Statements pairs (TS-FS)
...
VLM
TS FS video-language dataset specifically designed to assess complex visual reasoning and grounding.
This section presents the design principles, structure, and construction pipeline of both the MAIA dataset and the benchmark built upon it. In line with this, Figure 1 illustrates the overall workflow adopted for dataset creation, embedding it with the broader architectural framework of the benchmark, which also includes the downstream tasks the data is designed to support.
As shown, the dataset creation begins with the collection of short videos, each associated with twelve highlevel reasoning categories. These categories reflect different semantic phenomena and were chosen to ensure a rich and controlled testing environment for visual and linguistic reasoning. Based on these categories, we constructed our multimodal dataset by first collecting a set of questions that served as the conceptual backbone for the creation of the linguistic data, both manually (i.e. a set of answers) and automatically generated (i.e. True and False statements), as described in detail in Section 5. Figure 2 illustrates an example1 of a MAIA item and highlights the cascading logic behind the data creation process. This architecture supports the development of two aligned 1Although all source data are in Italian, examples are presented in English to enhance readability. evaluation tasks: a Visual Statement Verification task, using paired true/false statements to assess the model’s ability to distinguish accurate from misleading content in a multiple-choice format, and an open-ended Visual Question Answering task, where each question is matched with eight diferent human answers serving as a reference set to evaluate the quality of the response generated by the VLM. Each task will test diferent aspects of visual understanding and reasoning, all grounded in the same set of videos and categories.
Table 1 presents the structure of the MAIA dataset after the data creation and validation process.
Feature Videos Semantic Categories Questions (Q) Answers (A) True Statements (TS) False Statements (FS) n 100 12 2, 400 19, 200 19, 200 19, 200 3.1. Reasoning Categories We defined 12 reasoning categories as the outcome of two pilot studies conducted with a group of expert volunteer CAUSAL COUNTERFACTUAL IMPLICIT UNCERTAINTY OUT-OF-SCOPE PLANNING SENTIMENT SPATIAL TEMPORAL
Partial Total Partial Total Partial Duration Why is mozzarella melted?
The heat from the wood oven has melted it
Mozzarella is melted by the heat of the wood oven
Mozzarella is melted by the heat generated by the sun.
What would happen if the pizza chef dropped the pizza on the floor?
He would dirty the floor and would have to remake the pizza. pizza, he would dirty the floor and would have to remake the pizza.
If the pizza chef dropped the
If the pizza maker dropped the pizza,
he would not dirty the floor and would not have to remake the pizza.
Is the person who rolls out the pizza the same one who puts it in the oven? What is the function of all the wooden planks under the
wood oven? On average, how many pizzas does the pizza chef bake each day?
No, they are two different people.
They have to feed the fire.
I do not have enough data to know.
In the scene, the person who rolls out the pizza dough and the one who puts it in the oven are two distinct figures.
In the scene, the person who rolls out the pizza dough and the one who puts it in the oven are the
same person.
The wooden planks under the wood oven are for feeding the fire.
The wooden planks under the wood oven are for
decoration.
There is not enough data to determine the average number of pizzas a pizza maker cooks daily.
There is sufficient data to determine the average number of pizzas that the pizza maker cooks daily.
What is the cake made of?
I cannot see any cake.
There is no cake in the video.
There is a cake in the video.
What steps should the pizza maker take to revive the fire? What attitude does the pizza maker show while taking the pizza out of the oven? Where is the pizza placed after being taken out of the oven?
He should stir up the embers a bit and throw
some new wood.
The pizzaiolo looks focused.
The pizza is placed on a
plate.
Where is the pizza maker?
In the pizzeria in front of the oven
To revive the fire, the pizza maker should stir the embers
and add new wood.
In the video, the pizza maker looks focused while taking the pizza out of the oven.
After being taken out of the oven, the pizza is placed on a plate.
In the scene, the pizza maker is in the pizzeria in front of the oven
To revive the fire, the pizza maker should stir the embers and add
new water.
In the video, the pizza maker looks distracted while taking the pizza
out of the oven.
After being taken out of the oven, the pizza is placed on
the table.
In the scene, the pizza chef is in the pizzeria by the counter When does the pizzaiolo take the pizza out of the
oven? How long does it take to cook the pizza in the video?
When he considers it cooked, towards the end of the video.
The pizzaiolo takes the pizza out of the oven towards the end of the video when he considers it cooked.
Pizza baking time is approximately 30 seconds
The baking of the pizza in the video takes approximately 30 seconds
The pizzaiolo takes the pizza out of the oven towards the beginning of the video when he considers it
cooked.
The baking of the pizza in the video takes approximately 30 seconds annotators. These pilots aimed to identify the optimal categories, including their definitions and any associated number, type, and specificity of the categories needed to efectively probe the cognitive and linguistic abilities of VLMs on our videos. Based on the feedback received, some initially proposed categories were merged due to content overlap or redundancy. Conversely, other categories were added to enhance the granularity of reasoning assessment (e.g, we introduced a Planning category, as we consider it a meaningful expression of reasoning skills). These refinements allowed us to design a more robust and informative framework to explore the interplay between language and vision in multimodal processing.
The following paragraphs introduce the final macrosub-categories.
Causal
focuses on reasoning about the causes or effects of events depicted in the video. It includes two
2 subtypes , namely Implicit and Explicit , ofering a comprehensive test of a model’s ability to describe causality within events. The former involves inferring unobservable causes from visible efects in the scene, requiring logical reasoning beyond what is directly shown. The 2Unlike the following cases, these are not treated as distinct subcategories but as two equally represented subtypes of the same category latter concerns clearly observable cause-and-efect dy- entities or events in the scene, throughout the entire namics, where either the cause or the efect is directly video. A typical response to a sentiment question may identifiable from the video content. describe a specific sentiment, attitude, or emotion, or it may reflect a neutral stance. This category evaluates Counterfactual focuses on questions about hypo- the ability of the model to recognize and identify the thetical scenarios that do not actually occur in the video emotional state or attitude of characters based on visual but could take place under specific conditions. These cues. questions are based on entities or events visible in the video and explore the consequences of an event or situa- Spatial investigates the spatial relationships between tion that might happen in the video if a certain condition entities, objects, or events depicted in the video. It aims were met. This category tests the ability of a model to at assessing the model’s ability to infer both stable and reason about hypothetical scenarios grounded in the con- time-dependent spatial relationships, as well as the text of the video while deriving logical and plausible ability to determine relative positioning in space and to outcomes from such scenarios. rely on grounding competencies.
Implicit investigates entities, events, or their at- Total Spatial: focuses on position of entities in space tributes that are not explicitly visible in the video while (including their relation to other entities) that remains their presence or properties can be reasonably inferred constant throughout the whole video, disregarding from the context. It evaluates the ability of a model any temporal variations or minimal movements of the to infer implicit details based on context, whether the entity at diferent moments in the video. A typical target information was never shown or was previously response to this type of question provides general spatial visible but later obscured. information valid for the entire duration of the video. Total Implicit: involves entities or events that are never directly visible in the video but can be inferred from observable details. A typical answer provides the requested information based on logical inference.
Partial Implicit: involves entities or events that were visible earlier in the video but are no longer visible due to a shift in the scene or because they have moved out of the frame.
Partial Spatial: focuses on time-related positions of entities in space, takin into account events occurring in the scene. A typical answer to this question provides spatial information that is valid only for the requested time range in the video.
Temporal focuses on temporal information and studies the ability of a model to infer temporal relationships, sequence of events, and durations from visual content in a coherent manner.
Sentiment assesses sentiment, mood, attitude, or emotion displayed by characters in the video toward other Out-of-scope refers to entities or events entirely absent from the video, focusing on properties or details of Partial Temporal: focuses on the temporal properties these non-existent elements. Typical responses to out- and relationships between events in the video, excluding of-scope questions involve a negation, indicating that their duration. Questions target aspects such as when the referenced entity or event is not present in the scene. something happens or whether it occurs before or after Typical answers to this question types involve a negation, another event. Typical answers specify the event along signaling that the referenced content is not present. This with the requested temporal detail. category indirectly tests the ability of a model to detect multimodal hallucinations and an assertiveness tendency in its responses.
Duration Temporal: focuses on a specific property of events in the video: their duration. A typical answer to a question involves several ways to express the duraPlanning asks for actions needed to achieve a spe- tion of the event. cific goal related to the video. The typical response to a planning question is a sequence of actions that someone Uncertainty refers to entities or events present in the should perform in order to reach the desired outcome. video but lacking suficient information to answer the Such a category assesses the ability of the model to infer question precisely. Questions are inherently ambiguous, and plan the necessary steps to accomplish a goal based as the visual content does not fully support a definitive on the visual cues provided in the video. response. Answers may ofer plausible options, acknowledge uncertainty, or signal that the reply is a guess. This category tests a VLM in handling ambiguity and incomplete evidence, and in assessing its tendency to respond ahn ing umb n i d rsoen ircah adbg ltaeb llraeb liccye lttoeb cup rca oenh toab ckkap izzap lobw ltoapp lltrsab rsoeh itscae trcku ite p p l l e c c a b o p s u s
4.1. Video Selection A key design choice for the MAIA benchmark was to reflect Italian culture in real-world scenarios through a carefully curated selection of video clips. To ensure richness and variety, the selection process was based on the following thematic areas: Locations, Food, Sport, Job, Nature, Activities. Such topics allowed us to collect a dataset showing locations, iconic Italian cities, and daily activities (e.g., enjoying breakfast at a café, cooking pasta, attending a soccer match) or even typical events (e.g., Italian local festivals or weddings). This cultural focus was not intended to limit the generalizability of the benchmark, but rather to ofer a valuable opportunity to assess model performance on culturally grounded data, which is an aspect often underrepresented in existing multimodal resources. 4.2. Video Collection We collected a culturally representative set of 100 short videos (~30 seconds each) sourced from YouTube Italy.
Following the criteria described in Section 4.1, videos were retrieved using keyword-based queries across selected thematic areas. Only Creative Commons licensed content was included to ensure reproducibility. When necessary, longer videos were manually checked and cut to extract the most relevant 30-second segments, resulting in a uniform and culturally grounded video set. 4.3. Analysis of Videos
To better understand the visual content present in the MAIA benchmark, we conducted an object detection and classification analysis over the full set of videos using a YOLOv113 detection pipeline. For each video, we sampled 32 uniformly spaced frames and ran object detection on them. This analysis provides a high-level view of the 5. Curated Linguistic Data typical objects types in MAIA.
Figure 3 shows the frequency distribution of detected 5.1. Questions Collection object labels across all annotated frames. Person is by far the most common object class, reflecting the human- We created 12 diferent sets of guidelines, each assigned centered nature of most videos in the benchmark. How- to a diferent annotator via Google Forms in order to colever, the dataset also includes a wide variety of everyday lect two questions per reasoning category for each video. objects, suggesting a rich and diverse set of visual ele- Annotators were PhD students under 30 with specializaments. tions in Linguistics and Computational Linguistics4. To
Figure 4 shows the distribution of the number of de- ensure variability between the pair of questions about tected objects per frame. Most frames contain a moderate that video, annotators were asked to change the entities number of objects, typically between two and six. This indicates that the videos ofer a balance between visual simplicity and complexity, making them suitable for testing both low-level perception and high-level reasoning in VLMs. 3https://docs.ultralytics.com/it/models/yolo11/ 4Each annotator was paid 100 euros for generating questions, which were collected through the administration of 1, 200 forms (10 per annotator)
What role do the men in white shirts play? Che ruolo svolgono gli uomini con le maglie bianche?
The men in white shirts are the competition judges Gli uomini con le maglie bianche sono i giudici di gara
They observe who scores a point
Osservano chi fa punto Men in white give judgements on the competition
Gli uomini in bianco danno giudizi sulla gara
They seem to be the referees of this bocce game
Sembra che siano gli arbitri di questa partita a bocce They measure the distance of the thrown ball from the little one and determine the winner of the set
Misurano la distanza della boccia tirata dal boccino e decretano il vincitore del set
The men in white shirts are the referees of the match Gli uomini con le maglie bianche sono gli arbitri dellla partita
The men in white are the jury
Gli uomini in bianco sono i giudici
Men in white shirts play the role of refereeing the match
Gli uomini con le maglie bianche svolgono il compito di arbitrare la partita and/or events involved in both of them. Each provided form contained both the definition of the assigned semantic category with examples, and also general rules to be followed (see Appendix, Figure 8 for an example of the form used). Each question had to be generated naturally and as an open-ended question. Questions involving a ‘Yes/No’ answer (e.g. Is there a car in the video?) were not allowed. Finally, for the correct execution of the task, the audio of the video had to be ignored, as the VLMs to be tested could only work on the visual part. Subsequently, questions were manually reviewed to ensure quality and category alignment. 5.2. Answers Collection Italian as their first language, and had spent the majority of their first 18 years of life in Italy. As with the question collection step, we used Google Forms to provide the task6. Each form included 10 videos, and for each video, the annotators were asked to answer 12 questions, one per reasoning category (see Appendix, Figure 9 for an example of the form used). Annotators were encouraged to use their own world knowledge when interpreting the visual content of the video.
To guarantee high quality of the collected answers, we employed rigid control mechanisms based on sanity check questions. Answers were accepted only if the annotators correctly answered at least 90% of these control questions, otherwise their submissions were rejected and the task was reassigned to another annotator. In total, 2, 400 questions were paired with 8 answers each, resulting in 19, 200 responses. They were then further checked by a semi-automated two-step validation process based on GPT-4o with few-shot prompting: The goal of this phase was to collect 8 diferent answers for each question to ensure not only accuracy but also variability in responses. This choice is also supported by ifndings from Mañas et al. [33], who empirically show that using up to 8 demonstrations provides an efective trade-of between diversity, accuracy, and computational Semantic Consistency Check. Each response was eficiency in in-context learning with LLMs for VQA eval- evaluated for semantic consistency with the corresponduation. We used the Prolific platform5 and selected an- ing question. In cases where inconsistencies were denotators aged 25 to 80 who were born in Italy, spoke tected, the answers were manually reviewed to assess 5https://www.prolific.com 6Annotators were paid £7 per hour for answering questions
Given an Italian question Q and an answer A concerning a video, you must create a statement S based on A. While generating S, try not to alter the words composing A. If A includes first-person verbs or phrases (e.g., 'I think,' 'I believe'), rephrase S to be impersonal, avoiding a first-person perspective.
The statement should be a concise, declarative sentence.
Given an Italian caption (TS) regarding the position or location of someone or something, your task is to create its foil (FS) by changing only the spatial information.
Don't add other information respect to what is stated in TS. Here is an example to guide you: TS: La donna nel video è in un campo di papaveri.
FS: La donna nel video è in una classe.
Given an Italian caption (C) dealing with temporal information about events and its foil (F), your task is to assess whetthheerctohrereqcutneestsisoonfsFhboausldedboenreC-.answered by another with diferent wording. Following the same procedure annoTtoabtoervoarlidt,hFesrheospulodnesxepcreosuslda dstiOilelrbeentatcecmepptoerda.l iRnefoarlmautisoendwfoitrhtrheefepreonoclseotof 8threeospnoenesxepsr,ewsseepdeirnfoCr.med a quality inco nIfsFisitseancviaelsidwfoerile, gfoeunenrdatteo 'bceormreicnti'moathle(ir.we.i,sfee w'neort tchoarnrect'c.heck to ensure lexical variability within the 2400 pools 100Bout of 19, 200 responses). of true statements (TS) (see Section 5.6).
Your task is to determine the natural language inference (NLI) relationship between S1 and S2. The possible Conltarbaedliscatiroe:n Test. We checked whether, within each 5.4. False Statement Generation pool- oEfnt8airlmesepnot:nSs2eslotgoictahlley fsoalmlowesqfuroemstiSo1n., any of the respo- nCsoenstcraodnitcrtaiodnic:tSe2dctohnetoratdhiecrtss. SW1.e found that 90.25% The goal of this phase is to create a false statement (FS) of th-eN8e-uantrsawl: eSr2paonodlsS1exahreibrietlafutelldabgurtedemoneontt,eanstatihleoyrcdoontrafdoircteaecahchToSthalerre.ady collected, in order to form a minimal not cPoronvtiadienoannlyyocnoenlatrbaedliacstioountsp.uTt(hEentraeilmmaeinnti,nCgon9.t7ra5d%ictioTnS,-oFrSNpeauirt,raeln).abling controlled experiments and precise (234 cases) were manually reviewed by an additional analysis of a model’s behavior with respect to the reaannotator to resolve inconsistencies. soning categories. As for TSs, the FSs were automatically
A post-processing phase of the responses was imple- generated using GPT-4o for editing only the elements of mented to ensure a suficient degree of variability and the sentence related to that semantic category, an apreduce potential redundancy within each of the 2, 400 6pBroashcho winsspairperdo mbyptthuesecadpftoiornt-hfeoiFlSmsegtheonder[a1t4io].nF7i.gFuorer poolGsoivfe8naansqwueersst(isoene (SQec),tiaonca5.n6d).idFiagtuerean5sswhoewr s(Aa)n, and a set of 8 reference answers (R1–R8), your instance, taking into account the previous example in 5.3: examtapslekoisf otoned8e-taenrsmwienrepwoohleatshseorciAatiesdcwoirtrhecatv.iAdeiso considered correct if it aligns with at least one of and a question, after this refinement procedure described abovteh. e reference answers. In the video the boy is in the bathroom before running away.
Return only one label as output: 'Correct' or 'Incorrect. 5.3. True Statement Generation At this step we automatically generate a true statement (TS) for each question-answer pair collected in the previous phases. A TS consists of descriptive declarative sentences aligned with the visual content of the videos.
For example, if a video shows a boy who is initially in a kitchen and he hears a loud noise and runs away, a TS for the Spatial category could be: In the video, the boy is in the kitchen before running away.
To create TS we used GPT-4o, with the prompt in Figure 6A, leveraging the combination of each question and its answer to automatically generate 19, 200 true statements (TSs). As with the answers, the TS are organised into 2, 400 pools of 8 items, each expressing the same event Finally, we implemented two quality checks for FS using GPT-4o.
Structural Check aiming at automatically verifying that each FS aligns correctly with its corresponding TS according to its category. While the GPT-4o evaluation initially flagged 864 out of 19, 200 cases as incorrect, only 2.5% were ultimately confirmed as truly problematic and subsequently corrected through manual revision.
Contradiction Test performed by assuming that a correct FS must be in contradiction with the relevant TS.
We ran an NLI task to classify TS-FS pairs as Entailment, 7Due to space constraints, we could not include all the 12 prompts used for generating FSs specific to each reasoning category. However, the prompt shown here is representative of the adopted methodology.
Nouns in Q&A Across All Videos
As said in Section 5.2, we opted for a pool-based structure with 8 items per question in order to balance semantic consistency with lexical diversity both across answers and statements. To meet this requirement, we assessed and enhance lexical richness within our data. This phase was carried out in several incremental steps (i.e., a string based test, lexical overlap and Type-Token Ratio (TTR) 8Nouns from TS and FS were excluded, as those sentences are derived from Q&A and would result in redundant repetitions. 9https://spacy.io 10Since TSs are generated from an automatic rephrasing of Q&A pairs, we checked and improve their lexical diversity. This indirectly benefits the corresponding FSs, which difer by a single term from TS. This work has been carried out while Davide Testa was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome in collaboration with Fondazione Bruno Kessler (FBK). Giovanni Bonetta and Bernardo Magnini were supported by the PNRR MUR project PE0000013-FAIR (Spoke 2).
Alessandro Lenci and Alessandro Bondielli were supported by the PNRR MUR project PE0000013-FAIR (Spoke 1). Alessio Miaschi was supported by the PNRR MUR project PE0000013-FAIR (Spoke 5). Lucia Passaro was supported by the EU EIC project EMERGE (Grant No. 101070918).
The following figures show examples of the forms adopted for collecting the questions (Figure 8) and the corresponding answers (Figure 9).
General Task Privacy Policy
and Research Purposes
Category specific task
Example 2 Questions 2-Qgueensteiroantsion generation Declaration on Generative AI During the preparation of this work, the author(s) used ChatGPT (OpenAI) in order to: Paraphrase and reword, Improve writing style, and Grammar and spelling check. After using these tool(s)/service(s), the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.