Spatial Reasoning is an important component of human cognition and is an area in which the latest Visionlanguage models (VLMs) show signs of dificulty. The current analysis works use image captioning tasks and visual question answering. In this work, we propose using the Referring Expression Comprehension task instead as a platform for the evaluation of spatial reasoning by VLMs. This platform provides the opportunity for a deeper analysis of spatial comprehension and grounding abilities when there is 1) ambiguity in object detection, 2) complex spatial expressions with a longer sentence structure and multiple spatial relations, and 3) expressions with negation ('not'). In our analysis, we use task-specific architectures as well as large VLMs and highlight their strengths and weaknesses in dealing with these specific situations. While all these models face challenges with the task at hand, the relative behaviors depend on the underlying models and the specific categories of spatial semantics (topological, directional, proximal, etc.). Our results highlight these challenges and behaviors and provide insight into research gaps and future directions.
Vision-language model (VLM) research has boomed in the recent past, owing to the enhanced user
interaction and accessibility they provide. Models such as GPT 4o1, LLaVA [
Most of the previous works confine their analysis to testing which models work well for spatial relations. We go further to analyze the comparative performance of these models for spatial categories that represent diferent orientational and positional relations between objects. A novel aspect of our work is the analysis of the efect of varying spatial composition (number of spatial relations) in the expressions on the performance of the models.
Previous works focused on spatial analysis with image captioning-related tasks, thus failing to locate the source of error in the presence of visual and linguistic ambiguity. To avoid this, we adopt the Referring Expression Comprehension (REC) task for our analysis. The REC models output bounding boxes around the target entity based on a natural language expression, the analysis of which could reveal the parts of the input that the models fail to comprehend. Comprehension accuracy (or simply, https://www.cse.msu.edu/~kordjams/ (P. Kordjamshidi)
CEUR Workshop
ISSN1613-0073 (a) The white napkin that is wrapped around the hot dog (b) The white box that is around the mirror (c) The brown table that is to the left of the black cell phone (d) The sandy shore that is near the murky water (e) The baseball player that is to the left of the black helmet and to the right of the home plate (f) The large branch that is to the right of the log that is behind the large bear (g) The black monitor that is to the left of the keyboard or on the desk (h) The blanket that is not green and that is not on the bed (i) The fence that is not black and that is not to the left of the man accuracy) is a common metric for this task; it captures how often a model correctly outputs the bounding box around the target entity.
For our analysis, we use the CopsRef dataset [
We test two popular VLMs - LLaVA [
Some of our important findings are as follows:
(1) Referring expressions that include spatial relations, in addition to object attributes, result in higher
accuracy on the REC task compared to expressions with only attributes. (2) Increasing the spatial
complexity (no. of spatial relations) of an expression afects the performance of the VLMs, but models
with explicit compositional learning components maintain the performance. (3) Expressions involving
dynamic spatial relations yield low accuracy across all models, indicating the dificulty in modelling
these relations. (4) The task-specific trained models achieve higher accuracy for expressions with
geometric spatial relations (e.g., left of, right of) while the VLMs show relatively better accuracy for
expressions having ambiguous relations such as proximity. (5) The models fail to recognize negated
spatial relations in referring expressions in multiple instances, though the extent of this failure varies
across models.
2. Related Work
Previous works have conducted a broad analysis on the ability of VLMs to perform multimodal perception
and reasoning tasks, such as Spatial Reasoning, Multimodal conversation, etc. Many comprehensive
real-world benchmarks have been introduced to test multiple VLM capabilities. [
Some works [
Other closely aligned work includes Embodied Spatial Analysis, which focuses on the efects of
diferent perspectives and non-verbal cues on the spatial reasoning capabilities of VLMs [
RQ1. Which spatial relation categories result in low accuracy for REC models? RQ2. How do diferent model characteristics/architectures influence the REC task accuracy for certain spatial relation categories compared to the others? RQ3. Does the inclusion of spatial relations increase or decrease the accuracy of REC models? RQ4. How does the number of spatial relations in the expressions afect the accuracy across diferent types of models? RQ5. Do the REC models accurately recognize negated spatial relations in expressions?
To answer these questions, we explain our research methodology and the designed experiments in this section.
We select three distinct models for our analysis such that they difer in key components like architecture, pre-training tasks, and input formats.
MGA-Net. [
Grounding DINO. [
LLaVA. [
Model Diferences. A key diference in the three main models can be seen in their input format. While Grounding DINO and LLaVA take the entire image as the input and perform bounding box regression to get object proposals, MGA-Net directly takes the externally detected bounding boxes as the input. Grounding DINO and LLaVA also have similarities in their architectures, as they both have vision and language backbones that are fed the entire image and text inputs. This is unlike MGA-Net, which has dedicated transformer architecture modules for visual, linguistic, and relative location components. However, Grounding DINO and MGA-Net show similarities in having grounded training/pre-training tasks, while LLaVA only has general multimodal pre-training.
In addition to these three models, we also experimented with InstructBLIP [
We create the following dataset test splits for evaluation and answering the earlier mentioned research questions, RQ1-RQ5. 4.2.1. Fine-grained Spatial Relations Split In the test dataset, we split the expressions with 1 spatial relation using the categories shown in Table 2. Using the categories from Table 3, we split the remaining expressions based on the number of spatial relations they contain. Then, we rank the models based on their accuracy for each category.
To compare the models’ performances across the categories, we employ a statistical test known as the Kendall Tau Independence Test. It evaluates the degree of similarity between two sets of ranks given to the same set of objects. We calculate the Kendall rank coeficient ( ), which yields the correlation between two ranked lists. Given value, we calculate the statistic, which follows standard normal distribution, as: = 3 ∗ ∗ √( − 1)/ √2(2 + 5).
(1) Using the 2-tailed p-test at 0.05 level of significance, we test the following: Null hypothesis: There is no correlation between the two ranked lists. Alternative hypothesis: There is a correlation between the two ranked lists. 4.2.2. Visual Complexity Split To observe the efect of visual complexity on model performance, we split the test dataset into two parts. The first part has images that have multiple instances of one or more objects mentioned in the associated referring expressions. The second part has images with at most one instance of every object mentioned in the expression. We perform this splitting by first collecting the entities in each expression using spaCy2 and then employing Grounding DINO to find the number of instances in the image for each of the collected entities. 4.2.3. Negation Analysis Split In our analysis, we found that models have dificulties in grounding spatial expressions with negations. Therefore, we created a test split for a more accurate evaluation and a deeper analysis of negated spatial expressions. We collected expressions that include the keyword ‘not’ and divided them into two sets according to the number of occurring negations (1 or 2). Then, we collected those expressions for which all three models give an IoU of less than 0.5. For each expression, we perform a qualitative analysis to verify whether the errors are due to misinterpreting the negations or conflation of other errors. We limit our analysis to the results from the first run of the three models to facilitate the instance-wise analysis.
Hardware. For Grounding DINO, LLaVA, and the OWL-ViT baseline, we use the T4 GPU provided by Google Colaboratory for inference.3 For MGA-Net, we use the NVIDIA GeForce GTX 1650 GPU for training. We run each model three times (both training and testing for MGA-Net, and inference for the VLMs and the baseline) to ensure the statistical significance of our results.
Evaluation Metrics. We evaluate the models using the Intersection over Union (IoU) metric.
Following previous works [
From Table 4, we can observe that Grounding DINO and MGA-Net outperform the OWL-ViT baseline, with the former achieving the highest accuracy in grounding the referring expressions. However, we also tried training MGA-Net with the full ResNet-101 visual backbone (Full CNN) instead of the partial backbone (Partial CNN). We could only train this model for four epochs due to computational constraints. However, the model crossed 60% test accuracy in just four epochs and was monotonically increasing. This shows that MGA-Net could potentially provide a better performance using adequate computational resources. To avoid unfair comparisons due to the training discrepancies, we focus our results on the relative performances of each model across diferent spatial relation categories rather than comparing the absolute performances.
For LLaVA, we used the prompts explained in Section 4.1. The shorter prompt gave a slightly better accuracy than the longer prompt. Hence, we used the shorter prompt for further experiments. The accuracy of LLaVA is less than both the other models and the baseline. Possible reasons are the lack of both bounding box regression and visual grounding instructions during pre-training.
Since we trained/tested each model for three runs, we report the average accuracy of the three runs and the standard deviation in the table. Since we re-train MGA-Net for each of these runs, there is a noticeable diference in model predictions in each run, leading to a slightly high standard deviation. However, we test the VLMs and the baseline zero-shot, leading to zero or near-zero standard deviation in the accuracies. This also follows for the future result tables.
2https://spacy.io/ 3https://colab.research.google.com/
Table 6 shows the Kendall Tau Independence test results for the three pairs of VLMs. We can observe that while the category-wise ranks of the VLMs (Grounding DINO and LLaVA) are correlated, MGANet’s ranks aren’t correlated with them. This motivates us to study the possible reasons behind the diference in the category-wise performances of MGA-Net and the VLMs.
Among spatial categories of MGA-Net and VLMs, the major diference occurs with the Proximity and Projective categories. To answer RQ2, we can observe that the ‘Proximity’ category ranks third for both the VLMs but 8th for MGA-Net. On the other hand, ‘Projective’ has a higher rank for MGA-Net than both VLMs. We can see that MGA-Net prefers geometric spatial relations like left of, on top of, etc., as it takes the relative locations of bounding boxes as input, which helps represent such relations. On the other hand, the two VLMs have a better ranking than MGA-Net for ambiguous relation categories that do not specify a clear distance or geometric direction (e.g., by, close to). This is because the vision backbones of the VLMs utilize the entire image and help capture relations between a region in the image and its surrounding regions, unlike MGA-Net, which only receives the detected bounding boxes as input.
To study further diferences between MGA-Net and the VLMs, we design Table 7, which shows the performance of the three models and the OWL-ViT baseline for expressions having diferent numbers of spatial relations. We observe that VLMs perform considerably better for expressions with 0/1 spatial relations compared to expressions with 2/3 spatial relations. This proves that VLMs find it comparatively dificult to ground multiple spatial relations. However, MGA-Net takes advantage of its compositional learning architecture to handle multi-step reasoning, resulting in a similar performance for all categories.
An interesting observation is that the performance of the baseline considerably drops for the ‘Two’ and ‘Three’ categories, even though the spatial relations aren’t being passed as input to the baseline. The reason might be that 41.4% of these images have multiple instances of objects, the impact of which is explained in the next section.
From Table 7, we can also compare the performance of the models for expressions with none and one spatial relation. We observe that LLaVA performs better for the former and MGA-Net for the latter. Grounding DINO gives a similar performance for both.
Now, to answer RQ3, we observe in Table 5 that among the seven categories of single spatial relations, MGA-Net and Grounding DINO perform better for five of those compared to expressions with no spatial relations. LLaVA also performs better for four such categories. Thus, we can conclude that in a setup involving visual and linguistic ambiguity (such as ours), spatial relations along with visual attributes often aid the models in grounding the expressions, compared to the attributes alone. This is also reinforced by the results of the baseline. From Table 7, we can observe that while the baseline gives the second-best performance for expressions with no spatial relations, it drops to the third place for expressions with one spatial relation, with a 7.3% reduction in performance. This is because the baseline doesn’t have access to the spatial relations.
Finally, Table 7 helps us answer RQ4 as it shows the efect of increasing spatial relations on the performance of MGA-Net versus the VLMs (as discussed before).
Out of 12586 test data points, we found that in the images of 4730 data points, there are multiple instances of objects mentioned in the referring expressions. Table 8 shows the accuracies of the three models and the OWL-ViT baseline for images with a single instance (‘Accuracy Single’ column) and multiple instances (‘Accuracy Multi’ column). The models perform better for the single instance images by 5.4% on average compared to the multi-instance images. The 8.4% performance drop of the baseline for multi-instance images proves that the images are indeed complex and require more than just the label as the input for grounding the right object. However, the 7.3% performance drop of LLaVA, as compared to MGA-Net and Grounding DINO, shows that grounded pre-training also plays a crucial role in helping the models ground the right object instance in multi-instance images.
We obtained 36 expressions with 1 ‘not’ and 73 expressions with 2 ‘not’s for which all models
gave incorrect predictions. Table 9 shows the total number of expressions we obtained with 1 and 2
negations. The ‘Total failure’ row gives the number of instances for which models failed to recognize at
least 1 negation. We can observe that Grounding DINO has the highest number of failure instances.
LLaVA performs better than Grounding DINO, possibly due to the Vicuna [
Another interesting observation was for the outputs of MGA-Net and LLaVA models when they are close to the target object. From Table 10, we can see that while LLaVA has a better precision in such cases, MGA-Net has a better recall.
Here, we provide a qualitative analysis of certain issues faced by the models in handling referring expressions.
The expressions pertaining to Figures 1a and 1b consist of the same spatial relation (‘around’). In the ifrst figure, the wrapping of the napkin around the hot dog only makes the napkin partially visible. But in the second figure, the white box around the mirror is almost entirely visible. This shows how the interpretation of ‘around’ is highly dependent on the configuration of the involved objects. For the first image, LLaVA fails to precisely localize the object, while MGA-Net only returns a part of the napkin that is visible. In the second image, both models fail to localize the object.
For ‘Two-and’ category expressions, the models sometimes only satisfy one of the spatial clauses. This often happens if multiple objects of the same class are in the image. For example, in Figure 1e, the output baseball player is to the left of the black helmet but is not to the right of the home plate.
Similarly, for ’Two-chained’ category expressions, the models sometimes do not consider the entire expression. For example, in Figure 1f, MGA-Net and LLaVA return the ‘log that is behind the large bear’, and Grounding DINO returns the bear itself. None of the models consider the ‘large branch’ part of the expression, which should have been the output.
Finally, for ’Two-or’ category expressions, the model might pay attention to only one spatial clause. Consequently, it returns an object satisfying that clause but not the additional attributes mentioned in the expression. For example, in Figure 1g, the model returns the monitor, which is to the ‘left of the keyboard’, but it does not satisfy the color attribute.
Figures 1h and 1i show two cases where all models fail to recognize negation. In 1h, we can observe that while MGA-Net is wrong, LLaVA is close to the ground truth but partially covers the target object (high precision, low recall). In 1i, while LLaVA is wrong, MGA-Net is closest to the ground truth but covers an excess area (low precision, high recall).
Spatial reasoning is an integral aspect of cognitive reasoning and embodied AI tasks. However, recent studies have shown that state-of-the-art VLMs often fail to accurately comprehend spatial relations. To better understand the limitations of these models, we evaluate their spatial understanding using the referring expression comprehension task because it requires explicit grounding of complex linguistic expressions in the visual modality. We picked multiple models, including Vision-language models (LLaVA, Grounding DINO) as well as task-specific models (MGA-Net). We observed that the VLMs that are trained in the wild with visual and textual data perform worse in grounding. All models show low accuracy in grounding Directional relations. However, the VLMs do better in vague relations such as proximity, while the task-specific models are better in geometrically well-defined relations such as left and right. While using spatial relations increases the grounding accuracy, using multiple relations makes the reasoning more challenging for all models, with a higher impact on VLMs. However, unlike VLMs, MGA-Net maintains its performance for complex spatial expressions due to its compositional learning architecture. In the presence of visual complexity, the performance of all models drops, but LLaVA’s performance is afected the most due to a lack of grounded pre-training. Finally, both VLMs and task-specific models have failure cases when grounding expressions that include negation. These ifndings shed light on the gaps for future work on Vision-language models.
Although increasing the number of parameters of VLMs can improve their performance for expressions
with simple spatial relations, architectural changes are necessary if the VLMs are to maintain their
performance even for expressions with novel complex compositions of spatial relations. We observed
that MGA-Net maintains a consistent performance for expressions with varying spatial complexity
better than the VLMs due to its soft attention module, which decomposes the expression into its
semantic components for compositional reasoning. This highlights the decomposition of complex
spatial expressions as a potential path forward to help VLMs generalize. Alternative strategies [
Another issue to address is the VLMs’ inability to comprehend negations. MGA-Net’s improved performance over Grounding DINO due to the presence of negated expressions in the training data motivates us to explore the augmentation of training/instruction tuning data of VLMs with synthetically generated negated expressions. Additionally, we also plan to formulate contrastive learning objectives to penalize the model when it fails to comprehend negations.
For our analysis, we utilize the spatial categories introduced by [
E.g.: man on the right that is standing and wearing gray pant 2. Adjacency: Consists of relations that describe the close, side-by-side positioning of two objects.
They may or may not imply a particular direction.
E.g.: The large poster that is leaning against the wall 3. Directional: Consists of dynamic action verbs / directional relations. They describe the movement or change in position of an object relative to other objects in the image. The interpretation of these relations heavily relies on the configuration of the involved objects and/or the dynamic spatial relationship between them.
E.g.: The gray car that is driving down the road 4. Orientation: Consists of relations which describe the orientation of an object w.r.t another object.
E.g.: The sitting dog that is facing the window that is to the right of the mirror 5. Projective: Consists of relations that indicate the concrete spatial relationship between two objects, i.e., these relations can be quantified in terms of the coordinates of the two objects.
E.g.: The black oven that is above the drawer 6. Proximity: Consists of relations that indicate that two objects are near each other without giving a specific directional relationship.
E.g.: The blue chair that is close to the white monitor 7. Topological: Consists of relations that indicate the broader arrangement or the containment of an object w.r.t another object
E.g.: The silver train that is at the colorful station 8. Unallocated: Consists of relations that cannot be allocated to any of the above categories.
In our analysis, we also experimented with InstructBLIP [
For InstructBLIP, we designed three prompts for the REC task. They are as follows: In the prompts, the ‘bounding box list’ placeholder takes the coordinates of the detected bounding boxes in the image being passed as the input, along with indices for each bounding box, starting from ‘1’. But for the third prompt, the model has no access to pre-detected candidate bounding boxes in the image. While the expected output for the first prompt is the index of the correct bounding box, for the other 2 prompts it is the bounding box coordinates as the output.
The bounding box format is [x1, y1, x2, y2], where (x1, y1) is the bottom left corner and (x2, y2) is the top right corner of the box. The coordinate values are a fraction of the total length/width of the image according to the position of the coordinate.
Unfortunately, none of the prompts gave consistently correct outputs. The outputs were as follows: Prompt 1: The outputs were mostly incorrect. Sometimes, the model also gave ‘0’ as the output, even though it is not a valid index.
Prompt 2: The output did not return meaningful coordinates in most cases. But in the few instances that it did, they were mostly incorrect. Example outputs when the model could not return meaningful coordinates are: • {1: [0.16, 0.55], 2: [0.32, 0.47], 3: [0.55, 0.6], 4: [0.21, 0.06]} • [0.9, 0.53, 0.93, 0.57, 0.0, 0.39]
Prompt 3: The model could not understand the task, and it just paraphrased parts of the prompt instead of giving the coordinates as the output. Example prompts and outputs are: • Prompt: Provide the bounding box coordinates for: ”The large poster that is leaning against the wall”
Output: what is the bounding box coordinates for the large poster that is leaning against the wall • Prompt: Provide the bounding box coordinates for: ”The young man that is leaning against the wall” Output: is standing in an elevator. the young man that is leaning against the wall is standing in an elevator
Prompt 1: We tested all the prompts designed for OpenFlamingo in both 2 and 3-shot settings. • Example output format: <image>Bounding Boxes:bounding box list; Expression: Refexp;
Correct Bounding Box:”ID”<|endofchunk|> • Query format: <image>Bounding Boxes:bounding box list; Expression: Refexp; Correct Bounding Box:“ ‘bounding box list’ placeholder takes the list of candidate bounding boxes in the image as input, in the same format as InstructBLIP (discussed in the previous section). The expected output is the index of the correct bounding box. However, we observed that irrespective of the query, the model gave the same output index for the same set of prompting examples.
Prompt 2: • Example output format: <image>Expression: Refexp; Correct Bounding Box:[Bounding box coordinates]<|endofchunk|> ‘bounding box list’ placeholder takes the same input as explained for Prompt 1. But instead of expecting the index, we expect the coordinates of the bounding box as the output. The format of the bounding box is the same as explained for InstructBLIP in the previous section. However, the model failed to give meaningful coordinates as output in most cases. When it did give meaningful coordinates, the outputs were mostly incorrect.