Spatial reasoning poses a particular challenge for intelligent agents and is at the same time a prerequisite for their successful interaction and communication in the physical world. One such reasoning task is to describe the position of a target object with respect to the intrinsic orientation of some reference object via relative directions. In this paper, we introduce GRiD-A-3D, a novel diagnostic visual question-answering (VQA) dataset based on abstract objects. Our dataset allows for a ifne-grained analysis of end-to-end VQA models' capabilities to ground relative directions. At the same time, model training requires considerably fewer computational resources compared with existing datasets, yet yields a comparable or even higher performance. Along with the new dataset, we provide a thorough evaluation based on two widely known end-to-end VQA architectures trained on GRiD-A-3D. We demonstrate that within a few epochs, the subtasks required to reason over relative directions, such as recognizing and locating objects in a scene and estimating their intrinsic orientations, are learned in the order in which relative directions are intuitively processed.
Reasoning to solve complex spatial tasks like grounding directional relations in an intrinsic frame of reference can be decomposed into a set of subtasks that are hierarchically organized. Consider two objects 1 and 2 in an image, where each of the objects has a clear front side and orientation. Learning to answer whether the triple (1, , 2) holds for a given directional relation in a frame of reference that is intrinsic to 2 spans the following stages (see Fig. 1 for an example): 1. Both the target object and the reference object have to be recognized in the image (existence prediction). In other words, an agent must initially be capable of answering questions such as “Is 1 in the image?” or “Is 2 in the image?”. 2. Next, the object’s pose that defines the relative relation has to be discerned, enabling an agent to successfully respond to questions such as “What is the cardinal direction of 2?” (orientation prediction).
ing the two preceding competencies, allowing an agent to answer a question similar to “What is the relation between 1 and 2 from the perspective of 2?” (relation prediction). Likewise, predicting which target object is in a specific relation to some reference object (link prediction) can be answered, e.g., “Taking 2’s perspective, which object is in relation to it?”.
mine whether a specific directional relationship exists between the two objects (triple classification), thus successfully providing an answer to a question like “From 2’s perspective, is 1 left of 2?”.
tial bias that confounds the analysis of reasoning about relative directions.
In the present work, we introduce GRiD-A-3D, a novel
and simplified diagnostic VQA dataset, which allows for
a more eficient and targeted analysis of the
corresponding reasoning process by removing possible biases from
using real-world objects. Subsequently, we report the
performance of the two established end-to-end VQA models
MAC [
We summarize the contributions made in this paper as follows: • We complement our GRiD-3D benchmark suite1 with a novel GRiD-A-3D (Grounding Relative Directions with Abstract objects in 3D) dataset that enables a faster and less biased evaluation of spatial reasoning behavior in VQA compared with the original GRiD-3D dataset. • We verify our previous research findings with the new dataset, thus underpinning our hypothesis that multi-task learning enables neural models to learn to ground relative directions in VQA. • Furthermore, we add evidence to our hypothesis that during multi-task learning, spatial reasoning abilities of a neural model develop along the intuitive order of corresponding subtasks, thus forming an implicit curriculum.
Aiming to provide a suitable setup to assess the reasoning
capabilities of neural models on vision-language tasks,
diagnostic datasets have been introduced [
With the introduction of the GRiD-3D dataset [
Overview and statistics With our new GRiD-A-3D
dataset, we address the following six tasks: Existence
of the image. Yet taking into account more realistic sce- Prediction, Orientation Prediction, Link Prediction, Relation
narios such as multi-agent dialogue in a situated envi- Prediction, Counting, and Triple Classification. All 8 000
renronment, understanding relative directions is a prerequi- dered images are split without overlap into 6 400 for
trainsite for meaningful communication. As a consequence, ing, 800 for validation, and 800 for testing. The 432 948
early models to learn symbolic reasoning with relative corresponding input questions follow largely the same
directions have been proposed [
An early synthetic dataset providing a test bed for
grounding relative directions is Rel3D [
GRiD-3D is the first-of-its-kind to target multi-task Blender.2 We choose a
conlearning of relative directions in a controlled setting. sistent lighting setup across
With this dataset, it was shown that, before learning all images, add shadows to Figure 4: The six abstract
how to answer the question whether a triple (object1, each object, and restrict the objects used in
relation, object2) holds, neural end-to-end VQA models image generation to a fixed the GRiD-A-3D
rely on an implicit curriculum of related subtasks such camera angle, thus obtaining dataset.
as object detection, orientation estimation, and relation one image per scene.
prediction [
In this work, we aim to provide a variation of the origi- age. An overview of all six objects can be found in Fig. 4. nal dataset that ensures the elimination of such potential distortions (see Fig. 3 for examples), enabling a model Note that the overall object count in the original GRiD- image and question features are fed to special neural 3D dataset is 28, whereas GRiD-A-3D is restricted to six units called residual blocks (FiLM) or MAC cells (MAC). diferent objects. A chain of such units provides the core of the reasoning process.
We use existing PyTorch3 implementations of FiLM
and MAC with their default hyperparameters for the
published CLEVR [
We interpret our results in the following way: Existence and Orientation Prediction are learned earlier than other tasks. We explain this observation with the fact that these (a) FiLM (b) MAC tasks only require a model to focus on one single object.
For the most straightforward task of Existence Prediction, uFsigedurfeor5o:uNreeuxpraelriemnedn-ttos.-eTnhde VgeQnAermicoudneiltss F( hiLeMre caonldorMedAiCn we observe similar behavior for the two datasets: Both orange and blue, respectively) control how the question and converge to an accuracy of almost 100% at nearly the image features are being processed. same time. For the Orientation Prediction task, we observe convergence to an accuracy of over 80% for both datasets.
Noticeably, the learning happens faster for the abstract
GRiD-A-3D dataset. The shorter learning time can be
Question generation In addition to rendering the im- attributed to the more unequivocal identification of front
ages from our sampled scenes, we obtain scene graphs and back sides of the abstract objects due to the lack
equipped with ground truth information such as absolute of symmetry related noise as shown in Fig. 3. The fact
position, orientation, and relative directions of objects, that the accuracy on Orientation Prediction is capped at
that we use to generate questions related to the six tasks about 85% can be explained by objects placed close to the
contained in GRiD-A-3D. Our question generation builds border between two cardinal directions, as such cases are
upon the framework provided with CLEVR [
We follow the depth-first search strategy to determine higher accuracy. Similarly to the results on the Orientaand instantiate question-answer pairs that comply with tion Prediction task, the main reason for these observathe scene information and can therefore be considered tions may lie in the facilitated learning conditions due to valid. We set additional constraints to make sure that an- the lack of front-back symmetries or strong occlusions swers are uniformly distributed for each task. To ensure with the abstract objects. This efect is most pronounced a wide variety of natural language questions, we sample for Link Prediction, i.e., predicting which target object from a rich set of diferently phrased question templates is in a given relation to some reference object. We atfor each reasoning task and randomly omit utterances or tribute this observation to the smaller set of objects in replace words with suitable synonyms. the GRiD-A-3D dataset.
Finally, we observe a mixed result for the Counting task: 4. Evaluations While learning of both VQA models converges faster for the GRiD-A-3D dataset, higher accuracy is reached for the GRiD-3D dataset. We hypothesize that this higher accuracy stems from the more diverse-looking objects in the GRiD-3D dataset, facilitating the models to distinguish and thus count multiple objects in close proximity.
In summary, our results suggest the following two
facts: First, the abstract GRiD-A-3D dataset leads to faster
For our experiments, we train MAC [
20 FiLM on GRiD-3D MAC on GRiD-3D FiLM on GRiD-A-3D MAC on GRiD-A-3D 30 40
Epoch Counting
FiLM on GRiD-3D MAC on GRiD-3D FiLM on GRiD-A-3D MAC on GRiD-A-3D 30 40 learning and can thus enable more computationally ef- curriculum. ifcient experimentation while achieving comparable results to the original GRiD-3D dataset. Second, the results support our assumption of a chronology of subtasks, as Existence Prediction and Orientation Prediction are learned before the models can reason about relative directions.
relative directions with end-to-end neural VQA architectures. We provide a comprehensive, simplified GRiDA-3D dataset with abstract objects that shows similar behavior to the original GRiD-3D dataset when learned by the two established VQA models FiLM and MAC. With our experiments, we show that the learning of tasks that focus on a single object like object recognition and orientation prediction happens prior to learning to ground relative directions and object counting.
The abstract nature of the dataset eliminates approximate front-back object symmetries that can have a negative impact on object orientation prediction and all reasoning tasks about directional relations that build upon it. Furthermore, the simplification of the object set allows for conducting experiments with a more comprehensive dataset. In future work, this will allow us to conduct fast pilot studies on curriculum and transfer learning based on the intuitive dependency of the diferent spatial reasoning tasks on one another and the observed implicit