Spatial understanding is crucial for any agent that navigates
in a physical world. Computational and cognitive frameworks
often model spatial representations as spatial templates or
regions of acceptability for two objects under an explicit
spatial preposition such as “left” or “below”
Here, we propose the task of predicting the relative spatial
locations of two objects given a textual input of the form
(Subject, Relationship, Object). We report on initial
experiments with a simple neural network model with
distancebased supervision learned in annotated images that obtains
promising performance. Crucially, we show that the model
can reliably predict templates of unseen combinations, e.g.,
predicting (man, riding, elephant) without having seen such
scene before. Furthermore, by leveraging word embeddings
of objects and relationships, the model can correctly predict
spatial templates for unseen words. E.g., without having ever
seen “boots” before but only “sandals”, the model predicts
correctly the template of (person, wearing, boots) by
inferring that, since “boots” are similar to “sandals”, they must be
worn at the same position of the “person”’s body. Hence, the
model is able to leverage the learned common sense spatial
knowledge to generalize to unseen objects.
*The reader may refer to a full paper
Obj.
g Compose n i n r
man flying kite Subj. Rel. Obj. j)b j) (zeu rSeub S ( iS ten
C Obj.
Obj. g n i s s e c o r P e r P
coordinates Im
Proposed task and model 2.1
Proposed task We propose the task of predicting the 2D relative spatial arrangement of two objects under a relationship given a structured text input of the form (Subject, Relationship, Object)— abbreviated as (S, R, O). More precisely, the model predicts the Object’s box center and box size (output) given the structured text input (S, R, O) plus the center and size of the Subject’s box (Fig. 1). 2.2
Proposed model We employ a feed forward network with embeddings (Fig. 1). The embedding layer maps the input words (S,R,O) to their d-dimensional representations. The embeddings are then concatenated with the Subject’s box center and size. This vector is then fed into a fully connected layer to compose S, R, O into a joint representation. model predictions (Object’s center and size) are evaluated against ground truth with a mean squared error (MSE) loss.
3
Data. We use the Visual Genome
Data pre-processing. Coordinates are normalized by image width and height. Since right/left depends only on the camera viewpoint, we get rid of this arbitrariness by mirroring the image when the Object is on the left of the Subject. Evaluation metrics. We use standard regression metrics: (i) Mean Squared Error (MSE) between predicted and true Object center and size. (ii) Coefficient of Determination (R2) of model predictions and ground truth. (iii) Pearson Correlation (r) between predicted and true x-component of the Object center, and similarly for the y-component. We also consider the classification of above/below relative locations of the Object w.r.t. the Subject. We report (macro averaged) F1 (F1y) and accuracy (accy). 4
We test the following model variations. EMB denotes a model that uses pre-trained word embeddings1, RND a model with random normal embeddings, 1H employs one-hot embeddings and ctrl outputs random normal predictions. Overall, the preliminary results outlined below look promising. 4.1
Quantitative results
Evaluation with raw data. Table 1 shows that all methods
perform well in the Raw data. Remarkably, we see that
relative locations can be predicted from implicit spatial language
at least as accurately as from explicit spatial language.
Unseen combinations. All models perform well on unseen
combinations (table not shown), remarkably closely to their
1We use 300-d GloVe embeddings
Unseen Words. Contrarily, large differences in performance are observed with unseen words (table not shown) where the model that uses embeddings (EMB) performs significantly better than the rest.
person, holding, cat
man, following, elephant person, riding, elephant man, flying, kite man, holding, kite man, walking, dog Heat maps in Fig. 2 show regions of predicted high (red) and low (blue) probability. The “heat” of the objects is assumed to be normally distributed with equal to the object’s center and to the object’s size. The EMB model is able to infer both, relative locations and sizes, e.g., predicting correctly the size of a “cat” relative to a “person” even though the model has never seen a “cat” before. Notably, the model learns to compose the triplet as a whole, distinguishing, e.g., (man, flying, kite) from (man, holding, kite).
Acknowledgments This work has been supported by the CHIST-ERA EU project MUSTER.2
2http://www.chistera.eu/projects/muster