Spatial Reasoning is a fundamental aspect of human cognition to perform everyday activities. It is also an essential skill for machines to engage in human-like interactions with the environment. However, recent research shows that even state-of-the-art language models struggle in spatial reasoning, especially in unobserved situations with complex input compositions. This is attributed to not achieving the right level of abstraction required for their generalizability. To alleviate this issue, we propose training the language models with neuro-symbolic techniques to exploit the spatial logical rules of reasoning and provide an additional source of supervision to the models. Training models to adhere to spatial reasoning rules guides them to make more efective abstractions for generalizability and transfer learning. We evaluate our proposed technique on various benchmarks for spatial reasoning over text. Our results based on the multiple language model backbones show the efectiveness of our neuro-symbolic training in domain transfer and complex multi-hop spatial reasoning.
Spatial reasoning is essential for humans cognition and
also plays a crucial role in many AI applications, including
language grounding [
Large Language models have been widely applicable in
many of problems in these areas and, in some cases, show
human level performance [
In this paper, we address the issue of spatial reasoning
in LMs and their dificulty in obtaining the abstractions
required for generalizability in unobserved complex situations
employing a generic neuro-symbolic framework. We
propose to fine-tune the LMs with a neuro-symbolic technique
that exploits the spatial logical rules to guide the level of
abstraction captured during training. In particular, we train
the models to minimize both the cross-entropy loss and
the violation from logical constraints. We demonstrate the
efectiveness of our proposed framework in both
encoderbased and generative language models. For evaluation, we
use three Spatial Question Answering (SQA) benchmarks,
SpartQA-HUMAN [
The results show that our proposed method benefits both
LM types, especially when multiple hops of reasoning are
required. The performance improvements over multiple
domains confirm our hypothesis about the efectiveness of
neuro-symbolic training on generalizability.
2. Training with Spatial Logic
The spatial logical rules used in our framework are based
on the developed spatial logical knowledge base in [
R1 q3: Triangle below box
Converse: Above(X, Y) => Below(Y, X)
Topological: Below(X, Y) /\ Contain(Y, Z) =>
Below(X, Z)
Converse: CoveredBy(X, Y) =>
Contain(Y, X) +
R3 T: Triangle below
Square q2: Square is in box
Initial facts
R2 q4: Box contain square
Intermediate
Target Rules 1 2 3 Rules 1 2 3
Constraints in YN (1) ⇒ (3) (2) ⇒ (4) (3) ∧ (4) ⇒ ()
Constraints in FR (1, ) ⇒ (3, ) (2, ) ⇒ (4, ) (3, ) ∧ (4, ) ⇒ (, ) that if an object is above object , therefore, object is below object . The rest of the rules use a similar notation.
To apply training with Spatial Logic, we follow three steps. Firstly, we create example-specific rules based on the given Spatial Logic. This process is explained in the example of Figure 1.
We use the resolution tree, which provides the logical implication steps, to infer the answer to the final query from the input context. Note that our synthetic training data (e.g., SpaRTUN) provides the logical representation of the context. We start creating the tree using initial facts in the given context and a forward chaining approach to find the applicable rules. In this way, we obtain the intermediate inferred facts. We denote fact as and the sequence of all derived intermediate facts, including the target question, as -ℎ.
Secondly, we generate the consistency constraints between s given the -ℎ. To explain the consistency constraints between questions, we denote the answer to the YN questions as (), which will be True if the answer to is True. We denote the answer to the FR questions as (, ), which will be True if the specified relation exists in the set of answers to . We obtain a set of consistency rules per training example, as shown in Table 1. For example, in Figure 1, if 1: box above the square," is True, then 3: "triangle below box," should be True. The corresponding constraints for YN will be (1) ⇒ (3), and for the FR case, will be (1, ) ⇒ (3, ).
Lastly, after obtaining the consistency constraints, we
minimize the violation of the model from these constraints
by adding a corresponding term in the loss function
objective. However, we need to obtain a diferential form of
logic as a surrogate of the original logical constraints
violation to do this. We follow the previous research for this
goal [
We conduct two sets of experiments on realistic (ReSQ) and synthetic datasets (SpartQA, SpaRTUN, and StepGame). With these experiments, we empirically evaluate the impact of our proposed logic-based fine-tuning on small-scale language models and compare them to very large language models that merely use prompt engineering. We evaluate the performance of our proposed method on two types of language models, encoder-based and generative models. We select BERT as the baseline encoder-based and Flan-T5 as the baseline generative model.
We also report the results of basic fine-tuning with the 1https://hlr.github.io/domiknows SpaRTUN dataset in two, so-called, BERT-T and Flan-T5-T models.
Realistic Domain. ReSQ serves as the realistic SQA domain. As observed in Table 2, using the -ℎ is efective for both models (BERT and Flan-T5) with a notable improvement on Flan-T5. Particularly, Flan-T5-T+-ℎ (line 6) shows 2% improvement over Flan-T5-T (line 5).
For a deeper understanding of these results, we analyzed the performances on diferent splits of ReSQ. There are three splits based on the manually annotated depth of reasoning required to answer questions in ReSQ. The first two splits include questions that require one or two hops of reasoning, denoted as =1, and =2. The last type is unclassified , which covers questions where the depth of reasoning is dificult to determine. Those questions require more of commonsense knowledge. Our observations in Table 2 reveal that our model consistently improves on = 2 but adversely afects BERT’s performance on = 1 and the unclassified categories. According to this result, we conclude that when more hops of reasoning are required, logic-based tuning demonstrates significant improvement. However, our proposed tuning method is less efective in the unclassified class, which requires commonsense knowledge.
On the other hand, LLMs show superior performance on ReSQ compared to all fine-tuning results. The LLMs consistently exhibit around 2% to 13% higher performance compared to Flan-T5+T+-ℎ (lines 7 to 13). The performance is much higher on the unclassified subset of the dataset, which can be seen even with the zero-shot method. This implies that LLM’s out-performance is mainly due to their commonsense knowledge rather than their complex reasoning capability, in contrast to our proposed method, which deals with complex multi-hop reasoning.
Nevertheless, we observe that using logic-based finetuning yields a higher improvement over Flan-T5 compared to BERT on the unclassified subset. This indicates that the -ℎ approach can guide complex reasoning when applied to a model with more commonsense knowledge. Synthetic Domain with More Complex Logical Reasoning. SpartQA-Human and StepGame are synthetic domains used in our experiments. We consistently observe improvement with our proposed -ℎ in this domain, which typically requires many more hops of reasoning. As observed in Table 2, -ℎ consistently shows improvement in FlanT and BERT compared to fine-tuning without it. Moreover, the gap between small PLMs and LLMs is much smaller in this dataset compared to the realistic domain (ReSQ). This is expected since LLMs are better at commonsense than complex reasoning as previously presented.
The result is further supported when assessing the
proposed method on StepGame. As can be observed in Table 3,
the fine-tuning method consistently demonstrates
significant positive diferences in all reasoning steps compared to
LLMs. The struggle of GPT3 on reasoning over StepGame
is also investigated in [