Dual-Phase Models for Extracting Information and Symbolic Reasoning: A Case-Study in Spatial Reasoning

Abstract

Spatial reasoning over text is challenging as the models need to extract the direct spatial information from the text, reason over those, and infer implicit spatial relations. Recent studies highlight the struggles even large-scale language models encounter when it comes to performing spatial reasoning over text. In this paper, we explore the potential benefits of disentangling the processes of information extraction and reasoning in models to address this challenge. To explore this, we devise various models that disentangle extraction and reasoning (either symbolic or neural) and compare them with SOTA baselines with no explicit design for these parts. Our experimental results consistently demonstrate the eficacy of disentangling, showcasing its ability to enhance models' generalizability within realistic data domains.

Keywords

• eol>Spatial Reasoning
• Spatial Role Labeling
• Disentangling Extraction and Reasoning
• Pretrained Language Models

-

1. Introduction a) PLMs: End-to-End model # and QA+SpRL annotation, showcasing the advantage of the design of this model in utilizing QA and SpRL data within explicit extraction layers and the data preprocessing. Also, the better performance of this model compared to PistaQ demonstrates how the end-to-end structure of SREQA can handle the errors from the extraction part while capturing some rules and commonsense knowledge from ReSQ training data that are not explicitly supported in the symbolic reasoner.

Table 1 Table 3 Results on auto-generated datasets. We use the accuracy Result on ReSQ. *Further training on SpaRTUN. The metric for both YN (Yes/No) and FR (Find Relations) questions. _ℎ refers to evaluation without further training on ReSQ or CLEF training data.

Ttroaitnraining tohnetehxetrcaocrtrieosnpomnoddiunlgeso,rwaeuxaidliaaprtytdhaetmastehtsr.oTughhe Story: abepthwoeteonoafnadroaopmicwtuitrhe wonhittheewwaalllsl a,tbwovoestinhgelebebdesd.s with a night table in outcomes of our experiments provide noteworthy in- Question: Are the beds below the picture? Answer: Yes sights: Story BERT F0a:c['tasp:ircigtuhrte(2',,'t1h),ebbeelodws(']2,,20:)[,'an']e,a1r:([2'a,0p)icture', 'the wall'] ( 1 ) Tables 1 and 2 show the performance of models Facts: GPT3 3F:ac['ttsw:oasbionvgele(5b,e3d),sa',b'tohvee(b5e,d6s)'.].,.5: ['a picture'], 6: ['the wall', 'the beds'] SopnatRwToUdNataasnedtsSwp aitrhtcQoAnt-rAoulletdo.eAxpseirti misesnhtoawlcno,nPdiistitoanQs, Queries: GBPETR3T bbeellooww((30 ,, 50))??oorrbbeellooww((30,,61))?? voauttipoenrsfoirnmtshiasllsPetLtMingbadseemlionnesstarnatdeStRhEatQdAis.eOnutarnogblsinerg- Reasoning: GBPETR3T bbeellooww((30 ,, 50))==TFraulsee,,bbeeloloww(3(0, ,61))==FFaalslsee→→AnAsnwsewre=rY=eNso extraction and symbolic reasoning compared to PLMs Figure 2: An example of using BERT-based SpRL and GPT3 enhances the models’ reasoning capabilities, even with as information extraction in PistaQ on a ReSQ. comparable or reduced supervision. This result suggests that SpRL annotations are more efective in the PistaQ pipeline than when utilized in BERT-EQ in the form of ( 3 ) Recent research[1] show that powerful LLMs canQA supervision. Note that the BERT-EQ uses all the orig- not perform well on SQA task. Similarly, our experiments, inal dataset questions and extra questions created from as shown in Tables 2 and 3 show the lower performance the full SpRL annotations. of GPT3.5 compared to humans and our models PistaQ ( 2 ) We select ReSQ as an SQA dataset with realistic and SREQA. However, we show that harnessing LLMs’ settings and present the result of models on this dataset potentials in information extraction [8] can yield signifin Table 3. The low performance of PistaQ is attributed icant benefits within the disentangled structure of exto , first, the absence of integrating commonsense in- traction and reasoning. Figure 2 provides a comparison formation in this model and, second, the errors in the between the BERT-based SpRL extraction modules and extraction modules, which are passed to the reasoning GPT3.5 with _ℎ prompting in PistaQ. It shows modules. SREQA* surpasses the PLMs trained on QA that GPT3.5 extracts more accurate information, leading to correct answers from the reasoning phase.

References

[1] Y. Bang , S. Cahyawijaya , N. Lee , W. Dai , D. Su , B. Wilie , H. Lovenia , Z. Ji , T. Yu , W. Chung , Q. V. Do , Y. Xu , P. Fung , A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination , and interactivity, 2023 . arXiv: 2302 . 04023 .

[2] R. Mirzaee , H. Rajaby Faghihi , Q. Ning , P. Kordjamshidi, SPARTQA: A textual question answering benchmark for spatial reasoning, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics , Online, 2021 , pp. 4582 - 4598 . URL: https://aclanthology.org/ 2021 .naacl-main. 364 . doi: 10 .18653/v1/ 2021 . naacl-main. 364 .

[3] D. Mollá , M. Van Zaanen , D. Smith , et al., Named entity recognition for question answering ( 2006 ).

[4] A. C. Mendes , L. Coheur , P. V. Lobo , Named entity recognition in questions: Towards a golden collection ., in: LREC , 2010 .

[5] D. Shen , M. Lapata , Using semantic roles to improve question answering , in: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL ) , Association for Computational Linguistics , Prague, Czech Republic, 2007 , pp. 12 - 21 . URL: https://aclanthology. org/D07-1002.

[6] H. R. Faghihi , P. Kordjamshidi , C. M. Teng , J. Allen , The role of semantic parsing in understanding procedural text , arXiv preprint arXiv:2302.06829 ( 2023 ).

[7] R. Mirzaee , P. Kordjamshidi , Transfer learning with synthetic corpora for spatial role labeling and reasoning , in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022 , pp. 6148 - 6165 . URL: https://aclanthology.org/ 2022 .emnlp-main. 413 .

[8] T. Shen , G. Long , X. Geng , C. Tao , T. Zhou , D. Jiang , Large language models are strong zero-shot retriever , arXiv preprint arXiv:2304.14233 ( 2023 ).