The ABC Theory Repair System (ABC) has demonstrated its success in facilitating users to repair faulty theories utilizing distinct techniques. Yet, comprehending ABC-repaired theories becomes more challenging due to the presence of dummy constants or predicates introduced by ABC. In this paper, we propose a grounding system by incorporating Large Language Models (LLMs) to provide these dummy items with meaningful names. By applying ABC and grounding alternately, the resulting theory is both fault-free and semantically meaningful. Moreover, our study shows that LLMs without fine-tuning still exhibit capabilities of common knowledge, and their grounding performances are enhanced by providing suficient background or asking for more returns. Large language model, Closed-book question answering, Faulty logical theory repair, Automated theorem Logical theory stands as a reasoning tool in the field of artificial intelligence (AI), representing structured and precise representations of relationships among objects [1]. The theory needs to be refined to cope with new observations [ 2]. When users introduce novel information, the original theory may either make incorrect predictions or fail to predict the expected truth. [3].
CEUR ceur-ws.org
CEUR Workshop Proceedings dummy items. In the previous example, users can deduce that “dummyConst1” and “dummyConst2” are referring to a birth mother and a stepmother respectively. Yet, there are instances when users face uncertainty in naming these items. Since the current implementation of ABC does not encompass the consideration of their semantic implications, numerous generated repaired theories might be logically consistent but devoid of semantic meaning.
Example 1 : A Comparison of Original and Repaired Motherhood Theory Original Theory: ( , ) ∧ ( , ) ⟹ = Repaired Theory: ( , , 1 ⟹ ( , ) ⟹ (, ) ) ∧ ( , ,
The challenge of attributing meanings to meaningless symbols is known as the “symbol
grounding problem”, an important problem in Cognitive Science [
Our study demonstrates an application of LLMs to solve the symbol grounding problem.
The primary objective of this paper is to enrich the semantic content of the repaired theory by
utilizing LLMs to replace the names of dummy items with semantically meaningful content. We
hypothesize that the closed-book question answering (CBQA) task [
Figure 1 illustrates the process employed by the grounding system. We first parse the input
theory in Datalog, a subset of First Order Logic (FOL) [
Before grounding starts, ABC detects and repairs the fault in the input theory when it
conflicts with the given preferred structure ℙ , which represents users’ observations [
We design a heuristic to formulate a prompt in the ungrounded theory. We suspect that
LLMs perform grounding more accurately if we provide suficient knowledge in the prompt, so
we contain assertions without dummy items and multiple axioms with the same dummy item
in the same prompt. For each proposition with a maximum arity of 3 in axioms, we convert it
into natural language with the interpretation in Table 2 in the Appendix. The interpretation is
similar to [
For each grounding of dummy items, users are presented with one to two rounds of recommendations. The initial recommendations are from the “LLM-grounding Recommender”, a phase where the LLM’s suggestions are displayed. Users are provided with the option to directly select the LLM-suggested answers, retain the dummy name, or propose new names. The inclusion of the latter choice allows users to refine the grounding names based on the LLM-generated answers or to tailor them to their preferences. In the subsequent phase “Existeditems Recommender”, the system explores the presence of existing items within the theory that exhibit high similarity to the chosen or suggested groundings from the previous phase. This comparison process involves assessing the resemblance of all prior suggestions against constants or predicates within the theory, depending on the type of dummy item. This phase employs the F1 score of BERTScore vanilla (referred to as F1 BERTScore) [15] to gauge the similarity between items. BERTScore utilizes embeddings from the pre-trained BERT model, calculating the cosine similarity of embeddings to measure word matches between candidates and references [15]. Higher scores correspond to more significant similarity. Users also have the flexibility to set a threshold for the F1 BERTScore, enabling the system to recommend items that surpass the specified F1 BERTScore.
Example 2: Prompt Formulating in Repaired Tweety Theory ( ,
aNotice that the names of the penguins Tweety and Polly are not capitalized in the prompt.
An important consideration is that the grounding process has the potential to reintroduce faults into repaired theories. In response, we incorporate a safeguard as an extra feature by aiding users in re-running ABC following the exportation of all grounded theories. This validation step assesses whether the theories remain free from faults. This iterative approach involving repair and grounding persists until users attain a satisfactory theory. A practical example of the interplay between repair and grounding is depicted in the appendix.
We experimented with GPT-3.5 Turbo (4K context version) and GPT-4 (8K context version)
[
As ABC uses Datalog, a subset of FOL, the grounding system cannot be evaluated with major FOL datasets [17, 18]. We compromised to examine the performance of LLMs. We substituted some items with dummy names in the theories and studied if the LLMs could ground similar items as the original ones in our system. We constructed theories automatically from two knowledge bases, enriched WebNLG dataset [19] and excerpt of DART [20], and replaced some items with dummy names. These theories serve as simulations of the generated repaired theories, with both assertions and rules. Details of the construction of the evaluation dataset are in the project’s GitHub repository1.
We adopted two semantic-based metrics, F1 BERTScore [15] and SAS [21], to evaluate the semantic similarity of answers in LLM-grounding Recommender, as they are shown to have 1https://github.com/HistoChan/ABCGrounding a certain correlation between human judgement [21]. The former one is the same as the one used in the “Existed-items Recommender”. SAS, however, uses a pre-trained cross-encoder and applies the model by concatenating two texts with a separator token in between. Diferent from BERTScore, SAS considers two inputs together [21]. We calculated the scores of the LLM outcomes with the original items’ names. All the metrics values range from 0 to 1, with values closer to 1 indicating greater semantic similarity between candidates and references.
Prompt content
number of output basic w/ background w/ multi axioms w/ both w/ both
Table 1 is the statistics of the experiment, which shows that an LLM without any fine-tuning can still have an adequate grounding performance in zero-shot. The increase in the metrics confirms that the inclusion of additional background information in the prompt can obtain higher-quality grounding outcomes. The utilization of background information emerges as a more impactful hint for successful grounding, surpassing the efectiveness of querying multiple axioms in a single prompt. Moreover, the performances generally enlarge with model size, and an increase in the number of groundings would correspondingly enhance overall performance. We also adopted other models such as T5 models by Google [22], OpenLLaMA models from OpenLM Research [23], and Dolly 2.0 models by Databricks [24]. Their performances also support the above statements, which the project’s GitHub repository1 contains the statistics of their performances. Some case studies are in the appendix.
In this paper, we have proposed a system of symbol grounding for the ABC repair system. We formulate the grounding challenge into a CBQA task and require LLMs to return possible answers. The system also embraces user interactivity, in which users have a right to control the model use, types of formatting prompts and grounding results. This system not only helps to enhance the semantics in the repaired theory but also determines the rationality of the repair plan. Yet, we suspect that the grounding performance is limited by the quality of the prompt, which can be improved in the future. Additionally, we facilitate using LLMs without either ifne-tuning or few-shot learning for CBQA tasks by providing suficient background information and enhancing the number of outputs. Nonetheless, we do not deem that few-shot learning can be replaced by providing background information. It is worth studying if few-shot learning can achieve similar performance of fine-tuning, and if the performance enhancement with few-shot learning is limited by the scale of the model. [15] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, BERTScore: Evaluating Text Generation with BERT, arXiv preprint arXiv:1904.09675 (2019). URL: https://github.com/ Tiiiger/bert. [16] OpenAI, GPT-4 Technical Report, arXiv preprint arXiv:2303.08774 (2023). URL: http: //arxiv.org/abs/2303.08774. [17] S. Han, H. Schoelkopf, Y. Zhao, Z. Qi, M. Riddell, L. Benson, L. Sun, E. Zubova, Y. Qiao, M. Burtell, D. Peng, J. Fan, Y. Liu, B. Wong, M. Sailor, A. Ni, L. Nan, J. Kasai, T. Yu, R. Zhang, S. Joty, A. R. Fabbri, W. Kryscinski, X. V. Lin, C. Xiong, D. Radev, FOLIO: Natural Language Reasoning with First-Order Logic, arXiv preprint arXiv:2209.00840 (2022). URL: http://arxiv.org/abs/2209.00840. [18] J. Tian, Y. Li, W. Chen, L. Xiao, H. He, Y. Jin, Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (2021) 3738–3747. [19] T. C. Ferreira, D. Moussallem, S. Wubben, E. Krahmer, Enriching the WebNLG corpus, in: Proceedings of the 11th International Conference on Natural Language Generation, 2018, pp. 171–176. URL: http://data.statmt.org/wmt17_systems. [20] L. Nan, D. Radev, R. Zhang, A. Rau, A. Sivaprasad, C. Hsieh, X. Tang, A. Vyas, N. Verma, P. Krishna, Y. Liu, N. Irwanto, J. Pan, F. Rahman, A. Zaidi, M. Mutuma, Y. Tarabar, A. Gupta, T. Yu, Y. C. Tan, X. V. Lin, C. Xiong, R. Socher, N. F. Rajani, DART: Open-Domain Structured Data Record to Text Generation, in: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Online, 2021, pp. 432–447. URL: https://aclanthology.org/2021.naacl-main.37. doi:10.18653/v1/2021.naacl- main.37. [21] J. Risch, T. Möller, J. Gutsch, M. Pietsch, Semantic Answer Similarity for Evaluating Question Answering Models, arXiv preprint arXiv:2108.06130 (2021). URL: http://arxiv. org/abs/2108.06130. [22] C. Rafel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, Journal of Machine Learning Research 21 (2020) 1–67. URL: http://jmlr.org/papers/v21/20-074.html. [23] X. Geng, H. Liu, OpenLLaMA: An Open Reproduction of LLaMA, 2023. URL: https://github.
com/openlm-research/open_llama. [24] M. Conover, M. Hayes, A. Mathur, X. Meng, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, R. Xin, Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM, 2023. URL: https://www.databricks.com/blog/2023/04/12/ dolly-first-open-commercially-viable-instruction-tuned-llm. Example 3 (Continue): Repair and Grounding a Capital Theory Step 2: land, in ( , ) ∧ ( , )
ABC finds there is a fault in having two capitals in Scotwhich it suggests replacing “capitalOf” with “dummyPred”
( , ) , and it is grounded as ‘cityOf‘: ⟹ = ⟹ ( ℎ, ⟹ ⟹ (, ) ( , ) )
We compare the performance of grounding containing background information of two models in Example 4, which illustrates that the grounding would be more reasonable with providing background information.
Example 4: A comparison of grounding answers of Example 2 • Without extra content: What is a possible entity, such that opus is broken wing of the entity? Answer within 5 wordsa. • With background information: Given that opus is super penguin. What is a possible entity, such that opus is broken wing of the entity? Answer within 5 wordsa.
– GPT-3.5 Turbo: Defective wing. – T5 Large (NQ): Feathers are – GPT-3.5 Turbo: X is injured – T5 Large (NQ): Cannot fly aNotice that the name of the penguin Opus is not capitalized in the prompt.
We provide Example 5 as a comparison of grounding performance on diferent LLMs, in which the answers are more accurate with larger model sizes. Despite our explicit instruction to return answers within five words and request of the maximum output tokens as five, Dolly 2.0 and OpenLLaMA still have a high tendency to return a complete sentence.
Example 5: A comparison of grounding answers of a repaired Capital Theory Question: What is a possible entity such that edinburgh is cap of of the entity? Answer within 5 words.
• Dolly 2.0 3B: Edinburgh is the capital of • Dolly 2.0 7B: The answer is the Edinburgh • OpenLLaMA 3B: edinburgh is cap of • OpenLLaMA 7B: The answer is Scotland. • GPT-3.5 Turbo & T5 XL (NQ): Scotland • GPT-4: Scotland or United Kingdom. • T5 Small & Large : Edinburgh • T5 Small (NQ): other social entity • T5 Large (NQ): Kingdom of Scotland
We experimented on the efect of the number of groundings generated from LLM. This experiment lay in the diversity of the returned results. With open-ended questions like that in Example 6, the answers returned reflect the multifaceted nature of potential responses. The augmented number of returned answers not only aids in identifying high-quality grounding but also empowers users to brainstorm a broader spectrum of possible groundings.
Example 6: A comparison of answers of prompt from a repaired Tweety Theory Question: Given that tweety is penguin. What is a possible entity such that polly is bird of the entity, and In a FOL expression, if x is bird of the entity, then x is fly? Answer within 5 words.a.
• GPT-3.5 Turbo: “Airplane.”, “Flying creature like parrot”, “Fish” • GPT-4: “Sky or Air could be”, “Possible entity is ’the”, “The possible entity: magical” • Suggested Answer: “flying”