As Large Language Models gain prominence in the AI landscape, it is essential to understand their capabilities and limitations, among others in terms of reasoning. This paper is a first step towards understanding the capabilities in terms of defeasible rule-based reasoning. It presents results of initial experiments and discussed future research directions. RuleML+RR'23: 17th International Rule Challenge and 7th Doctoral Consortium, September 18-20, 2023, Oslo, Norway htp:/ceur-ws.org CEUR Workshop Proceedings (CEUR-WS.org) ISN1613-073
Large Language Models (LLMs) have risen in prominence recently due to their exceptional performance in various language related tasks since they are the underlying technology behind chat bots such as ChatGPT1. Large Language models such as LaMDA [1] and GPT [2] are based on training deep neural networks with billions of parameters over huge lexical datasets and often employing human judgment in a semi-supervised (e.g., reinforcement learning) training setting [3, 4]. The exceptional -human level- performance of LLMs in various tasks has led to a widespread discussion about the potential benefits and dangers of such technologies in various areas and human society in general including petitions to pause research on more capable LLMs [5].
Despite their impressive performance Large language Models, including their ability to demonstrate an emerging intelligent behaviour and reasoning capabilities, leading to the point of considering them forerunners of Artificial General Intelligence[ 6] several issues related to LLMs have been identified, such the energy cost of training LLMs[ 7, 8], dificulty to control their behaviour[9], ensure conformity with stakeholders requirements and norms and interpret their functionality [10]. The interpetability of LLMs is crucial issue since neural network based LLMs appear to be ‘black boxes’, in contract to logic based systems, and although various attempts exist to deal with this problem, including the use of LLMs to interpret LLMs [11], this is still an unresolved issue. In addition, since LLMs are based on vast amount of raw text CEUR Workshop Proceedings they tend to replicate their input rather that apply robust reasoning [12]. Based on raw text instead of structured knowledge bases integrating machine readable semantics, contributes to the dificulty to achieve eficient reasoning and this is an issue examined in various works such as [13] and surveyed in [14]. Various attempts to integrate Knowledge graphs (KGs) to LLMs have been proposed [15, 16] as a solution to the last issue, but recent advances in LLMs capabilities, including high performance on academic and professional exams [2], illustrated the need for an updated evaluation of the reasoning capabilities of LLMs. This updated evaluation should take into account the recent developments in the field, including the deployment of systems such as chatGPT employing the benefits of scalability [ 17] and the LLMs demonstrated ability to adjust to new tasks given just a small number of examples [18]. Furthermore LLMs capabilities with respect to important formalisms such as defeasible reasoning have not been examined in detail yet.
This work is an initial step towards developing a deep understanding of reasoning capabilities of LLMs with emphasis of nonmonotonic reasoning. For achieving this we conducted experiments on nonmonotonic rule-based reasoning using chatGPT and compare it with performance of dedicated reasoning systems and human users. This kind of reasoning is important for cases where knowledge is incomplete and conflicting which is the case in many application areas, including law and healthcare.
Defeasible reasoning is a form of nonmonotonic reasoning [19] that seeks to draw conclusions
in the absence of contrary information, conclusions that may be invalidated by later additions
to the knowledge base. A prototypical example regards the flying abilities of Tweety the bird:
(
One use of defeasible reasoning is reasoning with rules and exceptions, an exception seen as more reliable than the general rule for the cases it applies to. We used an example from the domain of law where rules with exceptions prevail [20], specifically the use case presented in [21]: User: So the setting is this: Suppose that a Privacy Act contains the following norms: Section 1. The collection of personal information is forbidden, unless acting on a court order authorising it. Section 2. The destruction of illegally collected personal information before accessing it is a defence against the illegal collection of the personal information. Section 3. The collection of medical information is forbidden, unless the entity collecting the medical information is permitted to collect personal information. Suppose an entity, subject to the Act, collects some personal information without being permitted to do so; at the same time they collect medical information. The entity recognises that they illegally collected personal information (i.e., they collected the information without being authorised to do so by a Court Order) and decides to remediate the illegal collection by destroying the information before accessing it. Is the entity compliant with the Privacy Act above? ChatGPT: Based on the information you’ve provided, the entity in question collected personal information without being permitted to do so and also collected medical information. According to Section 3 of the Privacy Act, the collection of medical information is forbidden unless the entity collecting it is permitted to collect personal information.
In addition, according to Section 1 of the Privacy Act, the collection of personal information is forbidden unless authorized by a court order. Therefore, the entity in question would be in violation of the Privacy Act for both the collection of personal information without a court order and the collection of medical information without being authorized to collect personal information.
While the entity’s decision to destroy the illegally collected personal information before accessing it may be seen as a defence under Section 2 of the Privacy Act, it’s important to note that this defence is only available if the collection of the personal information was illegal in the first place. In this case, the collection of personal information was indeed illegal, so the defence would apply. However, the entity would still be in violation of the Privacy Act for the collection of medical information without being permitted to do so. Therefore, the entity would not be fully compliant with the Privacy Act in this scenario.
This is an impressive reply from the LLM! It provides both a correct response and an informative analysis of the situation and justification for its conclusion.
Next we consider examples involving strict and defeasible rules, conflicts and where specificity may play a role. Given the already identified afinity of ChatGPT to use other background knowledge when predicates and atoms are real-world entities, we use imaginary names of species on an imaginary planet, following [22]. This way, we wanted to test pure reasoning capabilities that may be indicative of what to expect in completely new situations.
(
(
(
(
(
(
A graphical representation of questions (
The replies of ChatGPT show a general disregard of specificity. Seeking to further understand the reason, positive and negative conclusions were inverted, but all replies were consistent so no preference of positive over negative conclusions was observed. However, there was a dramatic change in behavior the moment the order of statements was inverted.
(
Now the answer b was provided! The explanation provided by ChatGPT provided the evidence supporting this conclusion. So, our initial investigation revealed serious weaknesses in terms of reasoning, in that the system seems to not appreciate the need to carefully weigh the pros and cons of a conclusion making use of implicit specificity.
But most interestingly, once ChatGPT is explicitly told about specificity, its result is updated
correctly. We observed that for example (
It is interesting to contrast the performance of LLMs not only against theories of defeasible
reasoning but also the way human actually reason. Although nonmonotonic reasoning was
meant to enable “jumping into conclusions”, the formal theories developed are very complex
and humans actually find this kind of reasoning challenging, as research into human cognition
has established. The examples (
The results demonstrate that the discrepancy of human reasoning from the outcomes postulated by KR is similar to the discrepancy between ChatGPT and KR. As we move to better understand defeasible reasoning capabilities of LLMs, in is very interesting to work out the 2Nonmonotonic systems like defeasible logics or default logic with priorities representing the concept of specificity.
3The input for question (
This is a first step towards our goal of understanding the reasoning capabilities of Large Language Models. In this paper we focused on defeasible reasoning and conducted some experiments with ChatGPT. The results show a mixed picture: basic principles are respected, explanation facilities are superb and performance on real-world entities is good. On the negative side, we observed varying behavior based on the names of predicates (real world or artificial) and reliance on syntactic aspects. Clearly, there is a lot of research to be done. In the following, we highlight a number of research directions.
Prompt engineering. We observed that using abstract or real-world names has an efect on the outcomes of the reasoning, similarly to the sequence of statements. Additionally, pointing to the principle of specificity had a desired change in outcome. All this points to the fact that we need a better understanding on how to interact with LLMs, an observation that has been made in many other areas of use of LLMs.
KR vs human cognition vs LLMs. Knowledge representation sets principles of sound reasoning but humans reason in diferent ways, indeed there is an increasing body of work seeking to understand the diferences between logical reasoning and human reasoning. LLMs bring an additional dimension, and it interesting to contrast the three approaches. Apart from this being a pure research question, we can hope to gain insights into how LLMs and humans can be taught reasoning, and into how conversational interaction about reasoning outcomes with LLMs and humans can take place.
Benchmarking reasoning capabilities of LLMs. Overall, there is a need for a systematic analysis of reasoning schemes, chains of reasoning etc. Developing relevant benchmarks is an important vehicle to this end and we intend to work on it. Our initial work reported here is about nonmonotonic rule-based reasoning but we have longer-term plans to investigate and benchmark other forms of reasoning, including first order reasoning, epistemic reasoning, reasoning about change, reasoning about action and reasoning about time.
Acknowledgments This research was funded by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKILabor with grant No. 01DD20003. [14] J. Huang, K. C.-C. Chang, Towards reasoning in large language models: A survey, arXiv preprint arXiv:2212.10403 (2022). [15] C. Zhen, Y. Shang, X. Liu, Y. Li, Y. Chen, D. Zhang, A survey on knowledge-enhanced pre-trained language models, arXiv preprint arXiv:2212.13428 (2022). [16] D. Yin, L. Dong, H. Cheng, X. Liu, K.-W. Chang, F. Wei, J. Gao, A survey of knowledgeintensive nlp with pre-trained language models, arXiv preprint arXiv:2202.08772 (2022). [17] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford,
J. Wu, D. Amodei, Scaling laws for neural language models, 2020. arXiv:2001.08361. [18] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei, Language models are few-shot learners, 2020. arXiv:2005.14165. [19] G. Antoniou, M.-A. Williams, Nonmonotonic reasoning, Mit Press, 1997. [20] G. Governatori, M. Hashmi, No time for compliance, in: 2015 IEEE 19th International
Enterprise Distributed Object Computing Conference, IEEE, 2015, pp. 9–18. [21] G. Governatori, The regorous approach to process compliance, in: 2015 IEEE 19th
International Enterprise Distributed Object Computing Workshop, IEEE, 2015, pp. 33–40. [22] M. Ford, D. Billington, Strategies in human nonmonotonic reasoning, Computational
Intelligence 16 (2000) 446–468. [23] C. K. Baker, C. Denny, P. Freund, T. Meyer, Cognitive defeasible reasoning: the extent to which forms of defeasible reasoning correspond with human reasoning, in: Artificial Intelligence Research: First Southern African Conference for AI Research, SACAIR 2020, Muldersdrift, South Africa, February 22-26, 2021, Proceedings 1, Springer, 2020, pp. 199–219. [24] M. Ford, Human nonmonotonic reasoning: The importance of seeing the logical strength of arguments, Synthese 146 (2005) 71–92. [25] M. Ragni, C. Eichhorn, T. Bock, G. Kern-Isberner, A. P. P. Tse, Formal nonmonotonic theories and properties of human defeasible reasoning, Minds and Machines 27 (2017) 79–117.