We explore whether Large Language Models (LLMs ) are capable of logical reasoning with distorted facts, which we call Deduction under Perturbed Evidence (DUPE). DUPE presents a unique challenge to LLMs since they typically rely on their parameters, which encode mostly accurate information, to reason and make inferences. However, in DUPE, LLMs must reason over manipulated or falsified evidence present in their prompts, which can result in false conclusions that are valid only under the manipulated evidence. Our goal with DUPE is to determine whether LLMs can arrive at these false conclusions and identify whether the dominant factor influencing the deduction process is the encoded data in the parameters or the manipulated evidence in the prompts. To evaluate the DUPE capabilities of LLMs, we create a DUPEd version of the StrategyQA dataset, where facts are manipulated to reverse the answer to the question. Our findings show that even the most advanced GPT models struggle to reason on manipulated facts - showcasing poor DUPE skills - with accuracy dropping by 45% compared to the original dataset. We also investigate prompt settings inspired from student simulation models a.k.a. knowledge tracing models, which mitigate the accuracy drop to some extent. Our findings have practical implications for understanding the performance of LLMs in real-world applications such as student simulation models that involve reasoning over inaccurate information. The prompts and dataset are available at https://github.com/lufycodes/gpt-knowledge-tracing .
Over the last several years, Transformer models have played a significant role in shaping the
ifeld of Natural Language Processing (NLP) [
In this paper, we investigate the reasoning abilities of LLMs models under a novel paradigm
we dub Deduction under Perturbed Evidence (DUPE for short). By testing LLMs’ capacity to
https://sites.google.com/view/shashanksonkar/ (S. Sonkar); https://richb.rice.edu/ (R. G. Baraniuk)
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
reason with flawed or perturbed evidence, we aim to determine whether LLMs can generate
logically sound yet erroneous conclusions when presented with misleading information. Strong
DUPE skills are critical in NLP applications like student simulations [
Contributions: This paper develops a novel reasoning paradigm – Deduction under Perturbed Evidence (DUPE) – to examine whether LLMs arrive at diferent conclusions when presented with distorted initial facts. To test the DUPE capabilities of LLMs, we create a DUPEd version of StrategyQA dataset (Figures 1, 2). StrategyQA [16] is an open-domain QA dataset that is characterized by its explicit provision of the necessary facts required to answer each yes-no question. In the DUPEd version of the dataset, we manipulate the facts provided in a way that results in a diferent answer to the original question.
Our findings reveal that state-of-the-art LLMs, , including GPT3.5 and GPT4, struggle significantly on the newly introduced DUPEd-StrategyQA dataset. The accuracy of these models dropped drastically by approximately 45%, falling from an impressive 91.9% on the original dataset to only 46.7% on the DUPEd-StrategyQA dataset. In addition, we conduct an ablation study on the DUPEd-StrategyQA dataset by categorizing it into two distinct parts based on the type of manipulation used – one involving language perturbations and the other involving mathematical manipulations. Furthermore, our results demonstrate that the accuracy drop can be mitigated by using prompt settings inspired by student simulation models. This approach reduced the accuracy drop to 29%, with the models achieving an accuracy of 62.7% on the DUPEd-StrategyQA dataset. Our findings carry crucial implications for practical LLMs applications, particularly in the realm of student simulation models that demand reasoning over erroneous information.
In this section, we overview the DUPE reasoning framework, provide details on the DUPEd version of AllenAI’s StrategyQA dataset, and then explore customized prompt settings designed to assess the DUPE skills of LLMs. 2.1. DUPE Given a true-false question , the correct response ∈ { , } and facts that determine the truth or falsehood of ( ), we change to ′ s.t. the correct response to flips to ¬ under altered facts ′,
DUPE((, , ) ) = (, ′, ′) s.t. ′ = ¬ , editdist( , ′) < , (1) where editdist ensures that the edit distance between the fact strings and ′ is less than a threshold . The threshold is generally set to two to three words to ensure minimal changes to underlying facts (examples in figure 2). The new DUPEd-tuple (, ′, ′) can be used to probe the DUPE capabilities of LLMs as shown in Figure 1.
We use AllenAI’s StrategyQA dataset [16] to assess the DUPE skills of LLMs. StrategyQA dataset provides explicit facts for answering open-domain questions. We create a DUPEd version of StrategyQA dataset composed of a total of 325 examples, of which 173 introduce natural language perturbations, while the remainder introduce mathematical errors (refer to examples in figure 2).
While designing the DUPEd version, we were careful to modify the facts in the most minimal way possible As a result, we made a conscious efort to only alter one or two words in the original facts whenever possible, in order to preserve the overall meaning and context of the original question. Additionally, we refrained from using explicit negation, such as the word not, to modify the facts, since our intent is not to evaluate the reasoning proficiency of LLMs in handling negation.
DUPE is highly relevant to student simulation models [
DUPE and Counterfactual Reasoning: Counterfactual reasoning and student simulation models require diferent types of reasoning. In counterfactual reasoning, the focus is on exploring hypothetical scenarios that may or may not correspond to actual reality. The fact that the information being considered is hypothetical or counterfactual is usually known beforehand.
In contrast, a student simulation model needs to reason about both true and false information, and may not know beforehand whether the information being considered is true or false. For example, in figure 2, the model lacks prior knowledge about which facts are true and which ones are perturbed. The model must identify incorrect answers from the student to make inferences about future questions, which requires robust and nuanced reasoning capabilities beyond those needed for counterfactual reasoning.
We evaluate the DUPE capabilities of the two largest GPT models – GPT3.5 (version gpt-3.5turbo-0301) and the latest GPT4 model (version gpt-4-0314) – via experiments under two diferent prompt settings, P1) “You are a question answering model. Your task is reason on provided evidence to answer a YES or NO question”, and P2) “You are a student simulation model. Your task is reason on student’s responses to accurately measure the student’s current knowledge state and predict the student’s response to a YES or NO question based on the student’s current knowledge state” from section 2.3. An example is illustrated in Figure 1.
In the prompt setting P1, both GPT3.5 and GPT4 performed poorly on the DUPEd version of the dataset, with a decrease in accuracy by 46.0%. and 45.2% respectively. As expected, the latest GPT4 model demonstrates superior performance to GPT3.5 on both the original and the DUPEd StrategyQA dataset.
Prompt P2 inspired by student simulation setting informs/ primes the models that the provided evidence may be incorrect since the evidence reflects the erroneous nature of students’ responses. We found that prompt setting P2 performs significantly better than P1 by a margin of 16.0% for the GPT4 model. However, there was still a significant 29.2% drop in accuracy compared to GPT4’s performance on the original dataset.
While curating the DUPEd-StrategyQA dataset, we divided the perturbations introduced into two distinct categories - one that involved language perturbations, while the other manipulated mathematical information (see figure 2). Our finding suggest that both GPT models are more resilient to math perturbations compared to language perturbations. E.g. for GPT3.5 there was accuracy drop of 58.7% and 32.4 for language and math Perturbations respectively, while for GPT4 the accuracy drops were 50.3% and 39.4.
To explain the GPT models’ poor performance on the DUPEd dataset, we need to identify the main factor influencing their reasoning process, i.e., whether it is the encoded information in parameters or the manipulated evidence in prompts. Recent studies have shed light on this issue, suggesting that factual information encoded in the parameters of LLMs plays a dominant role in governing the generated output. For instance, the feed-forward layers in transformer models function as key-value memories, which implies that they encode factual information, as noted by Geva et al. [19]. Moreover, Meng et al. [20] demonstrated that localized computations, such as Rank-One Model Editing (ROME), can modify these factual associations, leading to alternative conclusions. These findings suggest that the encoded information in parameters has a significant impact on LLMs’ reasoning process; further investigation is left for future work.
In this paper, we have introduced a new reasoning paradigm we call Deduction under Perturbed Evidence (DUPE for short). Through DUPE , we have assessed the ability of LLMs models to arrive at logically sound yet erroneous conclusions when faced with distorted initial facts. Our study, which used a carefully curated dataset to evaluate DUPE abilities, has revealed that even the most advanced GPT models struggle with logical reasoning in the presence of falsified information. Moving forward, we plan to investigate into the performance of diferent LLMs with our dataset in varied prompt settings.
Due to limitations in both financial and computational resources, we had to limit our testing to only the most advanced LLMs – the GPT models. Consequently, we directed our attention towards developing a dataset for evaluating proposed reasoning scenarios. As a result of these limitations, we chose to focus specifically on the evaluation of the two largest models ofered by OpenAI. While we recognize that other LLMs may produce diferent outcomes, we believe that our dataset could serve as a valuable resource for further research into the capabilities and limitations of LLMs .
This work was supported by NSF grants 1842378, ONR grant N0014-20-1-2534, AFOSR grant FA9550-22-1-0060, and a Vannevar Bush Faculty Fellowship, ONR grant N00014-18-1-2047. [15] N. Liu, Z. Wang, R. Baraniuk, A. Lan, Open-ended knowledge tracing for computer science education, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 3849–3862. [16] M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, J. Berant, Did Aristotle use a laptop? A Question Answering benchmark with implicit reasoning strategies, Transactions of the Association for Computational Linguistics 9 (2021) 346–361. [17] S. Sonkar, A. E. Waters, A. S. Lan, P. J. Grimaldi, R. G. Baraniuk, qdkt: Question-centric
Deep Knowledge Tracing, arXiv preprint arXiv:2005.12442 (2020). [18] M. Bommarito II, D. M. Katz, GPT takes the Bar Exam, arXiv preprint arXiv:2212.14402 (2022). [19] M. Geva, R. Schuster, J. Berant, O. Levy, Transformer feed-forward layers are key-value memories, arXiv preprint arXiv:2012.14913 (2020). [20] K. Meng, D. Bau, A. Andonian, Y. Belinkov, Locating and editing factual associations in gpt, Advances in Neural Information Processing Systems 35 (2022) 17359–17372.