Asking Human Reasoners to Judge Postulates of Belief Change for Plausibility

Abstract

Empirical methods have been used to test whether human reasoning conforms to models of reasoning in logic-based artificial intelligence. This work investigates through surveys whether postulates of belief revision and update are plausible with human reasoners. The results show that participants' reasoning tend to be consistent with the postulates of belief revision and belief update when judging the premises and conclusion of the postulate separately.

Keywords

• eol>revision postulates
• update postulates
• human reasoning
• survey

1. Introduction

human reasoning is consistent with postulates, advanced by Alchourrón, Gärdenfors and Makinson (AGM) [ 8 ], for It has been shown that human reasoning displays non- belief revision. To enable comparison, we also hypothemonotonicity, but the methodologies that test this rela- sise that human reasoning is consistent with postulates, tionship difer within the AI community. An example advanced by Katsuno and Mendelzon (KM) [ 9 ], for rea[ 1 ] of one approach is through surveys in which English soning with belief update. We investigate the hypotheses translations of the postulates of defeasible reasoning were at the postulate level and at the system level. We use judged for plausibility by human reasoners. As another the language of propositional logic in the formulation example, a combined approach [ 2 ] was also used to inves- of our postulates to construct logically closed belief sets. tigate the link between formal theories of non-monotonic Additionally, we note that once our hypotheses are tested reasoning and the extent to which humans reason defea- using propositional logic as the underlying language, our sibly. The combined approach involved a theoretical and results can be lifted to other forms of logic. This work empirical analysis. In the theoretical analysis, the predic- extends previous work that investigated postulates of tions of each system was compared using the Suppression defeasible reasoning [ 10 ] and belief change [ 11 ] with Task [ 3 ], a logical experiment used in the psychology human reasoners via surveys. Furthermore, this work is community in which subjects appear to retract valid logi- an extended abstracted of a paper that is currently under cal inferences when subjects gain new information. In the review for a special issue of the Journal of Applied Logic. empirical analysis, three experiments were used to test the predictions of each system, as well as the inferences of human reasoners, with strict and defeasible knowl- 2. Background edge. While there are empirical studies that investigated the relationship between non-monotonic reasoning with 2.1. Belief Revision human reasoning, the relationship between belief change and human reasoning has been primarily studied from a theoretical perspective, e.g. in classical logic [ 4 ], probability and possibility theory [ 5 ], ontologies [ 6 ] and abstract argumentation [ 7 ]. Our first hypothesis is that

The first form of belief change we investigated was revi

sion. It is an approach to reasoning with changing beliefs under the assumption that the world did not undergo a fundamental change. It is characterised by a belief set , a revision operation * and reasoning rules referred to as NMR’22: 20th International Workshop on Non-Monotonic Reasoning, postulates. A belief set is a set of propositional formulas August 07-09, 2022, Haifa, Israel closed under logical consequence. A revision operation * Corresponding author. allows a reasoner to add new information to his beliefs ($T. bMkercylear0)[e-mail] (C. K. Baker); [e-mail] if the new information is consistent with his beliefs. A € https://tinyurl.com/5n6vuyp7 (C. K. Baker); revision operation also allows a reasoner to add an exhttps://tinyurl.com/5ejbf2vr (T. Meyer) ception to his beliefs to account for the situation where 0000-0002-3157-9989 (C. K. Baker); 0000-0003-2204-6969 this exception or new information is inconsistent with (T. Meyer©) 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License his beliefs. Moreover, the result of a revision operation CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) must always be that a reasoner’s beliefs do not contradict (R3) If is satisfiable, then * is also satisfiable (R4) If 1 ≡ 2 and 1 ≡ 2, then 1 * 1 ≡ 2 * 2 (R5) ( * ) ∧ implies * ( ∧ ) (R6) If ( * ) ∧ is satisfiable, then implies ( * ) ∧ * 1 is equivalent to * 2 (R7) If * 1 implies 2 and * 2 implies 1, then (R8) ( * 1) ∧ ( * 2) implies * ( 1 ∨ 2) belief revision framework. (R1)–(R6) correspond to the one another. There are eight postulates in the AGM [ 8 ] then it is a partial preorder. A preorder is total if ≤ core rationality postulates and the (R7)–(R8) correspond postulates (R1)–(R6) using the notion of a total preorder to supplementary postulates.

(R1) * implies * ( ∧ ) duced by each model of , while for revision, only one ordering is induced by the whole of . or ≤

for all , ∈ . A revision operator satisfies on interpretations while an update operator satisfies postulates (U1)–(U6) using the notion of a partial preorder on interpretations. By replacing postulates (U6) and (U7) with a new postulate (U9), the class of update operators can be designed using total preorders. The second and more important diference between revision and update is that, in the case of update, a diferent ordering is in3.

Methodology

Our empirical investigation took place through four experiments. In the first experiment, we prepared a survey of 30 general statements about the world for participants to evaluate for clarity and bias. 7 participants had to complete a table in which they identified statements with ambiguous language and biased examples. In the second experiment, we prepared a survey of 30 general statements about the world taken from refining the material in the ifrst experiment. 30 participants evaluated the degree to which they believed each of the statements in the survey and explained their answers. In the third experiment, we prepared a survey of English statements corresponding to translations of the AGM postulates for belief revision. 35 participants on Mechanical Turk (MTurk) evaluated the degree to which they believed each statement in the survey. We tested our hypothesis statistically and determined whether the association between the premises and the conclusion for each postulate holds for the general English-speaking reasoner. In the last experiment, we used the same material from the belief revision experiment to instantiate the KM belief update postulates. The experimental setup followed a similar approach to the belief revision experiment. We obtained ethical clearance from the Faculty of Science Ethics Research Committee at the University of Cape Town. We include the consent forms and a link to our data management plan in our

Github project repository, linked in Appendix A. For

the bulk of our reasoning experiments, we used Google

Forms to design our surveys, and we used Mechanical Turk to crowdsource our data collection. satisfiable

2 ◇ 2 ◇ 1 ↔ ◇ 2 ◇ ( 1 ∨ 2) (U4) If 1 ↔ 2 and 1 ↔ 2 then 1 ◇ 1 ↔ (U5) ( ◇ ) ∧ implies ◇ ( ∧ ) (U6) If ◇ 1 implies 2 and ◇ 2 implies 1 then (U7) If is complete then ( ◇ 1) ∧ ( ◇ 2) implies 2.2. Belief Update The next form of belief change we investigated was update. It is an approach to reasoning with changing beliefs after some fundamental shift in the world occurred. It is characterised by a belief set , an update operation ◇ and postulates for reasoning. As with revision, refers to a logically closed set of propositional formulas. When we update with new information , we are saying that we used to believe , we know now that holds, and we need to modify by adding , acknowledging that we may have been wrong if contradicts . There are nine postulates in the KM [ 9 ] belief update framework.

(U1) ◇ implies (U2) If implies then ◇ is equivalent to (U3) If both and are satisfiable then ◇ is also (U8) (1 ∨ 2) ◇

↔ (1 ◇ ) ∨ (2 ◇ ) (U9) If is complete and ( ◇ ) ∧ is satisafible then

◇ ( ∧ ) implies ( ◇ ) ∧ Revision and update difer from non-monotonic logic using the concept of orders on interpretations. A homogeneous relation ≤ definition ≤ is some subset of × and the notation on some given set , so that by 4. Results ≤ is used in place of (, ) ∈ , is called a preorder In this work, we investigated the endorsements of each if the relation is also transitive and reflexive. A reflexive relation has the property that ≤

for all ∈ . A transitive relation has the property that if ≤ ≤ then ≤

for all , , ∈ . If a preorder is also anti-symmetric, that is, ≤ and ≤ implies = , and component of the AGM postulates when formulated as material implication statements. We found evidence for whether or not the participants found our concrete instantiations of the AGM postulates plausible. We determined whether the postulates hold in general. The results show that the participants’ reasoning tends to be consistent tionships. The empirical part will focus on identifying with the 9 AGM postulates, with the significance of the as- representations of the postulates that support both the sociation between the endorsement of the premises and theory and the beliefs of human reasoners. It will inthe endorsement of the conclusion ranging from non- volve the development of an online reasoning tool that significant to highly significant. The number of logical automates the production and presentation of structured violations per postulate is generally low with a range of reasoning examples in a survey setting. The structured 0 to 4 violations (< 12% of participants). The exception reasoning examples will depict the static and dynamic is postulate (R5) with 54,29% (19 participants) endorsing nature of changing beliefs in terms of a revision and an the premises, but not the conclusion. update, respectively. In turn, the tailored surveys created

We also investigated the endorsements of each compo- using the reasoning tool can be used to elicit responses nent of the KM postulates when formulated as material from human reasoners, the design of which improves implication statements. We found evidence for whether upon the limitations of question types from conventional or not the participants found our concrete instantiations survey platforms like Google Forms and Microsoft Forms. of the KM postulates plausible. We determined whether Furthermore, non-parameterised statistical methods, that the postulates hold in general. The results show that the is, methods that do not assume how the sample data is participants’ reasoning tends to be consistent with the 9 distributed, e.g. the Wilcoxon signed-rank test [ 12, 13 ], postulates of KM belief update, with the significance of will be used to interpret the significance of the postulates the association between the endorsement of the premises of belief change, as found by human reasoners. and the endorsement of the conclusion ranging from nonsignificant to highly significant. The number of logical violations per postulate is generally low as well with a Acknowledgments range of 0 to 8 violations (< 23% of participants). The exception is postulate (U8) with 48,57% (17 participants) endorsing the premises, but not the conclusion.

We wish to express our sincere gratitude and appreciation to the DSI – CSIR Interbursary Support (IBS) Programme and the Centre for Artificial Intelligence Research (CAIR) for financial support.

5. Conclusions and Future Work

Our work builds on previous empirical studies involving human subjects who are tasked with reasoning nonmonotonically. We created a reproducible approach for empirically investigating the plausibility of postulates of belief change. This approach accounts for the efect of the premises and conclusion of each postulate and determines whether the overall postulate is found plausible with statistically significant evidence. We applied this approach to the formal theory of belief revision and update. We hypothesised that human belief change is consistent with the AGM postulates of belief revision and the KM postulates of belief update. The results show that the participants’ reasoning tends to be consistent with the 8 AGM postulates, (R1)–(R8), with the significance of the association between the endorsement of the premises and the endorsement of the conclusion ranging from non-significant to highly significant. The results also show that the participants’ reasoning tends to be consistent with the 9 KM postulates, (U1)–(U9), with the significance of the association between the endorsement of the premises and the endorsement of the conclusion ranging from non-significant to highly significant.

In future work, we will refine our approach in the following way. We will conduct a theoretical and empirical investigation of the postulates of belief revision and update. The theoretical part will build on this work by exploring inter-postulate and inter-framework rela

A. Additional resources

The Github repository for this work, containing supplementary material and code scripts, can be accessed via this URL, https://tinyurl.com/2p98m76n.

References

[1] R. Neves , J. Bonnefon , E. Raufaste, An empirical test of patterns for nonmonotonic inference , Annals of Mathematics and Artificial Intelligence 34 ( 2002 ) 107 - 130 .

[2] M. Ragni , C. Eichhorn , T. Bock , G. Kern-Isberner , A. Tse , Formal nonmonotonic theories and properties of human defeasible reasoning , Minds and Machines 27 ( 2017 ) 79 - 117 . doi: 10 .1007/ s11023-016-9414-1.

[3] R. M. Byrne , Suppressing valid inferences with conditionals , Cognition 31 ( 1989 ) 61 - 83 .

[4] F. Lévy , A survey of belief revision and updating in classical logic , International Journal of Intelligent Systems 9 ( 1994 ) 29 - 59 .

[5] D. Dubois , H. Prade , A survey of belief revision and updating rules in various uncertainty models , International Journal of Intelligent Systems 9 ( 1994 ) 61 - 100 .

[6] C. Boutilier , N. Friedman , J. Y. Halpern , Belief revision with unreliable observations , in: AAAI/IAAI , 1998 , pp. 127 - 134 .

[7] S. Doutre , J.-G. Mailly, Constraints and changes: A survey of abstract argumentation dynamics , Argument & Computation 9 ( 2018 ) 223 - 248 .

[8] C. E. Alchourrón , P. Gärdenfors , D. Makinson , On the logic of theory change: Partial meet contraction and revision functions , Journal of Symbolic Logic 50 ( 1985 ) 510 - 530 . doi: 10 .2307/2274239.

[9] H. Katsuno , A. O. Mendelzon , On the diference between updating a knowledge base and revising it, Belief revision ( 1991 ) 183 .

[10] C. Baker , C. Denny , P. Freund , T. Meyer, Cognitive defeasible reasoning: the extent to which forms of defeasible reasoning correspond with human reasoning , in: Proceedings of the First Southern African Conference for Artificial Intelligence Research (SACAIR 2020 ), CCIS, Springer, 2020 , pp. 119 - 219 .

[11] C. Baker , T. Meyer, Belief change in human reasoning: An empirical investigation on mturk , in: Proceedings of the Second Southern African Conference for Artificial Intelligence Research (SACAIR 2021 ), 2021 , pp. 520 - 536 .

[12] X. Li , Y. Wu , M. Wei , Y. Guo , Z. Yu , H. Wang , Z. Li , H. Fan , A novel index of functional connectivity: phase lag based on wilcoxon signed rank test , Cognitive Neurodynamics 15 ( 2021 ) 621 - 636 .

[13] L. Zhou , Performance of corporate bankruptcy prediction models on imbalanced dataset: The efect of sampling methods , Knowledge-Based Systems 41 ( 2013 ) 16 - 25 .