Abstract

In this work we operate on the view that the pragmatic employment of objects by an agent is manifested via their afordances for that specific agent. What afordances can manifest themselves in a particular situation depends, in part, on the dispositions ofered by the particular objects involved. The contribution of this work is to construct and deploy a large scale formal model of dispositions for physical objects that can be employed to constrain and describe the roles they can play in narratives.

1. Introduction

will sufice: a disposition is a quality an object needs so that it can play a particular role in a particular action.

A simple question such as “what is a particular object” Dispositions tend to operate in, at least, pairs. Where turns out to have some complications if its answer is to the dispositions of knife and dough meet, there can be be relevant for human beings. Hobbs remarks, by way of cutting. A block of wood can be cut too, though it would illustration, that a road is a line when we plan a trip, a be hard to do so with a knife. There are more kinds of surface when we drive on it, and a volume when we hit a dispositions to being a cutting instrument, and not all pothole ([ 1 ], pg. 820). A road is always a road, of course, of them are good fits for all the dispositions to being but in our common way to talk we confuse between what a cutting patient. Therefore, also needed is knowledge something is, and what it is used as. This confusion sug- about what dispositions go together, to answer questions gests that the use is more immediately relevant than the such as “what can I cut apples with?”, “what can I contain ontological classification, at least where agents pursuing boiling water with?” and so on. pragmatic goals are concerned. If equipped with knowledge of object dispositions, and

The relevance of “use as” requires that ontological how these dispositions “match” and allow various afmodelling be able to describe both levels: of being, and fordances to manifest [ 4 ], an agent can tackle several of use – and of other ways to interact with the pragmatic problems it may encounter in its coping with the envigoals of an agent. One approach to achieve this works by ronment. It can select appropriate tools for tasks it needs also modelling the roles that (world) entities play in a nar- to perform, or seek passable substitutes. It can also look rative an agent constructs about its interactions with the at a scene and form an idea of what can happen, and how world [ 2 ]. The entities that appear in the agent’s narra- to work with, or against, such possibilities. If there is a tive are classifications of world entities or ground objects puddle of oil lying around, people tend to be more careful based on the roles they play, rather than the ground ob- with lit matches. jects themselves. A ground object such as a hammer can In our everyday coping with the world, we don’t seem play various roles, e.g., a murder weapon, a paper weight to usually tell ourselves stories about activities we master or a door stopper. It can even be a tool to drive nails. The and render routine ([ 5 ], chapter 1; [ 6 ]). We just follow potential classifications of a physical object – the ways our goals, mostly avoid dangerous configurations of the in which it can be narrativized – depend on its physical world, without thinking about our decisions for too long. dispositions. If however we had to narrativize – perhaps to teach some

A disposition is a quality an object may have, relevant one else, perhaps to correct a perceived flaw in our action for questions such as “what can this do?” and “what can or learn something new – we would use knowledge of be done to it?” A knife can cut – it has a disposition dispositions, and dispositional matching. We would asallowing it to be a cutting instrument. A ball of dough sert that some tool is appropriate for a task, or explain can be cut – it has a disposition to be a patient of cut- that we did something to prevent something else from ting. While the concept of disposition is complicated to happening. Presumably, a similar capability is useful also model [ 3 ], for our purposes here this basic understanding for robotic agents for doing household chores; if nothing else, they should be able to explain themselves to human users/observers.

Therefore, a large scale formal model of dispositions of physical objects will be useful to the domestic service IJCAI 2022: Workshop on semantic techniques for narrative-based understanding, July 24, 2022, Vienna, Austria $ [e-mail] (M. Pomarlan); [e-mail] (R. Porzel)

© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License robots of the future. The contribution of this work is to CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) construct and deploy such a model1 and show how it can be employed to constrain and describe the roles objects can play in an agent’s narratives.

3. Limits of Open Knowledge Sources

Given the abundance of large knowledge graphs that

2. State of the Art have some connection to commonsense, are more such graphs needed or useful? We argue here that they are. 2.1. Dispositions and Afordances We initially used OpenCyc as a foundation for representing knowledge for a robot doing household activity.

Afordances were informally defined by Gibson as what DOLCE UltraLite [ 27 ] provided a more cognitively motian environment ofers to an agent [ 7 ]. Formal accounts vated foundation, and we then developed an ontological have been proposed (afordances as qualities [ 8 ], as model on top of it to distinguish between object class and events [ 9 ], as designs [ 10 ], as relations [ 11 ], [ 4 ] etc.; object use [ 28, 4 ], a distinction which OpenCyc does not overview in [ 3 ]). Turvey proposed a theory [ 12 ] stat- make consistently. We will look again at the knowledge ing the disposition of a “bearer” can realize an event items in Open-/ResearchCyc in the future. In this pawhen it encounters a “trigger” disposition under some per we discuss the more recent, open-access knowledge “background” conditions. An ontological model for dis- graphs such as CSKG. positions based on Turvey’s insights was recently pro- Modern knowledge graph projects emphasize their posed [ 13 ], [ 3 ]. Learning afordances from interaction broad coverage, manifested in millions of triples. Indeed, has been investigated [ 14 ], [ 15 ], [ 16 ]. CSKG contains plenty of trivia knowledge. Knowledge for everyday activities is sparser. E.g., there are no Used2.2. Commonsense Knowledge Graphs For triples for wipe/v/wn/contact – there is no knowledge about what tools would be appropriate for an everyday Several benchmarks are available to test an agent’s ability task such as wiping. to answer commonsense queries, e.g. WinoGrande [ 17 ]. The large scale knowledge graphs of today would probState of the art AI either sits below human performance, ably not exist without automatic techniques and cheap or is not reliable in providing appropriate answers. crowd-sourcing. However, this is prone to introducing er

Embodying commonsense reasoning in a computa- rors2. We do not intend to be critical of the CSKG project, tional model is complex for many reasons [ 18 ], such which formed the basis of our own. We however became as the need for commonsense knowledge. To address very aware during our work of how problematic it is for this, several commonsense knowledge graphs have inference. been constructed: the CommonSense Knowledge Graph We position our work, relative to other knowledge (CSKG; [ 19 ]), itself a merge of among others Concept- graphs, as specialized on everyday knowledge and reaNet [ 20 ], ATOMIC [ 21 ], ImageNet [ 22 ]. The most fa- soning. More human attention was spent on vetting the mous commonsense knowledge graph was constructed items, with newer ontological modelling approaches [ 28 ] by Cyc [ 23 ], but unfortunately its open- and research and new sources of knowledge about object use obtained access versions have been discontinued. from games with a purpose [ 29 ].

Linguistic resources have also been used for commonsense reasoning: WordNet [ 24 ], FrameNet [ 25 ], VerbNet [ 26 ]. These capture relations between word mean- 4. SOMA_DFL ings, descriptions of scenarios, and thematic roles and their selectional restrictions. We now describe what are the sources for our knowledge

Our work builds from existing knowledge graphs by se- graph and how it is structured and organized into an lecting, correcting, and integrating knowledge items from ontology for the purposes of reasoning and answering them with new sources. We obtained a rich knowledge queries. store for answering questions about object capabilities and uses. 4.1. Knowledge Sources 1The knowledge graph is available at https://github.com/ease-crc/ ease_lexical_resources Our primary source is CSKG [ 19 ], which combines several knowledge resources. We have focused on the part of CSKG that is comprised of entities with an associated

2In CSKG, we find that amputate/v/wn/medicine is a manner of audio

tape/n/wn/artifact, authorize/v/wn/communication, fructify/v/wn/body, and several other such doubtful relations. English WordNet synset3. We selected the part of the assert that chewing is a kind of grinding and a disposition object taxonomy that refers to tools, buildings, and food. to be chewable is a disposition to be grindable. Some of these entities also have associated UsedFor and UsedFor triples are interpreted as asserting that an obCapableOf triples, where the third member of a triple cor- ject can play an instrumental role in a particular task. responds to an action. We have also selected MannerOf CapableOf triples are interpreted as asserting that an triples between actions in UsedFor and CapableOf triples. object can play a passive role, usually Patient. Triples deWe added triples linking actions to VerbNet 3.2 classes, scribing combinations of items are interpreted as general and so to thematic roles and selectional restrictions on assertions about all items of the categories mentioned in role fillers. We have done extensive manual corrections the triple. on the collected triples. We added triples describing what items can be used together during an action, Some of this knowledge comes from WordNet synset definitions, 5. Queries some from the work of our colleagues on games with a purpose [ 29 ].

Reasoning with SOMA_DFL consists in performing sub

sumption inference tasks between concepts in the ontology and query concepts defined by the user. We query 4.2. Structure SOMA_DFL with the reasoner Konclude. The ontology is For reasoning and query answering, the triples from somewhat large, with 22527 object classes coming from the knowledge graph are organized into an ontology. CSKG and related resources, and 45538 subclassof/equivWe now use OWL-DL, but plan to add support for non- alentclasses axioms. Further, the ontology uses the full monotonic inference as it resembles more the default- expressivity of OWL-2. Nonetheless, performing disposiwith-exceptions pattern that human rules often follow. tion queries is fast – less than a milisecond – as long as a All birds fly, except those that do not – and it is conve- cache is constructed first, which takes about 10 seconds nient both to keep the default rule as well as an open- using Konclude 0.7.0 on an Intel®Core™i5-7500 CPU @ ended list of exceptions. 3.40GHz with 8GB RAM. Table 1 shows some example

The ontology we produce is built on top of DUL [ 27 ] queries. and the SOcio-physical Model of Activity (SOMA [ 28 ]).

These provide higher-level knowledge, in particular that 6. Conclusion and Future Work Tasks are classify events and define Roles which can be iflled by appropriate Objects.

We have presented a knowledge graph that contains in

formation about object dispositions and possible com4.3. Axiomatization binations of objects that can be used to achieve a task. The graph focuses on knowledge useful for everyday acOur dispositional theory asserts that to play a role, an ob- tivities, and can be employed by a computational agent ject must have a disposition for it. From VerbNet knowl- to select appropriate tools or patients (objects to act on) edge, we produce axioms asserting that if an object fills a for a task, or to understand a scene in terms of what is particular role in a particular task then it must also obey possible for the objects in it to do together. Our graph the appropriate semantic restrictions. E.g., the following difers both in its focus – everyday activity knowledge axioms – and in its purpose – logical reasoning, as opposed to − 1 ( ⊓ ∃ − 1 ℎ) ⊑ information retrieval – from previous projects. ∃ℎ ℎ. We will add support for non-monotonic inference, be∃ℎ ℎ. ⊑ ⊓ cause it captures better the default with open-ended exassert that what plays the patient role for chewing must ception list way that human knowledge is often orgahave the chewable disposition, and therefore comestible nized, and will integrate knowledge about causal relaand solid. tions between events.

MannerOf triples are interpreted as describing a taxonomy, and the roles a manner defines are connected to the roles defined by the task it is a manner of. E.g., the Acknowledgments following axioms ℎ ⊑ ℎ. ⊑ .

This work was funded by the by the FET-Open Project

#951846 “MUHAI – Meaning and Understanding for Human-centric AI” by the EU Pathfinder and Horizon 2020 Program and by the German Research Foundation (DFG) as part of Collaborative Research Center (SFB) 3In CSKG, the names of such entities can be identified by a “/c/en/” prefix and a “/wn/” infix. An example of such an entity is /c/en/cut/v/wn/contact. In this paper we will omit the “/c/en/” prefix. What can you use (as object acted on) to – What can you do with – What can you use (as instrument) to – cut firewood contain a liquid cover a bowl What can you – open with scissors scoop with a ladle

References

[1] J. R. Hobbs , Sketch of an ontology underlying the way we talk about the world , Int. J. Hum. Comput. Stud . 43 ( 1995 ) 819 - 830 . URL: https://doi.org/ 10.1006/ijhc. 1995 . 1076 . doi: 10 .1006/ijhc. 1995 . 1076 .

[2] R. Porzel , On formalizing narratives , in: Proceedings of the JOWO - Ontology Workshops , Bolzano, Italy, 2021 .

[3] F. Toyoshima , A. Barton , J.-F. Ethier , Afordances and their ontological core , Journal of Applied Ontology 17 ( 2022 ) 285 - 320 .

[4] D. Beßler , R. Porzel , M. Pomarlan , M. Beetz , R. Malaka , J. Bateman , A Formal Model of Afordances for Flexible Robotic Task Execution , in: European Conference on Artificial Intelligence (ECAI) , 2020 .

[5] P. Agre , Computation and Human Experience , Learning in doing, Cambridge University Press, New York, 1997 . 00550 .

[6] D. Kahneman , Thinking, fast and slow, Farrar, Straus and Giroux, New York, 2011 . URL: https://www.amazon.de/ Thinking-Fast-Slow-Daniel-Kahneman/dp/ 0374275637/ref=wl_it_dp_ o_pdT1_nS_nC? ie=UTF8&colid=151193SNGKJT9&coliid= I3OCESLZCVDFL7.

[7] J. J. Gibson , The Ecological Approach to Visual Perception, Psychology Press Classic Editions, 1979 .

[8] J. Ortmann , W. Kuhn, Afordances as qualities , in: Proceedings of the 2010 Conference on Formal Ontology in Information Systems: Proceedings of the Sixth International Conference (FOIS 2010 ), IOS Press, Amsterdam, The Netherlands, The Netherlands, 2010 , pp. 117 - 130 .

[9] L. A. Moralez , Afordance ontology: towards a unified description of afordances as events , Res. Cogitans 7 ( 2016 ) 35 - 45 .

[10] I. Awaad , G. Kraetzschmar, J. Hertzberg , Challenges in finding ways to get the job done , in: 2nd Planning and Robotics (PlanRob) Workshop at 24th International Conference on Automated Planning and Scheduling (ICAPS) , 2014 .

[11] A. Chemero , An outline of a theory of afordances , Ecological Psychology 15 ( 2003 ) 181 - 195 .

[12] M. T. Turvey , Ecological foundations of cognition: Invariants of perception and action ., American Psychological Association ( 1992 ).

[13] F. Toyoshima , Modeling afordances with dispositions , in: Proceedings of the Joint Ontology Workshops , 2018 .

[14] B. Moldovan , P. Moreno , M. Van Otterlo , J. SantosVictor , L. De Raedt, Learning relational afordance models for robots in multi-object manipulation tasks , in: 2012 IEEE International Conference on Robotics and Automation , IEEE, 2012 , pp. 4373 - 4378 .

[15] R. Detry , E. Baseski, M. Popovic , Y. Touati , N. Kruger , O. Kroemer , J. Peters , J. Piater , Learning object-specific grasp afordance densities , 2009 . doi: 10 .1109/DEVLRN. 2009 . 5175520 .

[16] E. Ugur , J. H. Piater , Emergent structuring of interdependent afordance learning tasks using intrinsic motivation and empirical feature selection , IEEE Trans. Cogn. Dev. Syst . 9 ( 2017 ) 328 - 340 . URL: https://doi.org/10.1109/TCDS. 2016 . 2581307 . doi: 10 . 1109/TCDS. 2016 . 2581307 .

[17] K. Sakaguchi , R. L. Bras , C. Bhagavatula , Y. Choi , Winogrande: An adversarial winograd schema challenge at scale , ArXiv abs/ 1907 .10641 ( 2020 ).

[18] E. Davis , Logical formalizations of commonsense reasoning: A survey , J. Artif. Intell. Res . 59 ( 2017 ) 651 - 723 . URL: https://doi.org/10.1613/jair. 5339. doi: 10 .1613/jair.5339.

[19] F. Ilievski , P. A. Szekely , B. Zhang, CSKG: the commonsense knowledge graph , CoRR abs/ 2012 .11490 ( 2020 ). URL: https://arxiv.org/abs/ 2012 .11490. arXiv: 2012 .11490.

[20] R. Speer , J. Chin , C. Havasi , Conceptnet 5.5: An open multilingual graph of general knowledge , in: AAAI Conference on Artificial Intelligence , 2016 . URL: http://arxiv.org/abs/1612.03975.

[21] M. Sap , R. L. Bras , E. Allaway , C. Bhagavatula , N. Lourie , H. Rashkin , B. Roof , N. A. Smith , Y. Choi, ATOMIC: an atlas of machine commonsense for if-then reasoning , in: The Thirty-Third AAAI Conference on Artificial Intelligence , AAAI 2019 , The Thirty-First Innovative Applications of Artificial Intelligence Conference , IAAI 2019 , The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019 , Honolulu, Hawaii, USA, January 27 - February 1, 2019 , AAAI Press, 2019 , pp. 3027 - 3035 . URL: https://doi.org/ 10.1609/aaai.v33i01.33013027. doi: 10 .1609/aaai. v33i01. 33013027 .

[22] J. Deng , W. Dong , R. Socher , L.-J. Li , K. Li , L. FeiFei, Imagenet: A large-scale hierarchical image database , in: 2009 IEEE Conference on Computer Vision and Pattern Recognition , 2009 , pp. 248 - 255 . doi: 10 .1109/CVPR. 2009 . 5206848 .

[23] D. B. Lenat , Cyc: a large-scale investment in knowledge infrastructure , Commun. ACM 38 ( 1995 ) 32 - 38 .

[24] C. Fellbaum (Ed.), WordNet: an electronic lexical database , MIT Press, 1998 .

[25] J. Ruppenhofer , M. Ellsworth , M. Petruck , C. Johnson , J. Schefczyk, Framenet ii: Extended theory and practice ( 2010 ).

[26] K. K. Schuler , VerbNet: A Broad-Coverage , Comprehensive Verb Lexicon, Ph.D. thesis , University of Pennsylvania, 2006 . URL: http://verbs.colorado. edu/~kipper/Papers/dissertation.pdf.

[27] V. Mascardi , V. Cordì , P. Rosso , A comparison of upper ontologies ., 2007 , pp. 55 - 64 .

[28] D. Beßler , R. Porzel , M. Pomarlan , A. Vyas , S. Höfner , M. Beetz , R. Malaka , J. Bateman , Foundations of the socio-physical model of activities (soma) for autonomous robotic agents , in: B. Brodaric , F. Neuhaus (Eds.), Formal Ontology in Information Systems - Proceedings of the 12th International Conference, FOIS 2021 , Bozen- Bolzano , Italy, September 13-16 , 2021 , Frontiers in Artificial Intelligence and Applications , IOS Press, 2021 .

[29] J. Pfau , R. Malaka , We asked 100 people: How would you train our robot? , in: Extended Abstracts of the 2020 Annual Symposium on Computer-Human Interaction in Play , 2020 , pp. 335 - 339 .