=Paper=
{{Paper
|id=Vol-3322/short2
|storemode=property
|title=Narrative Objects
|pdfUrl=https://ceur-ws.org/Vol-3322/short2.pdf
|volume=Vol-3322
|authors=Mihai Pomarlan,Robert Porzel
|dblpUrl=https://dblp.org/rec/conf/ijcai/PomarlanP22
}}
==Narrative Objects==
https://ceur-ws.org/Vol-3322/short2.pdf
Narrative Objects
Mihai Pomarlan1 , Robert Porzel2
1
Applied Linguistics, University of Bremen, Bremen, Germany
2
Digital Media Lab, University of Bremen, Bremen, Germany
Abstract
In this work we operate on the view that the pragmatic employment of objects by an agent is manifested via their affordances
for that specific agent. What affordances can manifest themselves in a particular situation depends, in part, on the dispositions
offered by the particular objects involved. The contribution of this work is to construct and deploy a large scale formal model
of dispositions for physical objects that can be employed to constrain and describe the roles they can play in narratives.
1. Introduction will suffice: a disposition is a quality an object needs so
that it can play a particular role in a particular action.
A simple question such as “what is a particular object” Dispositions tend to operate in, at least, pairs. Where
turns out to have some complications if its answer is to the dispositions of knife and dough meet, there can be
be relevant for human beings. Hobbs remarks, by way of cutting. A block of wood can be cut too, though it would
illustration, that a road is a line when we plan a trip, a be hard to do so with a knife. There are more kinds of
surface when we drive on it, and a volume when we hit a dispositions to being a cutting instrument, and not all
pothole ([1], pg. 820). A road is always a road, of course, of them are good fits for all the dispositions to being
but in our common way to talk we confuse between what a cutting patient. Therefore, also needed is knowledge
something is, and what it is used as. This confusion sug- about what dispositions go together, to answer questions
gests that the use is more immediately relevant than the such as “what can I cut apples with?”, “what can I contain
ontological classification, at least where agents pursuing boiling water with?” and so on.
pragmatic goals are concerned. If equipped with knowledge of object dispositions, and
The relevance of “use as” requires that ontological how these dispositions “match” and allow various af-
modelling be able to describe both levels: of being, and fordances to manifest [4], an agent can tackle several
of use – and of other ways to interact with the pragmatic problems it may encounter in its coping with the envi-
goals of an agent. One approach to achieve this works by ronment. It can select appropriate tools for tasks it needs
also modelling the roles that (world) entities play in a nar- to perform, or seek passable substitutes. It can also look
rative an agent constructs about its interactions with the at a scene and form an idea of what can happen, and how
world [2]. The entities that appear in the agent’s narra- to work with, or against, such possibilities. If there is a
tive are classifications of world entities or ground objects puddle of oil lying around, people tend to be more careful
based on the roles they play, rather than the ground ob- with lit matches.
jects themselves. A ground object such as a hammer can In our everyday coping with the world, we don’t seem
play various roles, e.g., a murder weapon, a paper weight to usually tell ourselves stories about activities we master
or a door stopper. It can even be a tool to drive nails. The and render routine ([5], chapter 1; [6]). We just follow
potential classifications of a physical object – the ways our goals, mostly avoid dangerous configurations of the
in which it can be narrativized – depend on its physical world, without thinking about our decisions for too long.
dispositions. If however we had to narrativize – perhaps to teach some-
A disposition is a quality an object may have, relevant one else, perhaps to correct a perceived flaw in our action
for questions such as “what can this do?” and “what can or learn something new – we would use knowledge of
be done to it?” A knife can cut – it has a disposition dispositions, and dispositional matching. We would as-
allowing it to be a cutting instrument. A ball of dough sert that some tool is appropriate for a task, or explain
can be cut – it has a disposition to be a patient of cut- that we did something to prevent something else from
ting. While the concept of disposition is complicated to happening. Presumably, a similar capability is useful also
model [3], for our purposes here this basic understanding for robotic agents for doing household chores; if nothing
else, they should be able to explain themselves to human
IJCAI 2022: Workshop on semantic techniques for narrative-based users/observers.
understanding, July 24, 2022, Vienna, Austria Therefore, a large scale formal model of dispositions
$ pomarlan@uni-bremen.de (M. Pomarlan); of physical objects will be useful to the domestic service
porzel@uni-bremen.de (R. Porzel)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License robots of the future. The contribution of this work is to
Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
6
construct and deploy such a model1 and show how it can 3. Limits of Open Knowledge
be employed to constrain and describe the roles objects
can play in an agent’s narratives.
Sources
Given the abundance of large knowledge graphs that
have some connection to commonsense, are more such
2. State of the Art graphs needed or useful? We argue here that they are.
We initially used OpenCyc as a foundation for repre-
2.1. Dispositions and Affordances
senting knowledge for a robot doing household activity.
Affordances were informally defined by Gibson as what DOLCE UltraLite [27] provided a more cognitively moti-
an environment offers to an agent [7]. Formal accounts vated foundation, and we then developed an ontological
have been proposed (affordances as qualities [8], as model on top of it to distinguish between object class and
events [9], as designs [10], as relations [11], [4] etc.; object use [28, 4], a distinction which OpenCyc does not
overview in [3]). Turvey proposed a theory [12] stat- make consistently. We will look again at the knowledge
ing the disposition of a “bearer” can realize an event items in Open-/ResearchCyc in the future. In this pa-
when it encounters a “trigger” disposition under some per we discuss the more recent, open-access knowledge
“background” conditions. An ontological model for dis- graphs such as CSKG.
positions based on Turvey’s insights was recently pro- Modern knowledge graph projects emphasize their
posed [13], [3]. Learning affordances from interaction broad coverage, manifested in millions of triples. Indeed,
has been investigated [14], [15], [16]. CSKG contains plenty of trivia knowledge. Knowledge
for everyday activities is sparser. E.g., there are no Used-
2.2. Commonsense Knowledge Graphs For triples for wipe/v/wn/contact – there is no knowledge
about what tools would be appropriate for an everyday
Several benchmarks are available to test an agent’s ability task such as wiping.
to answer commonsense queries, e.g. WinoGrande [17]. The large scale knowledge graphs of today would prob-
State of the art AI either sits below human performance, ably not exist without automatic techniques and cheap
or is not reliable in providing appropriate answers. crowd-sourcing. However, this is prone to introducing er-
Embodying commonsense reasoning in a computa- rors2 . We do not intend to be critical of the CSKG project,
tional model is complex for many reasons [18], such which formed the basis of our own. We however became
as the need for commonsense knowledge. To address very aware during our work of how problematic it is for
this, several commonsense knowledge graphs have inference.
been constructed: the CommonSense Knowledge Graph We position our work, relative to other knowledge
(CSKG; [19]), itself a merge of among others Concept- graphs, as specialized on everyday knowledge and rea-
Net [20], ATOMIC [21], ImageNet [22]. The most fa- soning. More human attention was spent on vetting the
mous commonsense knowledge graph was constructed items, with newer ontological modelling approaches [28]
by Cyc [23], but unfortunately its open- and research and new sources of knowledge about object use obtained
access versions have been discontinued. from games with a purpose [29].
Linguistic resources have also been used for common-
sense reasoning: WordNet [24], FrameNet [25], Verb-
Net [26]. These capture relations between word mean- 4. SOMA_DFL
ings, descriptions of scenarios, and thematic roles and
their selectional restrictions. We now describe what are the sources for our knowledge
Our work builds from existing knowledge graphs by se- graph and how it is structured and organized into an
lecting, correcting, and integrating knowledge items from ontology for the purposes of reasoning and answering
them with new sources. We obtained a rich knowledge queries.
store for answering questions about object capabilities
and uses. 4.1. Knowledge Sources
Our primary source is CSKG [19], which combines sev-
eral knowledge resources. We have focused on the part
of CSKG that is comprised of entities with an associated
2
In CSKG, we find that amputate/v/wn/medicine is a manner of audio-
1
The knowledge graph is available at https://github.com/ease-crc/ tape/n/wn/artifact, authorize/v/wn/communication, fructify/v/wn/-
ease_lexical_resources body, and several other such doubtful relations.
7
English WordNet synset3 . We selected the part of the assert that chewing is a kind of grinding and a disposition
object taxonomy that refers to tools, buildings, and food. to be chewable is a disposition to be grindable.
Some of these entities also have associated UsedFor and UsedFor triples are interpreted as asserting that an ob-
CapableOf triples, where the third member of a triple cor- ject can play an instrumental role in a particular task.
responds to an action. We have also selected MannerOf CapableOf triples are interpreted as asserting that an
triples between actions in UsedFor and CapableOf triples. object can play a passive role, usually Patient. Triples de-
We added triples linking actions to VerbNet 3.2 classes, scribing combinations of items are interpreted as general
and so to thematic roles and selectional restrictions on assertions about all items of the categories mentioned in
role fillers. We have done extensive manual corrections the triple.
on the collected triples. We added triples describing what
items can be used together during an action, Some of
this knowledge comes from WordNet synset definitions, 5. Queries
some from the work of our colleagues on games with a
Reasoning with SOMA_DFL consists in performing sub-
purpose [29].
sumption inference tasks between concepts in the ontol-
ogy and query concepts defined by the user. We query
4.2. Structure SOMA_DFL with the reasoner Konclude. The ontology is
For reasoning and query answering, the triples from somewhat large, with 22527 object classes coming from
the knowledge graph are organized into an ontology. CSKG and related resources, and 45538 subclassof/equiv-
We now use OWL-DL, but plan to add support for non- alentclasses axioms. Further, the ontology uses the full
monotonic inference as it resembles more the default- expressivity of OWL-2. Nonetheless, performing disposi-
with-exceptions pattern that human rules often follow. tion queries is fast – less than a milisecond – as long as a
All birds fly, except those that do not – and it is conve- cache is constructed first, which takes about 10 seconds
nient both to keep the default rule as well as an open- using Konclude 0.7.0 on an Intel®Core™i5-7500 CPU @
ended list of exceptions. 3.40GHz with 8GB RAM. Table 1 shows some example
The ontology we produce is built on top of DUL [27] queries.
and the SOcio-physical Model of Activity (SOMA [28]).
These provide higher-level knowledge, in particular that 6. Conclusion and Future Work
Tasks are classify events and define Roles which can be
filled by appropriate Objects. We have presented a knowledge graph that contains in-
formation about object dispositions and possible com-
4.3. Axiomatization binations of objects that can be used to achieve a task.
The graph focuses on knowledge useful for everyday ac-
Our dispositional theory asserts that to play a role, an ob- tivities, and can be employed by a computational agent
ject must have a disposition for it. From VerbNet knowl- to select appropriate tools or patients (objects to act on)
edge, we produce axioms asserting that if an object fills a for a task, or to understand a scene in terms of what is
particular role in a particular task then it must also obey possible for the objects in it to do together. Our graph
the appropriate semantic restrictions. E.g., the following differs both in its focus – everyday activity knowledge
axioms – and in its purpose – logical reasoning, as opposed to
𝑐𝑙𝑎𝑠𝑠𝑖𝑓 𝑖𝑒𝑠−1 (𝑃 𝑎𝑡𝑖𝑒𝑛𝑡 ⊓ ∃𝑑𝑒𝑓 𝑖𝑛𝑒𝑠−1 𝑐ℎ𝑒𝑤) ⊑ information retrieval – from previous projects.
∃ℎ𝑎𝑠𝑄𝑢𝑎𝑙𝑖𝑡𝑦 𝑐ℎ𝑒𝑤.𝑃 𝑎𝑡𝑖𝑒𝑛𝑡 We will add support for non-monotonic inference, be-
∃ℎ𝑎𝑠𝑄𝑢𝑎𝑙𝑖𝑡𝑦 𝑐ℎ𝑒𝑤.𝑃 𝑎𝑡𝑖𝑒𝑛𝑡 ⊑ 𝑐𝑜𝑚𝑒𝑠𝑡𝑖𝑏𝑙𝑒 ⊓ 𝑠𝑜𝑙𝑖𝑑 cause it captures better the default with open-ended ex-
assert that what plays the patient role for chewing must ception list way that human knowledge is often orga-
have the chewable disposition, and therefore comestible nized, and will integrate knowledge about causal rela-
and solid. tions between events.
MannerOf triples are interpreted as describing a tax-
onomy, and the roles a manner defines are connected to
the roles defined by the task it is a manner of. E.g., the Acknowledgments
following axioms
𝑐ℎ𝑒𝑤 ⊑ 𝑔𝑟𝑖𝑛𝑑 This work was funded by the by the FET-Open Project
𝑐ℎ𝑒𝑤.𝑃 𝑎𝑡𝑖𝑒𝑛𝑡 ⊑ 𝑔𝑟𝑖𝑛𝑑.𝑃 𝑎𝑡𝑖𝑒𝑛𝑡 #951846 “MUHAI – Meaning and Understanding for
Human-centric AI” by the EU Pathfinder and Horizon
3
2020 Program and by the German Research Foundation
In CSKG, the names of such entities can be identified by a “/c/en/”
prefix and a “/wn/” infix. An example of such an entity is /c/en/-
(DFG) as part of Collaborative Research Center (SFB)
cut/v/wn/contact. In this paper we will omit the “/c/en/” prefix.
8
Table 1 [4] D. Beßler, R. Porzel, M. Pomarlan, M. Beetz,
Disposition queries, informally stated; concept names short- R. Malaka, J. Bateman, A Formal Model of Affor-
ened for space, responses in alphabetical order dances for Flexible Robotic Task Execution, in: Eu-
What can you use (as instrument) to – ropean Conference on Artificial Intelligence (ECAI),
broiler, dutch oven, gas oven, 2020.
bake
microwave, oven, rotisserie, tandoor [5] P. Agre, Computation and Human Experience,
adz, auger, ax, axe, backsaw, Learning in doing, Cambridge University Press,
cut New York, 1997. 00550.
(100 more)
cleaning pad, handkerchief, [6] D. Kahneman, Thinking, fast and slow,
wipe paper towel, piece of cloth, sponge, Farrar, Straus and Giroux, New York,
swab, towel 2011. URL: https://www.amazon.de/
What can you use (as object acted on) to – Thinking-Fast-Slow-Daniel-Kahneman/dp/
album, anthracite, art paper, 0374275637/ref=wl_it_dp_o_pdT1_nS_nC?
burn ie=UTF8&colid=151193SNGKJT9&coliid=
autograph, card, (204 more)
acerola, ackee, acorn squash, I3OCESLZCVDFL7.
eat
adobo, aitchbone, (1491 more) [7] J. J. Gibson, The Ecological Approach to Visual Per-
spill
absynth, acetone, acidophilus milk, ception, Psychology Press Classic Editions, 1979.
aioli, alcohol, (709 more) [8] J. Ortmann, W. Kuhn, Affordances as qualities, in:
What can you do with – Proceedings of the 2010 Conference on Formal On-
break, break into, change, tology in Information Systems: Proceedings of the
bowl Sixth International Conference (FOIS 2010), IOS
contain, cover, (25 more)
carve, change, cube Press, Amsterdam, The Netherlands, The Nether-
knife
cut, dice, (14 more) lands, 2010, pp. 117–130.
paper
burn, change, cover [9] L. A. Moralez, Affordance ontology: towards a
cut, extinguish, (12 more) unified description of affordances as events, Res.
What can you use (as instrument) to – Cogitans 7 (2016) 35–45.
ax, axe, backsaw, [10] I. Awaad, G. Kraetzschmar, J. Hertzberg, Challenges
cut firewood in finding ways to get the job done, in: 2nd Plan-
battle-axe, broad hatchet, (25 more)
alembic, amphora, ampulla, ning and Robotics (PlanRob) Workshop at 24th In-
contain a liquid
aspersorium, autoclave, (162 more) ternational Conference on Automated Planning and
cover a bowl
adhesive tape, book, duct tape, Scheduling (ICAPS), 2014.
foil, newspaper, (3 more) [11] A. Chemero, An outline of a theory of affordances,
What can you – Ecological Psychology 15 (2003) 181–195.
open with scissors envelope, letter, packet [12] M. T. Turvey, Ecological foundations of cognition:
alphabet soup, applejack, aqua vitae, Invariants of perception and action., American
scoop with a ladle
aquavit, armagnac, (75 more) Psychological Association (1992).
[13] F. Toyoshima, Modeling affordances with disposi-
tions, in: Proceedings of the Joint Ontology Work-
1320 EASE – Everyday Activity Science and Engineering, shops, 2018.
University of Bremen in subprojects P01 and P05. [14] B. Moldovan, P. Moreno, M. Van Otterlo, J. Santos-
Victor, L. De Raedt, Learning relational affordance
models for robots in multi-object manipulation
References tasks, in: 2012 IEEE International Conference on
Robotics and Automation, IEEE, 2012, pp. 4373–
[1] J. R. Hobbs, Sketch of an ontology underlying the 4378.
way we talk about the world, Int. J. Hum. Com- [15] R. Detry, E. Baseski, M. Popovic, Y. Touati,
put. Stud. 43 (1995) 819–830. URL: https://doi.org/ N. Kruger, O. Kroemer, J. Peters, J. Piater, Learn-
10.1006/ijhc.1995.1076. doi:10.1006/ijhc.1995. ing object-specific grasp affordance densities, 2009.
1076.
doi:10.1109/DEVLRN.2009.5175520.
[2] R. Porzel, On formalizing narratives, in: Proceed- [16] E. Ugur, J. H. Piater, Emergent structuring of in-
ings of the JOWO – Ontology Workshops, Bolzano, terdependent affordance learning tasks using in-
Italy, 2021. trinsic motivation and empirical feature selection,
[3] F. Toyoshima, A. Barton, J.-F. Ethier, Affordances IEEE Trans. Cogn. Dev. Syst. 9 (2017) 328–340. URL:
and their ontological core, Journal of Applied On- https://doi.org/10.1109/TCDS.2016.2581307. doi:10.
tology 17 (2022) 285–320.
9
1109/TCDS.2016.2581307. tional Conference, FOIS 2021, Bozen-Bolzano, Italy,
[17] K. Sakaguchi, R. L. Bras, C. Bhagavatula, Y. Choi, September 13-16, 2021, Frontiers in Artificial Intel-
Winogrande: An adversarial winograd schema chal- ligence and Applications, IOS Press, 2021.
lenge at scale, ArXiv abs/1907.10641 (2020). [29] J. Pfau, R. Malaka, We asked 100 people: How would
[18] E. Davis, Logical formalizations of commonsense you train our robot?, in: Extended Abstracts of
reasoning: A survey, J. Artif. Intell. Res. 59 the 2020 Annual Symposium on Computer-Human
(2017) 651–723. URL: https://doi.org/10.1613/jair. Interaction in Play, 2020, pp. 335–339.
5339. doi:10.1613/jair.5339.
[19] F. Ilievski, P. A. Szekely, B. Zhang, CSKG:
the commonsense knowledge graph, CoRR
abs/2012.11490 (2020). URL: https://arxiv.org/abs/
2012.11490. arXiv:2012.11490.
[20] R. Speer, J. Chin, C. Havasi, Conceptnet 5.5: An
open multilingual graph of general knowledge, in:
AAAI Conference on Artificial Intelligence, 2016.
URL: http://arxiv.org/abs/1612.03975.
[21] M. Sap, R. L. Bras, E. Allaway, C. Bhagavatula,
N. Lourie, H. Rashkin, B. Roof, N. A. Smith, Y. Choi,
ATOMIC: an atlas of machine commonsense for
if-then reasoning, in: The Thirty-Third AAAI
Conference on Artificial Intelligence, AAAI 2019,
The Thirty-First Innovative Applications of Ar-
tificial Intelligence Conference, IAAI 2019, The
Ninth AAAI Symposium on Educational Advances
in Artificial Intelligence, EAAI 2019, Honolulu,
Hawaii, USA, January 27 - February 1, 2019, AAAI
Press, 2019, pp. 3027–3035. URL: https://doi.org/
10.1609/aaai.v33i01.33013027. doi:10.1609/aaai.
v33i01.33013027.
[22] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, L. Fei-
Fei, Imagenet: A large-scale hierarchical image
database, in: 2009 IEEE Conference on Computer
Vision and Pattern Recognition, 2009, pp. 248–255.
doi:10.1109/CVPR.2009.5206848.
[23] D. B. Lenat, Cyc: a large-scale investment in knowl-
edge infrastructure, Commun. ACM 38 (1995) 32–
38.
[24] C. Fellbaum (Ed.), WordNet: an electronic lexical
database, MIT Press, 1998.
[25] J. Ruppenhofer, M. Ellsworth, M. Petruck, C. John-
son, J. Scheffczyk, Framenet ii: Extended theory
and practice (2010).
[26] K. K. Schuler, VerbNet: A Broad-Coverage, Com-
prehensive Verb Lexicon, Ph.D. thesis, University
of Pennsylvania, 2006. URL: http://verbs.colorado.
edu/~kipper/Papers/dissertation.pdf.
[27] V. Mascardi, V. Cordì, P. Rosso, A comparison of
upper ontologies., 2007, pp. 55–64.
[28] D. Beßler, R. Porzel, M. Pomarlan, A. Vyas,
S. Höffner, M. Beetz, R. Malaka, J. Bateman, Foun-
dations of the socio-physical model of activities
(soma) for autonomous robotic agents, in: B. Bro-
daric, F. Neuhaus (Eds.), Formal Ontology in Infor-
mation Systems - Proceedings of the 12th Interna-
10