Introduction

We envision household robots that can be placed in any kitchen to then be given a random recipe from the Web that they can understand and parse into action plans that can be broken down into executable body motions that can be performed with available objects in the environment. For this, robots need to be enabled to perform meal preparation tasks with any tool, on any available object and for a variation of tasks. This tutorial is based on prior research that proposed a methodology for creating actionable knowledge graphs [1], where a solution for creating knowledge graphs that link object to action and environment information and thus make them actionable is proposed, as well as a knowledge engineering methodology that is more specifically aligned to creating ontologies for meal preparation tasks that can be used to parameterise robot action plans in order to perform task variations of cutting actions [2].

There has been lots of research on creation of knowledge graphs, which has led to many domain knowledge graphs that have proven to be good in answering questions. Usually, these knowledge graphs contain object information (e.g. about food objects, recipes, people, books). To make such knowledge graphs actionable, it is important to link the contained object knowledge to environment knowledge. If robots shall use the knowledge graphs for action execution, they need to further include action knowledge.

ESWC 2024 Workshops and Tutorials Joint Proceedings, May 26-27, Heraklion, Greece beetz@cs.uni-bremen.de (M. Beetz); cimiano@techfak.uni-bielefeld.de (P. Cimiano); michaela.kuempel@uni-bremen.de (M. Kümpel); enrico.motta@open.ac.uk (E. Motta); i.tiddi@vu.nl (I. Tiddi); jtoeberg@techfak.uni-bielefeld.de (J. Töberg) 0000-0002-7888-7444 (M. Beetz); 0000-0002-4771-441X (P. Cimiano); 0000-0002-0408-3953 (M. Kümpel); 0000-0003-0015-1952 (E. Motta); 0000-0001-7116-9338 (I. Tiddi); 0000-0003-0434-6781 (J. Töberg) This implies that actionable knowledge graphs do not aim at perfectly modeling object knowledge, but instead focus on reuse of existing knowledge sources and modeling and linking of environment and action knowledge in order for making the contained knowledge applicable in agent applications. This tutorial will detail the necessary concepts for creating an actionable knowledge graph for the example domain of Cutting Fruits and Vegetables, which shall be used by robotic agents to be able to infer the correct body motions for quartering an apple or dicing a cucumber.

Structure of the Tutorial

The tutorial is centered around the knowledge engineering methodology introduced in [2] and its application on the exemplary task of Cutting Fruits & Vegetables. In general, the methodology consists of five steps to create actionable knowledge graphs that a robot can employ to handle manipulation tasks, as can be examined in Figure 1. In the following we present a brief summarisation of these steps:

1) Defining Motion Parameters: Definition of the domain-and action-dependent parameters influencing the execution of the target manipulation action. An example is the knife position for cutting tasks. 2) Collecting Knowledge Sources: Collection of different sources for three types of knowledge: action knowledge, object knowledge & knowledge for linking action and object knowledge 3a) Extraction of Action Groups & Affordances: Collect information about the manipulation action and its associated synonyms and hyponyms. This information is used to organize different action verbs into groups based on similarities in their motion parameters. For each so called action group, a representative is chosen and their affordances are created. 3b) Extraction of Object Knowledge & Dispositions: Collect information about objects participating in the manipulation action (e.g. tools, environments, targets). Then collect information and concrete values for the task-specific object properties that influence the action execution. This knowledge is represented through dispositions. 4) Relate Object to Action Knowledge: Relate the action affordances to the object dispositions in an ontology by re-using relations from the SOMA [3] ontology. 5) Link to Cognitive Architecture: Map concepts in the generalized manipulation plan to their representation in the ontology and use the architecture's perception system to ground objects and their properties. In this tutorial we present the whole methodology but focus on the steps 1), 3) and 4), which represent the knowledge collection and extraction from (Semantic) Web resources.

Defining Motion Parameters

In order to create an actionable knowledge graph for the domain of cutting fruits and vegetables, we first have to investigate motion parameters that influence action execution. For this, one can first investigate a lexical resource like WordNet [4] to find commonly used synonyms of cutting, such as slicing, dicing, or halving.

We then investigate how different action verbs influence task execution, which results in the following motion parameters:

-number of repetitions: Cutting tasks vary in the number of repetitions to be executed. Sometimes, a cut is only performed once, while other tasks require to cut the whole object. -cutting position: Cutting tasks also vary in the applied cutting position. Halving requires a different position than slicing, for example. -result object: Cutting tasks result in objects of different amount and shape.

-prior actions: Some objects require a prior action (such as peeling) to be executed.

-dependent tasks: Some tasks depend on prior tasks (i.e. quartering depends on halving).

Extraction of Relevant Action Knowledge from the Web

The relevant action knowledge we focus on consists of the different verbs that are associated with the manipulation action. This includes the main verb (e.g. cut) as well as all of its hyponyms and synonyms. Additionally, action knowledge covers the properties of the different verbs that distinguish their action execution and generally influence the manipulation action.

In the tutorial we showcase the action knowledge extraction for the exemplary task of Cutting. We begin by extracting all synonyms and hyponyms from WordNet [4] and VerbNet [5], two expertly created resources for lexical information and verb usage. For the verb cut, we extract 211 verbs from WordNet and 147 verbs from VerbNet. After pre-processing and duplicate removal, 181 verbs remain. These remaining verbs are then filtered based on their relevance for the domain using an instructionfocused corpus from WikiHow. We set a threshold of 100 occurrences in a specific part of an article across the whole corpus to warrant an inclusion of the verb in future steps. With this restriction, only 46 verbs remain. However, there is still a need for manual post-processing since some important verbs are missing (e.g. halve or quarter) or are very general and thus not relevant for cutting (e.g. make or pull).

Table 1

Comparison of different methods for extracting anatomical parts for a given fruit sorted based on their F1-score. In each column, we mark the three methods with the highest performance in bold.

Method

Extraction of Relevant Object Knowledge from the Web

For the object knowledge, we focus on information about objects involved in the manipulation action, their properties, usage and their specific purpose. In general we showcase a similar pipeline to the one explained in Section 2.2. We begin by extracting all relevant objects from domain-specific taxonomies.

For our focus on fruits and vegetables, we query the FoodOn [6] using SPARQL, resulting in 257 unique fruits and 31 unique vegetables. Since not all of these fruits and vegetables are equally relevant and we need enough information to exist to evaluate their task-specific properties, we again use instructionfocused corpora to filter them based on their occurrence data. In this case we also look at the recipe corpus Recipe1M+ [7] and only include fruits and vegetables that occur in 1% of any part of these two corpora. This filtering step results in 15 remaining fruits and one remaining vegetables. Lastly, we present our ongoing efforts in automating the extraction of task-specific object property values. For this, we compare three different pre-trained embeddings (GloVe [8], NASARI [9] and ConceptNet Numberbatch [10]), two large language models (ChatGPT and GPT-4) as well as two techniques for extracting this information from the Recipe1M+ on the task of extracting the existing anatomical parts for a given fruit. Our preliminary results and their condition can be examined in Table 1.

Linking Action to Object Knowledge in the Ontology

For connecting and linking the action to the object knowledge, we rely on the concepts of disposition and affordance. In general, a disposition describes the property of an object, thereby enabling an agent to perform a certain task [11] as in a knife can be used for cutting, whereas an affordance describes what an object or the environment offers an agent [12] as in an apple affords to be cut.

In recent works like SOMA [3], both concepts are set in relation by stating that dispositions allow objects to participate in events realizing affordances, which are more abstract descriptions of dispositions. This is achieved in the TBOX by using the affordsTask, affordsTrigger and hasDisposition relations from SOMA. An example for the disposition of Peelability can be examined in Section 2.4.

hasDisposition 𝑠𝑜𝑚𝑒

(𝑃 𝑒𝑒𝑙𝑎𝑏𝑖𝑙𝑖𝑡𝑦 𝑎𝑛𝑑 (affordsTask 𝑠𝑜𝑚𝑒 𝑃 𝑒𝑒𝑙𝑖𝑛𝑔) 𝑎𝑛𝑑 (affordsTrigger 𝑜𝑛𝑙𝑦 (𝑐𝑙𝑎𝑠𝑠𝑖𝑓 𝑖𝑒𝑠 𝑜𝑛𝑙𝑦 𝐻𝑎𝑛𝑑)))

Figure 2: Example for connecting an affordance ("Peeling with a hand") to a disposition ("Peelability") using relations from the SOMA ontology [3].

Tutorial Material

For the tutorial, we made our implementation available in Jupyter Notebooks found in a GitHub repository 1 . Participants are encouraged to download the notebooks and follow along, but the notebooks are presented in depth during the talks, so actual hands-on experience is optional.