Service robots need common-sense knowledge to help humans in everyday situations as it enables them to understand the context of their actions. However, approaches that use ontologies face a challenge because common-sense knowledge is often implicit, i.e., it is obvious to humans but not explicitly stated. This paper investigates if Large Language Models (LLMs) can fill this gap. Our experiments reveal limited efectiveness in the selective extraction of contextual action knowledge, suggesting that LLMs may not be suficient on their own. However, the large-scale extraction of general, actionable knowledge shows potential, indicating that LLMs can be a suitable tool for eficiently creating ontologies for robots. This paper shows that the technique used for knowledge extraction can be applied to populate a minimalist ontology, showcasing the potential of LLMs in synergy with formal knowledge representation.
CEUR ceur-ws.org
To assist humans efectively in everyday life, service robots need a sound foundation of
commonsense knowledge to guide their actions. While there has been significant research on
KnowledgeBased Systems (KBSs) for robotics [
Common-sense knowledge for robots involves actions necessary to reach a desired state, defined
by an action verb, an actor, an object, and a tool, forming an action pattern [
The application of LLMs to robotics leads to the emergence of embodied Artificial Intelligence
(AI). However, success rates of such systems are still limited, for instance, up to 85 % for tidying
up [
This work focuses on common-sense knowledge in the form of action patterns. An action pattern is represented by an action , which is executed by a set of Agents using a set of Tools to modify a set of Objects , cp. Equation (1). These patterns capture various scenarios a robot might encounter and how it can act to change these situations. This knowledge allows the robot to infer appropriate actions to fulfill its goals, such as serving bread. Note that the knowledge should be grounded in real-world settings and continuously tested, possibly resulting in the robot confirming, discarding, or refining its knowledge. Note that the states of all elements before and after the action can be described via a set of attributes and , respectively. In addition, the spatial relations ℛ between the elements involved may change.
= (, , , , , , )
The set comprises all valid action patterns in the sense that they are meaningful in real-world applications. For instance, when tasked with baking bread, a robot has to identify suitable tools, as illustrated by the incomplete action pattern in Equation (2). = (, { }, { }, {
$tool}, {}, {ℎ}, {})
Action patterns can be expressed using Web Ontology Language (OWL) 2 Description Logic (DL), a subset of predicate logic. Here, Actions are defined by their relations to the notions Object, State, Location, and Time, forming action patterns. These notions, rooted in previous work, cp. Section 2, can be aligned with top-level ontologies such as Basic Formal Ontology. An Object can also take the role of an Agent or a Tool, and spatial relations can be expressed using existing ontologies. Equation (3) shows the DL representation of the notions Action and Object. ⊑ ∃ ⩾ 1.ℎ
_. ⊓ ∃ ⩾ 1.ℎ ∃ = 1.ℎ _. ⊓ ∃ = 1.ℎ ⊑ ∃ ⩾ 1.ℎ _. _. ⊓ ∃ ⩾ 0.ℎ _. _. ⊓ (1) (2) (3)
In order to replace KBSs, LLMs would need to be capable of consistently answering queries such as “Which tool can I use to bake bread?”. This task can be posed generically using Prompt 1. Note that specifying candidates corresponds to a simple grounding in reality. Further, we modified Prompt 1 to extract the state of an object before and after a given action has been applied to it. The prompt can also be adapted to extract details about the spatial relationships between the object and the tool used.
Prompt 1. In the following, I will ask you a question. In your response, I want you to answer with nothing but a list of suitable comma-separated words sorted by relevance. Which tool can I use to $action $object? Choose only from the following candidates: $candidates.
Example: ... Which tool can I use to bake bread? Choose only from the following candidates: bowl, oven, knife, ...
The large-scale extraction of action patterns to incorporate into a knowledge base can be achieved using Prompt 2. Compared to Prompt 1, this prompt specifies the domain of interest and permits constraints within the action pattern, ofering additional flexibility. The information extracted by Prompt 2 may not be directly applicable, but it is well-suited for populating an ontology with broad common-sense knowledge on a large scale.
Prompt 2. Please respond with nothing but lists of the form ’(action, agent, object, tool)’. An action pattern is defined by an action, i.e., a verb, an agent who executes the action, an object, which is modified, and optionally a tool. Generate $number action patterns for the domain of interest ’$domain_of_interest’.
Example: ... Generate 100 action patterns for the domain of interest ’kitchen’.
To evaluate the LLMs’ suitability as a source of common-sense knowledge, we used a ground truth consisting of 97 action patterns. The data set was created in a study with 20 participants, who completed the action patterns by determining the states of each object before and after the action, identifying the tools used, and outlining the spatial relationships between the objects and tools. An excerpt of this ground truth for the action “cut bread” is shown in Listing 1.
Listing 1: Excerpt from the ground truth for action patterns.
We conducted preliminary experiments for choosing suitable LLMs, including ones with Vicuña’s 7B and 13B models and OpenAssistant. Despite performing well in chat-like scenarios, these models had dificulty returning structured results. OpenAI’s models appeared particularly promising, so we included them along with BLOOMZ as an open-source alternative and BERT as a baseline. For BERT, we rephrased the prompts as demasking tasks. We accessed gpt-3.5turbo and gpt-4 via the OpenAI API and BLOOMZ via the Hugging Face API, and deployed BERT locally using a Hugging Face transformers pipeline. While we experimented with various temperatures, we ended up using a temperature of 0 to ensure reproducability. Providing candidates leads to significantly better results when comparing the responses to the ground truth. However, it is valid to assume that candidates are available, as this corresponds to a grounding in the robot’s environment, which can be achieved using an appropriate perception module. The architecture we developed for the evaluation is shown in Figure 1. We also adapted this architecture to populate an ontology revolving around the notions described in Section 3.1 with the action patterns extracted to demonstrate the use of the information extracted, see Figure 1 bottom right.
Ground truth: action patterns
Prompt generation Ontology population process
Evaluation process
LLM Wrapper + fill_mask() + conversation()
BERT BLOOMZ ChatGPT
Evaluation Ontology population
Plots Ontology
Our experiments indicate that LLMs are a scalable source for general common-sense knowledge
in the form of action patterns, which is a valuable basis for an ontology usable by robots.
However, LLMs are still insuficiently reliable for providing actionable knowledge for robotics
applications by themselves. Thus, we suggest pursuing the combination of LLMs and knowledge
graphs for robotics, especially in the form of an integrated pipeline for ontology population
from LLMs, and validation and reasoning using symbolic AI. Future work should also address
the validation of newly extracted information using the existing knowledge graph in the spirit
of Pan et al. [
Supplementary Material: The ground truth dataset, the prompts, and a Python script for creating an OWL ontology from the action patterns extracted are available via GitHub1.