Robotic manipulation tasks face many challenges when dealing with real scenarios that require the perception and understanding of the environment, planning with procedures that can adapt to the actual situation, and monitoring and recovery capabilities to ensure successful execution despite uncertainties and errors. The use of knowledge in terms of ontologies can address these needs by providing reasoning capabilities that allow the robot to be aware of the situation and the task and to dynamically create execution structures that adapt its behaviors in a flexible and robust manner. In this line, this paper discusses the work done at the Institute of Industrial and Control Engineering (IOC-UPC) that uses ontologies combined with Large Language Models to plan and automatically generate Behavior Trees with monitoring and recovery capabilities, which allow for the control of robot execution for successful performance in real scenarios.
Robotic manipulation tasks in real-world scenarios, such as those faced by a dual-arm mobile manipulator performing assistive tasks in a human environment, pose significant challenges due to uncertainty and their dynamic nature. This demands intelligent systems capable of adapting planning and execution to unforeseen changes. That is, the robot needs to be aware of the situation, of its own capabilities, of the task particularities and of the goal state to be achieved, which will condition the planning. On the other hand, to execute the plan the robot needs to be able to flexibly configure execution structures that also account for monitoring actions and recovery strategies, in order to ensure a robust performance of the task.
Several tools can be used to achieve this objective, such as automated planning, Knowledge
Representation and Reasoning (KR&R), Large Language Models (LLMs) and Behavior Trees (BTs). Automated
planning [
In this paper, we discuss the use of these tools in pursuing autonomy in robots performing manipulation tasks in the real world. Section 2 reviews the proposal of a knowledge-based manipulation framework. Section 3 reviews planning approaches that combine ontologies and LLMs. Section 4 reviews the automatic execution configuration of BTs and proposes new BT structures to add monitoring and recovery strategies flexibly. Finally, Section 5 presents the conclusions and discusses the future research directions.
The BE-AWARE framework [
Fig. 1 (left) shows the main parts of the framework: (1) the Knowledge Representation and Reasoning
core (BE-AWARE contains Situation, Domain and Execution ontologies, which are built upon the
Autonomous Robotics Ontology, AuR [
Fig. 1 (right) shows the planning and execution pipeline, where the ontologies play a role in the planning design and in configuring the BTs for the execution, detailed below in Sections 3.1 and 4.1,
A situation ontology is included in the BE-AWARE framework in order to help the robot to understand
the scene, i.e. to have situation awareness. The ontology includes concepts for describing the relative
location between objects, the ontology instances being provided by a smart perception module [
The LLM instantiation feature is also exploited by BE-AWARE to enable the user to define the goal
state of a manipulation task. BE-AWARE assumes that the PDDL domain is fixed and given, and it uses
this domain, together with the situation ontology, to generate the PDDL problem file. This is done by
matching ontological concepts to PDDL types and predicates, resulting in a problem file that includes
the PDDL objects, the initial state, and the goal state (red blocks in Fig.2). In this way, BE-AWARE
equips the robot with the task awareness necessary to solve the task using classical task planners [
As an alternative to classical task planning, LLM-based approaches have been proposed to exploit the potential of LLMs, although the resulting plans have not always been accurate enough (with flaws such as incorrect temporal goal ordering) due to static and template-based prompting, which has problems in adapting to dynamic environments and complex task contexts. Following this line, and complementary to the proposal described in the previous section, we explored LLM-based task planning, but addressing its limitations with a novel ontology-driven prompt-tuning framework that employs knowledge-based reasoning to refine and expand user prompts with task contextual reasoning and knowledge-based environment state descriptions. The integration of domain-specific knowledge into the prompt ensure semantically accurate and context-aware task plans. As illustrated in Fig. 3, the elaborated prompt lead the LLM to resolve errors such as incorrect object placement order (e.g., placing food before crockery), improving the plan’s semantic integrity.
In order to execute the actions of a task plan, like Pick, Move, Place, an execution structure, such as BTs, is required. We have proposed a flexible, ontology-based automatic generation of BTs (without using LLMs), tailored to the current circumstances and specific requirements of a task, that encapsulate execution details, many of which require information not available at plan time.
BTs provide modularity and reactivity that are ideal for robot execution, but, traditionally
templatebased BTs are used, which prevent the robot to adapt to the current circumstances if they have not
been considered in the BT. To go beyond this limitation, the work in [
To enhance the execution layer with robustness and reactivity, we propose here the use of Behavior Blocks, modular BT substructures tailored for monitoring and recovery. These blocks are instantiated from templates and further customized using ontologically driven flavors, in a similar way as done in the previous section. Each monitored behavior is wrapped with conditions that assess the state of the system at three points: preconditions (before execution), hold conditions (during execution), and postconditions (after execution), as shown in Fig. 5. Upon failure, a diagnostic code is generated, triggering a recovery strategy. The basic template includes retries (executing the behavior up to N times), but more sophisticated strategies can be selected based on the failure type, such as motion replanning, alternative action selection, or environmental interaction (e.g., obstacle removal).
The instantiation and tuning of monitoring and recovery structures is driven by semantic reasoning over the ontologies, which takes into account factors such as sensor availability or object properties. For instance, if the object is a fragile glass containing liquid, the system may insert an orientation check in the monitoring phase (the function implementing this check will determine the failure occurrence and generate a diagnostic code); if a slippery or hazardous item is detected, the system can tighten trajectory tolerances or reduce motion speed proactively. Fig. 6 illustrates the Behavior Block used for monitoring and recovery in the Move Arm action.
The aim of this paper has been to illustrate the potential of using ontologies to enhance both planning and execution in robotic manipulation, in particular, making planning adaptive to the circumstances that the robot encounters and making execution robust to uncertainties, thus contributing to make robots truly autonomous. A unified framework has been conceived with several ontologies at its core, which allowed: (a) the context-aware instantiation of symbolic problem definitions; (b) the tuning of prompts for LLM-based task planning; (c) the generation of optimized execution structures tailored to the current situation and task; and, (d) the generation of monitoring and recovery execution structures, based on the actions to be performed and the features and circumstances of the actual execution.
Future work includes the expansion of the Flavor library to support a wider range of behavioral adaptations, the improvement of failure handling through better integration of sensory context and reasoning mechanisms, and the testing on real robots performing everyday tasks in human environments.
This work was partially supported by the European Commission’s Horizon Europe Framework Programme with the project IntelliMan (AI-Powered Manipulation System for Advanced Robotic Service, Manufacturing, and Prosthetics) under Grant Agreement 101070136.
During the preparation of this work, the author(s) used ChatGPT to improve the writing style and for grammar and spelling checks. After using these tool(s)/service(s), the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.