induced changes. Robots are increasingly being used in dynamic environments like workplaces, hospitals, and homes. As a result, interactions with robots must be simple and intuitive, with robots' perception adapting eficiently to humanThis paper presents a robot control architecture that addresses key challenges in human-robot interaction, with a particular focus on the dynamic creation and continuous update of the robot's state representation. The architecture uses Large Language Models to integrate diverse information sources, including natural language commands, robotic skills representation, real-time dynamic semantic mapping of the perceived scene. This enables flexible and adaptive robotic behavior in complex, dynamic environments. Traditional robotic systems often rely on static, pre-programmed instructions and settings, limiting their adaptability to dynamic environments and real-time collaboration. In contrast, this architecture uses LLMs to interpret complex, high-level instructions and generate actionable plans that enhance human-robot collaboration. At its core, the system's Perception Module generates and continuously updates a semantic scene graph using RGB-D sensor data, providing a detailed and structured representation of the environment. A particle filter is employed to ensure accurate object localization in dynamic, real-world settings. The Planner Module leverages this up-to-date semantic map to break down high-level tasks into sub-tasks and link them to robotic skills such as navigation, object manipulation (e.g., PICK and PLACE), and movement (e.g., By combining real-time perception, state tracking, and LLM-driven communication and task planning, the architecture enhances adaptability, task eficiency, and human-robot collaboration in dynamic environments. Human-Robot interaction, Robot task planning, Large language models, Scene graphs Workshop on Advanced AI Methods and Interfaces for Human-Centered Assistive and Rehabilitation Robotics (a Fit4MedRob Proceedings
GOTO).
Immediacy is crucial in assistive robotics [
The chief concern of the work is to exploit scene graphs as semantic maps providing a structured representation of spatial and semantic information of robot’s environment. This enables LLMs to generate plans based on this information. Indeed, via scene graphs robots can map the relationships between objects, their properties, and their spatial arrangements.
Here we address such limitations by representing the environment as a graph endowed with updatable
CEUR
ceur-ws.org semantics that language models can interpret. More precisely, the dynamics of the update is achieved via particle filtering to enhance the reliability and precision of real-time semantic mapping. The model adopted (PSGTR) is lightweight and can be easily utilized, making it suitable for live applications and accessible even on less powerful hardware. Using RoBee, the cognitive humanoid robot developed by Oversonic Robotics, the system dynamically updates the environment graph and replans in case of failure, overcoming challenges in long-term task planning.
A scene graph captures detailed scene semantics by explicitly modeling objects, their attributes, and the
relationships between paired objects (e.g., “blue bottle on the table”) [
Recent works, such as [
Our system is based on two components: • Perception Module: it is responsible for sensing and interpreting the environment and building a semantic map in the form of a directed graph that integrates both geometric and semantic information. Its architecture is explained in detail below. • Planner Module: it takes the information provided by the Perception Module to formulate plans and actions that allow the robot to perform specific tasks. It is composed by the following: – Task Planner: Translates user requests, expressed in natural language, into high-level skills. – Skill Planner: Translates high-level skills into specific, low-level executable actions. – Executor: Executes the low-level actions generated by the Skill Planner. – Controller: Monitors the execution of actions and manages any errors or unexpected events during the process. – Explainer: Interprets the reasons of execution failures by analyzing data received from the
Controller and provides suggestions to the Task Planner on how to adjust the plan.
These components interact to allow the robot to understand its environment and act accordingly to satisfy user requests. In what follows we specifically address the Perception Module while details on the planner will be provided in a separate article.
Robot Hardware. The system was implemented using RoBee, the cognitive humanoid robot developed
by Oversonic Robotics. RoBee, shown in Figure 3.1, stands 160 cm tall and weighs 60 kg. It features 32
degrees of freedom, and is equipped with cameras, microphones, and force sensors.
3.1. Perception module
The Perception Module is the component responsible for building a representation of the environment,
which the robot can use for task planning. The representation takes the form of a semantic map, a
graph that integrates both geometric and semantic information about the environment. To generate
the semantic map, the perception module uses data from various sensors. It requires RGB-D frames
obtained from the camera which are then processed using a scene graph generation model, such as
PSGTR [
The presence of room nodes is important because it facilitates the categorization of objects based on their respective rooms, which helps distinguish between objects with the same name and enhances the natural language description of the task, while room nodes enable the application of graph search algorithms for planning paths to objects. Room nodes are created based on the geometric map, while object nodes are generated following the steps explained below.
As to edges, more specifically:
• Edges between rooms directly connect two rooms and facilitate navigation between them.
• Edges between objects represent the relationships between objects and are directed, the direction
capturing the influence of one object on another; the label associated with each edge is derived
from the inferences made by the PSGTR model.
Generating and updating the semantic map The scene graph generation process is based on the
PSGTR model, a single-stage model built on the Transformer architecture [
The Perception Module uses the result of PSTGR and builds the semantic map following the steps below: 1. Reading RGB-D frames: The video frames from the robot’s cameras are sent to the model to be analyzed and used to generate the scene graph.
2. Reading robot poses: To generate the scene and semantic map, it is necessary to know the robot’s position relative to the geometric map, the camera’s position relative to the map, and the camera’s mounting position on the robot. 3. Inference: Each received frame is processed by the model. Results are information about detected objects, such as labels and masks, and the relationships between them, such as relationship labels and associated probabilities. 4. Graph construction: This step involves extracting data from the object returned by the model and computing values dependent on the robot system, such as the position of objects. At a finer level it consists of three sub-steps: a) Node construction: Classes and masks of detected objects are extracted. Next, the 3D position of each object is computed, starting in the pixel coordinate system, then transforming to the camera system, and finally to the robot’s map coordinate system. Nodes for the semantic scene and the semantic map are instantiated using the appropriate 3D coordinates. A distance-based filter is applied to prune objects that are too far from the robot to avoid issues with object detection and tracking. b) Edge construction: Data about relationships between objects are extracted. For each relationship, the source and target object indices are identified. If both objects meet distance constraints and the relationship probability exceeds a defined threshold, an edge is created between the corresponding nodes. c) Inference improvement through Particle Filter (PF): As the model’s output is not accurate regarding mask inference, this leads to errors in calculating the object’s centroid for obtaining its position relative to the map. A PF based on previous observations is applied to improve the accuracy of the result.
At the end of the process, the semantic map is updated with the new information, and the semantic scene is generated and provided to the planner module.
The PF is used to track the object masks in real-time, provided as output by the PSGTR model, and to improve the estimation of their position in space. During the update process, the filter uses information from frames acquired to refine the position estimate of the objects. The last object masks identified by the PSGTR model are compared with previous ones using the Intersection over Union (IoU) metrics and by applying the motion model, which can be defined as a transformation of the camera position relative to the map between two time instances. Denote the transformation matrices describing the camera position at time − 1 and at subsequent time , T−1 and T , respectively; then, the change in position and orientation can be expressed by the transformation matrix ΔT = T T−−11 . To associate objects between successive frames, we use an IoU matrix computed over segmentation masks. For two masks and , IoU is defined as IoU(, ) = ||∩∪|| , where | ∩ | represents the area of intersection between masks and , and | ∪ | represents the area of their union. To compare segmentation masks between two successive frames, we denote the segmentation mask at time − 1 as −1 and at time as . The transformation matrix ΔT is applied to the previous mask to obtain a transformed mask −′1 such that −1 = ΔT ⋅ −1 . The Intersection over Union (IoU) is then computed between the ′ transformed mask −′1 and the current mask as follows: IoU( −′1 , ) = to identify the same object across successive frames based on their masks. | −′1 ∪
| | −′1 ∩ | . This allows us
More formally, each object is represented by a set of particles, where each particle at time is a 3D vector representing a hypothesis about the object’s position: = [ , , ] , where = 1, … , . The particles are initialized with a normal distribution around the initially observed position 0 = [ 0, 0, 0] : 0 ∼ (
0, Σ0), where Σ0 = diag( 2, 2, 2) is the initial covariance matrix. Initial weights distribution structure used for initial particle initialization. Given a new observation are uniform: 0 = 1 , where = 1, … , . Prediction takes into account the camera motion. If −1, is the transformation matrix from frame − 1 to frame , each particle is updated as = −1, ⋅ −1 + 0, where 0 represents the noise added to account for uncertainties in motion, maintaining the same , the particle weights are updated based on the Euclidean distance between the predicted position and the observed one: = ‖ −
1 ‖2 and = 1+ . Weights are then normalized: = the object ̂ is estimated as the weighted mean of all the particles: ̂ = ∑=1 Update semantic map with ̂
Update weights: = ‖ − new‖2, = 1
1+ Apply transformation: −′1 = ΔT ⋅ −1 ▷ Transform previous masks ▷ Compute IoU between nodes and inference results
Scene graphs provide a structured representation that captures geometric and semantic information about the environment. This comprehensive understanding enables improved task planning with large language models, allowing robots to execute commands.
In this article we have shown how to use real-time sensor data to dynamically update semantic maps,
thus enabling the robot to adapt to ongoing changes in their environment, particularly in collaborative
settings influenced by human actions. Here, particle filtering is applied to improve geometric data
precision and semantic map accuracy. This can be particularly important also for social interaction and
intention prediction [
The issues addressed in this work are cogent. Indeed, the efectiveness of planners in translating complex instructions into actionable plans relies on a robust state representation. Without an accurate semantic map, planners risk generating plans that misalign with the actual environment, potentially leading to task failures. The integration of semantic and geometric insights permits robots to reason about their environment in a more informed and adaptive way, ensuring that they can operate efectively and responsively in dynamic environments.
The adoption of a semantic map containing rich spatial information combined with a flexible LLM
based planner can easily allow to explore in the future the introduction of new spatial relationships, e.g.
wrapped, stuck under, surrounding, aligned, that could support specific novel robot skills [
Special thanks to Oversonic Robotics for enabling the implementation of the system using their humanoid robot, RoBee.
More information about RoBee and Oversonic Robotics are available: • RoBee, • Oversonic Robotics