The aim of this work is to develop a task representation and execution framework for dual-arm manipulators operating in tabletop scenarios. In particular, we want to enforce robot's autonomy, robustness to failures, and (in perspective) a natural human-robot interaction. To this purpose, the framework integrates (i) point cloud perception, (ii) ontology-based knowledge representation, (iii) high-level task planning, as well as (iv) task execution and monitoring. Our main contribution is an open source, closed-loop hybrid architecture based on semantic knowledge and high-level reasoning to ground perception-based task representation, reasoning and execution. An ontology integrates perceptual cues with planning-relevant knowledge to automatically perform context assessment, infer when to act to modify the environment and to generate appropriate de nitions of domains and problems to solve. An interface between the ontology and low-level motion planners allows for updating the representation at run-time, thus enforcing robustness versus unmodelled traits of the environment. The framework has been validated using a Baxter dual-arm manipulator operating in a tabletop scenario.
In order to enforce a robot's autonomy and exibility, a closer coupling is needed between high-level task planning and low-level action execution. Such a coupling is expected to allow reactive behaviors to operate not only on the basis of runtime feedback, but also within a well-de ned context. High-level task planning is scarcely robust with respect to uncertainty and unmodeled traits of the environment. However, it allows for goal-driven and context-based behavior, and for dealing with knowledge at a high level of abstraction. Furthermore, for longterm autonomous operations, a robot's architecture must identify its own goals (in the form of norms or directives to satisfy) and recognize whether they are met: the robot must check unsatis ed norms, address violated directives, plan for a course of action solving such inconsistences, update its knowledge and, in case of failures, re ne its plan.
A number of hybrid reactive/deliberative architectures, with similar goals,
have been proposed [
To validate our ideas we designed a software architecture adopting the
wellknown ROS framework [
Scenario. Let us consider a dual-arm manipulator operating in a tabletop scenario, which mimics a smart factory industrial setting. The robot must satisy two operational requirements: (i) keeping the workspace tidy, e.g., assuring that a certain class of objects is always located on the left hand side of the table, and (ii) satisfying user requests, e.g., handling objects to human sta . In our simpli ed scenario, we discriminate objects only by their shape and size. We de ne a directive as a maintenance goal, one that must be satis ed at all times. Furthermore, let us assume the following situation: { The robot is in front of a table with spheres and cylinders only. { The directive is: all cylinders must be on the left hand side of the table. { A user asks the robot to pick up a sphere and to put it down inside a tray. { The sphere cannot be picked up because it is cluttered by other objects. { The goal position for the sphere is already occupied by another object. To ful ll the requests, all the four phases in Figure 1 are involved. Scene perception. The perceived scene is encoded in the ontology in terms of objects and their relationships to de ne predicates, e.g., On(blue-ball,table). We distinguish two kinds of predicates: descriptive predicates represent factual knowledge about the environment, and normative predicates encode the desired state. Both objects and predicates are encoded as individuals belonging to the appropriate class in the ontology. We consider only unary and binary relationships and express their arguments as individual's object properties.
The perception module is always executed in the background to track object locations and keep the Moveit! planning scene consistent with the environment. Periodically, the last perceived scene is encoded in the ontology and updates the previous one. All predicates originating from the last encoded scene are considered descriptive.
Goal generation. Goals and directives are provided by the human
interacting with the robot and are encoded as normative predicates. The system keeps
updating its knowledge till the sets of descriptive and normative predicates do
not mutually con ict. In our case, one directive is speci ed and it is satis ed till
the human asks the robot to pick up a sphere and put it down in a tray. The
con icting predicates are detected by SWRL rules and the Pellet reasoner [
Task planning and action execution. Execution is staged by a mission control module, which is organised as three submodules, namely plan parsing, action execution and planner-independent ontology interface. In the considered scenario, the robot generates a very simple plan, i.e., (i) freeing up the goal position by (ii) moving the occupying object to another free position and (iii) moving the sphere there. The plan instance is parsed and dispatched to Moveit! for execution. In the motion planning phase, Moveit! recognizes that one of the task would fail if executed, since the sphere is not reachable because of clutter. Hence, mission control stops the execution and the ontology is updated with a predicate specifying the cluttered situation. The system then moves back to the reasoning and goal generation phase, and a new plan is determined and executed, taking the clutter-related predicate into account. No further reasoning activity is required unless a new violation is detected. This may happen in case the plan would somehow fail. Once satis ed, temporary goals can be cleared from memory, while long-term directives can be kept to ensure robust operation. 3
In this paper, we propose a hybrid reactive/deliberative architecture for perception, knowledge representation and planning with a dual-arm manipulator. The framework has been evaluated in a scenario in which a Baxter robot (provided with an external RGB-D sensor) manipulates objects on a (possibly cluttered) table. Robot actions involve typical tabletop operations, such as pick and place, as well as the swap between the positions of any two objects. Preliminary results suggest that such an architecture can be a viable approach for robust and exible autonomous operations in smart factory scenarios, especially in those contexts in which human-robot cooperation plays an important role. Current work is focused on expanding the architecture to allow a human user to interact with the robot using a natural language interface. The framework is partially available open source1. 1 Please check: https://github.com/EmaroLab.