The ability to grasp and manipulate objects is crucial for performing several complex tasks and it is highly desirable to transfer this skill efectively and naturally to robotic systems. Learning by demonstration provides a particularly interesting and promising technique to address this problem, as it allows us to leverage the guidance of the demonstrations provided by an expert to speed up task learning. In this work, we tackle the problem of learning manipulation tasks by demonstration in a virtual environment. Our aim is to develop an incremental, generalizable, and robust method for learning robotic manipulation tasks using a limited number of demonstrations, while assuming minimal information about the objects to be manipulated. The developed method combines imitation learning and reinforcement learning by proposing an incremental approach in which the operator first demonstrates specialized tasks to the robotic system, and subsequently more complex tasks, exploiting the skills learned during the previous phases. The experimental evaluation shows the feasibility and advantage of the proposed method in terms of modularity, low number of demonstrations, and reliability of the trained system.
CEUR ceur-ws.org
In this work, we address the problem of learning robotic manipulation tasks by training the
system through demonstrations provided by a human operator in a virtual environment. In
particular, an incremental approach to training is proposed, allowing the operator to first
demonstrate specialized tasks in simple scenarios and then progressively train more complex
and articulated tasks, leveraging the behaviors already trained in previous stages. The objective
is to provide an incremental, generalizable, and robust method for learning robotic manipulation
tasks using a limited number of demonstrations in a virtual environment, assuming restricted
information about the objects to be manipulated. Another relevant aspect of the proposed
approach relies on the use of virtual reality to provide demonstrations to the robotic system in
a safe and simplified manner, in so avoiding the complexity of interacting with a real robotic
system. The proposed method combines reinforcement learning techniques [
In the proposed method for learning robotic manipulation tasks, we exploit the expert’s demonstrations provided in a virtual environment through an incremental approach. Specifically, we illustrate the approach at work considering two training phases (see Figure 1). In the first phase, a robotic hand is trained to grasp diferent types of objects from a proximal pose, without considering the movement of the robotic arm (proximity grasping). In the second phase, the hand trained during the first phase is exploited to train a more complex task, where the robotic arm is tasked to reach, grasp, and lift a specific object ( grasp-and-lift ). These two training phases are therefore sequential, as the result of the first phase is used for the second.
The environment for this training phase is inspired by the one proposed by [
Demonstrations Controler/Learning
Actions Reward Nextstate
Environment
Trainingand
Testing
equal to gravity acting on the object is applied in four directions: upwards, downwards, to the
right, and to the left. The episode ends with a positive reward if the distance between the hand
and the object never exceeds a certain threshold. Therefore, if the grasp is stable, after the 200
test steps, the object will be at a distance less than the predetermined threshold, and the episode
can be considered successfully completed.
Observations. The agent observes the rotations on the individual rotation axes of the 8
joints of the BarrettHand in reduced space. Additionally, the agent observes a measure of force
applied. The information regarding the object that the agent observes includes the bounding
box (front-top-left and back-bottom-right points, and its rotation), velocity, angular velocity,
and the object type. As hypothesized in [
Actions. In the proposed setting, the agent can rotate every joint of the robotic hand on a single axis. The action space is discrete and there are 3 possible actions for each of the 8 articulations: 0 to keep the rotation unchanged, 1 to increase the rotation, 2 to decrease the rotation. For each joint ∈ {0, … , 7} , the target angle is increased or decreased starting from the current target rotation, which is the rotation that the articulation drive aims to reach applying a force or torque, with a fixed rotation speed. This means that if the agent decides to increase the target rotation of a joint i, the variation of the angle is fixed and the agent cannot choose the amount of rotation. The agent can only decide to change the target angle or leave it at the same value.
Demonstrations. To record demonstrations by directly teleoperating the robotic hand in
simulation, a mapping has been proposed between Oculus hand tracking and the BarrettHand
robotic hand (see Figure 3). In the Unity environment, the operator can directly control the
robotic hand and show the grips for the objects used. Specifically, to record the increments or
decrements of each joint of the robotic hand, the diference between the angle on the rotation
axis of the human hand (tracked by the Oculus headset) and the target angle on the robotic
hand is calculated. The sign of this diference will indicate whether to record an increase, a
decrease, or no variation in the target angle.
Models and Training. A significant aspect of the proposed training method relies on a
specific combination of imitation learning and reinforcement learning techniques. More
precisely, we leverage Proximal Policy Optimization [
In this second stage of learning, our primary objective is to comprehensively train the robotic system’s full range of motion. Specifically, our aim is to facilitate the acquisition of skillful hand movements and positioning strategies in close proximity to the target object, enabling successful grasping. In this context, the robotic hand is attached to the robotic arm, i.e., the Barrett WAM Arm, positioned deliberately in front of a table designated for object placement. For this particular experimental scenario, we have opted to exclusively use cylindrical objects. In this setting, two distinct grasping techniques have been defined, depending on the task: ”top grasp” for grasping the cylinder from the top and ”side grasp” for grasping it from the side The environment designed for this second phase, as illustrated in Figure 4, includes a table on which the cylinder is be placed and the robotic arm positioned in front of it. At the beginning of each episode, the object is placed on the table in a random position within a defined area, and a scale for the cylinder is randomly selected within preset intervals. The agent is granted 1000 steps to position the hand within the designated zone according to the task and activate the grasp Once the grasp is initiated, the agent becomes ”locked,” meaning that no further progress is made in the steps for the robotic arm, no decisions are made, and the outcome of the grasp is awaited. The hand is allotted 800 steps to adjust the finger positions, after which the same 200-step force test, utilized in the first phase, is activated, this time in the following global directions: right, left, forward, backward. Furthermore, the cylinder is subjected to gravitational force from the beginning of the episode and also during the test. In fact, to enhance the test’s robustness, the hand (while holding the cylinder) is moved upward while the forces are applied. The episode concludes with a positive reward if the test succeeds, i.e., if the distance between the hand and the object is less than or equal to _ throughout or with a zero reward if one of the failure conditions is met.
Observations. To move the robotic hand, the agent controls the target position that the endefector must reach by manipulating the arm joints through inverse kinematics. The agent thus observes spatial information related to the end-efector, including position, rotation and target position, wrist rotation (yaw and pitch), hand velocity and hand angular velocity. Furthermore, the agent observes the type of task (top grasp or side grasp) and the same information about the object described in the previous section.
Actions. The displacement of the end-efector, and thus the hand, is achieved through inverse kinematics. The agent, therefore, controls the target position that the hand must reach. Additionally, the agent must rotate the wrist about two axes (yaw and pitch) and activate the grasp to complete tasks. The agent has the following branches of actions at its disposal: the displacement of the IK target along the x, y, and z axes, the pitch and yaw rotations of the wrist, and the activation of the grasp. Also in this case, the action space is discrete: actions for target displacement and yaw and pitch rotations can assume values of 0, 1, and 2 (no change, increase, decrease). The final action, to activate the grasp, can only take on values of 0 and 1, representing ”grasp not activated” and ”activate grasp,” respectively.
Demonstrations. During the demonstration phase, we employ the controller of the Meta Quest 2 headset to manipulate the target position of the end-efector, activate the grasp using the controller, and adjust wrist rotation using the stick. Given that we operate within a discrete action space, during the demonstration, we cannot directly record the controller’s position. Instead, we determine, for each coordinate, whether to increment, decrement, or maintain the value unchanged based on the controller’s motion. Thus, we calculate the direction vector = poscontroller −postarget between the controller’s position and the target position, and the sign of each coordinate dictates whether to increase, decrease, or leave unchanged the corresponding coordinate of the target position. Similarly, for pitch and yaw rotations, the motion is discretized, and actions are recorded by indicating an increment, a decrement, or no change based on the analog stick’s movement. In this case as well, the movement and rotation speeds are fixed. Models and Training. Analogously to the first phase, the algorithm use for training is PPO in combination with BC and GAIL. In this case, we recorded 30 episodes for demonstrations (approximately 9 minutes in total), consisting of 15 episodes for top grasp and 15 episodes for side grasp. Once again, it is worth noting that all of these episodes were successfully completed by the expert. The neural network model does not utilize LSTM, instead, it has a structure with 3 fully-connected layers, each consisting of 512 neurons. A diferent machine was used for the training in the second phase (i5-9300H and GTX 1050 Ti). In this setting, the best training process took approximately 71 hours to complete 50 million steps.
In this section, we discuss the performance of proposed method for each training phase.
To evaluate proximity grasping, two tests were conducted. The first test focuses on the same
objects used during the training phase. These objects, characterized by inherent variability and
primitive shapes, can serve as an approximation of more complex objects or object parts. The
second test is carried out using 3 objects from the dataset proposed in [
As for the first test, we compared the proposed combination of RL, BC, and GAIL with respect to other ways to combine these algorithms (i.e., RL, RL + BC, RL + GAIL, and RL + BC + GAIL with diferent parametrizations) showing that the proposed method (and setting) achieves a significantly higher success rate with respect to the alternatives. In the second test, the agent achieves a satisfactory success rate (about 75.3% ± 1.5) with the novel dataset. In particular, it attains a success rate of 66.5% ± 4 for bottles and 71.5% ± 6.9, including the ”awkward” positions, meaning with the neck of the bottle or the handle of the hammer facing the palm. However, performance improves (90% ± 2.2) when excluding these unnatural grasp positions for this type of object.
For the second phase, in addition to the success rate, the percentage of successful activations of the grasp when it is correctly activated has also been calculated. The ratio is then calculated between the number of episodes completed correctly and the number of grasps activated correctly. This constitutes an additional quality test for the grasp policy obtained in the first part and for the hand placement near the object executed by the trained agent in the second part. Also in this case, the combination of RL+GAIL+BC proves to be the most efective among those tested, achieving an overall success rate of 84.5% ± 1.3 in the test phase. Specifically, it achieves an 81% ± 5.5 success rate for the vertical grasp and 87.3% ± 3.4 for the horizontal grasp. The success rate for correctly activated grasps is at 94.1% ± 0.8. During the experimentation, it was observed that the top grasp is the most complex. Even when executed successfully, the applied grasp may not be entirely ”correct” or natural.
In this work, an incremental and modular method for learning object manipulation through demonstrations in a virtual environment has been presented. Specifically, the proposed approach addresses the problem of grasping and manipulating objects. The focus was initially on the problem of proximity grasping before addressing the task of grasp-and-lift. The system has been designed to allow the operator to interact easily with a simulated robotic system, thereby avoiding the complexity of interacting with a real robotic system. The defined setup enables the recording of demonstrations in a simple manner, using cost-efective hardware. The experimental results illustrate the efectiveness of the proposed approach and how the two algorithms, BC and GAIL, manage to balance the disadvantages of the two imitation learning techniques and make the most of their potential when used together.
The research leading to these results has been partially supported by MELODY, grant P2022XALNS, PRIN 2022 PNRR, European Union - NextGenerationEU; Harmony, grant 101017008, European Union’s Horizon 2020; Inverse, grant 101136067, and euROBIN, grant 101070596, European Union’s Horizon Europe.