This paper details our winning submission to Phase 1 of the 2021 Real Robot Challenge4; a challenge in which a three-fingered robot must carry a cube along specified goal trajectories. To solve Phase 1, we use a pure reinforcement learning approach which requires minimal expert knowledge of the robotic system or of robotic grasping in general. A sparse, goal-based reward is employed in conjunction with Hindsight Experience Replay to teach the control policy to move the cube to the desired x and y coordinates. Simultaneously, a dense distance-based reward is employed to teach the policy to lift the cube to the desired z coordinate. The policy is trained in simulation with domain randomisation before being transferred to the real robot for evaluation. Although performance tends to worsen after this transfer, our best policy can successfully lift the real cube along goal trajectories via an efective pinching grasp. Our approach5 outperforms all other submissions, including those leveraging more traditional robotic control techniques, and is the first pure learningbased method to solve this challenge.
Dexterous robotic manipulation is applicable in various industrial and domestic
settings. However, current state-of-the-art robotic control strategies generally
struggle in unstructured tasks which require high degrees of dexterity.
Datadriven learning methods are promising for these challenging manipulation tasks,
yet related research has been limited by the costly nature of real-robot
experimentation. In light of these issues, the Real Robot Challenge (RRC) [
The 2021 RRC consists of an initial qualifying Pre-Phase performed purely in simulation, followed by independent Phases 1 and 2, both performed on the real robot. Full details can be found in the ‘Protocol’ section of the RRC website4. This paper focuses solely on our approach to Phase 1 of the competition.
In Phase 1, participants are tasked with solving the challenging ‘Move Cube
on Trajectory’ task. In this task, a cube must be carried along a goal trajectory
(which specifies the coordinates at which the cube should be positioned at each
time-step) using the provided TriFinger robotic platform [
‘Move Cube on Trajectory’ requires a dexterous policy that can adapt to
the various goal and cube positions encountered during an evaluation episode.
Last year (2020), the winning solutions to this task consisted of structured
policies which relied heavily on inductive biases and task specific engineering [
Traditional robotic manipulation controllers often rely on solving inverse kinematic
equations [
Most successful approaches in previous editions of the Real Robot Challenge
make use of a combination of motion planning and motion primitives. The
winning team of the 2020 edition of the challenge [
Deep RL methods promise to allow learning of sophisticated, dexterous robotic manipulation strategies that would otherwise be impossible, or at least very dificult, to hand-engineer. However, the data ineficiency of RL is a major barrier to its application in real-world robotics: real robot data collection is time-consuming and expensive. Thus, much RL research to-date has focused on resolving or by-passing these data-eficiency issues.
Due to their generally improved sample complexity, of-policy RL methods
[
Provided with a simulated replica of the real robotic setup, but without access to prior data or expert demonstrations, we use sim-to-real transfer to bypass real-robot RL data-eficiency issues. 3
ments as a Markov decision process (MDP), defined by the tuple (S, A, G, p, r, γ, ρ 0). S, A, and G are the state, action and goal spaces, respectively. The state transition distribution is denoted as p(s′|s, a), the initial state distribution as ρ 0(s), and the reward function as r(s, g). γ ∈ (0, 1) discounts future rewards. The goal of the RL agent is to find the optimal policy π ∗ that maximizes the expected sum of discounted rewards in this MDP: π ∗ = argmaxπ Eπ [Pt∞=0 γ tr(st, gt)].
policy RL algorithm which, in the goal-based RL setting, maintains the following neural networks: a policy (actor) π : S × G → A , and an action-value function (critic) Q : S × G × A → R. The critic is trained to minimise the loss Lc = E(Q(st, gt, at) − yt)2, where yt = rt + γQ (st+1, gt+1, π (st+1, gt+1)). To stabilize the critics training, the targets yt are produced using slowly updated polyakaveraged versions of the main networks. The actor is trained to minimise the loss: La = − EsQ(s, g, π (s, g)), where gradients are computed by backpropagating through the combined critic and actor networks. For these updates, the transition tuples (st, gt, at, rt, st+1, gt+1) are sampled from a replay bufer which stores previously collected experiences (i.e., of-policy data).
Hindsight Experience Replay (HER). HER [
We train our control policy in simulation with RL before transferring it to the
real robot for evaluation. This allows for quicker and easier data collection versus
real robot training. To compensate for modelling errors in the simulator, we
randomise the simulation dynamics [
Actions and Observations. Pure torque control of the robot arms is employed with an action frequency of 20 Hz (i.e. each time-step in the environment is 0.05 seconds). The robot has three arms, with three motorised joints in each arm; thus the action space is 9-dimensional (and continuous). Observations include: (i) robot joint positions, velocities, and torques; (ii) the provided estimate of the cube’s pose (i.e. its estimated position and orientation), along with the diference between the current and previous time-step’s pose; and (iii) the goal coordinates at which the cube should currently be placed (i.e. the active goal of the trajectory). In total, the observation space has 44 dimensions.
Episodes. In each simulated training episode, the robot begins in its default position and the cube is placed in a uniformly random position on the arena floor. Episodes last for 90 time-steps, with the active goal of the randomly sampled goal trajectory changing every 30 time-steps.
Domain Randomisation. To help the learned policy generalize from an inaccurate simulation to the real environment, we used some basic domain randomisation (i.e., physics randomisation) during training6. This includes uniformly sampling, from a specified range, parameters of the simulation physics (e.g. robot mass, restitution, damping, friction; see our code for more details) and cube properties (mass and width) each episode. To account for noisy real-robot actuations and observations, uncorrelated noise is added to actions and observations within simulated episodes. 4.2
The goal-based nature of the ‘Move Cube on Trajectory’ task makes HER a
natural fit; HER has excelled in similar goal-based robotic tasks [
Rewards and HER. In our approach, the agent receives two reward components:
(i) a sparse reward based on the the cube’s x-y coordinates, rxy, and (ii) a dense
reward based on the cube’s z coordinate, rz (the coordinate frame can be seen in
Figure 1 (a)).
6 Our domain randomization implementation is based on the benchmark code from
the 2020 RRC [
The sparse x-y reward is calculated as: rxy = (0
if − 1 otherwise g′xy − gxy ≤ 2cm (1) (2) where g′xy are the x-y coordinates of the achieved goal (the actual x-y coordinates of the cube), and gxy are the x-y coordinates of the desired goal.
The dense z reward is defined as: − a| zcube − zgoal| if zcube < zgoal rz = − a 2 | zcube − zgoal| if zcube > zgoal where zcube and zgoal are the z-coordinates of the cube and goal, respectively, and a is a parameter which weights rz relative to rxy (we use a = 20).
We only apply HER to the x-y coordinates of the goal; i.e., the x-y coordinates of the goal can be altered in hindsight, but the z coordinate remains unchanged. Thus, our HER altered goals are: gˆ = (gx′, gy′, gz), meaning only rxy is recalculated after HER is applied to a transition sampled during policy updates. This reward system is motivated by the following: 1. Using rxy with HER allows the agent to learn to push the cube around in the early stages of training, even if it cannot yet lift the cube to reach the z-coordinate of the goal. As the agent learns to push the cube around in the x-y plane of the arena floor, it can then more easily stumble upon actions which lift it. Importantly, the rxy + HER approach requires no complicated reward engineering. 2. rz aims to explicitly teach the agent to lift the cube by encouraging minimisation of the vertical distance between the cube and the goal. It is less punishing when the cube is above the goal, serving to further encourage lifting behaviours. 3. In the early stages of training, the cube mostly remains on the floor. During these stages, most g′ sampled by HER will be on the floor. Thus, applying HER to rz could often lead to the agent being punished for briefly lifting the cube. Since we only apply HER to the x-y coordinates of the goal, our HER altered goals, gˆ, maintain their original z height. This leaves more room for the agent to be rewarded by rz for any cube lifting it performs.
Goal Trajectories. In each episode, the agent is faced with multiple goals; it must move the cube from one goal to the next along a given trajectory. To ensure the HER process remains meaningful in these multi-goal episodes, we only sample future achieved goals, g′, (to replace g) from the period of time in which g was active.
In our implementation, the agent is unaware that it is dealing with trajectories: when updating the policy with transitions (st, gt, at, rt, st+1, gt+1) we always set (a) Pushing (b) Cradling (c) Pinching gt+1 = gt, even if in reality gt+1 was diferent 8. Thus, the policy focuses solely on achieving the current active goal and is unconcerned by any future changes in the active goal.
Exploration vs Exploitation. We derive our DDPG + HER hyperparameters
from Plappert et al. [
Our method is highly efective in simulation. The algorithm can learn from scratch to proficiently grasp the cube and lift it along goal trajectories. Figure 3 compares the training performance of our final algorithm to that of standard HER 9. Our 8 Interestingly, we found that exposing the agent (during updates) to transitions in which gt+1 ̸= gt hurt performance significantly, perhaps due to the extra uncertainty this introduces to the DDPG action-value estimates. 9 These runs did not use domain randomization. Generally we trained from scratch in standard simulation before fine-tuning in a domain-randomized simulation algorithm converges in roughly 23 the time of standard HER, and is markedly improved in the the early stages of training; this allowed us to iteratively develop our approach more quickly. Throughout diferent training runs, our policies learned several diferent manipulation strategies, the most distinct of which included: (i) ‘pinching ’ the cube with two arm tips and supporting it with the third, and (ii) ‘cradling ’ the cube with all three of its forearms (see Figure 2). Simulation -20,399± 3,799 -6,349± 1,039 -6,198± 1,840 Real robot -22,137 ± 3,671 -14,207 ± 2,160 -11,489 ± 3,790 5.2
Our final policies transferred to the real robot with reasonable success. Table 1 displays the self-reported scores of our best pinching and cradling policies under RRC Phase 1 evaluation conditions. As a baseline comparison, we trained a simple ‘pushing’ policy which ignores the height component of the goal and simply learns to push the cube along the floor to the goal’s x-y coordinates. The pinching policy performed best on the real robot, and is capable of carrying the cube along goal trajectories for extended periods of time, and of recovering the cube when it is dropped. This policy was submitted for the oficial RRC Phase 1 final evaluation round and obtained the winning score (see https:// real-robot-challenge.com/leaderboard, username ‘thriftysnipe’).
The domain gap between simulation and reality was significant, and generally led to inferior scores on the real robot. Policies often struggled to gain control of the real cube which appeared to slide more freely than in simulation. Additionally, on the real robot policies could become stuck with an arm-tip pressing the cube into the wall. As a makeshift solution to this issue, we assumed the policy was stuck whenever the cube had not reached the goal’s x-y coordinates for 50 consecutive steps, then uniformly sampled random actions for 7 steps in an attempt to ‘free’ the policy from its stuck state. 6
Our relatively simple reinforcement learning approach fully solves the ‘Move Cube
on Trajectory’ task in simulation. Moreover, our learned policies can successfully
implement their sophisticated manipulation strategies on the real robot. Unlike
last years benchmark solutions [
Due to the large domain gap, our excellent performances in simulation were
not fully matched upon transfer to the real robot. Indeed, the main limitation
of our approach was the absence of any training on real-robot data. It is likely
that some fine-tuning of the policy on real data would greatly increase its
robustness in the real environment, and developing a technique which could do so
eficiently is one direction for future work. Similarly, the use of domain adaptation
techniques [
This publication has emanated from research supported by Science Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289_P2, co-funded by the European Regional Development Fund, by Science Foundation Ireland Future Research Leaders Award (17/FRL/4832), and by China Scholarship Council (CSC). We thank the Max Planck Institute for Intelligent Systems (Stuttgart, Germany) for organizing the challenge and providing the necessary software and hardware to run our experiments remotely on a real robot. We acknowledge the Research IT HPC Service at University College Dublin for providing computational facilities and support that contributed to the research results reported in this paper.