We present an improvement in visual object tracking and navigation for mobile robot implementing the advantage actorcritic (A2C) reinforcement learning architecture on top of the Gym-Gazebo framework. This work provides an easier way to integrate reinforcement learning algorithms for navigation and object tracking tasks in robotics field. We train the convolutional-recurrent model employed for the policy estimation in an end-to-end manner. The robot is able to follow a simulated human walking in an indoor environment by using the sequence of images provided by the robot camera. The input of the algorithm is acquired and processed directly in ROS-Gazebo environment. The policy learned by the robot agent proved to generalize well also in an environment with diferent size and shape with respect to the training one. Moreover, the policy allows the robot to avoid obstacles while following the tracking target. Thanks to these improvements, we can straightforwardly apply the tracking system in a real world robot for a person following task in indoor environments.
nition, information retrieval, automation and control. tion of the input frames. It fuses past recurrent states
The tracking procedure finds itself in many applications with current visual features to make predictions of the
like mobile robotics, solar forecasting, particle tracking target object’s movements along the input sequence of
in microscopy images, biological applications, surveil- frames over time. We employed an end-to-end algorithm
lance, to cite the most common ones [
Docker is a software platform that allows one to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. We man3. System set-up and simulation aged to build a complex Docker container, requiring environment to sync many packages and tools that are dependent to each other. For instance, TIAGo robot ROS package The first step in the development of this work consisted (tiago_public_ws) require ROS Melodic version, ROS in the setup of a simulation environment in which the Melodic requires Python v2.7 ; about the RL module, GymTIAGo robot model and a walking human model (ac- Gazebo requires PyTorch machine learning framework
and PyTorch requires Python v3.6. Thus we need to setup packages. The main features of the ROS infrastructure both python2 and python3 for adapting with the sys- are: Nodes. They are the executable processes that partictem. Having an NVIDIA GPU available, we leverage the ipate in the communication. They can be programmed in NVIDIA Container Toolkit that allows users to build and C ++ or Python. For this project, Python has been chosen run GPU-accelerated Docker containers. for its speed and simplicity and the built-in integration GPU-enabled applications need access to both kernel- with the PyTorch deep learning library. Topics. The level device drivers and user-level CUDA libraries, and inter-process communication has a Publish/Subscribe diferent applications may require diferent CUDA ver- model and the communication data are called Topics. sions. One way to solve this problem is to install the They are the communication channels, defined by a speGPU drivers inside the container and map the physical cific name and a single type of message that can be posted NVIDIA GPU device on the underlying Docker host (e.g., to them. Nodes can subscribe to topics to receive the in/dev/nvidia0) to the container. The problem with this formation published in them or publish information for approach is that the version of the driver and libraries other nodes to receive it. Callbacks. They are the ininside the container needs to precisely match. Otherwise, terruption service routines that are generated when a the application will fail. In such a case, users still have node subscribed to a topic detects that something has to worry about what drivers and libraries are installed been published on that channel. In this routine, the data on each host computer to ensure compatibility with con- processing is done, such as saving the position of the tainerized applications. robot in the callback generated by the topic where an NVIDIA Docker, instead, provides driver-agnostic CUDA odometry sensor publishes the readings. Launch. A images. This Docker plug-in enables GPU applications class of files that run and manage multiple nodes for the running in containers to share graphic acceleration de- robot and its sensors, the simulation environment and vices on the Docker host without worrying about version the data visualization software, such as Rviz or Gazebo mismatches between libraries and device drivers. In par- simulator. They are encoded in XML format. ticular, we employed the rocker toolkit [15] that, building URDF (Undefined Robot Description Format) . They upon the nvidia-docker2 package, provides an easy are definition files of a robot. The links of the robot’s way to run the docker container with graphic user inter- kinematic structure are connected to each other by joints. face and GPU acceleration. The structure of the docker They also define the dynamic, kinematic, visual and colcontainer is shown in detail in the diagram of Figure 2 lision properties of the robot. As launch files, they are programmed in XML too.
Gazebo is a simulator specifically designed for robotics. 3.2. ROS - Gazebo Its convenient design makes it possible to quickly test ROS is a collection of libraries, drivers, and tools for the algorithms, robots and AI applications. In Gazebo all the efective development and building of a robot systems. elements present in reality are simulated, the sensors and It has a Linux-like command tool, an inter-process com- actuators act according to the environment. In this work, munication system, and numerous application-related Gazebo has been used for the design of the virtual indoor environment, where all the experiments with the TIAGo robot are performed.
resources well fitting to deep learning applications. 3.3. TIAGo Robot
object tracking algorithm, we consider some robot platforms, such as TIAGo and Turltebot. Diferently from TIAGo, Turltebot provides with APIs, packages and open source ROS plugins. However, we chose TIAGo robot platform that, although it does not provide as many open source APIs as Turltebot, it is a more advanced and modern robot platform and its kinematic structure can better adapt to our target. Since the aim of our work is to make a mobile robot to be able to learn in an end-to-end manner how to actively detect and track a person walking in an indoor space, the height of TIAGo is more suitable because the camera mounted on the head of the robot can catch the full human body of the person to follow, thus providing better input data to the reinforcement learning algorithm. Moreover, the robot is a service robot specifically designed to work in indoor environments, so our application can be easily deployed on the real platform and in the real world.
Following the main components of the TIAGo robot are described (see Figure 3).
• The mobile base uses a diferential drive system and has a maximum speed of 1 m/s. It is designed for indoor operation. At the base is the laser sensor that has a variable sensing distance depending on the model (iron, steel or titanium), between 5.6 meters and 25 meters. To detect what lies behind the robot, there are 3 sonars of 1 meter of detection. • The body is the central part of the TIAGo robot and is made up of the arm and torso. The torso has a prismatic articulation that allows increasing the height of the robot by 35cm. The arm has 7 degrees of freedom, a length of 87 cm in its maximum extension, and a load capacity of 3kg. 3.4. Animated human model: actor Gazebo simulator allows defining animated model (called ’actor’ in Gazebo) which is useful if one wants to have entities following predefined paths in simulation without being afected by the physics engine. They have a 3D visualization that can be seen by RGB cameras, and 3D meshes which can be detected by GPU based depth sensors, so being suitable for computer vision applications. A closed-loop trajectory is defined for each of the considered train test cases. Additional plugin files are used to control animations based on feedback from the environment. Actors extend common models, adding animation capabilities. There are two types of animations that can be used separately or combined together: • Skeleton animation, which is relative motion
between links in one model; • Motion along a trajectory, which carries all of the actor’s links around the world, as one group; PAL Robotics ofers diferent ROS packages and libraries compatibility allowing TIAGo robot to perform complex perception, navigation, manipulation and human-robot interaction tasks. The platform is also equipped with a • The head comprises the neck, having 2 DoF that allow TIAGo to look in any direction. In the place Both types of motions can be combined to achieve a skeleof the eyes, there is an RGB-D camera that pro- ton animation that moves in the world. Gazebo supports vides color and depth images, being able to recre- two diferent skeleton animation file formats: COLLADA ate the environments using point clouds. The (.dae) and Biovision Hierarchy (.bvh). The actor model technology is the same as that used by the pop- defined in the project loads a COLLADA file described ular Kinect cameras. In particular, the robot is within the <skin> tag. Sometimes, it is useful to comequipped with a built-in Asus Xtion camera. bine diferent skins with diferent animations. Gazebo allows one to take the skin from one file, and the animation from another file, as long as they have compatible skeletons. Scripted trajectories represent the high-level animation type of actors, which consists of specifying a series of poses to be reached at specific times. Gazebo takes care of interpolating the motion between them so the movement is fluid. The trajectory is defined inside the .world file containing all the models which are visualized in simulation. Inside the <script> tag one can define some of the parameters for the desired trajectory, such as the start time and the if or not the motion has to be repeated after it ends. For our actor model’s trajectory we set the trajectory to loop forever and start playing as soon as the world is loaded. In particular, the following parameters are available: • loop: it is set to true for the script to be repeated in a loop. For a fluid, continuous motion is worth setting the last waypoint to match the first one; • delay_start: this is the time in seconds to wait before starting the script. If running in a loop, this time will be waited before starting each cycle; • auto_start: this attribute is set to true if the animation should start as soon as the simulation starts playing. It is useful to set this to false if the animation should only start playing only when triggered, for instance, by a plugin.
Finally, the actual trajectory is built by defining a sequence of waypoints: The parameter waypoint is described inside the <trajectory> tag and represents the intermediate targets the model has to reach along the trajectory path. Each waypoint consists of a time and a pose; deep reinforcement learning, and multi-agent learning.
For the visual object tracking task this project deals with, two Actor-Critic methods have been considered: A2C (Advantage Actor Critic) in [17] and A3C (Asynchronous Advantage Actor Critic) in [18]. We decided to deploy the latter since it best fitted the available computational resources and the environmental settings. Thus the environment setup can be outlined as follows: a virtual environment implemented in Gazebo is combined with a reinforcement learning algorithm developed in ROS framework, encapsulating the agent description and reward updating in Gym-Gazebo package. Finally, a reimplementation of A2C algorithm using PyTorch library is used to define the policy learning method and manage the RL environment. • time the time in seconds, counted from the be- 4.1. Gym gazebo ginning of the script, when the pose should be reached;
ous applications in robotic fields, such as planning, control, air-based, under-water, land-based, etc. Moreover, a diferent state of the art RL techniques ad algorithms has been employed in this field, including actor-critic,
learning of robotics applications that relies on ROS • pose: the pose which should be reached. and Gazebo. This package follows the same baseline structure characterizing OpenAI Gym, and builds a Once trajectories are created and static animations are ROS/Gazebo environment on top of that. Indeed, in the loaded, the final step is to combine them in full synchro- past years, non-profit AI research companies, such as nized animation and trajectory. Our skeleton animation OpenAI, have created a generic set of algorithm and contains a transnational component in the − , environment interfaces. In OpenAI’s Gym, agent-state as we could notice by running the animation without combinations encapsulate information environments, any trajectory. But that animation is not yet synchro- which will be able to make use of all the available nized with our trajectory until we enable that by set- algorithms and tools. This abstraction allows easier ting <interpolate_x> to true inside <animation> implementation and tune of the RL algorithms, but most tag. Figure 4 shows how an actor is modeled and an- importantly, it creates the possibility of using any kind imated in Gazebo. of virtual agent. This includes robotics, which Gym is already supporting with several environments on their 4. Reinforcement learning for roster. Gym-Gazebo toolkit aims to integrate the Gym APIs with robotic hardware, validating reinforcement robotics learning algorithms in real environments. Real-world operation is achieved by combining Gazebo simulator with ROS, so providing a set of libraries and tools that help software developers to create robotic applications.
The architecture consists of three main software blocks:
OpenAI Gym, ROS and Gazebo. Environments developed
in OpenAI Gym interact with the Robot Operating System, which is the connection between the Gym itself and Gazebo simulator. Gazebo provides a robust physics engine, high-quality graphics, and convenient programmatic and graphical interfaces.
tasks and encapsulated it into Gym-gazebo package.
There are two main observations to do for this step: ifrst, we needed to correctly build the environment class following OpenAI Gym format; second, we had to fully integrate our own environment file inside the Gym-Gazebo toolkit, by inheriting the properties of python language. A common interface of OpenAI Gym environment provides three main methods to be implemented: reset, step, render. The function reset is used to re-initialize parameters for new episodes; the step function is called during training, and the render function is used to run Gazebo server. In Figure 5 it shows the architecture of the Gym-Gazebo toolkit and how it is extended for our implementation. 4.2. Reinforcement learning Reinforcement Learning (RL) is an area of machine learning where a software agent learns by interacting with an environment, observing the results of these interactions with the aim of achieving the maximum possible cumulative reward. This imitates the trial-and-error method used by humans to learn, which consists of taking actions and receiving positive or negative feedback. In general, agent learning is based on a policy function that maps states to an action at each time step. The agent learns how to maximize the total rewards returned from the environment. State space - For the considered object tracking and human following task, the state consists in a sequence of RGB images, acquired from an action camera mounted on the robot head, and collected over time. The input sequence of images includes 15 camera frames sampled from a list collected in 0.5 seconds. Images are pre-processed, reshaped to size 128x128x3 and normalized before to be provided as input to the model.
Action space - The agent action space is discrete and consists in three control commands: go forward, rotate left and rotate right, as described in more detail in the table Table 1, while the human walking speed along the trajectory is set to 0.55 m/s. Network model - The neural architecture defined for the implemented RL algorithm is shown in Figure 1. We experimented a more complex and deep neural model with respect to the network defined in [ 19], among the improvements proposed, the observation encoder contains more convolutional layer and also the size of features maps and kernels are changed. Moreover, the actor-critic framework is modified to be more suitable for our task.
The detailed description of network architecture is provided in Table 2.
The observation encoder covers layers 1 to 4, the sequence encoder consists in the recurrent layer 5 and the actorcritic network is represented by the two fully-connected layers 6. The observation encoder extracts the features map of input images acquired from the environment; in- obtained reward is maximized. If the human moves away put features are collected over time by an LSTM recurrent and changes it orientation w.r.t. the robot, the reward layer. reduces, so the robot has to learn how to move such that the human is always in front of the robot and at the right 4.3. Advantage actor-critic architecture distance (since also in case the human is too close to the robot, the reward value gets worse).
(A2C) From the reward function in Equation 1 it is computed In the developed RL-A2C learning framework, the obser- the discount reward :∞ = + * +1 + 2 * +2 +..., vation encoder () extracts from the raw images a fea- that is used to generate the value function for the actorture vector which is then used as input for a sequence critic learning algorithm. encoder. The latter is implemented by a LSTM layer as a Actor-Critic Networks - In A2C, the advantage funcfunction = (1, 2, ...) of the observation history tion (, ) = (, ) − () is approximated to a (1, 2, ...). After the new feature representation linear function by neural network. The action value funcis extracted from the observation, it is provided as in- tion (, ) = [:∞| = , = , ] stands for put to both the actor and critic networks. The target of the expected future reward of choosing an action at a parLSTM layer is to organize the process state over time, in ticular state, and the value function () = [:∞| = this case the information of human position in the image , ] expresses the value of being in a specific state. The frame acquired by the robot camera sensor. Each time critic network approximates the value function () , the human moves, the agent needs to collect information which provides the expected future reward. The actor to choose a suitable action. Processing the input state network outputs the policy distribution (· ; ) used for as a sequence of images provides much information and action decision. In A2C, () and (· ; ) are used to opspeeds up the network update. In this project, the encoder timize the network weights during the training process. function is re-implemented in the form of the function Thus the value and policy functions are evaluated on ℎ() = (ℎ( − 1), ) where ℎ() is the hidden state the updated model parameters (, ′) and (|; ′). of the recurrent network at time step . Reward func- Then, the networks can learn by stochastic policy gradition - The reward function used to evaluate the outcome ent of the actions chosen by the robot during the learning episodes is based on the transform relationship between the robot and human coordinates reference frames (RF).
In particular the following parameters are considered: ← + ∇ log (|; )(, )+ ∇ ( (|; )) (2) • is the relative angle orientation of the human w.r.t. the robot (the angle between the x-axis of the human RF and the x-axis of the robot RF). • is the desired distance between the center of the human RF and the center of the robot RF).
(3) Where :+− 1 is the discounted reward for steps; is the discount factor; is an entropy regularization factor; is the networks regularization factor; ′− is the model parameters set in the previous time step.
= − * ( − √︁2ℎ + ℎ2)2 − * | | (1) Where is the maximal reward; is a scale parameter for the human position w.r.t robot frame; is a scale parameter for the relative orientation of the human w.r.t robot; ℎ and ℎ are the position coordinates of the human in the robot RF. The reward shows that if the human is in front of the robot (so fully captured by the camera sensor) and at most two meters far from the robot, the
ing pre-trained model weights. In the training process, the human moving is set up to follow the trajectory showed in figure Figure 6. Two training runs consisting of 1000 and 2000 epochs respectively are performed.
The Adam optimizer for the training process is used to speed up and stabilize convergence to the global minimum in the gradient descent phase. For further details about hyper-parameters refer to Table 3. 5.1. Training results During the training process, the accumulated reward per episode is collected. Figure 7 shows the trend of total reward increasing during the training epochs. This demonstrates that A2C framework learned well how to select robot actions to accomplish the considered visual object tracking task. At the end of the training process, the robot can follow the human along the whole trajectory.
One of the main improvements in the training process 6. Testing process achieved in this project is the number of episodes required to get the cumulative reward converging to high In this section it is described the testing framework used values, that is smaller with respect to the number of to get quantitative and qualitative evaluations of the trainepochs required to the training of RL algorithms for anal- ing process. The test process is articulated as follows: ogous tasks. This is mainly due to the use of the robot model, instead of the camera view, in the RL process, that • Three testing trajectories are defined to analyze is more suitable for robotics applications, and to have set and evaluate the performances of the trained the empty room to make the learning process converging model to track the human target in the same inquickly. But our training process also has disadvantage door environment used for training. that is the time of training. There are two main reasons for that: in Gazebo simulator is not easy to change the time speed as in game engines (frequently used as simulation environments for RL applications) and the low computational resources available. • A new 3D simulation-world is created to test the tracking performances of the trained model when the robot has to move in an indoor environment showing structural and textural diferences with respect to the training one. • Diferent values of robot speed, trajectory (path and duration) for single action steps, reward function parameters are also evaluated during test process. 6.1. Trajectories settings The trajectories produced for the test process are designed to show an increasing level of diversity with respect to the trajectory defined for the training process, which in turn also determines a gradual increase in the complexity of the active tracking task to be learned by the agent. Below a schematic view of the designed trajectories and an overview of the main features are presented: Base squared trajectory - A first evaluation of the experimental results is performed on the same trajectory used in the training process, aiming to validate the learning results achieved in the training procedure. In this case, the diference with respect to the training trajectory is determined by changes made in the set-up of agent parameters (speed, action steps) and environment parameters (speed of human targets, obstacles, light conditions).
See Figure 8(a).
Squared rotated and reshaped trajectory A second level of evaluation is executed on a trajectory which start showing changes in the shape and in the orientation of the path with respect to the environment coordinate frame. Also in this case a diferent set-up of robot and world parameters is exploited to prove the robustness of the model learning state. In particular, setting diferent human target speed and robot action steps showed an improvement in evaluation results. Refer to Figure 8(b).
Short and trapezoidal trajectory A further step in the evaluation process is performed on a new trajectory which shows no correlation with the training one. The starting and final position of the agent is changed, the shape and orientation of the path in the world frame are diferent.
The main features of the trajectory set-up are the length of the path that is less than the previous two test cases also due to a very acute angle of the first path curve that enhance the complexity of the visual tracking task.
Suitable settings of the agent action steps and of the human speed produced good results for this test case. See Figure 8(c) Long eight-shaped trajectory Finally, the task complexity is further increased defining a test case on a new long trajectory characterized by: diferent start and end position of the agent, complex shape with more curvatures with acute and wider angles, orientation and shaped changed with respect to the trajectory used in all the previous test cases. Also in this case it has been possible to define a suitable set-up for the agent and environment parameters allowing the model to successfully track most of the human target motion along the whole trajectory. Refer to Figure 8(d)
51–63 (a) Base squared trajectory (b) Rotated & reshaped squared trajec
tory (c) New short & trapezoidal trajectory (d) New long & height-shaped trajec
tory 6.2. New test 3D simulation world model To better analyse and validate experimental results, a new simulation indoor world environment is created. The new space is slightly wider in terms of squared meters area, with respect to the world model used for training and for the test study cases described in paragraph 6.1. Moreover, it has a diferent shapes, with a more irregular intersection of the perimeter walls and also the presence of some obstacles which can considerably afect the noisy of the input images processed by the neural model and obstruct the robot motion. Due to the strong diferences in the world setting and also the environmental simplifications adopted in training process to speed up learning convergence, the experimental results achieved in this test case are not as concrete as for the results obtained using the original world simulation set-up, but they allow to show anyway some interesting generalization properties acquired by the neural network model in the policy learning process. The new experimental environment and the related test trajectory are shown in Figure 9 • Failure Rate: number of fails (end-of-episode) over the total number of episodes. This measure is analogous to the homonyms metric used for evaluation in the conventional tracking literature. 6.4. Experimental Results • Episode Length: measures the duration of good tracking, which shares the same spirit as the Successfully Tracked Frames 2 used in conventional tracking applications.
1In single-target short-term tracking Precision is defined as the average overlap, i.e., the average value in the sequence overlap between predicted target’s region form the tracker and the groundtruth region.
2This measure reports the number of successfully tracked frames from the tracker’s initialization to its (first) failure. The choice of the failure criterion may impact the evaluation result.
(a) Cumulative-per-episode rewards (orange) & mean cumulative reward over the whole test run (blue) (b) Average episodes lengths over the whole test run (b) Average episodes lengths
over the whole test run (a) Cumulative-per-episode rewards (orange) & mean cumulative reward over the whole test run (blue) and follow the human motion for more than a half of the total trajectory while also avoiding the obstacles displaced in the traversed regions of the room (Figure 14) The strong performance achieved in tracking and obstacle avoidance task are further proved by the relatively low failure rate of the test runs on the new environment, as can be observed by comparing the values in Table 5 and Table 6 (a) Cumulative-per-episode rewards (orange) & mean cumulative reward over the whole test run (blue) (b) Average episodes lengths
over the whole test run Irregular-shaped with obstacles room environment Test environment Total episodes Failure rate In the end, a further evaluation case is defined, to test the Irregular-shaped generalization capabilities of the trained model and the with obstacles 100 0.59 possibility to extend it also to a new environment with a higher degree of complexity (characterized by an irregular perimeter shape and the presence of obstacles). To let the model process in a good way the input (noisier 7. Conclusions due to the obstacles appearing in the camera frames and the larger field of view required in the considered wider In this project, we provided an improvement in visual space, we run 100 test episodes, during which both the object tracking problem for robotics application. Our mean length of the test run and the cumulative reward aim was to accomplish the task of person following for a gradually increase, reaching peak values comparable with mobile manipulator in an indoor environment for healththe ones obtained for the much simpler test cases con- care purposes. We modeled the learning task as a Markov sidered in 6.1. In particular the robot was able to track Decision Process and exploited the use of robot APIs, of