Training agents in hard exploration, sparse reward environments is a di cult task since the reward feedback is insu cient for meaningful learning. In this work, we propose a new technique, called Directed Curiosity, that is a hybrid of Curiosity-Driven Exploration and distancebased reward shaping. The technique is evaluated in a custom navigation task where an agent tries to learn the shortest path to a distant target, in environments of varying di culty. The technique is compared to agents trained with only a shaped reward signal, a curiosity signal as well as a sparse reward signal. It is shown that directed curiosity is the most successful in hard exploration environments, with the bene ts of the approach being highlighted in environments with numerous obstacles and decision points. The limitations of the shaped reward function are also discussed.
A reinforcement learning agent learns how to behave based on rewards and
punishments it receives through interactions with an environment [
Closely related to the sparse rewards problem is the issue of exploration.
Exploration algorithms aim to reduce the uncertainty of an agents understanding
of its environment [
These types of environments are prevalent in the real-world [
A popular approach to learning in these environments is reward shaping,
which guides the learning process by augmenting the reward signal with
supplemental rewards for intermediate actions that lead to success [
Intrinsic rewards that replace or augment extrinsic rewards is another area
of research that has exhibited promising results [
In this research, we present Directed Curiosity : a new technique that
hybridises reward shaping and Curiosity-Driven Exploration [
Learning in hard exploration, sparse reward environments is a well-studied area
in reinforcement learning. Reward shaping is a popular approach that augments
the reward signal with additional rewards to enable learning in sparse reward
environments. It is a means of introducing prior knowledge to reduce the number
of suboptimal actions [
Potential-Based Reward Shaping has been proven to preserve the optimal
policy of a task [
The restriction on the form of the reward shaping signal limits its
expressiveness [
It is di cult to manually engineer reward functions for each new
environment [
An alternative to \shaping" an extrinsic reward is to supplement it with
intrinsic rewards [
Classic work in [
Approximating these counts in large state spaces is a di cult task [
Other methods of exploration include maximising empowerment [
We propose a new reward function that is made up of two constituents: a distance-based shaped extrinsic reward and a curiosity-based intrinsic reward. 3.1
Shaping rewards is a fragile process since small changes in the reward function
result in signi cant changes to the learned policy [
Various functions were engineered and compared. It is essential that the
positive and negative rewards are balanced. In an episode, the agent should not
receive more positive rewards for moving closer to the target, or more negative
rewards for moving further away, so as not to introduce loopholes for the agent
to exploit. If the weighting of positive rewards is too high, the agent learns to
game the system by delaying reaching the target to gain more positive rewards
in an episode. If the weighting of the negative rewards is too high, the agent
does not receive su cient positive reinforcement to nd the target. This means
that the shaped rewards alter the optimal policy of the original task [
The shaped reward should encourage the agent to keep advancing towards the target by favouring consecutive positive moves and punishing consecutive negative ones. It must not dominate the terminal reward such that the agent is no longer incentivised to nd the target and its motivations become polluted. To overcome these issues, a shaped reward function based on relative distance between target and agent is used.
Input: Agent position Pagent, target position Ptarget, maximum distance Dmax, previous distance Dprev, reward coe cient C Calculate distance Dcurrent distance(Pagent; Ptarget) Calculate reward signal: R Dcurrent=Dmax if Dcurrent < Dprev then
return C (1 R) else
return C ( R) end if
There are various bene ts to Algorithm 1. The agent is penalised if it stays still and the shaped reward signal can be controlled using the reward coe cient C. This ensures that the episodic shaped rewards cannot exceed terminal positive reward. There is a balance between positive and negative rewards since they are both relative to the change in distance. The agent receives the highest reward when it moves closest to the target and the highest penalty when it moves furthest away. This means that the shaped reward function is policy invariant i.e. it does not alter the goal of the agent to learn the optimal path to the target.
Since the rewards are shaped exclusively based on distance metrics that do not take into account the speci c dynamics of the environment, the same function can be used across di erent environments, and in general, for navigation tasks. A limitation of this approach is that the target location needs to be known. We have investigated using ray casts to nd the location of the target if it is unknown, however, the scope of this research is to teach an agent to navigate past obstacles and nd an optimal path, given a starting point and a destination. The de nition of the task changes drastically, from a navigation-based one to a goal- nding or search task, when the location is unknown. This is a possible area for future work. 3.2
Pathak et al. [
rit = k ^(st+1)
(st+1)k22 2
In order to generate a curiosity reward signal, the inverse and forward dynamics models' loss functions are jointly optimised i.e. curiosity is de ned as the di erence between the predicted feature vector of the next state and the real feature vector of the next state. is a scaling factor.
As an agents explores, it learns more about its environment and becomes less curious. A major bene t of this approach is that it is robust: by combining the two models, the reward only captures surprising states that have come about directly as a result of the agents actions. (1)
We propose hybridising curiosity [
Directed Curiosity simultaneously maximises two reward signals. The reward function components are somewhat con icting so it is essential to nd a balance between them. The agent needs su cient time to explore the environment, while also ensuring that it does not converge to a suboptimal policy too quickly. This is similar to the exploration vs exploitation Problem in RL. We balance the reward by manually tuning weights attached to both the constituent reward signals. In future work, we wish to nd a means of dynamically weighting the reward signals during training. We also wish to investigate alternative means of combining them.
Input: Initial policy 0, extrinsic reward weighting we, intrinsic reward weighting wi, max steps T , decision frequency D for i 0 to T do
Run policy i for D timesteps Calculate distance-based shaped reward ret (Algorithm 1) Calculate intrinsic reward rit (Equation 1) Compute total rewards rt = wi rit + we ret
Take policy step from i to i+1, using PPO [
PPO [
A custom testing environment was created to analyse the performance of our technique, based on the principal of path nding. It consists of a ball and a target. The ball is an agent that must learn to navigate to the target, in the shortest possible time (see Fig. 1). The agent is penalised every time it falls o the platform, since there are no walls along the boundaries and it receives a positive reward upon reaching the target. An episode terminates upon falling o the platform, reaching the target, or after a maximum number of steps.
The bene t of this environment is that it de nes a simple base task of nding an optimal path to a target. This allows us to perform thorough analysis of the algorithm by continuously increasing the di culty of the task. In this way, we are able to identify its limitations and strengths. The environment represents a generalisation for navigation tasks wherein an agent only receives positive feedback upon reaching its destination.
The agent is equipped with a set of discrete actions. Action 1 de nes forward and backward movement while action 2 de nes left and right movement. Simultaneously choosing the actions allows the agent to move diagonally. The agent's observations are vectors representing its current position and the target position. It is not given any information about the dynamics of the environment. The agent must learn an optimal policy that nds the shortest path to the target.
The baseline reward function was carefully tuned: a +100 reward is received for nding the target, -100 penalty for falling o the platform and -0.01 penalty every timestep. The reasoning behind the selected values is to remove bias from the experiments. An agent cannot fall into a local optimum by favouring a single suboptimal policy. This is because a policy that immediately falls o the platform and a policy that learns to remain on the platform for the entire episode, without nding the goal, will both return roughly the same episodic reward. This function was used as a baseline that was tuned for each new environment. (a) BasicNav (b) HardNav (c) ObstacleNav (d) MazeNav1
For the simplest version of the task, the agent and target are placed at xed locations, on the opposite sides of the platform, without any obstacles between them. We term this an easy exploration task since it is possible for an agent trained with only the sparse reward function to nd the target. This is achieved by tuning the oor to agent ratio and agent speed. The shaped reward coe cient C in Algorithm 1 was ampli ed to 0:1 due to the simplicity of the environment. This is referred to as BasicNav (see Fig. 1a).
The next environment, termed HardNav (see Fig. 1b), is signi cantly larger.
It is a hard exploration environment [
We also perform testing in environments with walls that block the direct path to the goal and make nding the target more di cult. ObstacleNav (see Fig. 1c) has a single obstacle that is deliberately placed perpendicular to the optimal path to the target, forcing the agent to have to learn to move around the obstacle. The agent is never explicitly given any information about the obstacle. This environment was designed to test the limitations of Directed Curiosity since shaping the reward to minimise the distance to goal is counter-intuitive because it leads the agent directly into the obstacle. The coe cient C in Algorithm 1 was dampened to 0:001.
The remaining set of environments contain multiple walls and obstacles in a maze-like structure. These environments were designed to investigate if the agent can learn to move further away from the target at the current timestep, in order to pass obstacles and reach the target at a later timestep i.e. it needs foresight to succeed. We term the rst maze as MazeNav1 (see Fig. 1d).
The last environment is the most di cult version of the task since it has deadends and multiple possible paths to the goal. This allows us to investigate the robustness of Directed Curiosity. Even after nding the target, it is di cult to generalise a path from the starting point to the destination since it is easy for the agent to get stuck in dead-ends or behind obstacles. We term this environment as MazeNav2 (see Fig. 1e). Due to the increased complexities, the terminal reward was increased to +1000 and the shaped reward coe cient C in Algorithm 1 was dampened to 0:000001. 4.2
It is important to carefully tune the hyperparameters for each environment. The
success of the algorithms hinge on these values. Although literature guided this
process, the hyperparameters were manually optimised, since the experiments
were performed in custom environments. The base hyperparameters were found
in BasicNav and then ne-tuned for all other environments, in order to cater
for the increased complexities. PPO is a robust learning algorithm that did not
require signi cant tuning [
Hyperparameter tuning was essential in ensuring that the algorithms were able to perform meaningful learning. By attempting to tune the parameters to the best possible values, we were able to perform a fair comparison. The notable parameters are a batch size of 32, experience bu er size of 256 and a learning rate of 1:0e 5. The strength of the entropy regularization is 5:0e 3 and the discount factor for both the curiosity and extrinsic reward is 0:99. The extrinsic reward weighting is 1:0 and the curiosity weighting is 0:1. The network has 2 hidden layers with 128 units. The baseline parameters were adjusted for each environment: the maximum training steps is 50000 in BasicNav, 250000 in HardNav, 750000 in ObstacleNav and 1000000 in MazeNav1 and MazeNav2. 5
Each algorithm was run ve times in every environment. 30 parallel instances of the same environment are used for data collection during training. (b) HardNav (c) ObstacleNav (d) MazeNav1
The sparse rewards agent does not perform consistently in BasicNav. The agent is able to nd the target and learn an optimal policy on some runs only. This is the reason for the high variance in Fig. 2a. In the hard exploration environments, the agent learns to avoid falling o the platform but is unable to nd the target on all runs and therefore receives no positive rewards in training. This highlights the need for an exploration strategy.
The reward shaping agent performs well in BasicNav. This is because the shaped rewards act as a de nition of the task since there are no obstacles blocking the direct path to the goal. Continuously moving closer to the target on every timestep leads the agent to the goal in the shortest time. Even in HardNav, the agent is able to learn an optimal policy very quickly, for the same reasons.
The de ciencies of using the shaped reward only are exposed when obstacles are introduced (see Fig. 2c, Fig. 2d). The agent fails to nd the target on all runs in ObstacleNav and MazeNav1 and gets stuck behind obstacles. This is because the shaped reward function is a greedy approach and the agent is not equipped with the foresight to learn to move around the obstacles. It cannot learn to move further away from the target at the current time, in order to reach the target at a later stage.
In MazeNav2 (see Fig. 2e), the agent was able to nd the target on two runs. Even though there are multiple obstacles, the optimal path to the goal in MazeNav2 is similar to that in HardNav. The agent \ignores" the obstacles and avoids dead-ends by acting simplistically. By the end of training, however, the agent was unable to converge to an optimal policy on any of the runs.
These results show that distance-based reward shaping provides the agent with some valuable feedback, but without an intuitive exploration strategy, the agent lacks the foresight needed to moves past obstacles that block it's path to the target.
The curiosity agent was able to consistently learn an optimal policy in the environments without obstacles. However, Fig. 2a shows that the curiosity agent takes longer to converge to an optimal policy in BasicNav. This highlights that curiosity is not necessary in environments that are not hard exploration. In HardNav (see Fig. 2b), the curiosity agent is still able to nd an optimal policy on all runs, but it is signi cantly slower than the shaped reward function.
The necessity of the curiosity signal is highlighted when obstacles are introduced. Not only does it enable the agent to nd the distant target, it also implicitly learns about the dynamics of the environment, allowing the agent to learn how to move past multiple obstacles.
In ObstacleNav (see Fig. 2c), the agent is still able to learn an optimal policy on most runs. The performance of the agent is not as successful in MazeNav1 (see Fig. 2d) and MazeNav2 (see Fig. 2e).The agent successfully learns an optimal policy on two of the runs. In these environments, it is di cult to converge to an optimal policy, once the target has been found. One reason for this is that the agent keeps exploring after initially nding the target and gets stuck behind obstacles and in dead-ends, eventually converging to an unsuccessful policy, without being able to reach the target again. The curiosity signal is insu cient to direct the agent back to the target and learn a path to the destination. This is the reason for the increase of the average reward in the early stages of training and the subsequent drop thereafter in Fig. 2e.
These results indicate that curiosity equips an agent with the ability to nd a target in hard exploration environments with obstacles, but the agent requires additional feedback to consistently learn a path from the start point to the destination.
The Directed Curiosity agent is shown to be the most robust technique. Fig. 2a and Fig. 2b show that Directed Curiosity always nds an optimal policy in BasicNav and HardNav. It converges to a solution faster than the curiosity agent in HardNav, due to the additional shaped reward feedback.
The hard exploration environments highlight the bene ts of the technique. It is the only technique that converges to an optimal solution on all runs in ObstacleNav (see Fig. 2c). Curiosity enables the agent to nd the target and move past the obstacle, while the shaped rewards provide additional feedback that allows the agent to learn an optimal path to the target, once it has been found.
MazeNav1 (see Fig. 2d) and MazeNav2 (see Fig. 2e) exhibit promising results since the Directed Curiosity agent learns an optimal policy on more runs than any other technique i.e. on three of the ve runs. Training is more stable than the Curiosity agent. The agent always nds the target during training, however, it is unable to consistently nd an optimal policy on all runs. A major reason is due to the limitations we have highlighted with the shaped reward function. In future work, we wish to investigate a more intuitive reward function that has foresight. Another reason is due to the complexities we have introduced in these environments. The reward feedback is not su cient to guide the agent out of dead-ends back to the target. However, these results indicate that the two components of Directed Curiosity, when balanced correctly, allow the agent to learn in a more directed and intuitive manner.
For all algorithms, the variance of the results increase with the di culty of the task since the agents do not always converge to an optimal policy i.e. when the agent does not learn a path to the target, it does not receive the terminal reward and hence its episodic rewards are signi cantly lower. PPO learns a stochastic policy, hence, even on the successful runs, the algorithms converge at di erent times. Due to the inherent randomness in the algorithm, the agent explores di erently on every run and thus visits states in a di erent order. 6
A new approach to learning in hard exploration, sparse reward environments,
that maximises a reward signal made up of a hybrid of Curiosity-Driven
Exploration [
The Directed Curiosity agent was the most robust technique. It was able to consistently learn an optimal policy in hard exploration environments with a single obstacle, and learned optimal polices more often then the other techniques, in hard exploration environments with multiple obstacles and dead-ends.
In future work, we wish to investigate alternative reward functions that are more exible than the current greedy approach. We would like to perform further testing in existing benchmarked environments and in domains other than navigation. This requires further research into \intelligent exploration", through hybridising di erent shaped reward signals and exploration strategies. Another interesting direction is to create environments with multiple targets and agents.