This paper presents an Imitation Learning (IL) based visual navigation system, which could guide the robots navigating from some start position to a goal location without any explicit map. We pay close attention to the safety issue due to partially-observability and data distribution mismatching-when the robot meets some incomplete or unfamiliar states, it probably performs an unsafe action, making it hard to work on lifelong robot navigation. In this paper, a sequenceto-sequence (Seq2seq) deep neural network is built to enhance the agent's context-awareness in partially-observable conditions and boost the model's adaptability to unseen scenarios. Additionally, we propose Uncertainty-Aware Imitation Learning (UAIL) by explicitly estimating model uncertainty and actively request experts for labeling samples according to the uncertainty with On-Policy IL. Simulations demonstrated that the combined method-Safety-Aware Imitation Learning (SAIL) in goal-driven visual navigation achieves 35.6% shorter expected moving steps and 22% fewer collisions compared with current counterparts. With the learned safer policy, SAIL had be successfully adapted to unseen environments with minimal navigation performance loss.
Considering a task of navigating from a current location to find a specific goal. Classical geometry-based methods,
such as Simultaneous Localization and Mapping (SLAM) [
Alternatively, Imitation Learning (IL) [
This paper presents Safety-Aware Imitation Learning (SAIL) framework by addressing both partially observability and data distribution mismatching in IL. Firstly, to enhance the context-awareness in partially observable environments and facilitate the adaptability to unseen scenarios and goals, we build a sequence-to-sequence deep neural network with Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). In this network, both spatial relevance and temporal relevance are taken into consideration, which could significantly enhance the agent’s context-awareness and improve the model’s generation performance toward unseen scenarios and goals. Secondly, UAIL is proposed using Bayesian approximation with MC-Dropout [42]. For those potentially uncertain or unsafe actions that occurred in some unseen or unfamiliar scenarios, rather than to perform it anyway, UAIL request for expert’s advising (similar to active learning) for whether to act or label it. We predict the model uncertainty with MC-Dropout, a Bayesian approximation for uncertainty estimation in deep learning. This uncertainty has been used to improve the safety and efficiency of On-Policy Imitation Learning such as DAgger. Extensive experiments in the simulator have been conducted to compare the combined SAIL method with the vanilla IL approach, SAIL shows better navigation performance (35.6% fewer expected moving steps) and safer policy (22% lower collision rates). Additionally, with the learned safer policy, SAIL had shown successfully adapted to unseen scenarios and goals with minimal navigation performance loss. The main contributions of our work are listed as followings:
(1) We built a Seq2seq deep neural network to enhance the agent’s context-awareness in partially-observable scenarios and facilitate the model’s adaptability to unseen scenarios and goals.
(2) We presented Uncertainty-Aware Imitation Learning (UAIL) approach by estimating model uncertainty with MC-Dropout and combining it with On-policy IL method, which significantly improves the safety of IL.
(3) We proposed SAIL by combining UAIL and the Seq2seq network and done extensive experiments and evaluations, including navigation performance, safety, and the adaptability to unseen environments and goals.
This work is relevant to past literature in the domains of imitation learning, learning-based navigation and uncertainty estimation. In this section, these areas’ related work is reviewed respectively. 2.1
One of the commonly used solutions for IL is Behavioral Cloning (BC) [
Learning-based techniques, especially the deep learning, approach the visual navigation problem in an end-to-end
fashion. There are mainly two types of learning based visual navigation methods: 1) RL based methods [
There are two types of uncertainty in deep learning: 1) Aleatoric uncertainty, or statistical uncertainty, is resulted from the intrinsic data distribution themselves [46,47]. There are some approaches [48] had been used to model aleatoric uncertainty, but it cannot be alleviated by simply adding more samples or prior knowledge to the deep learning system. 2) Epistemic uncertainty, as known as systematic uncertainty, come from the model parameters themselves due to limited it theoretically could be eliminated by giving additional training samples or offering more prior knowledge. [50] provide a deep analysis of estimating uncertainty in deep neural networks. One of the most popularly used frameworks for estimating epistemic uncertainty in deep learning is Bayesian approximation. Such as MC-dropout [46, 49] and Bayesian ensemble [51]. In this work, we applied MC- Dropout to estimate the model uncertainty due to its plug-and- play advantages in existing deep network architectures. The only thing we need to do is to pass the same input to the deep network multiple times with random dropout rate and compute the entropy or variance of the outputs, which could be used to represent the uncertainty. 3
In this section, the SAIL framework for context-adaptive navigation is described in detail. Firstly, we introduce context-aware deep neural network design for goal-driven navigation, then the safety aware imitation learning using uncertainty estimation with MC-Dropout is presented. 3.1
For network design, as shown in Fig.2, we consider the spatial relevance between the observations and goal image.
In each time step, both of the current observation and goal image are fed into the network. The intuition of doing
this is inspired by [
CNN Feature Extractor: To extract the image features of observations and targets respectively. we use
two separate weights-shared CNN streams—ResNet-50 [
Embedding and FC Layer: this layer is used to fuse the observation feature and goal image feature to a 1024- dimension joint representation and then projected it to a new 512-dimension vector on fully-connected fusion layers. We use two separate fully-connected layers to learn the joint feature representation of observation and goal image. We can predict an optimal action at∧ with Pθ(at|ht). where where Given the training set D = (si, ai) The goal is to maximize the log-likelihood of the output action sequences.
at = πθ(st, g) st+1 = env(st, at, g) ht =
s0 fθ(ht1 , st) t = 0 t >= 1 at∧ = arg max Pθ(at|ht)
a
Pθ(at|ht) = sof tmax(ht) θ∗ = arg max log θ
X (si,ai)∈DPθ(ai|si) log Pθ(a|s) =
X log Pθ(at|st) t (1) (2) (3) (4) (5) (6) (7) (8) RNN Context Modeling: To learn the temporal relevance of observation sequences. RNN Layers is added right after the embedding and FC layer. Gradient Vanishing is a key problem of RNN with long-term dependencies. Although LSTMs [38] are more prevalent in addressing this problem in past literature, we make use of GRUs [37] that have smaller number of parameters, simpler to use and are generally faster to train than LSTMs.
A sequence-to-sequence IL model is built as Fig 2. At the time step t = 0, an initial goal g and a start state s0 are fed into the network, then in each time step t > 0, the agent takes an action at with its current policy πθ.
Then the environment changes its state to a new state st+1 according to the transition dynamic env.
The current hidden vector ht is a function fθ on current observation st, current goal g and the previous hidden vector ht−1, θ is the parameter of function fθ.
We minimize the cross-entropy loss between predicted action and corrective action made by experts with Adam optimization algorithm.
L(θ) = X
X[πE (a|s, g) log(πθ(a|s, g))] τinD a 3.2
Considering the safety issues of off-policy IL due to the data distribution mismatching. UAIL was proposed
by considering model uncertainty and combining it with vanilla on-policy training methods—Data Aggregation
(DAgger) [
Uncertainty Estimation with MC-Dropout: To bridge the gap between uncertainty estimation and deep neural network, we use MC-Dropout, a Bayesian approximation of uncertainty estimation in deep learning to represent the confidence of action to be executed. Basically, it has been demonstrated that using Dropout at inference time in the deep neural network is equivalent to doing Bayesian approximation in Bayesian deep learning. The key idea here is letting dropout doing the same thing in both training and testing time. At test time, we will repeat beta times in passing the same input to the network with a random dropout value. MCDropout provides an efficient way to estimate uncertainty with minimal changes in most existing deep networks. It provides a plug-and-play module to deep learning for uncertainty estimation.
Safety-Aware Imitation Learning: To improve the training efficiency of the on-policy training approach, we intend to reduce the number of queries and aggregation times for data aggregation. We propose an uncertaintyaware imitation learning method (similar to active learning). It initializes model’s weights on initial dataset with supervised learning, then executes the current policy until the learner’s confidence falls below a solid threshold, at which point it queries the expert for corrective action. Uncertainty based approach may decide to stop asking for queries once the confidence exceeds the threshold in all states. which make use of the complementary advantages of humans and robots. Algorithm 1 summarizes the training procedure of Safe IL with uncertainty estimation.
4.1
In this section, we evaluate the SAIL approach in the simulation environment—AI2-THOR1 [
In this experiment, the authors use 4 different scenes to train our models (bathrooms, bedrooms, kitchens, and living rooms). For expert’s policy generation, we use the A* search algorithm to find the shortest path from the start location to the target location while avoiding collisions simultaneously. In order to narrow the situation gap between different room types, we trained our model on each scene separately. In each scene, the authors set 10 remarkable goal locations, such as the laptops, chairs and table lamps. For each goal, this work runs 50 stochastic starting points to evaluate the performance. To validate the proposed approach, the authors compare and evaluate the following 4 baselines and 2 proposed methods.
(1) BC (off-policy): an off-policy IL algorithm that learns a mapping from states to actions directly by supervised learning without any demonstrations sampling.
(2) DART (off-policy): an improved BC method that randomly injects noise into the expert’s trajectory. (3) DAgger (on-policy): an on-policy training approach by iteratively querying experts for the new correct sample and aggregating the new dataset for training.
(4) SaferDAgger (on-policy): an extension of vanilla DAgger method that minimizes the number of queries to a reference policy both during training and testing stage. 4.3
The ultimate goal of goal-driven navigation is to find a given target with minimum steps while avoiding collisions simultaneously. Although these two facts can be affected by each other, having fewer moving steps is not equal to having fewer collisions in real scenarios. Therefore, we evaluate SAIL’s performance from several different metrics.
Expected Trajectory Steps. We evaluate the navigation performance by expected trajectory steps, the expected trajectory steps are defined as the total number of steps taken to navigate from an initial start position to a given target location. The goal of this task is to minimize the expected trajectory steps, the closer the expected navigation steps to the shortest path is, the better the navigation’s performance is. For four different scenes, the authors compare the training results of 4 different scenes with shortest path steps, as shown in Table 1. It can be seen that SAIL with MC-Dropout achieve shorter expected moving steps than vanilla off-policy and on-policy IL approaches.
Navigation Safety Evaluation. In order to evaluate the robot’s safety awareness in goal-directed navigation, the expected collision rate is defined as the mean percentages of robot’s collisions with obstacles when navigating from different start locations to different target positions. In this work, the agent is allowed to collide with the environment, when the agent collides with the environment, we recognize it as one navigation step. As shown in Table 2. SAIL with MC-Dropout can significantly reduce the collision rate of goal-driven navigation.
Navigation Success Rate in Unseen Scenes. Another metric to evaluate navigation performance is the navigation success rate in unseen scenes. Since there are 120 different scenes in each types of environments, we set 100 scenes as training scenes and 20 scenes as testing scenes. Experiment result (see Table 3) show that the performance of SAIL in unseen scenes was farther markedly improved than the baseline IL model. This can be explained by the safer policy learned by SAIL and improved perception ability due to context-aware neural network design.
This work proposes the SAIL approach for goal-driven context-adaptive visual navigation. This approach addresses the robot’s safety issues due to data distribution mismatch and poor performance on partially-observable visual navigation. To alleviate the safety issue, we model the policy uncertainty in deep learning with MCDropout and combine it with the on-policy IL approach. To enhance the agent’s perception capability and context awareness in partially-observable environments, we build a sequence-to-sequence deep neural network with both spatial and temporal relevance taken into consideration. Experiments on simulator demonstrated that the proposed SAIL method had better navigation performance on several evaluation metrics, compared with the state-of-the-art IL methods. With the safer policy, SAIL had shown successfully be adapted to unseen scenarios and goals with fewer collisions and minimal navigation performance loss.
We are thankful to Ai Shi who moderated this paper and, in that line improved the manuscript significantly. [38] Sak H, Senior A, Beaufays F. Long short-term memory recurrent neural network architectures for large scale acoustic modeling[C]//Fifteenth annual conference of the international speech communication association. 2014.