State estimation is an essential part of any control system. A perception system estimates a representation of the states using sensors data. Recently, an increasing interest in exploiting machine learning techniques to learn state representation. Representation learning allows estimating states in real-world scenarios. We are interested in learning representation from RGB images extracted from videos. And use the state representation to compute cost/reward suitable for control and reinforcement learning. We propose a method in which the user has to provide just a couple of videos demonstrating the task. Our approach uses a sequential contrastive loss to learn a latent space mapping, and descriptors of the taskrelated objects. Our framework serves robotics control scenarios, especially model-based reinforcement learning algorithms. The resulted representation eliminates the need for engineered reward functions or any explicit access to positioning systems, aiming to improve the applicability of learning to control physical systems. Our framework allows for reducing the learning time and working with low-resource scenarios.
Optimal control[Zho96, Cam13] deals with finding the best possible control action by optimizing a criterion. The criterion is a cost function that is a function to the state and the control. Optimal control is beneficial in the presence of constraints on the control variables or the states. In optimal control, we use the system equation to predict the next state according to the current state and the executed control action, which means a need for a state estimation system, to find the current state.
Reinforcement learning [Sut98] describes the system as a Markov decision process. An agent interacts with an environment by choosing actions that maximize a reward. Actions are produced by a policy, which is a function of the environment state. In the Markov decision process notion, the sensor data defines the observations, which used to estimate the states. Based on the states, we compute the reward function.
In simple cases, the state is proportional to the sensor data. For more complex cases, states computed using the dynamical model ( forward kinematic model for the position of the endefector [Jaz10], odometry for a mobile robot). In compound tasks, external sensors are used (positioning systems[Mer17], RGBD cameras[Sch15]) to get a relative position of an object in the surroundings. Forming representative states in such cases needs tremendous efort and sufers from inaccuracy. This fact leads to exploiting machine learning to generate relevant state representations, to be used in control and reinforcement learning. We will concentrate on the learning representations from visual input to control physical systems.
Feature extraction from images [Nix19] is a well-studied topic in computer vision literature, which aims to extract features that characterize an image. Deep learning allows automatically learning the extraction of low-dimensional features from high-dimensional images. State representation learning [Les18] is a case of feature learning that has additional requirements to encode information about the time and interactions with the environment.
End-to-end learning to control [Lev16] maps observations to actions, by feeding observation to a deep neural network. A convolutional part is essential when the observations are images (learning from pixels), the output of this convolutional part is a low-dimensional vector. This vector representation is not interpretable and can’t be used to compute the cost/reward. Endto-end learning needs an external reward signal. This type of learning sufers from a long training time as the representation is learned from scratch.
Autoencoders [Bal12] and variational autoencoders [Pu16] can be applied to learn lowdimensional representations without supervision. Autoencoders used to aid visuomotor policy learning. Special types of encoders allow controlling the distribution of features in the latent space [Mak15], while others provide information about the spatial information of the states [Fin16]. Autoencoders reduce the training time and give task-agnostic representation. Their problem lies in the dificulties of using the learned features to compute the cost/reward. The usual solution is comparing states with given goal images to compute a sparse reward.
Time-Contrastive Network [Sem18, Sem17, Dwi18] uses a self-supervised learning approach to learn representations entirely from unlabeled videos. TCN makes use of a triplet loss [Soh16] combined with a multi-view metric to ensure that the features disentangled in the latent space following the task progress. The reward /cost after learning is the Euclidean distance between the current state and the target state in the latent space. Building on the TCN, we are proposing a sequential training procedure.
Dense object nets [Flo18] use a pixel contrastive loss [Sch16] to learn descriptors of an image, these descriptors provide information about the objects in the scene. The position of the descriptors can serve as features in visuomotor policy learning [Flo19]. Unlike DON, we won’t use RGBD cameras, we will learn from RGB images. We are proposing a triplet pixel loss to learn descriptors of the task-related object in a self-supervised way.
Contributions. Our primary contribution is (1) a novel formulation for self-supervised representation learning for low-resource scenarios (RGB cameras, single GPU). (2) our model will output an embedding of the states in the latent space, alongside dense descriptor image of the task-related objects. (3) we are proposing a new sequential contrastive loss and a contrastive pixel loss. The framework and the experiments is beneficial for robotics manipulation tasks (similar to pig-in-hole insertion and pick-and-place).
Consider we have an input image ∈ ℝ × ×3. The model will have two outputs; a descriptor image ∈ ℝ × × and an embedding of the image in a latent space ∈ ℝ , , are the width and the height of the image, is the depth of the descriptor, the size of the embedding vector . The base CNN network is taken from the inception model [Sze17] pre-trained on ImageNet [Den09], the parameters of this part is frozen and won’t be re-trained. We add two additional convolutional layers, followed by a spatial softmax layer. Spatial features ( ( )) are the output of this part. A fully connected layer is added to get the embedding of the input image = ( ). The parameter to train embedding model are the parameters of the convolutional layers, and the parameters of the fully connected layer.
Additional descriptor model takes the spatial feature from the embedding model as input. The descriptor model consists of two transposed convolutional layers, followed by a bilinear interpolation operation to upscale the output to the same of the input image. The output of the descriptor model is a descriptor image = ( ), where ( ) is the output of the spatial softmax operation from a pretrained embedding model. a random coloring and random rotation.
The dataset is formed by extracting images from video files and form triplets for learning. In contrastive learning, the process starts by sampling an image from a video, this image is called anchor, a positive range is defined as all the images that far from the anchor by less than a positive margin, the negative margin is the rest of the images. one positive and one negative images are sampled from the positive and negative ranges respectively. The triplet consists of an anchor , a positive image and a negative image .
All images normalized and augmented before feeding them to the network. The normalization is need because the pretrained part trained on normalized images. The augmentation was done by using a color jitter (random coloring) and random rotation. The augmentation is useful for transfer learning, and to avoid over-fitting to the training dataset.
In our method - Sequential contrastive learning, we introduced a scheduled update for the positive margin, and instead of having just one negative margin, we used near and far negative margins. All margins will be updated during the training, alternating between tight positive range, and tight, not-that-far negative range to wider positive range, and wide, for negative range. Our goal is to encourage disentangling near embeddings before separating them into clusters, which gives better distribution in the latent space.
Given a triplet of images; an anchor , positive image , and negative image , the timecontrastive loss is the loss tries to ensure that the anchor and the positive images are closer to each other in the latent space than the negative image. i.e. the aim is to learn an embedding such that:
|| ( ) − ( )||22 + < || ( ) − ( )||22 sentations in accordance with the task progress (time). where is a margin that is enforced between and . I.e. trying to disentangle the repre
In our method - Sequential contrastive learning, we are proposing using a scheduled update of the margin between and . The update of the loss margin will be synchronized with the update of the positive and negative margins of the sampling from the dataset. The goal of this update is to encourage making the distance in the latent space between the clusters of similar images according to the task progress. At the same time, it will ensure disentangling embeddings inside the clusters.
The sequential time contastive loss is defined as: =
Pixel-wise contrastive loss is defined depending on the correspondences between the pixels of a pair of RGB images , ∈ ℝ
× ×3. In Dense Object Networks [Flo18] they used depth information, a pixel ∈ matches a pixel ∈ if they correspond to the same vertex of the dense 3D reconstruction. While this assumption is accurate and gives satisfying results, we claim that the 3D reconstruction works but it is laborious.
In our method, we won’t use depth information, only RGB cameras will be used. Our workaround is to making use of the spatial features learned in the embedding model training process, and define a sequential pixel-wise contrastive loss.
Given two RGB images 1, 2 ∈ ℝ
× ×3 (we will sample a triplet from our dataset, and use the anchor and the negative images as 1, 2 respectively). We will feed each of the images to a trained embedding model, instead of using the output of the fully connected layer, we use the output of the softmax layer a.k.a. spatial features. The results are two spatial features ( 1), ( 2), by feeding each of them to the descriptor model we will get two descriptor ( 1)), 2 = (
( 2)) images 1 = ( image 1 =
To find the pixel-wise contrastive loss, we extracted the indices of the features from each 2[
]||22. The pixel-wise contrastive loss will be defined: Canny edge detector. Pixels in the output descriptor images correspond to the indices will be used to compute the positive distance || 1[ 1] − 2[ 2]||22. Negative points randomly from the second image, will be used to compute the negative distance || 1[ 1] − will be sampled ( 1), 2 =
( 1), the features in our case was the edges detected by the
Where is a scheduled margin, the sequential update of the margin will encourage the descriptor model to diferentiate the close images from far images, which leads to better descriptors for task-related objects.
Firstly we have to define the margins (in the following we present the best margins according to our experiments):
L- the number of frames extracted from each video
pos_margins = [L/4, L/5, L/10, L/20, 2]
neg_margin_near = [L/3, L/4, L/5, L/10, L/20]
neg_margin_far = [L, L/2, L/3, L/4, L/5]
Loss margins
STCL_margins = [12.5, 10, 7.5, 3.5, 1]
SPCL_margins = [
The function update margins() updates the sampling margins of the dataset (pos margins, (STCL and SPCL margins). neg margins near and neg margins far), and the values of variables for each of the losses
Training of the embedding model, starts by sampling a triplet of augmented images from the dataset, feeding them to the model, and use the output to compute the sequential time contrastive loss, the loss then is used to update the parameters of the model. The margins are updated every while.
2 Feed images to the model , ,
1 Sample triplets from the dataset , , ∼
3 Compute the loss 4 Update the parameters 2 Get the spatial features 1, 2 = ( 1), ( 23) Feed the spatial features to the model wise contrastive loss. 1 Sample from the dataset 1,, 2 ∼
1, 2 = ( 4 Compute the loss 1), ( 2) 3. Experiment ing videos from many viewpoints, and if possible add a video of a robotic manipulator moves
pos margins near negative margins far negative margins time contrastive margins pixel-wise contrastive margins randomly to train the descriptor model.
When running the framework, 100 frames will be extracted from each video. Frames will be resized and saved to a list. The dataset class will sample an anchor, positive and negative images (in accordance with the margins), normalize, and augment them when the data loader asks for samples. The hardware used for training is a PC with a single GPU NVIDIA GTX 1050Ti. The usage of memory in our case was less than 1 GB.
The embedding model consists of a pre-trained inception model (until Mixed_5d layer), followed by a convolutional layer with 100 filters, a batch normalization layer, second convolutional layer (kernel size 3, stride 1) with filters with a number equal to the size of the spatial softmax layer, followed by second batch normalization layer. After that, we have a spatial softmax layer, and lastly a fully connected layer with output size equal to the desired embedding size (in our experiment was 32).
The descriptor model consists of a transposed convolution layer (kernel size 5, stride 2), its input has the same number of the channel of the size of the spatial features, the output has 6 channels, followed by another transposed convolutional layer with 3 output channel (to be visualized as an RGB image - we can use other number to learn densely descriptors). Lastly, we use a bilinear interpolation operation to scale up the result to the same size as the input image.
The sequential learning is performed by updating the margins every-while during training. Empirical results led to choosing dataset margins (positive and negative margins) proportional to the number of frames extracted from each video. Margins of the sequential contrastive (a) Latent space before training (b) Reward before training (c) Latent space - no sequential training
(d) Reward - no sequential training (e) Latent space - sequential learning (f) Reward - sequential learning losses chosen proportional to the gap between the positive and the near negative margins of the dataset. Margins of the sequential pixel-wise contrastive loss should be chosen big enough to enforce contrast. Margins associated with the best performance during our experiments listed in Table 1.
The embedding model was trained for 100 epochs (10000 triplets), margins start to be updated after 25 epochs (2500 triplets). and then, all margins of the losses and the database updated every 5 epochs (500 triplets). Alongside the visualization of the latent space (embeddings) in Figure 4, we have plotted a reward function depends on the distance to the target image in latent space:
( ) = −|| ( ) − ( )||22 For each video file, the reward function should be monotonically increasing to zero, smoother function means better performance. To judge the benefit of using sequential learning, we plotted the latent space and the reward function before using the sequential updates. We can notice, the sequential learning led to better intra-class distribution in the latent space, i.e. better rewards.
To validate the trained model, we have tested it on a video from outside the training dataset, to harden the evaluation, we used a diferent USB flash for the new video. Plotting the reward function for the evaluation video1 (Fig. 3) shows a reward function with suitable values. (a) a (b) b
Training the descriptor model took 20 minutes (10000 triplets = 30000 images). In fig.5 we have visualized the descriptor image at the beginning and at the end of the training procedure. the model can distinguish the task-related parts and give them distinguishable descriptors even when the image is augmented, which gives a sign of the robustness of our method.
The visualization of descriptor images is a proof-of-concept. We need additional processing to extract the reward from them, which is out of coverage in this paper.
We have presented a self-supervised representation learning framework for robotics manipulation tasks. The framework makes use of sequential and contrastive learning to master a 1The training and evaluation videos are available on the project page: https://alonso94.github.io/SCL/ better distribution of the images embeddings in the latent space. The distribution corresponds to the task progress. Also, pixel-wise training was used to learn the descriptor representation of the image, the resulted model was able to concentrate on the task-related object even when working with augmented images (color changing and random rotation).
The training time for a task is relatively short (around one hour), and the results could be improved by further learning. We have demonstrated by experiments the ability to learn reasonable results during these time, which gives a good usability feature to the framework.
The framework provides a promising way to run robotics experiments, as the user has to collect some videos (possibly with a smartphone) demonstrating the task. The framework better to be integrated with model-based reinforcement learning, or other optimal control algorithms to provide fully automated experiments.
Open-source code is made available for reproducibility and validation https://alonso94.github.io/SCL/.
The reported study was supported by RFBR, research Project No. 18-29-22027.