Deep learning has introduced a lot of successful approaches in a lot of supervised learning areas including computer vision. Modern neural network-based models have proved a human-level performance accuracy. The one limitation that has come along with deep learning is - the requirement of large-scale datasets to train such models. Despite the large-scale datasets like ImageNet or OpenImages, a huge amount of classes are left uncovered. In order to extend the existing model to one more class a lot of data collection and annotation is required. Few-shot learning approaches tackle the issue of large-scale dataset requirements. Most of the few-shot learning approaches tackle the problem of identity recognition (like face recognition etc). Novel object classification remains a challenging task. In this work, we built metric learning based deep learning model based on triplet loss. We explore how the triplet loss-driven model may be applicable to image recognition in a case of a lack of data. Our experiments show that such kind of model leads up to 83% accuracy using only a few samples per class.
Deep learning models recently have achieved great success in various computer vision tasks
like image classification, segmentation, object detection, etc. A lot of progress has been achieved due
to increasing model capacity - researchers came up with models like ResNeXt[
The approach of synthetic data generating may be used to generate synthetic data using
generative adversarial networks[
Transfer learning is a widely used approach to fine-tune existing pretrained models on a small
dataset. In computer vision, it was usually done by cutting off head leyers and replacing them with
newly created ones. It is proven that fine-tuning only the last layers gives a significant improvement
by transferring knowledge of the base model to the tuned model[
Described research trend is extensive development rather than intensive in terms of model flexibility. It is impossible to extend the classification model class without ● Model retraining ● Extending a large-scale dataset
Moreover, even satisfaction of the requirements described above does not guarantee a successful model retraining. It is worth mentioning that a lot of business cases of deep learning model applications do not accept manual work like data annotation. Others do not support handling computation resources on their own. A typical example of such a business case - is intelligent cashier-less checkout in the supermarket.
Few-shot learning is a novel approach to deep learning. The main idea of this concept - is to replace classical supervised learning tasks. Instead of forcing the model to generalize the training dataset (or memorize in case of overfitting), we would train the model's ability to learn. It is natural that a baby is able to recognize a car by seeing only several cars in its life. It is done by memorizing several samples of cars and then just performing an intelligent comparison of new objects with existing ones. This is exactly what metric learning does.
A lot of few-shot research works are focused on individual recognition rather than different object recognition (e.g. face recognition is the main focus of few-shot learning since all faces share the same shape but differ in details). In this paper, we move the focus of few-shot object classification from individual recognition to different object recognition. In this work, we explore the capacity of few-shot classification approaches in terms of the replacement of traditional deep learning classification approaches. In summary the contribution of this paper: 1. Build a few-shot 1000-class classification deep learning pipeline 2. Explore strengths and weaknesses of our model on routine object classification 3. Compare results with traditional approaches of image classification
Traditional classification approaches use softmax activation along with cross-entropy loss. That means that each model output of i-th neuron is responsible for the probability of i-th class. Obviously, such models have a constant number of classes that they deal with.
Metric learning suggests as to take another approach - instead of directly predicting the class
of input pictures we calculate the distance between picture pairs. In other words, we build a novel
metric function that is able to calculate the distance in picture space. Such a function should satisfy
the following requirements:
1. Distance between pictures that belong to the same class should be minimal
2. Distance between different pictures should be as high as possible
3. Distance between the same pictures should be zero
Obviously, comparing pictures pixel-wise leads to poor results. Usually, such metrics are constructed
by the deep learning model backbone (feature extractor) and some predefined metric (such as
L2-distance or cosine distance). In that setup backbone is a trainable part, which learns the
embeddings of pictures.
The simple loss that forces the model to learn such embeddings is a contrast loss[
Further study showed that pairwise loss is not the best option. Triplet loss performs much
better in a lot of cases[
However, selection of random triples is not the best option. We will show that it leads to
optimisation the model to a local minimum in the next sections. In work[
Centre loss[
There are a lot of techniques of few-shot learning. One of the most popular - using support set. Let’s define an N-way-K-Shot classification problem. In that setup, the support set contains samples of N-classes. So, during forward pass such models could classify only 1 class of N. There are a lot of papers and benchmarks tackling that problem.
It is natural that an increasing number of classes N will decrease the overall accuracy of our
model. So, that approach leads to pure results while working with a huge amount of classes (like
MiniImage[
In this work, we are going to build a model that will classify images along with 1000 classes where only 10 images per class are available. Since we have a limited amount of data per class, traditional classification will lead to poor results.
Our work is mainly based on an FSS-1000 dataset[
FSS-1000 provides us with instance segmentation masks. Since this work is focused on a classification task we crop each image by its bounding box mask. We calculate the bounding box mask as a minimum and maximum coordinate of the mask both for X and Y axes. Each cropped sample was resized to the standard resolution of 224x224 in order to match the expected resolution of pretrained backbone models.
Resizing of cropped samples provides us with additional distortion. In figure 5 we can observe the level of such distortions. However, according to the original aspect ratio exploration, most of the samples have a 1:1 aspect ratio which illuminates any distortion while resizing to 224x224.
In our experiments we propose two deep learning pipelines. One is suitable for binary classification (e.g. decide if two given images are similar or not). The second one - traditional image classification on 1000 classes. Actually, both pipelines are quite similar. The main difference lies in embedding the processor module.
Since the dataset we are working with is quite limited we decide to use a model from the
EfficientNet family. In our experiments, we used EfficientNet B2[
A fully connected layer is initialized randomly. L2 normalized is used in order to keep a meaningful scale of extracted embeddings.
For a 1000-class classification task we start from the pipeline defined above. First of all, we cut off a similarity estimation model. It was replaced by a traditional K-Nearest neighbors model. Since the KNN model cannot be trained using gradient descent methods we introduce a 2-stage training pipeline. At the first stage, we train a backbone model using triplet loss. Moreover, we may reuse the backbone extracted trained for the binary classification task. In the second stage, we calculate embedding for each sample in the training dataset. Afterward, extracted embeddings are feed to the KNN model along with class labels. So, the backbone model extracts embedding from the input image. The second KNN predicts class labels based on nearest neighbors.
All training and evaluations have been performed on an FSS-1000 dataset. We split the dataset randomly per class. We take 7 samples of each class for training. The rest part is used for validation purposes. We follow a traditional approach of 80% to 20% dataset split but shuffling has been performed in respect of class balance. As a result, we came up with a perfectly balanced dataset with exactly the same number of samples per class both in training and validation splits. 4.2
We define use embedding size of 256 (i.e. fully connected layer size is set to 256). In order to keep backbone weights healthy, we start training with frozen backbone weights. After reaching the loss plateau we unfreeze backbone weights and continue training. Such setup helps us to increase final accuracy by ~2%.
Triplet loss is used as the main loss to optimize during training. We use a semi-hard triplet
mining policy[
We use the KNN model with K=7. That number of the nearest neighbors is chosen according to the number of train samples per class. We fit the KNN model to the training embeddings.
It is very important to select a triplets mining policy if using triplet loss. Deep models tend to overfit training data (especially if training on small datasets). Using triplet loss allows us to increase the number of samples exponentially (by incorporating three samples there are a lot of different combinations). Despite a large number of different triplets, there is still a chance to overfit the model. This is caused by the fact that model optimizes all distances between samples despite having very good distances between easy samples. In order to prevent such behavior, a triplet mining policy is implemented.
As we can see, random triplet mining tends to generate very good embeddings between some classes, but embeddings of the rest are awful. Such setup may produce nice scores for binary classification tasks, but for 1000-class classification, only 0.122 accuracy was obtained.
Traditional approaches of deep learning model cames up with a model that is able to classify images end-to-end. Actually, such models implicitly consist of two parts - feature extraction and feature classification. The goal of classification layers is to attribute input features to one of the classes. Part of the feature extraction module - is to produce such features that classification layers will be able to classify. Thus, combining these two modules in one model along with gradient descent optimization methods gives us an end-to-end trainable model. Since those modules are training simultaneously feature extractor tries to produce features that are easy to classify. So, there are no constraints for such embeddings. Thus, the model pays attention to a unique part of the object despite the real importance of such part (for example, the glass may be treated as the most important part of the object to be classified as a car, despite numerous other objects with glasses).
In contrast to that approaches, the same networks explicitly find separable embeddings. By separable here we mean that model is forced to calculate features that are distinctive from other class embeddings but not suitable. In other words, the model doesn’t operate with class labels at all.
In Figure 9 we show how well our embeddings are separable. There is a tiny overlap between positive and negative distances which stands for hard negative and hard positive pairs. In this overlap, we are likely to give a wrong prediction on classification but having a distance value we can evaluate a confidence level and refuse to give any prediction or make a bias toward one of the classes (which class actually depends on the task). In Figure 9 we also plot a distribution for training and validation test sets, so there is a good fit for validation. It indicates that there was no overfit. The same conclusion is proven by observing a validation loss following the training loss during the training process.
Having such embeddings we can apply a distance-based classification model to it. We find the K-Nearest neighbor classifier the best candidate for classification. K-Fold on the training set (with K=5) gives us the best parameters for KNN - 7 neighbors and weighted distance comparison. Evaluation of such a model on the validation sets proves a significant performance in comparison with the traditional model. Especially, it gives us more than a 10% of accuracy boost in comparison with EfficientNet B2 followed by classification layers. We also evaluate a top 5 accuracy and got relatively the same boost performance.
We also want to emphasize the gap between training and validation accuracies. The traditional model overfit training data completely while our ending extractor model still avoids overfitting and produces better performance. Note, that we do not consider the K-Nearest neighbor model overfit since it should overfit training data by the definition.
The key difference between the few-shot model and the traditional end-to-end model is e semantic of extracted embeddings. In a triplet loss based model, we can only find a similar embedding in the database while features don’t provide any class-specific representation. That’s why such a relatively small amount of data is enough to fit the model while the traditional approach leads to overfitting.
Classical approaches in deep learning image classifications usually consist of two modules implicitly tightened together - the feature extraction module and embedding classification layers. Training of such models leads to making feature extraction models find class-specific embeddings. Since there are no specific contains for extracted embeddings they are not separable in general.
In this work, we have tackled the issue of the large-scale classification task while dealing with a small amount of data. The proposed two-stage classification model is very promising in terms of class capacity, resistance to overfitting, and ability to fit unseen classes. Especially, it gives us a significant accuracy boost (more than 10% of top 1 accuracy) while keeping off overfitting. Distances between produced embeddings satisfy normal distribution without any outliers except easy positive pairs (very similar images in simple words). It indicates that such a model may be easily extended to predict novel classes just as is. There is enough to precalculate embeddings of unseen classes and such a model is likely to classify unseen images correctly.
The few-shot model seems to be a promising approach for large-scale image classification in
terms of a number of internal parameters as well. Our model contains less than 9M of parameters. The
smallest model with a similar performance on ImageNet 1000-class classification contains up to 66M
parameters[
We plan some additional steps for the further research. In particular, we are going to experiment with embedding size, triplet loss margin, etc. We believe that better embeddings may be obtained by ensembling embeddings from different models.