The typical classification task is based on the assumption that the model will later only encounter examples of classes available during its training. In practice, this is often not a realistic assumption, because of limitations in obtaining enough labeled training data. This contribution is focused on the case where the model might encounter a sample belonging to a class diferent from the classes seen in the training phase. The goal is to reject examples of unseen classes with the option of later adding them as representatives of new classes without the need to retrain the backbone model. This is important because the end-user might not be able to re-train the model for any reason. The presented approach is based on metric learning combined with the meta-classifier similar to the approach of Xu et al. [ 1]. Classified examples are first embedded in a vector space through an encoder trained to capture similarities in the input data. The classification itself is then performed by , where is the number of known classes, binary decisions. For each decision, the tested example is compared to the closest examples from the given class. If the model does not decide that the example belongs to any class, this example is rejected as possibly unknown. The method is tested in a visual data classification task.
backbone model. This requirement arises because retraining might not be a preferable way for the end-user.
Especially for a user with limited deep learning knowledge and inaccessibility to the original data or hardware to perform the training.
Many machine learning applications do not rely only on the used model, but arguably more importantly on the data used for training. The principal requirement for a supervised image classification task, solved by a deep neural network, is that the model will not just fit the training data, but generalize to the distribution from 2. Problem description which the data are sampled. For this purpose, many methods, including data augmentations, have been developed We will assume that for a given classification problem, [2]. there exists a finite 1 set of all possible class labels de
For simplicity, it could be assumed that the data seen noted by = { | 1 ≤ ≤ }, where is the during model testing or deployment are drawn from the number of all existing classes. Furthermore, there are same distribution as the training data [3]. However, this two types of classes, known classes, with labels in the assumption can make the model unreliable and prone set ⊆ and unknown classes, with labels in the set to failure when data instances diferent from what the ⊆ . Those sets are disjoint and each class is known model had seen during its training are encountered. or unknown, that is, ∩ = ∅ ∧ ∪ = .
When dealing with classification tasks, one way to For training, we are provided with a dataset = handle this problem is to use a method that can identify {(1, 1), (2, 2), ..., (, )}, where is the -th unfamiliar examples and reject them, that is, not classify image representation and ∈ its respective label them to any known class. The issue could be that the end- from the set of all possible labels. The set of all initial user might want to be able to adapt the classifier, allowing known class labels is induced by the training dataset. it to recognize some rejected examples as instances of a = { | ∃ ∈ N : (, ) ∈ }. For practical purposes, new class. In this case, a direct solution is to retrain the we will add a rejected class label to a set of known classes classifier with respect to all newly available data. and denote it = ∪ {}.
In our work, our goal was to build an image classifier The desired classification model : R → must that is able to reject unseen class examples but at the be able to assign a known label to any input image represame time enrich the known class set by a new class sentation ∈ R or reject it in the following way: when a user desires to, without the need to retrain the () = ⎧⎨, max ( | ) ≤ ∈ ⎩ar g∈max ( | ), ℎ, ℎ (1)
As mentioned above, the framework consists of two deep learning models with trained weights. In the following sections, both are described, including the way in which they are trained. 3.2.1. The Encoder 3.2. Separate training of encoder and classifier is a probability threshold that is set to a suitable value; for example, could be 0.5. During the deployment of the model, the user is allowed to add a new class to a set of known classes (and remove it from ), and the model should retain the classification ability given by 1. The key to the problem is the way we train the model to estimate ( | ). We will describe it in the next section.
The purpose of an encoder is to find a way in which images could be represented as low-dimensional vectors that somehow capture their most important features. For example, an encoder can take an image of size 224 × 224 × 3 and produce its representation in dimensions 1280, : R224× 224× 3 → R1280. 3. Proposed approach These feature vectors can be passed to an ML algorithm such as SVM or MLP to assign the original input image In this section, we will describe how the whole classifica- class. On the one hand, the desired feature vector should tion process works. be invariant to certain transformations of the input data, which are not important to the classifier decision. On the 3.1. Overview other hand, these representations must be suficiently The whole framework has three crucial components. The discriminative for diferent classes. ifrst is the encoder or embedding network whose task is At the moment, one of the most widely used tools for to take an input image and output its low-dimensional feature extraction in the context of computer vision are feature vector. Since we are dealing with an image clas- Convolutional Neural Networks (CNNs), that can learn to sification task, the suitable option for the encoder is a recognize features by adjusting weights without handdeep convolutional neural network (CNN). Our approach crafting 2. The weights in CNNs are typically trained is specific in the way it handles the encoder training, as end-to-end to perform a specific task on the training data. will be described later. For example, supervised classification with a softmax
The second is the memory of all known classes. In the classifier, object detection, or semantic segmentation. A memory, samples of each class are stored in the form simple way to obtain a feature representation of the input of their feature vectors. Memory is the adaptable part image is by taking one of the hidden representations in of the framework. It can accept a new class simply by a CNN. Typically, outputs of the last hidden layer are adding enough examples. A class could also be removed taken. similarly. The management of the known class set is in Training an encoder whose output can capture simithe hands of the user, which means that in the framework, larities and dissimilarities in the domain of input data is no part would automatically update this memory. The crucial to our task. In our case, the classification is done storage of only image feature vectors is also beneficial by comparing the input image vector representations due to the small amount of disk space required. with the closest classes and images in the memory.
The third and final point is the classifier , which classi- There are several ways to measure the distances beifes a tested example by comparing it with selected sets tween points in R, for example, the cosine distance or of known classes. In principle, it will take the feature the Euclidean distance. Learning input representations vector of a query image and evaluate it as positive (the for a specific distance calculation is called metric learnsame class) or negative (a diferent class) with respect ing [5]. In our work, we measure the distances between to the selected feature vectors of one class from the classified input and image stored in the memory by the memory. cosine distance in the feature space: a sTooluotiuornkfnoorwthleedpgreo,bthleemcloissethsterfercaemnetwwoorrkk, tohraitgionfearlsly (1, 2) = 1 − ‖11‖· ‖22‖ , proposed by Xu et al. [1] named L2AC (Learning to ‖·‖ is the Euclidean norm, and "· " in the numerator deAccept Class). Their approach was successfully tested in notes the standard dot product. For that reason, we dethe context of text classification. In this work, we study cided to train an encoder with respect to that distance by the option of utilizing a similar approach for images and 2However, transformers [4] are performing very well and could propose some changes to the model as well as to the replace CNNs, due to their eficiency and accuracy. Their drawback training procedure. is that they are more expensive to train. adding the triplet loss, with the cosine distance function, 1280). For the implementation, the feature vectors are component to the overall loss when training the encoder not just directly concatenated, but the input representaCNN. tions are obtained by concatenating their element-wise
Our encoder is trained with two objectives. One is a absolute diference and sum. Each of these representraditional softmax classifier performed to classify the tations is then transformed by a fully connected linear input into one class present in the training data with layer (() = + ) with shared weights and is the Cross-Entropy loss and the second is the Siamese followed by ReLu [7], which transforms the input into network trained with the triplet loss [6]. Siamese network the dimension 256. refers to a deep neural network that runs inputs through Applying the linear layer to all representations yields itself with the same weights to get multiple outputs and new representations, which must be converted to a perform an operation on them. In our case, the shared single probability score. For this purpose, we deploy CNN calculates embeddings for the selected image from transformer. A transformer treats those representathe training data (an anchor ), one image from the tions as node features of a fully connected graph and same class (positive ), and one from a diferent class updates them in a given number of steps corresponding (negative ). The triplet loss, with some margin ( > to the number of layers in the transformer model. Up0), can be calculated as follows: dates are performed with respect to all other nodes with the attention mechanism. In the end, the representations created by the transformer are pooled by the mean oper(, , ) = max{(, ) − (, ) + , 0}. ator, and the final representation is then processed by a linear layer with the desired output dimension for binary 3.2.2. Classifier classification (either 1 for a sigmoid or 2 for softmax). The classifier works with the feature representations pro- The advantage of this approach is that the score can duced by the encoder. Its goal is to calculate the proba- be calculated for any class label in the memory, that bility that a given query image (with representation ) is, for any class for which there are enough instances to belongs to a class with label : ( | ). This probabil- compare the query image. On the contrary, a traditional ity is estimated by comparing it to its closest image CNN classifier has a fixed output size, which can include representations from a class , which are stored in the the unit for the unknown class, but adapting it to a new memory. The procedure of calculating the probability class requires additional training. score is shown in Figure 1. Since the encoder model is trained separately, the fea
Our goal is to train this classifier so that it generalizes ture vectors of the images in the training set will not both for seen and unseen classes, and the decision is change during classifier training. It enables us to calcuindependent of the number of currently known classes. late them only once and pre-calculate all negative and For that reason, the classification of an image is always positive training pairs (pair creation is depicted in the done with respect to one class. Figure 2).
To make a decision about one class, the query im- For each training image, there is one positive trainage is first concatenated into each of the closest ing pair and negative training pairs. The is another feature representations (we use feature vectors of size hyperparameter and corresponds to the number of closest classes that will be selected as negatives. Negative classes are selected by calculating the distance between the image feature representations and the mean feature representation for each diferent training class. Selecting the closest examples makes sense because those should be the hardest for the classifier to distinguish.
where is the triplet loss described earlier, is the cross-entropy loss of supervised classification and , are pair classifier crossentropy losses for positive and negative pairs. Cross Entropy loss is widely used for training multi-class classification models and its description can be found in the PyTorch [8] documentation. PyTorch was also used for the implementation of this work.
Training the model in this way enables the use of image augmentations throughout the whole training. When trained separately, the input images were only augmented during encoder training. This type of training is much slower, and when trained on a single GPU, the possible batch size drops significantly.
In this section, we are going to describe an experiment performed on the selected dataset. We are comparing two diferent ways of training the classifier and the encoder.
Until now, both models had been trained separately. An- In both cases, the encoder backbone model is other option is to connect both trained components and EficientNet-B0 [ 9] with pre-trained weights. The softlearn the entire model end-to-end. This means that the max classification layer was removed and the model outloss function could be assembled from multiple compo- put is a feature vector of size 1280. nents: triplet loss, Cross-Entropy loss for classification, and Cross-Entropy loss for binary query-class evaluation. 4.1. Dataset
This setup requires more computing resources since the input consists of several raw images. During learning, To see whether the approach works, we tested it on imthe distances between the examples change continuously ages from the BIRDS 400 - SPECIES IMAGE CLASSIand should be reflected in the selection of pairs for the FICATION dataset 3. The data set was selected due to classifier. The joint model accepts an input image with a the properties required to test the approach. First, the label (used for supervised classification and as an anchor dataset should have enough classes so that we can make a for triplet loss computation), a set of positive images split between seen and unseen classes, and have enough (which are used to create a positive pair, and one of them known samples to train the model for feature extraction. is selected as positive for triplet loss), and a set of negative Secondly, we would like to use the dataset in which clasimages from one selected class (to create a negative pair, sified objects are from a similar domain since we want to and one of them is selected as negative for triplet loss). learn and leverage intra-class and inter-class similarities
This time, we simplified the selection of positive and and diferences in the feature space. It could be assumed negative pairs in the joint training setup. After each that if the unseen class has features never seen during epoch, we recalculate all the embeddings and the class training on known classes, the encoder network might means. For a selected image, we choose = 5 positive not be able to capture them. The whole approach is built images from the same class to make a positive pair and on the idea that features from known classes can be trans = 5 random negative images from a diferent class ferred to unknown classes. Figure 3 shows a few samples that are randomly selected from = 5 classes with the from the dataset used. closest mean in the epoch. The loss of the joint model is calculated as follows: = + + + ,
gpiosenka/100-bird-species 4.1.1. Dataset Split We did not use the original dataset train/validation/test split, but merged these splits together. For the purpose of this work, only selected classes are used. Of all 400 classes, 69 were randomly chosen as known and 10 as unknown. We divide them into training, validation, and test sets in the ratio 0.8 : 0.1 : 0.1. 4.1.2. Separate training
Lookahead optimizer (with Adam, = 0.001) [10]. The batch size was set to 4 and the model was trained for a supervised classification task with 69 training classes.
The loss function consists of the Cross-Entropy loss and Figure 3: The sample from the dataset used for framework the triplet loss calculated on triplets sampled from the evaluation. The classes have similar features and the classified same set of classes. We trained the model for 20 epochs. object is the most salient object in the image. The size of all
After that, all images in the training dataset were trans- images is 224 × 224 × 3. formed into their feature representations, and the training pairs were saved. These training embeddings were also used as a part of the memory for the classification of rejection rates for unknown and known classes. The the test set. For each image in the training dataset, one results for the case where the 10 unseen classes were positive pair and = 5 negative pairs from the closest added to the memory are shown in Tables 2 and 4. In classes (based on their mean feature embedding) were this case, the classification accuracy of the known and sampled. For each negative class, the = 5 closest ex- unknown classes is being measured, together with the amples to the given image were taken. This resulted in a frequency of rejected examples (now the rejection rate total of 58572 pairs of which 9762 were positive. is desired to be low) and the overall classification error
Training of the classification network was performed (for both known and unknown classes). The results are for 30 epochs with all negative and positive pairs and obtained for diferent rejection thresholds. with Cross entropy loss for binary classification. The batch size was set at 32 and the SGD optimizer was used ( = 0.0001).
4.1.3. Joint training For joint training, the images were sampled directly and no embeddings were stored. We trained both the encoder with the triplet loss and the cross-entropy loss for supervised known image classification and the binary classifier simultaneously. The sampling process was previously described. The model was trained with batch size 4 for 20 epochs and the SGD optimizer ( = 0.0001, because higher learning rates did not lead to generalization). The images were augmented before they were passed to the network.
We show the results obtained from the classification of the test set for both training approaches. The results of the separate training are shown in Tables 1 and 2, and the results of the joint training are given in Tables 3 and 4. Tables 1 and 3 show the classification with only known classes in memory. In that case, we are interested in the classification of known classes and the
L2AC [1], which follows the DOC [11] open-world classiifer with the reject option for text document classification. One of the diferences is the joint training in our case. Another related work is OpenMax [12], a classifier that can reject examples of unseen images or images made to fool a trained network. It calculates the likelihood of an input producing certain activation patterns in the penultimate layer of the network. Inputs whose activation patterns are distant from the activation patterns of known classes are rejected.
The problem of open set recognition (OSR) was formalized by Scheirer et al. [13] [14]. They introduced the concept of open space risk and relating its minimization to the solution of the OSR problem.
These approaches, as well as ours, do not deal with the problem of novel class discovery, i.e., finding new classes between rejected examples. The possible approach to ifnding new classes among rejected examples uses clustering performed on rejected examples [15] [16] [17].
Cao et al. [18] proposed a unified end-to-end framework, with the objective of classifying examples in an unlabeled dataset to one of the classes present in the labeled dataset along with the discovery of new classes in the unlabeled dataset. [18]. The discovery of new classes (in the context of unlabeled videos) was also studied by Hůla et al. [19].
Our work can be seen as part of a broader research direction focused on anomalous or out-of-distribution (OOD) data [3] [20]. We have focused on the setup in which OOD data belong to a diferent category from in-distribution data (ID). More generally, the problem of anomaly detection deals with data that are diferent from those seen by the model during training. Apart from a diferent class, this diference could be caused, for example, by faulty sensors.
The purpose of this initial study was to test whether the proposed approach is suitable for image classification tasks in an open world setting. We have shown that the model is able to accept new classes with good accuracy, while the ability to classify examples from the original known classes is preserved. The classification of known and unknown classes after their addition to memory was better after joint training. However, with separate training, the model was able to reject unknown class examples more reliably (with a lower threshold).
In the planned follow-up study, we will train the models Table 3 on a larger dataset with a more extensive hyperparam- Joint training - Classification before adding unseen classes eter search. We are especially interested in improving into the memory. the pipeline and training the model simultaneously for 0.5 0.8 0.9 0.9T5hres0h.9o9ld 0.995 0.998 0.999 all tasks with proper sampling (metric learning, feature Known classified (%) 97.3 97.3 97.3 97.1 96.4 96 95.2 93.6 extraction, and classification). Furthermore, testing the RRejeejcetcetdedUKnnkonwownsns(%(%)) 00 60 90.3 105.5.3 13.56 392.9 542.1.3 655.7.3 approach on more benchmarks is required. For the purpose of this work, we tested only three distance metrics (cosine, Euclidean, and manhattan) with similar performance. More work on the most suitable distance metric search is also ongoing. [15] L. Shu, H. Xu, B. Liu, Unseen class discovery in open-world classification, arXiv preprint [1] H. Xu, B. Liu, L. Shu, P. Yu, Open-world learning arXiv:1801.05609 (2018).
and application to product classification, in: The [16] E. Fini, E. Sangineto, S. Lathuilière, Z. Zhong, World Wide Web Conference, 2019, pp. 3413–3419. M. Nabi, E. Ricci, A unified objective for novel class [2] X. Ying, An overview of overfitting and its solutions, discovery, in: Proceedings of the IEEE/CVF Interin: Journal of physics: Conference series, volume national Conference on Computer Vision, 2021, pp. 1168, IOP Publishing, 2019, p. 022022. 9284–9292. [3] J. Yang, K. Zhou, Y. Li, Z. Liu, Generalized out-of- [17] K. Han, A. Vedaldi, A. Zisserman, Learning to disdistribution detection: A survey, arXiv preprint cover novel visual categories via deep transfer clusarXiv:2110.11334 (2021). tering, in: Proceedings of the IEEE/CVF Interna[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, tional Conference on Computer Vision, 2019, pp.
L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, At- 8401–8409. tention is all you need, Advances in Neural Infor- [18] K. Cao, M. Brbić, J. Leskovec, Open-world mation Processing Systems 30 (2017). semi-supervised learning, arXiv preprint [5] L. Yang, R. Jin, Distance metric learning: A compre- arXiv:2102.03526 (2021).
hensive survey, Michigan State Universiy 2 (2006) [19] J. Hůla, D. Adamczyk, D. Mojžíšek, V. Molek, Seg4. menting out generic objects in monocular videos., [6] A. Hermans, L. Beyer, B. Leibe, In defense of in: Proceedings of the 21st Conference Informathe triplet loss for person re-identification, arXiv tion Technologies – Applications and Theory (ITAT preprint arXiv:1703.07737 (2017). 2021), 2021, pp. 123–129. [7] J. Schmidhuber, Deep learning in neural networks: [20] C. Geng, S.-j. Huang, S. Chen, Recent advances in
An overview, Neural networks 61 (2015) 85–117. open set recognition: A survey, IEEE Transactions [8] A. e. a. Paszke, Pytorch: An imperative style, on Pattern Analysis and Machine Intelligence 43 high-performance deep learning library, in: (2020) 3614–3631.
H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035.
URL: http://papers.neurips.cc/paper/9015-pytorchan-imperative-style-high-performance-deeplearning-library.pdf . [9] M. Tan, Q. Le, Eficientnet: Rethinking model scaling for convolutional neural networks, in: International Conference on Machine Learning, PMLR, 2019, pp. 6105–6114. [10] M. Zhang, J. Lucas, G. E. Hinton, J. Ba, Lookahead optimizer: steps forward, 1 step back, Advances in Neural Information Processing Systems 32 (2019). [11] L. Shu, H. Xu, B. Liu, Doc: Deep open classification of text documents, arXiv preprint arXiv:1709.08716 (2017). [12] A. Bendale, T. E. Boult, Towards open set deep networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1563–1572. [13] W. J. Scheirer, L. P. Jain, T. E. Boult, Probability models for open set recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (2014) 2317–2324. [14] W. J. Scheirer, A. de Rezende Rocha, A. Sapkota,
T. E. Boult, Toward open set recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence 35 (2013) 1757–1772. doi:10.1109/ TPAMI.2012.256.