Consumer-to-shop clothes retrieval has recently emerged in computer vision and multimedia communities with the development of architectures that can find similar in-shop clothing images given a query photo. Due to its nature, the main challenge lies in the domain gap between user-acquired and in-shop images. In this paper, we follow the most recent successful research in this area employing convolutional neural networks as feature extractors and propose to enhance the training supervision through a modified triplet loss that takes into account hard negative examples. We test the proposed approach on the Street2Shop dataset, achieving results comparable to state-of-the-art solutions and demonstrating good generalization properties when dealing with diferent settings and clothing categories.
The visual search of an image from a database of several items is becoming a fundamental
task for many diferent applications in the fields of information retrieval, computer vision,
and multimedia. Typically, the task consists in finding the most similar images to a given
query, which can be either another image [
In the last few years, much research efort [
Pooling Strategy Pooling Strategy
Embedding Space
query images
shop images
works [
Given a query image of a fashion item and a corresponding in-shop image, these are fed through a CNN followed by a pooling strategy to extract a 1D feature vector for each image. Then, the obtained feature vectors can be compared through a similarity function that measures the similarity between the two images. An overview of the proposed approach is shown in Fig. 1. Extracting image features. Both query and in-shop images are processed through a CNN that extracts a 3D tensor for each image of × × dimensions, where , , and are respectively the output tensor height, width, and number of channels. The 3D tensor can be seen as a set of 2D features channel responses = {} where = {1, . . . , }, is the 2D tensor representing the responses of the -th feature channel over the set Ω of spatial locations, and () is the response at a particular position .
To obtain a single 2D tensor for each image, we employ two diferent pooling functions:
a standard average pooling and R-MAC descriptors [
= [ℛ,1...ℛ,...ℛ,]⊤, where ℛ, = max∈ℛ () is the maximum activation of the -th channel of ℛ. Each region ℛ is detected through a square grid of variable dimensions applied at diferent scales. After extracting a feature vector for each region, they are processed using ℓ2-normalization, PCA, and another ℓ2-normalization. Finally, the region feature vectors are summed and ℓ2-normalized to form a single feature vector for each image.
Training with hard negatives. Once the descriptor of the query and in-shop images are obtained, they are compared using a similarity function. Note that the descriptors embedding space is learned according to the loss function used in the backbone training phase. To extract similar descriptors from similar images, a standard hinge-based triplet ranking loss is usually employed and defined as: (, ) = ∑︁[ − (, ) + (, ^)]+
^ where []+ = (, 0) and is a similarity function (i.e. the cosine similarity in our experiments). In the equation above, (, ) is a matching image pair composed of a usergenerated image and a shop image (such that contains the same fashion item depicted in ), while ^ is a negative shop image with respect to (such that ′ contains a diferent fashion item). The sum term in the equation requires that the diference in similarity between the matching and the non-matching pair is higher than a margin .
As demonstrated in previous works [
In this section, we evaluate the performance of our approach and describe the dataset and implementation details used in our experiments.
Dataset and implementation details. We train and test our model on Street2Shop [
Bags Belts Dresses Eyewear Footwear Hats Leggings Outerwear Pants Skirts Tops
To encode images, we use two diferent CNNs ( i.e. ResNet-50 and ResNet-101 [22]) pre-trained on ImageNet [23]. We resize and crop all images to 224 × 224 and obtain a 2048-dimensional feature vector for each encoded image using both average pooling and R-MAC descriptors. In the case of R-MAC, we extract region feature vectors at 3 diferent scales. To train all models, we use Adam [24] as optimizer with a learning rate equal to 0.0001 decreased by a factor of 10 every 10 epochs. In all experiments, we use a batch size of 50 and a margin equal to 0.1. Experimental results. To evaluate the efectiveness of our approach, we report rank-based performance metrics @ ( = 1, 5, 10, 20) for consumer-to-shop clothes retrieval. Specifically, @ computes the percentage of test queries for which at least one correct result is found among the top- retrieved shop items. Table 1 shows the results using all shop images as retrievable items on the Street2Shop test set, without filtering the images by category. We report the retrieval performance of both ResNet-50 and ResNet-101 backbones while extracting image feature vectors either using average pooling or R-MAC descriptors. We compare the results of our approach, in which we finetune the backbone using the hinge-based triplet loss with hard negatives, with those obtained by finetuning the CNNs with a standard triplet loss and those extracted by using the CNNs pre-trained on ImageNet without finetuning. As it can be seen, ifnetuning the backbone leads to a noteworthy gain in performance on all considered settings. Also, the modified triplet loss further improves the final performance using both ResNet-50 and ResNet-101 as backbone and employing both pooling strategies.
In Table 2, we report the performance on each of the 11 clothing categories of the Street2Shop
ResNet-101 + R-MAC (Finetuned)
ResNet-101 + R-MAC (Finetuned with HN) dataset. These results are obtained by performing the retrieval on a subset of in-shop images, ifltered by query category. As it can be noticed, the use of hard negatives generally increases the network performance, leading to better results on almost all clothing categories. Finally, Fig. 2 shows sample query images along with the corresponding top-3 shop images retrieved by the ResNet-101 model using R-MAC descriptors and finetuned with and without the use of hard negatives in the training loss function.
In this work, we have tackled the task of consumer-to-shop clothes retrieval where the goal is to ifnd the most similar clothing item from a catalog of shop images using a user-generated photo as query. To address the task, we have employed a CNN-based feature extraction network and two pooling mechanisms to extract compact feature vectors from images and have proposed to train the network with a modified hinge-based triplet ranking loss that takes into account hard negative examples. Experiments, performed on the Street2Shop dataset, have shown that the proposed loss function can efectively improve the retrieval results in all tested settings.
This work has been partially supported by YOOX NET-A-PORTER Group and the “SUPER - Supercomputing Unified Platform” project (POR FESR 2014-2020 DGR 1383/2018 - CUP E81F18000330007), co-funded by Emilia Romagna region. reasoning networks on a similarity pyramid, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. [16] X. Zhao, H. Qi, R. Luo, L. Davis, A Weakly Supervised Adaptive Triplet Loss for Deep Metric Learning, in: Proceedings of the European Conference on Computer Vision Workshops, 2019. [17] A. Chopra, A. Sinha, H. Gupta, M. Sarkar, K. Ayush, B. Krishnamurthy, Powering robust fashion retrieval with information rich feature embeddings, in: Proceedings of the IEEE/CFV Conference on Computer Vision and Pattern Recognition Workshops, 2019. [18] A. D’Innocente, N. Garg, Y. Zhang, L. Bazzani, M. Donoser, Localized Triplet Loss for Fine-Grained Fashion Image Retrieval, in: Proceedings of the IEEE/CFV Conference on Computer Vision and Pattern Recognition Workshops, 2021. [19] L. Baraldi, M. Cornia, C. Grana, R. Cucchiara, Aligning text and document illustrations: towards visually explainable digital humanities, in: Proceedings of the International Conference on Pattern Recognition, 2018. [20] M. Stefanini, M. Cornia, L. Baraldi, M. Corsini, R. Cucchiara, Artpedia: A new visualsemantic dataset with visual and contextual sentences in the artistic domain, in: Proceedings of the International Conference on Image Analysis and Processing, 2019. [21] M. Cornia, L. Baraldi, H. R. Tavakoli, R. Cucchiara, A unified cycle-consistent neural model for text and image retrieval, Multimedia Tools and Applications 79 (2020) 25697–25721. [22] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016. [23] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision 115 (2015) 211–252. [24] D. Kingma, J. Ba, Adam: a method for stochastic optimization, in: Proceedings of the
International Conference on Learning Representations, 2015. [25] X. Wang, Z. Sun, W. Zhang, Y. Zhou, Y.-G. Jiang, Matching user photos to online products with robust deep features, in: Proceedings of the ACM International Conference on Multimedia Retrieval, 2016.