-We present an empirical study of applying deep Convolutional Neural Networks (CNN) to the task of fashion and apparel image classification to improve meta-data enrichment of e-commerce applications. Five different CNN architectures were analyzed using clean and pre-trained models. The models were evaluated in three different tasks person detection, product and gender classification, on two small and large scale datasets.
The recent progress in the image retrieval domain provides new possibilities for a vertical integration of research results into industrial or commercial applications. Based on the remarkable success of Deep Neural Networks (DNN) applied to image processing tasks, this study focuses on the task of fashion image classification. Photographs of clothes and apparels have to be classified into a set of pre-annotated categories such as skirt, jeans or sport-shoes. Online e-commerce companies such as Asos-EU 1, Farfetch 2 or Zalando 3 provide access to the data of their products in stock including item-metadata and images. Especially the provided meta-data varies in quality, granularity and taxonomy. Although, most of the companies provide categorical descriptions of their products, the applied terminology varies as well as the depth of the categorical hierarchy. Fashion image classification is thus used to consolidate the meta-data by enriching it with new generalized categorical labels.
This is a traditional image processing task with domain
specific challenges of large variating styles, textures, shapes
and colors. A major advantage is the image quality which
are professionally produced high quality and high resolution
images. There are generally two categories of photographs.
The first arranges products in front of a white background. The
second portraits a person or parts of a person who is wearing
the products. While the first category reduces semantic noise
of the images, the second one introduces it, because a person
wearing multiple items such as jeans, t-shirt, shoes and belt is
only assigned to a single label. Clothing and apparel retrieval
has been addressed to find clothes similar to a photograph
[
In this study we present an empirical evaluation of various DNN architectures concerning their classification accuracy in different classification tasks. These tasks are evaluated on two different datasets on further two different scales. First, a wide evaluation is performed on a smaller scale dataset and the best performing models are then applied to large scale datasets. The remainder of this paper is organized as follows. In Section II we review related work. In Section III the datasets used for the evaluation are presented. Section IV provides an overview of the evaluated neural network architectures. Section V describes the evaluation setup and summarizes as well as discusses the results. Finally, conclusions and outlooks to future work are given in Section VI.
Recently, CBIR has experienced remarkable progress in the
fields of image recognition by adopting methods from the area
of deep learning using convolutional neural networks (CNNs).
A full review of deep learning and convolutional neural
networks is provided by [
The data provided was retrieved from online e-commerce companies such as Asos-EU, Farfetch or Zalando. Person: The persons dataset consists of 7833 images and the corresponding ground truth assignments. 5.669 images are labeled as Person and 2.164 images are labeled as Products. Products: The product dataset consists of 234.884 images and their corresponding ground-truth assignments. These images belong to 39.474 products where each product is described by 5,95 images on average. Ground-truth labels are provided for categories category, gender and age. All labels, including age, are provided on a categorical scale. The provided ground-truth assignments consists of 43 classes for the category attribute. These categories are based on a hirarchical taxonnomy. The hirarchy for the provided dataset is depicted in Figure 1. Its largest class SPORTS SHOES contains 66.439 images (10.807 products) and its smallest class JUMPSUITS 6 images (1 product). To facilitate more rapid experimentation, the provided dataset was sub-sampled to approximately 10% of its initial size. Further, due to the class imbalance of the provided category labels, an artificial threshold has been applied to the class sizes of the assignments. All classes with less than 100 images have been skipped. The remaining classes have been sub-sampled to a 10% subset. The sub-sampling adhered to further restriction. First, stratification was used to ensure that the frequency distribution of class labels in the subsample corresponds to that of the original one. Second, subsampling was performed on product-level. This ensured the consistency of product-images and that there are no products with only one image. Finally, sub-sampling of a class was stopped when a minimum of 100 images was reached. This resulted in a subset of 23.305 instances, ranging from 5.659 images for SPORTS SHOES (922 products) and 103 images for STRAIGHT LEG TROUSERS (19 products).
In this study we compared vfie different DNN architectures
which varied in depth and number of trainable parameters,
including three winning contributions to the ImageNet Large
Scale Visual Recognition Challenge (ILSVRC) [
Vgg16 and Vgg19: very deep convolutional neural
networks (VGGnet) [
InceptionV3: high performance network at a relatively
modest computational cost [
The Convolutional Neural Networks were evaluated towards their classification accuracies in the tasks of differentiating persons from products as well as classifying product images according their product category and gender. We performed three-fold cross-evaluation and calculated the accuracies on a per-image and a per-product scale. To calculate the per-product
P
P e > 0.9 rs o n accuracy the cumulative maximum of all predicted product images was taken into account.
Person detection was introduced based on the observation that products are presented in two general types. First, there are images of products placed in front of a white background or table. The other type of images are worn products. Because persons on these images are wearing more than one product such as trousers, shirts, shoes and belts, it is hard for a classifier to learn the right boundaries. Thus, the intention was to train a person detector and to either filter person images, or to use this additional information as input for further models.
We applied a custom VGG-like CNN with three layers of batch-normalized stacked convolution layers with 32, 64 and 64 3x3 filters and a 256 units fully connected layer with 0.5 dropout. We realized this task as a binary classification problem by using a sigmoid activation function for the output layer. Predictions greater or equal 0.5 were classified as persons. This approach already provided an accuracy of 91.07% on the person dataset. Figure 2 shows example images of the person detection model. Images on the bottom row were predicted with values below 0.1 and are thus categorized as products, whereas images in the top-row are considered to be persons.
The product classification experiments were conducted using the different CNN architectures presented in Section IV on two different scales. First, a broad evaluation was performed on the small-scale subset of 23.305 images. Then, the best performing models were evaluated on the large scale 234.408 images dataset. All models, except those where explicitly mentioned, were trained using image data augmentation, including horizontal flipping of the image, shifting it by 25% in height and width as well as a 25% zoom range.
ation deals with the question of whether to train a model from scratch or to fine-tune a pre-trained model. The availability of a large collection of high quality images and a relative small number of classes suggests that models can be effectively fitted according the specific domain.
The results presented in Table I show that pre-trained models outperform clean models that have been specifically trained from scratch using only the images of the fashion image collection. Additionally, we evaluated the two different types of applying pre-trained models: a) resetting and training only the top fully connected layers while keeping all other parameters fixed, and b) continued fitting of all parameters on the new data - which is also referred to as fine-tuning. In either way the 1000 unit output layer of the pre-trained models had to be replaced with a 30 units layer representing the 30 product categories.The results of the evaluation show that finetuning outperforms the fitting of clean fully connected layers by 5.9% (VGG16) to 7.9% (InceptionV3). The smaller custom models did provide an advantage concerning processing time of fitting and applying the model, but their accuracy results differ by 16.1% to the top performing fine-tuned model.
Figure 3 shows the prediction accuracy per class for the best model (fine-tuned IncepionV3) on the 234.408 images dataset. The most reliably predicted classes are SPORT SHOES,
These results indicate the problem of different granularity within the provided ground-truth assignments. Root- and leafbest fold 0.706 0.658 0.646 0.569 0.579 0.564 0.578 0.603 0.585 0.640 0.492 0.506 0.463 0.438 0.439 0.437 0.819 0.798 0.762 best fold cum max 0.794 0.729 0.711 0.685 0.673 0.673 0.669 0.652 0.650 0.636 0.594 0.568 0.556 0.549 0.455 0.447 0.887 0.863 0.846 nodes are used interchangeably which results from the aggregation of different e-commerce catalogs using different taxonomies. Although confusion a child- with a parent-class is semantically not wrong, but the trained models do not take this hierarchy into account and predict each label individually. Thiseffect can be seen in the confusion matrix in Figure 4 where spezialized classes such as JEANS and SKINNY JEANS or SKINNY and SKINNY JEANS or are confused frequently.
The aim of the gender prediction task was to predict the intended gender of the product into the classes MALE, FEMALE and UNISEX. The results are comparable to the product classification task in the sense that pre-trained and fine-tuned models provide the highest accuracies with a best performing value of 88%.
In this study we presented an empirical evaluation of different Convolutional Neural Network (CNN) architectures concerning their performance in different tasks in the domain of fashion image classification. The experiments indicated that dispite the large amount and high quality of provided fashion images, pre-trained and fine-tuned models outperform those which were trained on the given collections alone. Future work will concentrate on analyzing models on a scale of two million images.