In eCommerce websites, products are annotated with various metadata such as a category by human sellers. Automatic item categorization is useful to reduce this cost and have been well researched. This paper describes how we won the 2nd place (weighted F1 at Stage 1 and Stage 2 are 0.8421 and 0.8399) at SIGIR eCom DataChallenge 2018, whose goal is to predict each product's category by its title. We formulate the task as a simple classification problem of all leaf categories in a given dataset. The key features of our methods are combining of Convolutional Neural Network and Bidirectional LSTM and using ad-hoc features from an external dataset (i.e. not given in this contest). An error analysis is also employed and some cases which are hard to predict accurately are revealed.
In eCommerce websites, products are registered with metadata (e.g.
title, category, etc.) by human sellers. Annotating products with
those metadata is hard job, and therefore automatic predictions
of metadata can reduce the cost[
SIGIR 2018 eCom Data Challenge, July 2018, Ann Arbor, Michigan, USA © 2018 Copyright held by the owner/author(s).
ACM ISBN 123-4567-24-567/08/06. https://doi.org/10.475/123_4 categorized incorrectly because product taxonomies are large. Furthermore, if two diferent sellers annotate a same product with a title, the result should be diferent. Those dificulties cause a noisy dataset, and therefore automatic item categorization is dificult task.
At this challenge of SIGIR eCom DataChallenge 2018, Rakuten Institute of Technology provides train and test datasets. The train dataset is composed of product titles and category ID paths, and the test dataset contains only product titles. The goal of participants is to predict the category ID path for each product title in the test dataset. This challenge is more dificult than the previous item categorization problems for following two reasons: (1) The metadata of products is only title and any other information (e.g. price, image, etc.) is not included. (2) The dataset contains not category “name” paths, but category “ID” paths. This causes dificulty in using prior knowledge of each category.
This paper describes how we won the 2nd place (weighted F1
at Stage 1 and Stage 2 are 0.8421 and 0.8399) at SIGIR eCom
DataChallenge 2018. We formulate the task as a simple classification
problem of all leaf categories in the given dataset. The key features
of our methods are following two parts: (1) Convolutional Neural
Network and Bidirectional LSTM are used together. This technique
may be useful because two models are diferent in structure. (2)
Amazon Product Data[
In the rest of the paper, the detail of our system is described in Section 2. Section 3 describes error analysis of our model. Finally, we present the conclusion in Section 4. 2
An overview of our system is given in Figure1. A product title is fed to the system and a category for the product is predicted by following procedures: (1) The product title is split into words and they are normalized. (2) Each word is converted into an embedding vector. (3) The embedding vectors are input into “Multi-kernel CNN module” and “BiLSTM module”. Each module outputs a flattened vector. (4) Ad-hoc features are generated for each word. Each ad-hoc feature is fed into a multi layer perceptron and a flattened vector is gotten. (5) Three vectors from step (3) and (4) are concatenated into a flattened vector and passed into a last fully connected layer. Probabilities of all leaf categories are output from the fully connected layer.
In the following of this section, we describe the detail of those procedures. 2.1
First, the input sequence (i.e. product title) is split into some words by a space character. Then, symbol characters (e.g. %, #, etc.) in each word are removed. Finally, each of the words is converted into lowercase. 2.2
We used word2vec implemented by gensim[
In general, a word embedding is used in the area of natural language processing. In addition to this, we generated other embeddings as follows.
• pos tags • stemmed word • lemma of a word • hypernym of a word
These embeddings are useful for unknown words in the test dataset. 2.3
We used Convolutional Neural Network with multiple kernels
(Multi-kernel CNN)[
The input of this module is embedding vectors described in Section 2.2 and the output is a vector whose elements correspond to probabilities of each leaf category. First, the input is passed into one-dimensional convolutional layers and the outputs are feature maps. We used multiple convolutional layers diferent in kernel size (e.g. 2, 3, 4 and 5). Next, feature maps are flattened into a vector and it is passed into a last fully connected layer.
2.3.2 Bidirectional LSTM with Soft Atention. In recent years,
Recurrent Neural Networks have been used in the area of natural
language processing. In the field of neural machine translation, it
is reported that attention mechanism is efective technique[
We employed bidirectional LSTM with Soft Attention[
The input and the output is same as “Multi-kernel CNN” module. First, the input is passed into a Bidirectional LSTM layer and the output is encoded sequence. Then, the encoded sequence is passed into a soft attention layer and the output is probability distribution over all leaf categories. 2.4
Ad-hoc features such as length of a title are useful to improve the accuracy. The list of ad-hoc features is as follows.
• title length • uppercase rate • alphabet/non-alphabet/digits rate count • space character rate/count • max length of words • unique word rate • a number of filtered symbols • a number of words • histogram of word length • histogram of pos tags %" %" !" %# %# !# %$ %$ !$ %" %# %$
In addition to ad-hoc features as below, metadata of products
from an external dataset, Amazon Product Data[
Figure 4 shows the procedure of generating the ad-hoc features from Amazon Product Data. (1) Training a model whose input is a title and output is a category with Amazon Product Data. (2) The product titles of the datasets given by this contest and Amazon Product Data are fed into the model of (1) and embedding vectors are gotten from an embedding layer. (3) Prices and categories of 20 products in Amazon Product Data which are similar to each product in the dataset given by this contest are fetched. The similarity of two products is the euclidean distance between two embedding vectors from (2).
We get three flattened vectors from “Multi-kernel CNN”, “Bidirectional LSTM” and the multi layer perceptron with ad-hoc features. In this part, these vectors are concatenated into a flattened vector and it is passed into a fully connected layer whose output is as same as the previous module (e.g. probabilities of each leaf category).
In training, we oversampled data by shufling words in titles for categories which have less than 50 products. It solves a problem of an imbalance of categories.
We split the train dataset by this contest into two parts: train and validation part to check a performance of the proposed system. … …
First, we show the accuracy of top level category prediction. Figure 5 shows the confusion matrix of top level category prediction. It is found that products whose top level category is “1208” are often miss classified as “4015”. In Table 1, examples which are miss classified “1208” as “4015” and products correspond to miss classified category are shown in Table 2. In the first and second lines in Table 1, true and predicted category are seem to be similar. More specifically, “1208>310>1629>1513>3369” and “4015>4454>473” seem to be a food category. “1208>546>4262>572”
F1 score for each category 1.0 0.8
In this sections, dificult categories are explored. Figure6 shows F1 scores for each category. It can be said that it is dificult to predict accurately for categories which few products are correspond to.
Furthermore, we focus on categories that are hard to predict in categories close to the number of products (i.e. bottom of the
In this paper, we describe how we tackle SIGIR eCom DataChallenge 2018. Our proposed model is combined Convolutional Neural Network and Bidirectional LSTM. We also used Amazon Product Data to generate ad-hoc features. In error analysis, it is found that two categories which are similar or share same words in titles are hard to distinguish. It is also found that media categories are hard to distinguish because each title of products in those does not have enough information. We believe that high prediction accuracy came from proposed deep learning models and the external dataset, but human prior knowledge for each category is useful to get better performance.