E-commerce product categorization is an important topic, and its quality directly affects subsequent search, recommendations and related personalized services. E-commerce product classification is challenging due to the large scale and complexity of the product information and categories. In the E-Commerce Text Classification Challenge, we combine machine learning, deep learning, and natural language processing to propose a multi-level and multi-class deep learning tree method. Our method constructs multiple models based on single-label and multi-level label predictions as well as the characteristics of the product tree structure and combines the multiple models to generate a new classification model. The proposed classification model is tested on the online test dataset. The accuracy, recall, and F1 score are 0.8552, 0.8389, 0.8404 in leaderboard(Stage 1) and 0.8397, 0.8428, 0.8379 in leaderboard(Stage 2) respectively, ranking among top 3 scorers.
CCS CONCEPTS •Applied computing organization infrastructure; Supervised learning •Computing methodologies → processing → E-commerce by classification; Natural language Deep learning, Hierarchical search tree, Text classification
The task of the big data challenge is to predict the e-commerce product category according to the given product title. The challenges of this task are as follows.
(1) The distribution of product categories is extremely unbalanced. For example, there are thousands of product titles related to some merchandise categories, but some product categories only have 1 to 2 titles in the training dataset. (2) Offline training data and online test data are very different. The analysis shows that the training dataset and the test dataset involve about 220,000 and 580,0000 different words respectively, which means that the online test dataset contains a large number of new words.
(3) The levels of category to be predicted is complex. If a category label such as "A>B>C" is defined as the third category level, the product category level in the training dataset can reach 8.
(4) The large number of product categories to be predicted greatly increases the complexity of the classification problem. If we consider each label as a category in the training data set, there are 3008 categories in total. If we look at categories at different levels, we can reach 1600 categories at the most levels.
To address these challenges, we develop classification strategies based on the characteristics of the data. We merge the training dataset and the test dataset to construct word vectors for textual expressions so that semantic similarity can be used to process new words in the test dataset. Sampling and data enhancement techniques are used to address the unbalanced category issue. To deal with the complexity of product classification, we construct eight sample datasets according to the category hierarchy and develop two classification algorithms to build classification models for different levels and search paths using category trees. 2
Text classification is an important topic in natural language
processing. It is widely used in information retrieval, search
recommendation, news classification, anti-spam, public opinion
analysis and other fields. A large number of text classification
methods have already been pro-posed in the past. Conventional
feature processing constructs TF-IDF and other features[
Compared with the prior work, we evaluate the effectiveness of different feature extraction methods and classification algorithms, and then combine multiple models to develop a new product category classification method.
All experiments use Tensorflow and Fasttext. In the classification experiment, we divide the data into three sets: training set, verification set, and test set. The data volume are 720,000, 80,000 and 200,000 respectively. In order to evaluate the impact of each step on the final test, we first analyze the distributions of the word frequency, product label hierarchy and word count, etc. The effectiveness of different text preprocessing methods, including noun extraction, stemming and excluding stop words, and different word vector models is evaluated. We also evaluate the impact of Fasttext, TextCNN, TextRNN , directory tree and AbLSTM models. Based on the results, we choose the best models and combine them to classify the online test dataset. 3.1
We first analyze the given data. The result shows in Fig1.
We find that the distributions of the title length are centered around 10 words and the length of the titles in the training dataset shows a longer-tail distribution from Fig1.b and Fig1.c. Fig1.d shows that the number of products of each category in the training set varies from 1 to 70,000, an extremely unbalanced distribution. We further study the distribution that 3rd, 4th and 5th categories account for the largest proportions of the data both in titles and subcategories from Fig1.e and Fig1.f. We analyze the category labels obtained by the classification model and investigate why some categories are misclassified on the validation dataset.
#e offline training dataset is divided into a standard training dataset with a size of 720,000 and a validation dataset with a size of 80,000. We set the ngram parameter to 2 and the word window length to 100. Using the Fas%ext model with the standard training dataset, the model's prediction performance on the validation dataset is shown in Table1. #e interval indicates the predicted probability interval. Inconsistent and Consistent indicate the number of prediction category labels that are consistent and inconsistent with the real category labels respectively. Total is the total number of titles included in the prediction probability interval.
The number of titles with a prediction probability higher than 0.93 is 65,374, accounting for 81.7% of all 80,000 titles in the validation dataset. The result is consistent basically with the calculated precision as Table2 shows. If the predicted probability is greater than 0.93, there is a high probability that the predicted label is consistent with the reference label. Otherwise, the prediction is erroneous and we need to study it further.
To analyze the erroneous results, we first obtain the data that does not match the actual prediction labels in the validation dataset, and then use the similarity method to perform an approximate match detection.
For example, for the title "Fuel Pressure Regulator Carter 404500HP" , the Fasttext model’s prediction is 2199> 661> 4498> 343 while the actual result is 2199> 4592>12. We search the most similar product titles as 2199>661>4498>343 and 2199>4592>12 in the training dataset. The most similar title as the former and the latter are "Fuel Pressure Regulator: Belt/Hex Drive Pump EFI Regulator" and "Edelbrock 1727 Fuel Pressure Regulator" respectively. This may be the cause of misclassification. 3.3
In the field of natural language processing, feature engineering includes text preprocessing, feature extraction and text expression. Feature extraction process plays a very important role in text classification. The classification task is mainly composed of two parts, converting data to features and features to classification. The former process determines the upper limit of the classification model performance.
3.3.1 Text preprocessing. We try different text preprocessing methods including removing stop words, stemming and nouns extraction to evaluate their accuracy using the Fasttext algorithm. The result on the validation dataset is shown in Table 2. We can see that the more we modify the title, the worse the resulting accuracy is. We hence decide to use the original title as input in the experiments later. 3.3.2 Feature extraction ant text extraction. Text representation is a way to convert text information into machine understandable language. The traditional methods for text representation are word-bag model or vector space model. The word-bag model is characterized by high dimensionality and sparsity. It cannot express the semantic information very well. In order to perform text representation better, we extract some features to enhance text information. Common feature extraction methods include mutual information, information gain, and TFIDF method. However, these feature extraction methods can only reflect the features of specific words, but can not express the context and semantic similarity. Semantics-based text representation transforms text into word vector. Word2vec, glove and Fasttext can perform text representation based on semantics. We use Fasttext to calculate the word vector on the 1 million dataset in our work. 3.4
Traditional text classification methods such as naive Bayes and
nearest neighbor classification do not take into account contextual
correlation. Therefore, the performance is poor when used in
large-scale multi-classification problems. Deep neural networks
may be better because it calculates the local correlation[
If we consider "3292>114>123" a single label, it’s a single-label classification problem. If we consider "3292>114>123" a combination of "3292", "114" and "123", it becomes a multi-label classification. We discuss the two different models next.
3.4.1 Single-lable prediction model. We initialize the word
embedding matrix using the Fasttext model. The word embedding
dimension is 100 and N-gram is 2. We also use TextCNN,
TextRNN, AbLSTM and other models to conduct the text
classification experiments [
We analyze the category labels and find that a category number belongs to a single level at a time. For example, number 11 never appears under the first-level and second-level categories at the same time. Based on this observation, we generate a multi-level tree using the product category labels for the offline training data. A multi-level tree example is shown in Fig2.
Different data training sets for different category levels are generated. Since the levels are up to 8, 8 models are generated. Each model predicts a label, and searches the tree in a top-down order. The probability of a multi-level category label is the average of the probability of the labels on the search path.
Fig.3 shows the multi-level tree classification model training process. First, the original data is divided into eight sets according to different hierarchical category labels, and a classification model corresponding to per level is trained on each data set.
Fig.4 depicts the multi-level tree classification prediction process. The multi-level tree classification model consists of an input layer, a word vector layer, a classification model layer, and a tree search layer. For a new product title, according to the pretraining word vector model, the word vector layer is converted into a vector form that can be calculated by the computer, and then according to different hierarchical models, the top three most probable labels are calculated, and finally the categories are combined. In the tree, all possible paths are calculated, and the path with the highest average probability is selected as the output value.
For each test sample, assuming that the maximum level of the hierarchical tree is k , the maximum possible path is calculated as follows : !"#ℎ% = [( )+ *, ( )+ - , … ( )+ 01 *, ( )+ 0]
7 !% = 3 !) 4+5!) 6
) 89 !"#ℎ):; = !"#ℎ:<=):; (?@) (3) where ( )+ 0 represents a random label in the kth category and Pathh is a collection of all paths made up of different levels of category elements. !) 4+5 and !) 6 represent the probability corresponding to the ith label at the m-1th level , and the jth label at the mth level respectively. Ph is the product of the probability of each node on a path and Pathmax is the path with the highest probability .
With the previous experience of the hierarchical tree model, we try a new sample tagging method, which replaces tag with a combination of the tags of current hierarchical level and all of its superior. For example, if the predicted target is '3292>114>123', we will get '3292', '3292>114' and '3292>114>123’. In order to distinguish it from the previous hierarchical tree, we name the new tree model SP tree. With different classification algorithms, different models including Fasttext+SP TREE (FST) and AbLSTM+ SP TREE (AST) are constructed. We choose the deepest level of the label set containing the hierarchical prediction label from top to bottom, and take the label with the highest probability in the set as the prediction result. Equation (4) and (5) describe the above calculation process, where BCDEF7 represents the sequence number corresponding to the prediction result with the highest probability at the kth level.!"#ℎ:<=):; (?(?:G%01 *)) represents the most probable prediction in the k-1th level. !"#ℎ7 denotes all predictions for the kth level. !"#ℎ7):; indicates the maximum probability label of the kth level which contains its superior labels.
BCDEFH = "IJK"F!(!"#ℎ"IJK"F(!(!"#ℎH−1)) ∈ !"#ℎH) (1) (2) (4) (5) !"#ℎ7):; = !"#ℎOPQR;0
The Fasttext and AbLSTM methods are used for classification according to the results of the single model. The models are submitted online for evaluating the test dataset. The result is shown in Table 4.
The model based on the single-label prediction achieves higher recall rates and F1 scores while the model based on multi-label prediction achieves higher precision. For model fusion, we use both the simple voting method and the weighted voting method. The simple voting method refers to voting on the results according to multiple models, and determining the category based on the voting results. The weighted voting method refers to adding up the predicted probability values of multiple models, and choosing the prediction with the highest probability value. The results of online data test using the simple voting method and the weighted voting method are shown in Table 5. The classification of product categories based on the product titles is an important and challenging problem. We propose a method that combines different classification models.
1. More data preprocessing such as excluding stop words, stemming, and extracting nouns can lead to worse performance. This observation suggests the use of the original titles as the input.
2. We build two types of models using single-label prediction and multi-level label prediction respectively. For single-label prediction, we use Fasttext, Text-CNN, Text-RNN, VDCNN and AbLSTM. The results show that Fasttext and AbLSTM perform better than the others. For multi-level label prediction, according to the different processing methods of sample tags, we construct hierarchical search tree model and short path tree model . For hierarchical search tree, we first extract the category tree structure in the training dataset and use different classification algorithms to predict the top three labels with the highest probability in each level. We then choose the path with the highest probability as the prediction result according to the category tree. Compared with the hierarchical tree, the short path tree appends all parent tags to the current level tag when processing the sample tag
The model based on single-label prediction achieves higher recall rates and F1 scores while the model based on multi-label prediction gets higher precision. These results imply that the classification model can be improved by combing different models.
3. Our approach combines the results from Fasttext, AbLSTM, Fasttext-Tree(FT) AbLSTM-Tree(AT) , Fasttext-SP-Tree(FST) and AbLSTM-SP-Tree(AST) with weighting strategy and achieves a precision, recall and F1 score of 0.8552, 0.8389, 0.8404 in leaderboard(Stage 1) and 0.8397 , 0.8428 , 0.8379 in leaderboard (Stage 2) respectively . The proposed approach could also be used for other text classification tasks such as movie, music, fresh, etc.
#is work was partially supported by the SIGIR eCom`18 Project "Taxonomy Classification for eCommerce-scale Product Catalogs" and by Rakuten Institute of Technology Boston (RIT-Boston).