ProBERT: Product Data Classification with Fine-tuning BERT Model

Abstract

In this paper, we describe our submission to the semantic web challenge on mining the product data in websites (MWPD2020). The dataset provided 19K instances of product data collected from various websites. The task is to predict the category, defined as hierarchical taxonomy as provided in the training set, of the product titles in the test set. In our approach, we present a simple BERT-based model (dubbed ProBERT) for classifying product data into one or more categories. We trained our system on products titles and descriptions to learn semantic representation. The participated systems are evaluated using weighted-average precision, recall and F1-score.

1 Introduction

Recently, many e-commerce websites are embedding structured product data into their content; according to the statistics from web data common1, there are 37% of web pages or 30% of websites contain structured data. Consequently, these structured data can be used for product data integration and optimize product search service [ 8 ]. In addition, product categorization becomes essential in providing personalized recommendations and targeting advertisements. However, classifying product data is a challenging task due to the intrinsic noisy nature of the product labels, the size of modern e-commerce catalogues. In addition, each website has its a di erent structure of their product data, we refer to it as site-specific annotation [ 5, 1 ]. For example, one product like a T-shirt can have di erent annotation labels in di erent websites (College>T-Shirts,Clothing >Tops>Shirts,Clothing accessories>Clothing>Tops). To train robust models in these cases, we need large amount of training data with balanced classes. Therefore, automated product classification is need to further organize these data semantically into a universal categorization system regardless of their site-specific annotation.

In this paper, we explain our method to solve this problem through the semantic web challenge on mining HTML-embedded product data (MWPD20202). The challenge aims to mining product data embedded into websites content. Previous studies [ 3, 6 ] focused on categorizing product data on a single e-commerce website and sensitive to it’s site-specific content. In this challenge, the goal is to predict each product’s categories based on datasets from di erent websites. We address this task as a multi-label classification problem, where each product can be assigned more than one class (i.e., label or category) simultaneously.

The latest development in language models (e.g., BERT) have shown impressive gains in a wide variety of natural language tasks ranging from sentence classification to sequence labeling. In our approach, we propose a BERT-based neural model to categorize a product based on it’s meta-data such as product name, description or site-specific annotation. In particular, we employ a fine-tune BERT model to represent product data as low-dimensional contextualized vector. We feed our model with product name and description to capture semantic representation for product information. We summarize our main contributions in this paper as follows: – We presented ProBERT, a BERT-based model for multi-label product classification based on product meta data (e.g name, description and site annotations). – We conducted di erent experiments to benchmark the impact of di erent embeddings approaches. The result indicates that our method can be a good baseline with contextualized embedding (BERT) for product classification.

The rest of this paper is organized as follows: We first explore the dataset used in the challenge in section 2. Then, we present our proposed approach and the o cial results in sections 3 and 4 respectively. In section 5, we conclude the paper with some discussion about future work. 2

Dataset

The dataset is provided in the JSON format and divided into three subsets: (1) training contains approximately 10k product instances, (2) validation contains 3k instances and (3) 3k instances used for evaluated and testing the submitted systems. The product attributes in the dataset as follows: – ID: refers to the product identification number. – Name: is the product name (can be an empty string if unavailable). – Description: is the description of product (truncated to a maximum of 5k characters.

can be an empty string if unavailable). – CategoryText: is the website-specific category for a product, or breadcrum (an empty string if unavailable).

– URL: refers to the original web page URL of the product.

Each product may be assigned one or more from the following classification levels, corresponding to the three GS1 GPC classification levels: – lvl1: the level 1 GS1 GPC classification. – lvl2: the level 2 GS1 GPC classification.

– lvl3: the level 3 GS1 GPC classification.

Approach

In this section, we present ProBERT, our simple BERT-based model for multi-label product classification. BERT is a pre-trained transformer network [ 2 ], which set for various NLP tasks new state-of-the-art results including text classification [ 7 ] and natural language understanding [ 4 ]. When we adopt BERT to NLP tasks in a target domain, a proper fine-tuning strategy, where a task-specific layer is added on top of BERT architecture. In this work, we leverage the BERT-Base pre-trained model with these details: Uncased: 12-layer, 768-hidden, 12-heads, 110M parameters. Then, we add a fullyconnected layer (i.e Dense). For multilabel classification purpose, we use binary-crossentropy as in Eq.1 loss function and sigmoid activation function to replace the original softmax. All hyper-parameters remain as default values, except we set max seq length as 30 words per input sequence.

L = yi log(H(xi))) + (1 yi) log(1

H(xi)) (1) where yi and H(xi) denote ground-truth and predicted categories for each product. xi refers to the feature vector obtained from the BERT model.

C E[CLS] CLS

T1 E1 W1

Product Classes

Sigmoid

Fully-connected Layer

Feature Vector

T2

T[SEP] BERT E2 CLS

E[SEP] SEP

Tx' Ex' Wx

TM' EM' Wm Fig. 1: ProBERT: A Fine-tuned BERT Model for Multi-label Product Categorization.

The general architecture of BERT is shown in Figure 1. We use a combined text of product title and description as an input features. Then, we do standard preprocessing which lower-casing and lemmatization of text. Then, a special preprocessing is performed for BERT processing; first inserting two special tokens. (CLS) is appended to the beginning of the text, another special token (SEP) is inserted after each sentence as an indicator of sentence boundary. The modified text is then represented as a sequence of tokens X = [w1; w2; : : : ; wn]. Each token wi is assigned three kinds of embeddings: token embedding, segmentation embedding and position embedding. These three embeddings are summed to a single input vector (C), which captures the overall meaning of the input. 4 4.1

Experiments

Evaluation The evaluation metrics used in this challenge are precision, recall and F1. F1-score in Eq. 2 is the harmonic mean of precision and recall scores. The organizers used macroaveraged F1 score as the main metric to compare and rank the participating systems. The organizers provided an overview of the performance of baselines with di erent embedding approaches (FastText, CBOW and Skipgram) on the validation dataset. As shown in Table 1, the baselines are evaluated based on both weighted-average and macro-average F1-scores. The experimental results are promising and shows that the systems based on embedding methods can achieve good F1-scores. Hence, we proposed our approach to employ the state-of-art contextualized embedding such as BERT to benchmark the system performance.

The results are reported in terms of three evaluation metrics: (precision, recall and F1-score). F1-score is the score ultimately used to compare and rank the participating systems. Table 2 shows the results of five participating teams and the baseline (FastText). Our team (DICE UPB) submitted one system based on fine-tuning BERT model. The performance is close to the baseline system in terms of F1 score (81:84% compared to baseline 84:26%). However, we found that feature engineering needs a special preprocessing rather than the standard preprocessing, due to the nature of product data such as: highly imbalanced in labels as shown in Figures 2a and 2b; noisiness in the descriptions. We suggest to perform the same preprocessing as [ 8 ] and change our strategy of fine-tuning BERT model to address these challenges properly. In this paper, we described our approach (ProBert) to classify product data based on micro annotations. Our approach leverage a simple BERT model that represents a single feature vector from product’s title and description, then predicts it’s categories. Our experiments suggest that ProBERT is a good baseline to benchmark the task of automatic products classification. In the future, we plan to re-evaluate our approach with di erent preprocessing and fine-tuning strategies. Also, we will investigate more deep models with di erent architectures (e.g., graph-based neural model).

Acknowledgments This work has been supported by the EU H2020 project KnowGraphs (GA no. 860801) as well as the BMVI projects LIMBO (GA no. 19F2029C) and OPAL (GA no. 19F2028A). 0 0 0 0 0 0 9 7 s e l b a t e g e V _ 0 0 0 0 6 2 0 5

Fig. 2: Label distributions (log scaled) in the training dataset. _ 0 0 0 0 7 1 0 5 /trssPeeeeuSdd lr_scooeeeuhud00H ii_saud00040860V ilt_aeEund00001AVmr_sckPPaaan00306009V /trr_sckyPoaeeudd80000501BB ii//tttrrrssssscvvxPoaaeeeeEnnSg ii/_Fyyaeeennbg04500010B liliit_caeEng78030000 itrscPa53104000 ryaaeb54110000Wr_Fooae630100w il/t_sFeEbd00005O trrsckyyPaaaaeebVC i/ttrrsssssccoeeenuAmii/ffrrccyyaeeenhnOM li/tttrssTaeeehndAm/ssyToae0010006Gm/r_ssoeeeSSdp700000 il/rsLaaeennudppSGw l/ftrrr_sFToaaeeeunSg ltr_ssccyooaaendCM r_svaeeeg20005000B il/tr_soaaaeeudggSAH liitrssckyPaaeupSpV tr_scyooudd000200B iiltrssFaaeunhnngml)(ttrssaenuuhnAGG iiii/tttrscoooenunnnbD i/ttr_scyPooahhpgp0O lli/rsskoeeeuSppdw li/rssccoanuSpppO l/t_yPoaeu00040 /()trssyyTaeEpmi/tttsssseeEuugSbg liitssccoaeenppAmliitssccoaeenppAmiit_sscooaengHDmiirssscckooeengAmliItr_scyPoaaenn0 i_ang00010047Cm/t_sFkPooend000D ilI-trr_scccaeEnn00 ii/trrccyveeeunuSS ittrscPoeenuS lrrscauuhgD iiittoonnngdC itrscPonugd soonudpCmitrsckPonud ilisLvanAmtcaaeenM irskoohdp isoaeedVGmiili/rannggbWtsooennpCmlit_ohng000C itrscPoungd trscPoenud iitscoanunmttrscPoeund irscaeendM ilrsssccoeA ltrrooenngC rsFvooeedd taeEnnudqH iI/ttcyPYoenD li/frscaeeeedRM r_Pae001010C /)rrrrsssFoaeeeehnndppU i/tr_PToooeeEnpq0000wmtcaaeeeEnnnhh51120000milii/tr_sscFFxTaeeunnhgb200000 i/trr_sssccFoaaeeng35160000Cmittr_soeEnuqSpp07100100 iil_sccaeeed00511500DM e ie tr t sse itn e u lf p ir n A g h d e t m o lla d mv g o r ta te ( rcPo rFu lauM iuqp itye r H L r h e

e P rn a i e in s g it raepd i/ffO iod isu ts a a ed trc _H _B 00 ils a _ M ra / en cae i/rA laen itve _S _ e rse ipmtroe ren 00 ce W05 se eh r ll e ca itcoa 6701 _u3010000B lr_syPoaengH _o60610000Cmt_saaaeeng00MmWit_cehn00003074K r_sPoaen00006401 llI//tt_sscPeen00020 /rr_Paeedp000019 l/_sToo1000000 f_aS01008500 il/ttrxPTaeenu

A P mn p u lt oa up itr i A o l

N a 2 /C e D D 00 /S 0 d ls _H ir / xoeB /trsuC j_oa0M _aS0m511300 ccTooab 502330

a 0 /N 0 s 0 0 e 1 8 01101 030068 /ftssyaeenS /rryoaeungSw _Pa00006151 ilii/ttoaaenngV _07140000C i/ttrrsccPoaen 53130000 89020000 iittrsssccvoooaeeendAml/tr_TooegS84010000W/tr_so00650100Cmll_ceE07804000C itrccoounnnCmm0 lid ie _A ra laug /Yog 000 000 00 d _ 0 C a m10 20 00 /H a 2 2 1 e re 7 7 2 g C 0 ra /r 5 8 0 0 0 0 U 1 81 00 0 5 35 12 17 05 8 _0 f) 28 7 7 4 0 a 4 0 e 0 L 1 n 0 o 0 N 6 ( y r e n i h c a M e c i f f O / y r e n o i t a t S _ 0 0 0 0 7 0 2 6 9 0 001000 itreeng 0 0 3 0 2 0 7 _ t_oS tteu 8 7 G 0 0 000 il/kB 0 0 0 20 _M 5 2 0 0 6 2 0 9 0 0 3 1 0 5 b u is it e _L /euB fcen i/ngH 0000

b 1 u om_oC m 08 _A

lu 8 0 _1030000H 50160000 0 1

P 0 _ 0 0 0 0 0 0 7 1 7 0 9 9

7

References

[1] Cevahir , A. , Murakami , K. : Large-scale multi-class and hierarchical product categorization for an e-commerce giant . In: Proceedings of COLING 2016 , the 26th International Conference on Computational Linguistics: Technical Papers . pp. 525 - 535 ( 2016 )

[2] Devlin , J. , Chang , M.W. , Lee , K. , Toutanova , K. : Bert: Pre-training of deep bidirectional transformers for language understanding . arXiv preprint arXiv: 1810 . 04805 ( 2018 )

[3] Gupta , V. , Karnick , H. , Bansal , A. , Jhala , P. : Product classification in e-commerce using distributional semantics . arXiv preprint arXiv:1606.06083 ( 2016 )

[4] Jiao , X. , Yin , Y. , Shang , L. , Jiang , X. , Chen , X. , Li , L. , Wang , F. , Liu , Q. : Tinybert: Distilling bert for natural language understanding . arXiv preprint arXiv: 1909 . 10351 ( 2019 )

[5] Kozareva , Z. : Everyone likes shopping! multi-class product categorization for ecommerce . In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies . pp. 1329 - 1333 ( 2015 )

[6] Xia , Y. , Levine , A. , Das , P. , Di Fabbrizio , G. , Shinzato , K. , Datta , A. : Large-scale categorization of japanese product titles using neural attention models . In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2 , Short Papers . pp. 663 - 668 ( 2017 )

[7] Zahera , H.M. , Elgendy , I.A. , Jalota , R. , Sherif , M.A. : Fine-tuned bert model for multi-label tweets classification . In: TREC ( 2019 )

[8] Zhang , Z. , Paramita , M. : Product classification using microdata annotations . In: International Semantic Web Conference . pp. 716 - 732 . Springer ( 2019 )