E-Commerce companies face a number of challenges in return requests. Claims of missing-items is one such challenge, where customer claims that main product is missing from shipment through return comments. It is observed that dominant part of such claims are inadvertent given the limited literacy of customers. Some of them have fraud intent. At Flipkart, such claims are evaluated manually to examine whether the comment relates to missing item. Classification of the claim intent automatically saves human bandwidth and provides good customer experience by reducing the turn around time to customers. However, this is challenging as comments are replete with spell variations, non-English vernacular words, and are often incomplete and short. This is compounded by noisy labeling of such comments due to human bias and manual errors. To classify the claim intent, we apply conventional as well as deep learning methods. To handle label noise, we employed stateof-the-art noise-aware techniques, which fail to perform due to pattern specific label noise. Motivated by the wide pattern specific label noise, we encode domain heuristics as labeling functions (LFs) which label subsets of the data. However, LFs may conflict and prone to noise. We address the conflict by defining a conflict-score to rank the LFs. Proposed method of noise handling with LFs out performs all the state-of-the-art noise-aware baselines.
E-commerce companies face a large number of return requests of various types (reason-codes). Missing-item is one such reason-code where customer claims that main product is missing from shipment. ∗Equal Contributions Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).
SIGIR 2018 eCom, July 2018, Ann Arbor, Michigan, USA © 2018 Copyright held by the owner/author(s).
ACM ISBN 123-4567-24-567/08/06. https://doi.org/10.475/123_4 Unlike a usual return, product pick up from customer is avoided in a missing-item return. Example of a missing-item return is customer ordered a handset and received a stone in place of the handset. A confirmed case of missing item results in loss to the company since there is a definite fraud with one of the stakeholders such as buyer, seller or delivery team. Hence, a careful scrutiny is necessary before the approval of any missing-item returns.
A missing-item return request is generated broadly for two reasons. The first can be due to definite fraud with one of the stakeholders. Secondly, given limited literacy levels of customers, it is observed that the claims do not always refer to missing-item but inadvertently claimed as missing-item. For example, a missing-item return with customer comment ‘I did not like the item.’ clearly indicates that the return belong to a return category other than missingitem. Because here customer received the main product. Hence, the return should be cancelled. We call this a comment-mismatch (customer’s comment does not match with the return reason-code). On the other hand, a missing-item return with customer comment ‘I ordered a phone but received an empty box.’ clearly indicates that the return belong to the missing-item category. This return should be approved. This is referred by non-comment-mismatch (customer’s comment matches with the return reason-code).
At Flipkart, a dedicated operation team assesses the compatibility of customers’ comments on missing-item return with the missingitem reason-code. Each missing-item return – passes through this process and – is rejected when the return comment is incompatible with the missing-item reason-code, otherwise approved. This process wastes lot of human bandwidth and is prone manual mistakes. Moreover, it hampers the customer experience due to the lag between the return placement time and its status update to the customer. Hence, we want to automate this process. In terms of Machine Learning problem, given a missing-item return comment, we want to predict whether it is a comment-mismatch (positive class) or non-comment-mismatch (negative class).
Customer comments are generally very noisy mainly because
of three reasons: a) spelling mistakes: empty is misspelled in
comment ‘Emety box’, b) usage of regional languages: comment ‘Galt
order ho gya h’, which means I ordered wrong product, uses Hindi
language, and c) varied comment length: token counts in a
comment ranges [
Also, the manual label generation process, which marks a
missingitem return comment as comment-mismatch/non-comment-mismatch,
is very noisy. The sources of noise are manual error, human bias,
and lack of well calibrated operation team. The label noise varies
depending on the patterns and is non-iid. This is clearly visible by
the fact that the overall label noise is ∼ 15% and a specific pattern
‘I ordered x quantity of an item but received y quantity’ has 50%
noise. Due to this reason, the state-of-the-art noise-aware [
The pattern specific noise variation and high noise on certain
patterns motivate us to use noise correction based on domain heuristics.
Like data programming[
In a typical data programming setting, only data points are available and LFs are created to generate labels. The LFs may conflict on certain data points and have varying error rates. To handle it, data programming defines a generative process over the LFs to learn the correctness probability of each LF on each data instance. This information is then used to fit a noise-aware classifier.
The key diference between our setting and the data programming is the availability of noisy labels (we call it as true LF) in our case. Unlike data programming, we would like introduce less noisy LFs compared to the true LF. Due to all these reasons, instead of applying data programming directly, here resort to a simple method. and promising methods to correct label noise using LFs, and leave out the exploration of data programming as future work.
We define multiple LFs to alter the noisy labels in our dataset. However, applying them directly to flip the noisy labels is impossible because the LFs conflict with each other - same data instance is labeled as positive by a LF and negative by another LF. This requires us to generate a ranked list of LFs. To do that, we define a conflict score, which captures how less a LF conflicts with other LFs. Then, the ranked LFs are applied to alter the true labels. In case of conflict between LFs, the LF with the least conflict score is chosen to alter the noisy label. We show that proposed method of noise handling with LFs out performs all the state-of-the-art noise-aware baselines as well as vanilla baselines.
We integrate the following aspects in the paper.
• Explanation of a real world problem and its challenges • Exploratory data analysis and feature engineering • State-of-the-art baselines - xgboost and LSTM and their noise-aware variants • Noise handling with labeling functions • Proposed conflict-score to handle conflicts • Shown superior performance of the proposed method Section 2 contains insights into data. Related work, feature engineering, modeling, experimentation, and conclusion are described in Section 3, 4, 5, 6, and 7, respectively. 2 2.1
The data in our study comes from the customer comments on
missing-item return requests. We use one year of customer
comments: April 2017 - Mar 2018. We consider a comment-mismatch as
positive label and a not-comment-mismatch as negative label. Table
1 describes the dataset statistics. We can observe ∼ 41% are positive
labels. We count the number of words in a comment by tokenizing
it. Table 1 also shows that the word count of comments ranges
[
Table 2 shows example of customer comments with diferent word count. Interestingly, we can observe the presence of spell error - ‘My mistek’ and regional language - ‘Sir khali box mila h’ (Hindi). Often, comments are very noisy and do not adhere to grammar rules - ‘Not working Good; tow time same product bye mistack accepted so remove it’. To provide more insight on the data, we show a word cloud of our dataset in Figure 2. Some of the important keywords are cx (customer), product, mobile, box, missing, and empty. The cx words occurs in the comments when customer calls customer care to place the return request. Recall, non-comment-mismatch implies a genuine return request of type missing-item and comment-mismatch implies a return request anything other than missing-item type. Given a missing item return comment, our operation team at Flipkart mark it as either commentmismatch (positive) or non-comment-mismatch (negative). This label generation process is very noisy due to manual error, human bias, and lack of well calibrated operation team. Table 3 shows example of noisy labels for comments with diferent word count. Actual label and true labels represents the label in the dataset and the expected label, respectively. We show noisy labels from one, two, four, and eight word count comments. For each word count, we show one comment whose label should be positive but marked as negative and one comment whose label should be negative but marked as positive. 2.3
To estimate the overall noise in our dataset, we manually relabeled 3k randomly chosen examples and consider it as test dataset. We calculate the noise by considering mismatch between the actual label and true label on the test dataset. The overall noise is estimated to be 15.65%. Also, the positive and negative classes have 10.06% and 19.45% noise, respectively. 3
Preprocessing is an important step for text classification. Two
important blocks [
Then, the preprocessed text is converted into feature vectors. One
of the widely used model for feature generation is the bag-of-words
(BOW) model [
Support vector machines (SVMs) [
Recently deep learning algorithms[
Noisy label can be handled broadly by three approaches [
As creating labeled training data is dificult and time consuming,
many approaches are developed to generate training data
automatically, such as distant supervision [
In this Section, we will discuss all the hand crafted features which are used for conventional Machine Learning models. 4.1
We construct nine meta features as shown in Figure 3. Word count, char count, alpha count, digit count, and non-alphanumeric count compute the number of words, characters, alphabets, digits, and non-alphanumeric characters in a comment. To show the discriminating power of each meta feature, in Figure 3, we show histogram of each feature w.r.t. both positive and negative class. In each subplot, the blue and green histogram represents the distribution for negative and positive class, respectively.
In each subplot, the blue distribution is right shifted. It implies that in general comments from the negative class are longer, have more alphabets, digits and non-alphanumeric characters compare to the comments from the positive class. This is explained by the fact that short comments lack descriptive ability, and thus have higher chance of being comment-mismatch. Interestingly, unique character count and unique alphabet count are the most discriminating features - there is a clear separation between the distribution of positive and negative class for these two features. 4.2
We observed that our dataset is very noisy. For example, a keyword
like ‘missing’ has numerous variations - missing, misssing, misisng,
missig, etc. Moreover, there are semantically similar words, such as
‘empty’ and ‘khali’ both means empty. Also, customers use regional
language, such as hindi (‘mujhe product nahi mila’). To handle
these complexities, we train a word2vec model with 200 dimension
on one year customer’s return comments data from all the return
reason-codes. The number of comments are O(10M). We train the
word2vec by considering words which occurs at-least 25 times in
our corpus. For tokenization, we use Gensim [
Table 4 shows similar words from the word2vec model for four keywords - missing, mobile, product, and ordered. We can observe that the word2vec model is able to capture spelling variations. It is also able to capture semantically similar word, such as missing and empty, mobile and handset, device and phone, and ordered and purchased. Moreover, the regional language variation is also captured, such as missing and khali (hindi), mobile and fone (hindi), and missing and illa (tamil).
4.2.1 Sum of Word2vec Features: We sum the individual 200dimensional embedding vector for each word in a comment and use it as the final feature in model. To handle out-of-vocabulary word, we fall back to the 200-dimensional zero vector.
4.2.2 Weighted Word2vec Features: We fit a tfidf model on the training data. Then, we calculate a weighted sum of the individual 200-dimensional embedding vector for each word in a comment. Where weight of individual word embedding is assigned from the tfidf score of the word. 4.3
Each comment in the training dataset is preprocessed with Gensim preprocessing. Then, a bag-of-words (BOW) model is trained with 5k vocabulary size and with English stop words removal. 5 5.1
We tried multiple conventional Machine Learning algorithms widely used for text classification, such as SVM, naive Bayes, logistic regression, random forest, and xgboost. In our data, around 50% of the comments are long and we found that often long comments have sequential information, such as ‘customer ordered 2 items but received only 1’. To capture the sequential information, we experimented with diferent RNN based models, such as RNN, LSTM, Multi-layer LSTM, and Bi-directional LSTM (BLSTM). In below, we only describe the best performing models from each of this approach.
5.1.1 Xgboost: A xgboost model is trained with all the features described in Section 4. The model parameter is tuned by grid search. While performing the grid search, we restrict the max depth of individual tree to 8 and max number of estimators to 500. The best parameters are chosen using 5-fold cross-validation.
5.1.2 BLSTM:. We used BLSTM which avoids feature engineering. In BLSTM, both past and future information are preserved using two hidden states, and it helps to learn the context better. We experimented with word2vec pretrained embedding from the word2vec model as well as learning the embedding from scratch in the network itself. We tune the number of neurons in the BLSTM. We also experimented by adding fully connected relu layes in the network before the output layer. The best parameters are tuned based on a validation set. 5.2
We tried two approaches to handle noisy labels: 1) data cleansing method and 2) label noise-tolerant algorithm.
5.2.1 Data Cleansing Method: The goal of such methods is to
iflter out data points which appears to be mislabeled. Here, we
apply a model prediction based filtering method [
In this filtering approach, we divide the training dataset into k-folds (five-folds). We train a xgboost model (with the best parameters found in 5.1.1) on k-1 folds and apply it to predict labels for k-th fold. This process is repeated k times to get labels for the entire training dataset. Then we filter out instances with disagreement between the actual label and the predicted label. On the filtered training data, a xgboost model (with same parameter) is trained which forms the final model. We refer this method as xgboost+filtering.
5.2.2 Label Noise-Tolerant Algorithm: In label-noise tolerant
algorithm, noise is handled explicitly in the modeling step. Here, a
probabilistic neural-network based framework [
Assuming the non-linear function applied on an input x be h = h(x ), the true label y is modeled by: p(y = i |x ; w) = Ík l =1 exp(uTl h + bl ) exp(uTi h + bi ) , i = 1, ..., k (1)
Where k is the number classes and w is the network parameterset (including the softmax layer). Next a softmax output layer is added to predict the noisy label z based on both the true label and the input features: (2) (3) (4) p(z = j |y = i, x ; wnoise) = Ík l =1 exp(uTil h + bil ) exp(uTil h + bil ) p(z = j |x ) = p(z = j |y = i, x )p(y = i |x )
Where, wnoise represents parameters in the second softmax layer. Given n training data points with feature vectors x1, x2, ..., xn with corresponding labels z1, z2, ..., zn and true labels y1, y2, ..., yn , the log likelihood term of the model parameters is written as: Õ log p(zt |xt ), t = 1, ..., n t
This modeling approach is named as c-model and learned using a neural-network training. For our experiment, the h function is considered to be a BLSTM, with the same parameter as in Section 5.1.2. This model is referred as BLSTM+noise-aware. def lambdapartial(x ): return 1 if # of positive integer tokens in x >= 2 else 0 def lambdasoap(x ): return -1 if IsTokenPresent(soap, x ) else 0 5.3
We encode domain heuristics as labeling functions (LFs) [
We consider that the actual labels came from a LF named λt rue , which has 100% coverage. As introduction of a more noisy LF than λt rue will increase overall noise in the dataset, unlike data programming, we introduce less noisy LFs than λt rue . Lets consider λ denotes the m newly created LFs - {λi }im=1, each of which looks at the domain specific patterns to label data points.
A specific pattern is observed in the comments - ‘I ordered x quantity of an item but received y quantity’. This is partial delivery, as the customer has received part of the order, and should be marked as comment-mismatch. With this pattern, a LF lambdapar t ial is defined in Table 5. We also observed that customer often writes they have received soap instead of a mobile phone. This is a genuine missing-item request, and should be marked as non-commentmismatch. With this pattern, a LF lambdasoap is also defined in Table 5. The IsTokenPresent function returns true when soap is one of the token of comment x . Table 6 describes the statistics of all the LFs. Where coverage represents the percentage of instances with at least one label, conflict represents the percentage of instances with conflicting labels, and overlap depicts the percentage of instances with more than one labels.
To deep dive, we show conflict and overlap count in Table 7 and Table 8, respectively. Where a cell (i, j) in Table 7 defines the number data instances where (λi == 1 and λj == −1) or (λi == −1 and λj == 1). Similarly, a cell (i, j) in Table 8 defines the number data instances where (λi == 1 and λj == 1) or (λi == −1 and λj == −1).
As LFs conflicts with each other applying them directly to flip the true labels is impossible. This requires us to generate a ranked list of LFs. To do that, for each λi ∈ λ, we define a conflict-score as below. Note, conflict count of λi is calculated by summing the number of conflict between λi and each rule from λ − λi . # of LFs 17
100
40
Conflict (%) 9 true partial phone missing soap true partial phone missing soap # unique conflicts of λi with other LFs
Coverage of λi
Intuitively, the cf_score captures how much a LF conflicts with other LFs. With this score, Algorithm 1 denoise the training data. λsor t ed contains the sorted list of m newly created rules in ascending order w.r.t the cf_scoreλi .
Algorithm 1 Label Denoising with Labeling Functions (LFs)
After denoising the training data with Algorithm 1, we apply the xgboost and BLSTM model on the denoised training data. This two approaches are named as xgboost+best-sequence and BLSTM+bestsequence.
The complete data consists of O(100k) instances out of which randomly chosen 3k instances forms the test data and rest forms the train data. The test dataset is manually relabeled to generate clean labels. Table 9 shows train and test data statistics. Note, the test dataset size is small because manual relabeling is time consuming.
We compare performance of xgboost+best-sequence and BLSTM+bestsequence against the baselines - xgboost, BLSTM, xgboost+filtering, and BLSTM+noise-aware. Moreover, to showcase the benefits of the conflict handling with the cf_score, we compare the performance of xgboost+best-sequence and BLSTM+best-sequence with xgboost+random-sequence and BLSTM+random-sequence. For randomsequence, λsor t ed consists of a of random permutation of m newly created rules. Model performance varies for diferent random sequences. Hence, for both xgboost+random-sequence and BLSTM+bestrandom, we repeat experiments 10 times with diferent random permutation of λ and report the mean and standard deviation. For all the models, we fix random-state to 2018. As our dataset is well balanced, we use accuracy as the evaluation metric.
Table 10 shows the performance comparison between xgboost, BLSTM, and their noise-aware variants. BLSTM is performing best with an accuracy of 87.43. This proves that using sequence information indeed benefits on our dataset. BLSTM and xgboost are performing better than BLSTM+noise-aware and xgboost+filtering, respectively. We can observe that the state-of-the-art noise-aware algorithms are hurting the performance. We think that the reason for such a performance degradation is due to the wide pattern specific noise variation. is quite low, and still BLSTM+best-sequence beats the BLSTM+randomsequence, again proving conflict handling with cf_score helps.
Overall, BLSTM+best-sequence performs best, proving the efifcacy of our proposed approach. Best-sequence provide benefits over random-sequence proving the benefits of conflict handling among LFs by cf_score. Sequence model BLSTM is able to provide benefits over xgboost. We were able to improve the accuracy from 86.90% to 90.04% with the help of BLSTM, LFs, and cf_score.
We discussed an important problem of classifying missing-item return comments into comment-mismatch/non-comment-mismatch. We highlighted the data and noise related challenges in both comments and labels. We have experimented with the state-of-the-art Machine Learning and Deep Learning methods as well as their noise-aware variants. We have proposed a simple method with labeling functions (LFs) to denoise the training dataset. A conflictscore is defined to handle the conflicts between LFs. Empirically, we have shown eficacy of our approach over the baselines.
As future work, we intend to explore the complete data programming framework to handle noisy labels.