When an expectation does not meet reality in a real-world situation, the difference is usually expressed and communicated via an act which is complaint. Customers often post reviews on the products or services they purchase on the retailer websites and different social media platforms, and the reviews may reflect complaints about the products or services. Automatic recognition of customers' complaints on products or services that they purchase can be crucial for the organisations, multinationals and online retailers since they can exploit this information to fulfil the customers' expectations including managing and resolving the complaints. In this work, we present the supervised and semi-supervised learning strategies to identify users' complaints from the language they use to post their reviews. In other words, we automatically identify complaints from the opinionated texts (reviews) about products posted in Hindi. For this, first we automatically crawled the Hindi reviews on different products from the the websites of the retail giant Amazon and the popular social media platform YouTube, and prepared a gold-standard data set via a systematic manual annotation process. We use state-ofthe-art classification algorithms for the complaints identification task and our classification models achieve reasonable classification accuracy (F1 = 68.38%) on a gold-standard evaluation test set.
Text classification is an active field of natural language processing (NLP) and data mining. Almost all online retailers allow users to freely express their opinions and thoughts on products via their websites and the relevant social media platforms. The customers who intend to purchase a product may take purchasing decisions based on the reviews of the product. Accordingly, the commercial and retail companies considers product reviews as an important source of information, and could exploit this information to build their marketing tool and strategy, and to resolve any issues in relation to the product. This could also benefit the users with the suggestions on the quality of the products or services that they want to purchase. As for the number of reviews of a product posted by the users, they could range from several hundreds to tens of thousands. The e-commerce companies and online retailers want to identify complaints given the reviews of a product for their own benefit. Likewise, the customers who want to buy a product or service may need such information while avoiding having to go through thousands of reviews about the product.
In this context, Gupta et al. [5] identified the relationship between users' purchase intent from their social media forums such as Quora 1 and Yahoo! Answers. 2 They primarily carried out text analysis to detect purchase intent from user-generated content (UGC). Wang et al. [13] investigated the problem of identifying purchase intent. In particular, the authors proposed a graph-based learning approach to identify intent tweets and classify them into six categories. For this, they retrieved tweets with a bootstrap method, with using a list of seed intent-indicators (e.g. 'want to'), and manually created training examples from the collected tweets. Haque et al. [6] extends the work of Wang et al. [13] while increasing the coverage of the purchase intent indicators with the distributed vector representation of words using the continuous skip-gram model [10].
Recently, Preotiuc-Pietro et al. [11] automatically identify complaints from the tweets posted by the social media users and the potential customers. To the best of our knowledge, the most relevant works to ours come from Preotiuc-Pietro et al. [11]. In fact, to a certain extent, our proposed methods can be viewed as the extension of Preotiuc-Pietro et al. [11] with mainly the following additions as far as this task is concerned: (i) we considered product reviews instead of tweets as in Preotiuc-Pietro et al. [11] and (ii) we explore a resourcepoor and less-explored language, Hindi, for our investigation, (iii) we applied a state-of-the-art sampling strategy [2] in order to encounter class imbalance problem in the training data, and (iv) we explore applying a semi-supervised classification algorithm in the complaint identification task. Moreover, one of the key contributions to this work is creation of the gold-standard dataset in Hindi.
The remainder of the paper is organised as follows. In Section 2, we detail how we semi-automatically created training data for our experiments. In Section 3, we present our experimental methodology and setups. Section 4 presents our evaluation results, with some discussions. Section 5 concludes and provides avenues for further work.
This section details the creation of training data that has been used in this task.
To the best of our knowledge, there is no existing (freely available) annotated review data (complaint and non-complaint) for Hindi. For this task we needed an annotated review dataset for Hindi. In order to create an annotated review dataset for Hindi, we first collected Hindi reviews posted online. The reviews were taken from two different sources: (i) the websites of the retail giant Amazon, and (ii) YouTube. The users usually post their Hindi reviews on these two platforms.
In order to collect the reviews from Amazon, we used amazon-reviews-scraper Python library 3 which takes a product name as input and provides the reviews about the product across the different languages. Similarly, in order to collect the reviews from YouTube, we used youtube-comment-downloader Python library. 4This script provided us reviews on the products across the different languages.
In order to remove noise (e.g. HTML tags, special characters) from reviews, we applied a number cleaning scripts including a language identifier. 5 We also removed emojis from the review texts. Each of the collected clean reviews is manually tagged with a particular category, namely complaint or non-complaint. The annotation scheme and results are presented in the next section.
The annotation task is performed in-house by our three undergraduate students. Each reviews is presented to the annotators. The annotators are expected to answer two questions for a given review. While the first question is related to the decision as to whether a review is complaints or non-complaints, the motivation behind the second question is to collect a more fine-grained (book, phone, tv etc.) gold standard dataset. 6To have a concrete idea about the agreement between annotators, we calculated the majority class for each review in our dataset. A review belongs to a majority class k if the most frequent annotation for the review was selected by at least k annotators. As a consequence, a large percentage of review belonging to high majority classes are symptomatic of good inter-annotator agreement. Similar to earlier studies, we consider all annotations with a majority class greater than 2 as reliable. In this case, for the complaints or non-complaints annotation scheme, over 88% of the review were annotated identically by the majority of annotators, while for fine-grained annotation scheme,7 over 85% of the annotations fell into this case. As such, we can be confident that the annotation process was successful and the dataset is reliable. A sample of our annotated dataset is presented in Table 1, and statistics about the dataset are presented in the next section.
We report the statistics of the our gold standard data set in Table 2. The table shows the numbers of the complaint and non-complaint reviews which are listed according to different product types. 3 Methodology
Nowadays, recurrent neural network (RNN), in particular with long-short term memory (LSTM) [7] hidden units, has been proved to be an effective model for many classification tasks in NLP, e.g. sentiment analysis [14], text classification [8,15]. RNN is an extension of the feed-forward neural network (NN), which has the gradient vanishing or exploding problems. LSTM deals with the exploding and vanishing gradient problems of RNN. An RNN composed of LSTM hidden units is often called an LSTM network. A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. More formally, each cell in LSTM can be computed as follows:
where
biases of LSTM, which need to be learned during training, parameterising the transformations of the input, forget and output gates, respectively. σ is the sigmoid function, and ⊙ stands for element-wise multiplication. x t includes the inputs of LSTM cell unit. The vector of hidden layer is h t . The final hidden vector h N represents the whole input review, which is passed to softmax layer after linearising it into a vector whose length is equal to the number of class labels. In our work, the set of class labels includes complaint and non-complaint categories.
We used fastText [8], a fast and efficient classifier, in our task. fastText is often on par with the state-of-the-art deep learning classifiers in terms of accuracy. It is also found to be faster for training and evaluation. fastText has many advance features, e.g. hierarchical softmax, hidden states shared among features and classes, and bag of n-grams features.
We also used a semi-supervised classification algorithm, i.e. random walk on graph. A graph, G is a pair of vertices V and edges E. We use a weighted graph for the classification. A weighted graph is a graph where each edge, E i , is linked to a numeric value W ij where i, j ∈ V . For our approach, it is necessary that
Random walk is based on the concept of randomly determined processes. It considers a transition probability of the random walker that it will reach from a vertex V i to the other vertex V j with each step being either +1 or -1 with equal probability. As for implementing random walk for this task, we follow the approach and python implementation of the algorithm described in [12].
Furthermore, we compare the deep learning model (the LSTM network), fastText and semi-supervised classification model (random walk) presented above with the classical supervised classification models. We employ the following classical supervised classification techniques in our task:
-Logistic Regression (LR) -Decision Tree (DT) -Random Forest (RF) -Naïve Bayes (NB) -Support Vector Machine (SVM) These classical learning models (LR, DT, RF, NB and SVM) can be viewed as the baselines in this task. Thus, we obtain a comparative overview on the performances of different supervised and semi-supervised classification models including the LSTM network.
In order to build LR, DT, RF and NB classification models, we use the wellknown scikit-learn machine learning library, 8 and performed all the experiments with default parameters set by scikit-learn. As for the representation space, each review was represented as a vector of word unigrams weighted by their frequency in the reviews.
For the classifiers based on the neural networks, we use a 300-Dimensional word embeddings from fastText. We use sigmoid activation function with Adam optimizer [9] and binary cross entropy loss function. The size of input layer of the NN is 300. We employ layer normalisation [1] in the model. Dropout [4] between layers is set to 0.10. The size of embedding and hidden layers are 300. The learning-rate is set to 0.0003, and the training examples were reshuffled for each epoch.
As for the random walk algorithm, we use the similarity between the sentences as the transition probability. Again, we create a network with the training examples. We take a set of labelled vertices L and unlabelled vertices U , where (L, U ) ⊆ G (G : set of all the labels). The random walk algorithm considers the vertices of the network as the states of a Markov chain. We calculate the similarity measure between the data points with the Dice coefficient (DC) [3] and reach to the final decision according to the probability. The DC measure has been widely used for evaluating the degree of association between words.
We recall Tables 2 and 3 where we can see the presence of class imbalance in the training data. In order to encounter the class imbalance problem in our training data, we followed Chawla et al. [2] who presented a combination of oversampling of minority class with under-sampling of majority class. This is also called SMOTE which is a kind of synthetic sampling technique and found to be effective in our problem.
We evaluate the performance our classifiers on the gold-standard test set (cf. Table 3) and report the evaluation results in this section. In order to measure classifier's accuracy on the test set, we use three widely-used evaluation metrics: precision, recall and F 1 measures. The results obtained are reported in Table 4. The first five rows of Table 4 represent our baseline classifiers (i.e. the classical supervised classification models). The next row represents the random walk algorithm. We see from the table that these classifiers performs below par and SVM is the best-performing method among them (SVM: a 47.94 F 1 score)) according to the F 1 scores.
As for NN-based classifiers, the LSTM network trained on fastText embeddings performed reasonable as we see from Table 4 that it produces a moderate F 1 score (68.38 F 1 ) on the test set. The fastText classifier also performs reasonably well; however, it could not surpass the LSTM network.
In Table 5, we show the F 1 scores of the best-performing classifiers of the complaint identification task presented in [11]. As can be seen from Table 5, 4). Note that, unlike us, Preotiuc-Pietro et al. [11] carried out experiments on English and the data sets including the evaluation test set are different too. Moreover, the nature of data is also different to us. Preotiuc-Pietro et al. [11] focused on identifying complaints on tweets and we focused on identifying the same on the product reviews posted on Amazon and YouTube. Naturally, the scores presented in Preotiuc-Pietro et al. [11] cannot be directly compared to those presented in this paper. Given the fact that English is a high resource language and there are a plenty of linguistic tools and resources freely available in English, Preotiuc-Pietro et al. [11] exploited many linguistic resources, models and tools in their task for feature engineering, e.g. sentiment and emotion analysis, temporal model, part-of-speech information. In Hindi, it is difficult to obtain such resources and standard tools, and many such tools are not available to use. Moreover, unlike English, Hindi is a morphological complex and highly inflected language. Therefore, we believe that identifying complaints on Hindi texts is to be more challenging in comparison to that in English or other high resource languages.
In this paper, we presented supervised and semi-supervised learning models to identify customers' complaints from the review data in Hindi, a low-resource, less-explored, and morphological rich highly inflected Indic language. There is no publicly available gold-standard training data as far as identifying complaints from user generated Hindi texts about products is concerned. Accordingly, we semiautomatically created a gold-standard dataset for complaints identification on Hindi. We conducted our experiments with state-of-the-art LSTM classifier and classical supervised classification models. With our LSTM classifier, we achieved a competent accuracy (a F 1 score of 68.38) on the gold-standard evaluation test set. There are limited linguistic resources and tools freely available for research in Hindi in comparison to many high-resource languages including English. We believe that this work would add an additional value to the social media analytics research in low-resource scenarios.
In future, we intend to test our method on different low-resource and non-English languages. We also plan to investigate applying more sophisticated and linguistic features in our model, e.g. part-of-speech information. We removed emojis from reviews at the time data prereprocessing. Since emojis may capture polarity and encode information regarding users' experiences and complaints, in future, we aim to carry out experiments while keeping emojis in the reviews. Our classifiers were trained on the data which is a mixture of reviews from different product types. We intend to train classifiers on reviews from specific product type, i.e. book or phone reviews. By this, we can compare the classifiers trained on the data from individual product type to the one trained on the data consisting reviews from all product types.