Individuals and organizations rely heavily on social media these days for consumer reviews in their decision-making on purchases. However, for personal gains such as profit or fame, people post fake reviews to promote or demote certain target products as well as to deceive the reader. To get genuine user experiences and opinions, there is a need to detect such spam or fake reviews. This paper presents a study that aims to detect truthful, useful reviews and ranks them. An effective supervised learning technique is proposed to detect truthful and useful reviews and rank them, using a 'deceptive' classifier, 'useful' classifier, and a 'ranking' model respectively. Deceptive and nonuseful consumer reviews from online review communities such as amazon.com and Epinions.com are used. The proposed method first uses the 'deceptive' classifier to find truthful reviews followed by the 'useful' classifier to find whether a review is useful or not. Manually labeling individual reviews is very difficult and time consuming. We incorporate a dictionary that makes it easy to label reviews. We present the experimental results of our proposed approach using our dictionary with 'deceptive' classifier and 'useful' classifier.
Nowadays, consumers looking to buy a product increasingly rely on user-generated online reviews to make or reverse their purchase decisions. Positive reviews of a product greatly influence the person’s decision to buy the product. However, if one sees many negative reviews, he/she will most likely choose a different product. The outcome of positive reviews gives significant profit and advertizing for the seller and their organization. This in turn creates a market for incentivizing opinion spam. This has resulted in more and more people trying to game the system by writing fake reviews to harm or promote some products or services. A fake review means that it is either a positive review written by the business owners themselves (or people they contract to write reviews) or a negative review written by a business’s competitors. Those fake reviews try to deliberately mislead readers by giving fake reviews to some entities (e.g. products) in order to promote them or to damage their reputation.
Opinion spamming refers to writing fake reviews that try to deliberately mislead
human readers. The focus of spam research in the context of online reviews has been
primarily on detection. Cornell University has developed a model to spot fake,
nonfake review for hotels [
The rest of the document is organized as follows: Section 2 presents related work. Background material related to this project is presented in Section 3. Our proposed approach and its implementation is presented in Section 4. Section 5 presents the experiments and analysis followed by conclusions and future work in Section 6.
Web spam and email spam have been investigated extensively. The objective of
Web spam is to make search engines rank the target pages high in order to attract
people to visit these pages. Web spam can be categorized into two main types: content
spam and link spam. Link spam is spam on hyperlinks that are placed between pages,
which does not exist in reviews as usually there are no links within them. Content
spam tries to add irrelevant or remotely relevant words in target pages to fool search
engines to rank the target pages high. Another related research is email spam [
A preliminary study was reported in [
Some of the related research addresses the problem of review spammer detection, or
finding users who are the source of spam reviews. Reviews usually come with ratings.
Detecting unfair ratings has been studied in several works including [
Review helpfulness prediction is closely related to review spam detection described
in above. A helpful review is one that is informative and useful to the readers. The
purpose of predicting review helpfulness is to faciliate review sites to provide feedback
to the review contributors and to help readers choose and read high quality reviews. A
classification approach to solving helpfulness prediction using review content and
meta-data features was developed in [
Amazon.com allows users to vote if a review is helpful or not. These helpfulness
votes are manually assigned and are thus subjective and possibly abused.
DanescuNiculescu-Mizil et. al found that a strong correlation between proportion of helpful
votes of reviews and the deviation of the review ratings from the average ratings of
products [
Detecting spam and predicting helpfulness are two separate problems since notuseful reviews are not necessarily fake. A poorly written review may be not-useful but is not fake. Spam reviews usually target specific products while not-useful votes may be given to any products. Given the motive driven nature of spamming activities, review spam detection will therefore require an approach different from not-useful review detection. Our proposed technique aims to detect truthful, useful reviews and provide a ranking of the reviews.
A computer system learns from training data that represents some “past experiences” of an application domain. In this section, we briefly describe the various classification methods used in order to categorize reviews into deceptive, truthful and useful, not-useful. Classification involves labeling of the data (observations, measurements) with pre-defined classes. We have used three supervised learning algorithms: Support Vector Machine, Naïve Bayes, and K-Nearest Neighbor.
Support Vector Machines [
The Naïve Bayes classifier [
The Naïve Bayes classifier will consider each of these attributes separately when classifying a new instance. So, when checking to see if the new instance is an elephant, the Naïve Bayes classifier will not check whether it has a trunk and has huge tusks and is large. Rather, it will separately check whether the new instance has a trunk, whether it has tusks, whether it is large, etc. It works under the assumption that one attribute works independently of the other attributes contained by the sample. In our experiments, it is seen that the Naïve Bayes classifier shows a drop in performance, when compared with K-NN and Support Vector Machines.
The K-nearest neighbor [
The Community Edition of RapidMiner [
Rapid Analytics [
In this section, we present our approach that includes (i) preparing a custom dictionary to label reviews as truthful or deceptive; (ii) the ‘deceptive’ classifier to predict testing data as a deceptive or truthful (iii) PHP script to label review as useful or not-useful; (iv) ‘useful’ classifier to predict testing data is either useful or not-useful; (v) “ranking” model to rank the reviews.
In general, spam review detection can be regarded as a classification problem with two classes, fake and non-fake. Machine learning models can be built to classify each review as deceptive or truthful. To build a classification model, we need labeled training examples of both classes. There was no labeled dataset for product opinion spam prior to this project. Recognizing whether a review is a deceptive opinion spam is extremely difficult if it has to be done manually reading the review because one can carefully craft a spam review which is just like any other genuine review. We prepared the dictionary for fake and non-fake reviews by adding knowledge from the dataset which is available on http://www.cs.uic.edu/~liub/FBS/CustomerReviewData.zip and using Cornell model. To prepare dictionary we passed reviews through Cornell model that tokenizes words based on specialized characters (like space, full stop, exclamation, question mark etc.) in each sentence and puts it into any one of the appropriate category along with weight like high positive (+3), moderate positive (+2), low positive (+1), neutral (0), high negative (-3), moderate negative (-2) or low negative (1). Some of words from neutral category of Cornell model are important for our domain and we placed those important words into positive or negative category with weight from http://www.cs.uic.edu/~liub/FBS/CustomerReviewData. After putting each word of each sentence into any one of six categories along with weight, we calculated final weight for each unique word based on our formula as follows: Weight of each word = ∗ ℎ
More precisely we can say that, Weight of each non-fake word = ∗ 3 + ∗ 2 + ∗ 1
where is the count of a particular word in high positve category, is the count of a particular word in medium positve category, is the count of a particular word in low positve category.
Weight of each fake word = ∗ −3 + ∗ −2 + ∗ −1
where is the count of a particular word in high negative category, is the count of a particular word in medium negative category is the count of a particular word in low negative category \Using above formula, we prepared two wordlists for fake and non-fake reviews along with their corresponding weights. We called that dictionary through a php script to label the review as fake or non-fake based on final summation of all words in each review. If final summation of weight for fake and non-fake words of a review are positive then it is labeled as “non-fake” otherwise it is labeled as “fake”.
The first component of the framework is the ‘deception’ classifier, which predicts whether each unlabeled review is non-fake (truthful) or fake (deceptive). As mentioned previously, we labeled training review as deceptive or truthful, so that we can train ‘deception’ classifiers using a supervised learning algorithm. We tried three supervised learning algorithms: support vector machine (SVM), K-NN, Naive Bayes to classify product review using two pre-classified training sets: deceptive and truthful. Our work has shown that SVM trained and performs well in deception detection tasks. We found that SVM creates a hyper plane to best separate the two planes and it outperforms the other two classifiers. We trained SVM classifiers using software package of RapidMiner tool. Results of evaluation are presented in the next section.
In general, useful review detection can be regarded as a classification problem with two classes, useful and not-useful. Machine learning models can be built to classify each review as useful or not-useful. To build a classification model, we need labeled training examples of both useful and not-useful class. There was no labeled dataset for product opinions as useful and not-useful at the time of project (to the best of our knowledge). However, to recognize review is useful or not, we considered reader’s rating on consumer’s review. Using php we labeled reviews as useful if reader’s rating is greater than 40% or as a not-useful review, if reader’s rating is less than 40%.
The second component of the framework is ‘useful’ classifier, which predicts whether each unlabeled review is Useful or Not-Useful. As mentioned above, we labeled training data, so that we can train ‘useful’ classifiers using a supervised learning algorithm. We tried different supervised algorithms like Naïve Bayes, K-NN, and SVM. Our work has shown that SVM trained and performs well in useful or notuseful detection tasks as compared to other algorithms. We train SVM classifiers using the software package of RapidMiner tool. Results of the evaluation are presented in the next section.
The last component of the framework is the ‘Ranking’ Model. This model takes the output from the ‘deceptive’ classifier and ‘useful’ classifier as input to rearrange the reviews based on weight (confidence) of fake, non-fake, useful, and not-useful. Higher sort priority is given to deceptive/truthful reviews and then to useful/not-useful reviews. Results of evaluation of the ‘ranking’ model are presented in the next section.
For the implementation of our approach we used RapidMiner, XAMPP, Rapid Analytics tools. We created a PHP script to collect product (e.g. camera) reviews from amazon and Epinion sites. To label training data, we prepared dictionary of words for deceptive/truthful reviews and labeled the reviews by using the dictionary in the PHP script. We created another PHP script to label training set as useful or notuseful based on reader’s rating. We utilized RapidMiner tool and its supervised learning method, e.g. SVM, for building the “deceptive” classification model and “useful” classification model as well as “ranking” model. For testing purpose, we designed HTML page to enter a product review. This review is stored in a database and when the RapidMiner process is executed, it will fetch reviews from the database and based on the classifier it is processed and results (reviews with classification) are stored in the database. Using the HTML page, the result of both classifiers can be displayed. 5
For evaluation, we trained both our models using different types of datasets such as balanced and imbalanced. The training dataset for ‘deceptive’ classifier had 1348 reviews in the imbalanced dataset and 140 reviews in the balanced dataset. The training dataset for the ‘useful’ classifier had 5003 reviews in the balanced dataset and 5103 in the imbalanced dataset. The following experimental result show that ‘deceptive’ classifier gives better performance using imbalance dataset and ‘useful’ classifier performs well using balanced dataset with SVM classification algorithm. We calculated the performance of our models using the following formula.
Performance, G = √( ∗ ) where Sn is the sensitivity and Sp is the specificity Sn = and Sp =
+ + where TP is the number of true positives
TN is the number of true negatives FP is the number of false positives FN is the number of false negatives We observed that SVM trained and performed well in deception detection tasks. We found that SVM creates a hyper plane to best separate the two planes and it outperforms the other two classifiers with an accuracy peak at about 66%. Crossvalidated classifier performance results are presented in Table 1.
We tried different supervised algorithms like Naïve Bayes, K-NN, and SVM for “Useful” classifier. Evaluation results show that SVM trained and performed well in useful or not-useful detection tasks as compared to other algorithms. This approach has been evaluated to be nearly 78% accurate at detecting useful or notuseful in a balanced dataset. Cross-validated classifier performance results are presented in Table 2. Results of the ranking model are presented in Table 3.
As individuals and businesses are increasingly using reviews for their decisionmaking, it is critical to detect spam reviews. We presented our approach for detecting spam, not-useful reviews and prioritization of the reviews based on their weight (confidence). The evaluation shows that ‘deceptive’ classifier and ‘useful’ classifier is nearly 66% and 78% accurate respectively. Various supervised learning methods were used and we observed that SVM worked best as it is an immensely powerful classifier and it is well suited for 2-class problem. In addition, we compared experimentally SVM, Naïve Bayes and K-NN in performance and concluded that SVM has very good predictive power. Online reviews are worthless if they are not honest opinion. Our models, can give an idea to users on which reviews are non-fake and useful as well as which reviews should be completely ignored in product purchase decision-making thereby helping choose the right product. Future work might explore other methods for labeling online reviews, and will focus on improving the accuracy and more sopphisticated techniques for detecting spam reviews.