Sentiment Analysis has nowadays a crucial role in social media analysis and, more generally, in analysing user opinions about general topics or user reviews about product/services, enabling a huge number of applications. Many methods and software implementing different approaches exist and there is not a clear best approach for Sentiment classification/quantification. We believe that performance reached by machine learning approaches is a key advantage to apply to sentiment analysis in order to reach a performance which is very close to the one obtained by group of humans, who evaluate subjective sentences such as user reviews. In this paper, we present the App2Check system, developed mainly applying supervised learning techniques, and the results of our experimental evaluation, showing that App2Check outperforms state-of-the-art research tools on user reviews in Italian language related to the evaluation of apps published to app stores.
Sentiment Analysis has nowadays a crucial role in social
media analysis and, more generally, in analysing user
opinions about general topics or user reviews about
product/services, enabling a huge number of applications.
For instance, sentiment analysis can be applied to
monitoring the reputation or opinion of a company or a
brand with the analysis of reviews of consumer products or
services [1]. Moreover, it can also provide analytical
perspectives for financial investors who want to discover
and respond to market opinions [
App stores can be seen as another, not yet well explored, field of application of sentiment analysis. Indeed, they are another social media where users can freely express their own opinion through app reviews about a product, i.e. the specific app under evaluation, or a service, to which the considered app is connecting the user (e.g., a mobile banking app connects users to mobile banking services). In addition, reading user reviews on app stores shows that people frequently talk about and evaluate also the brand associated to the app under review: thus, it is possible to extract people opinion about a brand or the sentiment about a company or the provided service quality.
In this paper, we focus on the app store as a social media platform and on the sentiment evaluation in app reviews, which are examples of reviews related to a product, or a service or the associated brand. App reviews are a very interesting application in our opinion because they have not been extensively explored yet [6], and also because the sentiment score detected in a comment can significantly differ from the score assigned by the user to the app under evaluation. For example, a user can assign his good score to the app (i.e. assigning 5 stars) but also express in natural language some suggestions or highlight some –even important– bugs that, if they may not influence the user overall app evaluation, from the perspective of the developer or app producers are very important. For example, the comment 1 “Great app to be honest, but it freezes while scanning the code of the pre-printed payment slip, it crashes, and it closes. Do something!!” was rated 4 stars by the user. However, the comment describes a severe bug that causes an app crash and we can agree that this comment has an overall negative sentiment, especially from the perspective of developers. Vice versa, the user can assign, in general, a low rating but highlight some good features. All of this non-structured information is fully missing by only superficially evaluating an app through a 1 to 5 overall score –or any other product evaluated by the user with both sentences and a score –.
About the methods of processing user reviews, many
methods and software implementing different approaches
exist and there is not a clear best approach for Sentiment
classification/quantification [7,8,9]. In [
In this paper, we present the App2Check system, developed mainly applying supervised learning techniques and focused – in this first release – on Italian language, and present the results of our experimental evaluation showing that App2Check (version 1.0) outperforms state-of-the-art research software on user reviews in Italian language related to apps (which are, at the moment, our main target application). We considered research tools for our experimental evaluation, since the current state-of-the-art commercial tools recently included strict restrictions related to the possibility to run them for competitive analysis or benchmarking. In particular, since there are not so many research tools managing natively the Italian language, we applied the approach already shown in [7] where the iFeel research platform has been presented. iFeel performs the promising approach to translate sentences into English before running 19 state-of-the-art research tools. In order to make a fair comparison, we also included in our comparison a research tool that natively manages the Italian language: to the best of our knowledge, it is the only research tool with this feature that is available for download.
The structure of the paper is the following. After the current introduction about the main paper topics, in section 2, we report a description of the research tools we used to perform the comparison. In section 3, we briefly describe our system App2Check; in section 4, we present and discuss our experimental evaluation and, in section 5, we provide the paper conclusions.
In this section, we describe the research tools that will be mentioned in the following sections and included in our experimental evaluation: iFeel, a platform developed at Federal University of Minas Gerais and running 19 research tools, and SentiStrength version for Italian language. 2.1 iFeel iFeel is a research web platform [10] allowing to run 19 state-of-the art research tools for sentiment analysis on the specified list of sentences. It allows to natively run tools supporting English and to first translate sentences from other languages into English and then run the underlying tools on the English translated sentences. It has been experimentally shown in [7] that well known language specific methods do not have a significant advantage over a simple machine translation approach.
The tools included in iFeel are the following (in alphabetical order): AFINN, Emolex, Emoticon DS, Emoticons, Happiness Index, NRC Hashtag, Opinion Finder, Opinion Lexicon, Panas-t, SANN, SASA, Senticnet, Sentiment140, SentiStrength, SentiWordNet, SO-CAL, Stanford Deep Learning, Umigon, Vader. We report in the following a few sentences from [13] describing each tool included in iFeel, in order to give an insight of the techniques implemented in these tools. 2.1.1. AFINN It is a lexicon-based approach described in [14] and uses a Twitter based sentiment lexicon including Internet slangs and obscene words. AFINN can be considered as an expansion of ANEW, a dictionary created to provide emotional ratings for English words. ANEW dictionary rates words in terms of pleasure, arousal and dominance. 2.1.2. Emolex It is a lexicon-based approach described in [17]. It uses a general sentiment lexicon supported by crowdsourcing. Each entry lists the association of a token with 8 basic sentiments: joy, sadness, anger, etc.
It is a lexicon-based approach described in [18] and creates a scored lexicon based on a large dataset of tweets. It is based on how the frequency each lexicon occurs with positive or negative emotions.
It is a lexicon-based approach described in [15] where messages containing positive/negative emoticons are simply associated to a positive/negative sentiment, respectively. Messages without emoticons are not classified.
It is a lexicon-based approach described in [19] and consists in a measure evaluating the psychological valence (happiness) distribution for words in the Affective Norms for English Words (ANEW). For each text, it is thus possible to compute the weighted average of the valence of the ANEW study words in a given text.
It is a lexicon-based approach described in [20] and it builds a lexicon dictionary using a Distant Supervised Approach. It uses known hashtags (i.e. #joy, #happy, etc.) to ‘classify’ the tweet. Afterwards, it verifies frequency each specific n-gram occurs in an emotion and calculates its Strength of Association with that emotion.
It performs both a lexicon-based and a machine learningbased approach, as described in [21]. It performs subjectivity analysis through a framework that applies before lexical analysis and then a machine learning algorithm.
It is a lexicon-based approach described in [22] and it focuses on product reviews. It builds a lexicon to predict the polarity of product features that are summarized to provide an overall score to that product feature. 2.1.9. Panas-t It is a lexicon-based approach described in [23] that detects mood fluctuations of users on Twitter. The method consists of an adapted version (PANAS) Positive Affect Negative Affect Scale of a well-known method in psychology with a large set of words, each of them associated with one from eleven moods such as surprise, fear, guilt, etc. 2.1.10. SANN It performs both a lexicon-based and a machine learning-based approach, as described in [24]. It infers additional user ratings by performing sentiment analysis (SA) of user comments and integrating its output in a nearest neighbor (NN) model. 2.1.11. SASA It is a machine learning-based approach, as described in [25], and it detects public sentiments on Twitter during the 2012 U.S. presidential election. It is based on the statistical model obtained from the Naive Bayes classifier on unigram features. It also explores emoticons and exclamations.
It is a lexicon-based approach described in [26]. It applies dimensionality reduction to infer the polarity of common sense concepts and hence provide a resource for mining opinions from text at a semantic level. 2.1.13. Sentiment140 It is a machine learning-based approach described in [27]. Sentiment140 is an ensemble of three classifiers (Naive Bayes, Maximum Entropy, and SVM) built with a huge amount of tweets containing emoticons collected by the authors.
It performs both a lexicon-based and a machine learning-based approach, as described in [11]. It uses a lexicon dictionary annotated by humans and improved with the use of machine learning. We provide more details in section 2.2.
It performs both a lexicon-based and a machine learning-based approach, as described in [28]. It uses a lexical resource for opinion mining based on WordNet. The authors grouped adjectives, nouns, etc. in synonym sets (synsets) and associated three polarity scores (positive, negative and neutral) for each one. 2.1.16. SO-CAL It is a lexicon-based approach described in [29]. It creates a new lexicon with unigrams (verbs, adverbs, nouns and adjectives) and multi-grams (phrasal verbs and intensifiers) hand ranked with scale +5 (strongly positive) to –5 (strongly negative). The authors also included part of speech processing, negation and intensifiers.
It is a machine learning-based approach described in [30]. It applies a model called Recursive Neural Tensor Network (RNTN) that processes all sentences dealing with their structures and compute the interactions between them. The RNTN approach takes into account the order of words in a sentence, which is ignored in most of the methods. 2.1.18. Umigon It is a lexicon-based approach described in [31] that disambiguates tweets using lexicon with heuristics to detect negations plus elongated words and hashtags evaluation. 2.1.19. VADER It is a lexicon-based approach described in [32]. It is a human-validated sentiment analysis method developed for Twitter and social media contexts. VADER was created from a generalizable, valence-based, human-curated gold standard sentiment lexicon.
SentiStrength was produced as part of the CyberEmotions project, supported by EU FP7. It estimates the strength of positive and negative sentiment in short texts, even for informal language. According to the authors, it has humanlevel accuracy for short social web texts in English, except political texts [11]. SentiStrength authors make available a version of the tool which natively manages Italian language. All tests have been carried out with both average emotion and strongest emotion options, but in this paper we only report the results obtained with the latter option turned on, due to better performance. Since the English version of SentiStrength is also included in iFeel, we ran on our own the Italian version and we will call it in the following SentiStrengthIta.
App2Check is our system using an approach in which supervised learning methods are applied in order to build a predictive model for sentiment quantification. The training of the model is performed by considering a huge variety of language domains and different kinds of user reviews. App2Check provides, as answer to a sentence in Italian language, a quantification of the sentiment polarity scored from 1 to 5, according to the most recent trend shown in the last sentiment evaluation SemEval [12], where tracks considering quantification have been introduced. Thus, we consider the following quantification: as “positive”, sentences with score 4 (positive) or 5 (very positive); as “negative”, sentences with score 1 (very negative) or 2 (negative); as “neutral”, sentences with score 3. In order to compute the final answer, App2Check does not use just the prediction coming from the predictive model, but it applies also a set of algorithms which take into account some natural language processing techniques, allowing e.g. to also automatically perform topic/named entity extraction. It is not possible to give more details about the engine due to non-disclosure restrictions.
App2Check is not only constituted by a web service providing access to the sentiment prediction of sentences, but it is also a full user-friendly web application allowing (more features in next release) in the current release 1.0 to: a) Search for the app a user wants to monitor on the Apple
App store, Google Play store or Microsoft Marketplace b) Show the main topics discussed in user reviews which are both comment-specific, associated to a specific month or evaluated to overall the app life c) Show the sentiment about the former extracted topics, including in the topics –if discussed in user comments– also the company brand and the provided service level d) Show a sentiment comparison on the app time horizon between apps owned by different app publishers (even market competitors).
A demo of the App2Check is available after sending a request by email to the first author of the paper.
In our experimental evaluation we considered user reviews of apps from Apple App store and Google Play store. More specifically, we focused on two different sets of comments. Test set A is made of 10 thousands comments from 10 different very popular apps (one thousand comments per app). These comments are associated only to an overall score for the app, called app rating in the app stores. Test set B is made of 1 thousand comments from the famous Candy Crush Saga app: in this case, we performed a manual quantification (in the 1-5 range) of the sentiment (from now on called human sentiment classification or HSC).
We ran App2Check, iFeel and SentiStrengthIta on these user reviews, in order to evaluate: on test set A, their relative performance using the app rating as a reference indicator, i.e. as an approximation of the user sentiment so that we avoid to manually classify the sentiment for 10 thousand comments. Of course, considering a single comment, as already said, in general, the score/rating expressed by a user respect to an app can be substantially different respect to the sentiment expressed by a human. However, we experienced that the average score/rating of many (hundreds of) comments can be an approximation of the average sentiment expressed by a human on the same set. In Table 1 we show this phenomenon: human sentiment classification (performed by only one person trained with guidelines and examples) agrees with rating on 79.8% of cases with app rating. on test set B, the performance of the three systems is compared respect to the sentiment manually classified/quantified by a person on 1 thousand reviews of Candy Crush app (his classification is made publicly available). Thus, in this case we compare systems on a reference that is not approximated.
All of the user reviews together with a limited demo access to the prediction web service, are made available by contacting the authors, in order to make the experiments repeatable. Saga app: accuracy is 79.8%, other measures in table.
Tool App2Check Umigon SentiWordNet Sentiment140 SentiStrength AFINN Stanford DL Op. Lexicon NRC Hashtag Emolex SASA Vader Senticnet SO-CAL SentiStrengthIta H. Index Emoticon DS Op. Finder SANN Panas-t Emoticons In all of the following tables we show macro F1 (MF1), accuracy (Acc), F1 on the negative class (F1(-)), F1 on the neutral class (F1(x)), and F1 on the positive class (F1(+)). We highlight in bold the best value per column. In Table 2 we compare the tools on test set B (1 thousand user reviews from the popular Candy Crush app) with respect to the app rating. It shows that App2Check has the highest macro F1 (59.2%) and the highest accuracy (78.3%), calculated using app rating as a reference. The second and third accuracy is obtained by Sentiment140 and SentiWordNet, respectively. SentiStrengthIta produced a bad performance with respect to the English version of the same tool. In Table 3, we make a comparison with respect to the human sentiment classification. App2Check wins again here, showing the highest macro F1 (65.8%) and accuracy (81.8%); we see that it is even higher than the one calculated in Table 2 using app rating as a reference. This indicates that App2Check is closer to the human sentiment classification (which is our goal) than to just the app rating. In Table 3 we can also see that Sentiment140 and SentiWordNet have the second and third macro F1 and accuracy, respectively. Almost all of the tools show the same pattern and we obtain almost the same chart, thus by confirming that, if we consider hundreds of comments, using app rating becomes –overall and on average– an approximation of the user sentiment. The latter result enables us to use app rating in the following experiments as a reference approximating the sentiment expressed by one single person on the test set.
In Table 4 we show the results of the systems on 10 thousand reviews, selected considering 1 thousand reviews per each of the following popular apps: Angry Birds, Banco Posta, Facebook, Fruit Ninja, Gmail, Mobile Banking Unicredit, My Vodafone, PayPal, Twitter, Whatsapp.
Considering app rating as a reference, we clearly see that App2Check outperforms all of the other tools, reaching an accuracy of about 86%. In order to better analyze App2Check performance, in Figure 1 we show a plot of the average sentiment per month of all user reviews (1 for positive, 0 for neutral and -1 for negative sentiment). In the plot we include app rating as a reference, App2Check and SentiStrengthIta (SS. ITA in the plot), since they natively support Italian, and the two best tools according to accuracy from Table 4: SentiWordNet (SWN), which is also the best according to macro-F1, and Emoticon DS (Emo DS). Emoticon DS assign too often a positive score, in fact its graph is very close to 1, even where the average rating is negative. In fact, the high accuracy reached by this tool is a consequence of the number of positive documents in this testset. It is clear that the other tools follow quite well the trend of the rating plot. Both SentiStrengthIta and SentiWordNet, instead, are closer each other and to the app rating, but their evaluation is under the reference plot. App2Check is the closest to the app rating, but in certain areas it differs from the rating, especially when the score provided by the user is on average far away from the sentiment expressed in the review. In our opinion, App2Check would have even higher accuracy on these 10 thousand instances, considering as a reference the human sentiment classification: this is made clear while using the web application and evaluating the answer of the system on every single user comment. In this paper we presented App2Check, a machine learningbased system performing sentiment classification or quantification on user reviews in Italian language. We evaluated it on 11 thousand user reviews related to apps published in app stores. Results show that App2Check outperforms state-of-the-art research tools on this test set. As future work, we want to extend the system to work on more languages and we want to extend the system evaluation on different kind of user reviews and on user feedbacks from Twitter.
Bibliographical References