User reviews play a vital role in mobile application development. New users can grasp the pros and cons of an app from user reviews, and developers can improve the app by addressing the issues mentioned in the user reviews. However, scanning and analyzing massive user reviews is always a challenging and time-consuming task for both users and developers. This paper proposes a solution to build a tool for analyzing and organizing user reviews. To analyze user reviews, we classify the sentences in the reviews into prede ned categories by using a machine learning algorithm; then, we apply text classi cation techniques to determine the review sentence's polarity and nally to mine key phrases from the sentences. We conducted an experiment using user reviews for two messaging apps. The experimental results demonstrate that we can organize the core information of each review and present the information to users and developers in respectively di erent ways.
give a perfect rating to an application at the beginning, but the user may nd some issues afterward. If the user later modi es the review message without updating the rating, the perfect rating then will hide the potential issues.
The existing mobile application stores do not provide well-organized tools to assist people in scan reviews. In this study, we propose a tool for analyzing and organizing reviews, and therefore provide a solution to meet di erent kinds of demands from users and developers. When users and developers are scanning reviews, their focus could be di erent. Users like to browse a group of reviews together, which makes them focus on the collective ideas from a group. On the other hand, developers want to check the reviews successively, because each review could contain di erent issues with each other. With machine learning techniques, our tool is able to provide summarized information to users and developers in respectively di erent ways.
The rest of the paper is organized as follows: Section 2 sketches related work. Section 3 presents methods to design and construct the tool. Section 4 describes data collection process, and Section 5 explains our analyzing process. Section 6 demonstrates how we organize the results and present it to users and developers. Finally, Section 7 concludes with future work. 2
Many studies demonstrated how machine learning techniques could help users and developers analyze and organize data. Rajeev et al. [RR15] and Priyanka et al. [PTB15] show solutions to advise users in nding online products. Chen et al. [CSM+18] present a tool to assist developers in software development. With practical examples, these studies show that machine learning could be a powerful means which helps users and developers to manipulate various kinds of data e ciently.
There has been a trend towards analyzing and organizing reviews with machine learning technologies. Many studies focus on text classi cation, data extraction, and information summarization. A standard method of analyzing user reviews is to determine whether a review presents a positive or negative attitude. Pang et al. [PLV02] and Turney et al. [Tur02] show that document-level sentimental analysis reaches a good result and even performs better than human-produced baselines. Tanawongsuwan et al. [Tan10] demonstrate how to process the sentiment classi cation on a product review through the analysis of parts of speech of the textual content. Instead of merely determining whether a review has a positive or negative tone, Maalej et al. [MN15] concentrate on review tagging. It proposed a method to determine whether a review is bug report, feature request, or praise.
Data extraction is also a pivotal process in analyzing reviews. Somprasertsri et al. [SL10] develop a method to extract product features and associated opinions from reviews through syntactic information based on dependency analysis. Dave et al. [DLP03] propose a method for extracting a product attribute and aggregating opinions about each of them; besides, this method can automatically di erentiate positive reviews and negative reviews. Kim et al. [KH06] represent a method not only to extract a products' pros and cons from reviews but also to mine the sentences which account for the reviewer's preferences.
Beyond text classi cation and data extraction in user reviews, Hu et al. [HL04] focus on mining and summarizing reviews by extracting opinion sentences about product features. Blair-Goldensohn et al. [BGHM+08] concentrate on aspect-based summarization models, where a summary is produced by extracting relevant aspects of a local service, aggregating the sentiment per aspect, lastly selecting aspect-relevant text.
The previous studies mostly concentrated on analyzing reviews to help users. They target on-line store products and local service reviews, such as banks, restaurants, movies, and travel destinations. Our study di ers in essential ways from previous ones: it focuses on mobile application user reviews. In recent years, there are also a number of studies concentrate on mining useful information from user reviews. Vu et al. [VNPN15] propose a framework to help analysts to collect and mine user opinions from reviews. Guzman et al. [GM14] propose an automated approach to help developers to lter and analyze user reviews. Our study is di erent as we are interested in building a tool that aims to analyze reviews for both users and developers. 3
Our tool aims to analyze reviews and adequately organize them for users and developers. The overall goal of the tool is to help users and developers accelerate the review scanning process. In this study, we target on the sentence-level analysis, since each review may contain various opinions. Also, these opinions could cover di erent aspects of an app; for example, an opinion can discuss a speci c function feature, or it can express a general option towards the entire app. Consequently, we target the sentence-level analysis and focus on analyzing user reviews in the following perspectives.
We use reviews from mobile messaging applications as examples to demonstrate the analyzing and organizing processes and tool implementation. Two popular messaging applications, WeChat1 and LINE2, are used in this study. They are available at the Google Play store and have su cient user reviews. WeChat is a messaging and social media app - it is a lifestyle for one billion users across the world. It provides not only chatting and calling features but also gaming, mobile payment features, and many other features. In total, WeChat had 100, 000, 000+ installations and 5, 000, 000+ reviews. Another messaging application, LINE reshapes communication around the globe, allowing users to enjoy not only messaging but also free voice and video calls. LINE had 500, 000, 000+ installations and 11, 000, 000+ reviews.
Sometimes an app's update may introduce new bugs and unsatis ed feature changes, and then it can cause many alike user reviews. For the variety of data maintenance concerns, we collected review data from a one-year period (April 2018 to April 2019) and selected the rst 100 English reviews from each month. In such a way, we can avoid similar bug reports or feature complaint reviews within a short period. Also, as suggested by Dave and Lawrence [DLP03], we ltered out the reviews, which contain less than three words or primarily written with symbols to avoid sparse data issue, from the collected data set.
1https://play.google.com/store/apps/details?id=com.tencent.mm\&hl=en 2https://play.google.com/store/apps/details?id=jp.naver.line.android\&hl=en
Category classi cation. We employ text classi cation techniques to classify review sentences to category tags. In the classi cation, the pre-processed review textual content is used as the feature.
Review sentence tagging. The tagging was performed manually in the category text classi cation process. The prede ned categories for the messaging application reviews model include General opinion, F unctional f eature, and Out of domain. General opinion indicates a review sentence that has a broad opinion towards the application, and it generally presents whether users like the application or not. F unctional f eature indicates a review sentence that directly discusses a speci c function or a feature of the application. Out of domain indicates a phrase that is neither talking about a speci c function or a general review, and it describes a type of phrase that usually contains a single word or an unrelated discussion. Both WeChat and LINE's review data are annotated with the above three categories. Table 1 shows the total number of sentences, the numbers of sentences tagged with di erent categories, and the number of vocabularies appearing in the review data.
The text pre-processing. Considering that users might write non-standard English words, such as emoji and online slang, in reviews, we simplify and clean the text before training. In our study, the processing steps are: lowering case conversion, converting numbers to words, tokenizing sentence, removing stop words, and reducing word forms. We use Natural Language Toolkit (NLTK) 3 library to break a sentence into words; this library has a Tweet Tokenizer module which can recognize online slang and diminishes the word length; for example, \waaaaayyyy" is reduced to \waaayyy".
Convolutional neural networks. In this study, the text classi cation algorithm is based on characterlevel Convolutional Neural Networks (CNN). Recent studies proved that CNN works well for problems in the natural language processing [Kim14] [ZZL15], although CNN is more often apply to solve machine learning image problems. Gradually the CNN model had become a standard baseline for new text classi cation architectures. Unlike image pixels that are used as input in image problems, the input in our classi cation problem is a review sentence represented as matrix. Each row of the matrix is a vector that represents a word; moreover, the vector is the index of a word into vocabularies appearing in our collected data.
Category classi cation cross-validation. To experimentally evaluate the performance of the category classi cation process, we conducted a 3-fold cross validation using the reviews we collected. The whole data set was shu ed and equally divided into three parts in the evaluation process. Each set was used as the testing set once, and the other remaining two sets were used as the training set. As a result, three classi cation models were developed for a given collection of data in the experiment. Table 2 presents classi cation results of using only WeChat data, Table 3 presents the classi cation results of using only LINE data, and Table 4 shows the classi cation results of using data from both WeChat and LINE.
All the tables show that classi cation models have satisfying prediction scores on these three categories. The f unctional f eature category has the best results, and the out of domain has the least prediction rate. The reason could be that the f unctional f eature has the most extensive training set size, whereas the out of domain has the least. Although WeChat had a slightly smaller number of reviews, each review from WeChat seems to
contain more sentences and generate a large data set. As a result, the WeChat classi cation model generally performs better than the LINE model. Since the same tagging rules were applied to WeChat and LINE review data, we used the mixed data together to train another classi cation model. Appropriately, this classi cation model also ended up satisfying classi cation results.
Sentiment analysis. After determining a review sentence category, we examine each sentence on whether its attitude is positive or negative. For instance, consider the following review examples, \WeChat send messages fast" and \WeChat lagging on taking pictures." They both discuss functional features, but they have very di erent attitudes towards the features. Knowing a sentence's polarity undoubtedly helps users and developers accelerate the scanning process. In this study, we use the NLTK library again for sentiment classi cation. It can determine a review sentence, whether it expresses positive sentiment, negative sentiment, or if it is neutral. This library uses hierarchical classi cation, where a sentence's neutrality is checked rst, and then the polarity is determined.
Key phrases mining. To help developers speed up to discover any issue from reviews, we reduce the text content amount developers need to read in reviews. To this end, we mine key phrases from review sentences. We use a library called RAKE-NLTK (RAKE stands for Rapid Automatic Keyword Extraction algorithm 4), to mine key phrases. RAKE is a domain-independent keyword extraction algorithm, and it tries to discover key phrases from the text body by analyzing the frequency of word appearance and word's co-occurrence with other
words in the text [RECC10]. The mining algorithm takes a text sentence as an input and outputs a list of text phrases with related scores. In this study, a simple lter algorithm was applied to the mining results. We rank the results by their scores from the highest to lowest, and we lter out the phrases whose scores are equal to the lowest score in the results. 6
In this section, we describe how the analyzed information is presented into two kinds of output for users and developers.
The user version. This version of output is organized to help new users to quickly answer the following two questions: 1) From what aspect did other users review the app? and 2) What are the polarities of these aspects? For this reason, we select the review category tag and the polarity tag information from the analyzed result database and draw them in the form of bar graphs for users. Figure 2 shows an example of analyzed WeChat reviews from April 2019. The graph illustrates the number of aspects, which are out of domain, general opinion, and f unctional f eature, mentioned in the reviews. The polarity information is colored and visibly displayed. More importantly, the original review sentence can be traversed by clicking the corresponding chart bar. Additionally, user version output supports the comparison mode so that users can compare the analyzed reviews from two apps. Figure 3 shows an example of comparing results between WeChat and LINE; in the gure, two analyzed results are horizontally presented side by side in one output. The output can e ciently show the di erence to users, and therefore clearly present which app is preferred by users in a given period of time.
ReviewsAnalysis Result: UserVersion Reviews Search By
Monthly 2019/04 Compare with : none Total Numberof Reviews Total Numberof Sentences 100 363 Category Out of Domain General
Opinion Total 78 129
Functional Feature 155
Total Numberof Reviews Total Numberof Sentences 100 275 Category Out of Domain General
Opinion Total 71 109
Functional Feature 95
The developer version. This version of output aims to assist developers to discover what kind of issue users have when using the app. For this, each review is transformed into a compact version in the output. The review sentence is replaced with a category tag, a corresponding polarity tag, and a list of key phrases. The polarity tag combined with the list of key phrases can reveal the potential issues users mentioned in the review, and the category tag indicates what aspect the issues mentioned. Similar to the user version's output, the original review text can be traversed by clicking the corresponding tags. Figure 4 shows an example of developer version output for WeChat, and Figure 5 shows the di erence between an original review and the related developer version output. The developer version output reduced many texts from the original review, but it still retains the core information. By reading the output, developers are able to develop an abstract understanding of the original review. In this paper, we proposed a method to build a tool for analyzing and organizing user reviews. This tool can generate two kinds of output in order to meet di erent demands from users and developers. The user version Review
1 / 5 Date output focuses on presenting the character of a group of data, and the developer version output focuses on shrinking the review texts. Our tool consists of two primary parts: using machine learning techniques to analyze user reviews, and organizing analyzed results for users and developers separately. We keep working on the following tasks to build a more sophisticated tool in the future.
In future work, an actual human evaluation should be involved. We will give the generated outputs to real developers and ordinary volunteers, and ask them to provide con dent rates for the analyzed and organized results. Another work could be to improve the accuracy of key phrase mining. Currently, the algorithm of ltering the mining result is naive. Additional techniques, such as removing the stop words and unnecessary adjectives, could make the mining results clearer and more reliable. [BGHM+08] Sasha Blair-Goldensohn, Kerry Hannan, Ryan McDonald, Tyler Neylon, George Reis, and Je Reynar. Building a sentiment summarizer for local service reviews. In WWW Workshop on NLP Challenges in the Information Explosion Era (NLPIX), 2008. [CSM+18]
Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. From ui design image to gui skeleton: A neural machine translator to bootstrap mobile gui implementation. In Proceedings of the 40th International Conference on Software Engineering, ICSE '18, pages 665{676, New York, NY, USA, 2018. ACM.
Kushal Dave, Steve Lawrence, and David Pennock. Mining the peanut gallery: Opinion extraction and semantic classi cation of product reviews. Mining the Peanut Gallery: Opinion Extraction and Semantic Classi cation of Product Reviews, 775152, 10 2003.
E. Guzman and W. Maalej. How do users like this feature? a ne grained sentiment analysis of app reviews. In 2014 IEEE 22nd International Requirements Engineering Conference (RE), pages 153{162, Aug 2014.
Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Won Kim, Ron Kohavi, Johannes Gehrke, and William DuMouchel, editors, Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Seattle, Washington, USA, August 22-25, 2004, pages 168{177. ACM, 2004. [KM17] [MN15] [PLV02] [PTB15] [RECC10] [RR15] [SJ15] [SL10] [Tan06] [Tan10] [Tur02] [VNPN15] [ZZL15]
Soo-Min Kim and Eduard H. Hovy. Automatic identi cation of pro and con reasons in online reviews. In ACL, 2006.
Yoon Kim. Convolutional neural networks for sentence classi cation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746{1751. ACL, 2014.
Z. Kurtanovic and W. Maalej. Mining user rationale from software reviews. In 2017 IEEE 25th International Requirements Engineering Conference (RE), pages 61{70, Sep. 2017.
W. Maalej and H. Nabil. Bug report, feature request, or simply praise? on automatically classifying
a
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. Thumbs up? sentiment classi cation using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 79{86. Association for Computational Linguistics, July 2002.
P. Tanawongsuwan. Relation between a book review content and its rating. In International
Conference on Computer Information Systems and Industrial A
Peter D. Turney. Thumbs up or thumbs down? semantic orientation applied to unsupervised classi cation of reviews. CoRR, cs.LG/0212032, 2002.