Mobile application stores enable end-users of software to directly express their needs and share their experience with mobile apps in the form of textual reviews. These reviews often contain important user feedback that can be leveraged by app developers to help them understand their end-user needs. However, such information are not readily available, and vetting individual reviews manually can be a tedious task. To alleviate this e ort, we introduce MARC, a Mobile Application Review Classi er. MARC is a stand-alone automated solution that enables developers to extract and classify user reviews into ne-grained software maintenance requests, including bug reports and user requirements. MARC is equipped with a set of con guration features to enable practitioners and researchers to classify user reviews under di erent settings. A dataset of app reviews sampled from three apps are used to evaluate the performance of MARC. The results show that MARC achieves accuracy levels that can be adequate for practical and research applications Copyright 2017 for this paper by its authors. Copying permitted for private and academic purposes
As of March 2015, the Apple App Store alone has reported around 2.25 million
active apps, growing by over 1,000 apps per day. This scale of app production has
resulted in an unprecedented level of competition in the app market, forcing
software creators to look beyond traditional time-consuming software engineering
practices into a new paradigm of methods that enable a more responsive
software production process. Recent analysis of large datasets of app user reviews
has revealed that almost one third of these reviews contain useful information
to app developers [
MARC1 supports two classi cation approaches, including the Bag-of-Words
(BOW) and the Bag-of-Frames (BOF) [
MARC is also equipped with several text pre-processing features to allow both researchers and practitioners to classify app reviews under di erent con guration settings. The performance of MARC is evaluated using user reviews collected from three iOS apps from di erent application domains. In what follows, we describe MARC, its basic features, classi cation engine, and our evaluation process in greater detail. 2
MARC provides a set of features that enable users to automatically extract reviews from the iOS App Store and experiment with di erent classi cation settings. In what follows, we describe these features in greater detail. 2.1
MARC provides a feature for extracting recent user reviews from the iOS App Store. The user can select any app through its unique App Store ID. MARC then makes a web request to the App Store's RSS feed generator. The generated JSON pages are parsed to extract the selected app's reviews. The user can extract between 50 and 500 reviews at a time. For example, to get the list of most recent Gmail app reviews, MARC makes the following Web request: 2.2
It is not uncommon in text classi cation tasks to use text reduction strategies to
minimize the number of classi cation features (words). The objective is to only
keep important words that have an actual impact on the predictive capabilities
of the classi er [
The core feature of MARC is to classify technically informative user reviews into user requirements and bug reports. To facilitate this process, MARC supports two di erent classi cation techniques, including Bag-of-Words and Bagof-Frames. The following is a description of MARC's classi cation engine and its classi cation techniques. 3.1
In addition the classical BOW classi cation approach, MARC supports a more
semantically-aware form of classi cation, known as the Bag-of-Frames (BOF).
This approach relies on the notion of Semantic Role Labeling (SRL) [
To support frame semantics, MARC uses SEMAFOR2|a probabilistic frame
semantic parser to parse each review [
MARC uses Support Vector Machines (SVM) and Naive Bayes (NB) to classify
app reviews. These two classi ers have been shown to be very e ective in app
review classi cation tasks, detecting di erent types of user reviews at decent
levels of accuracy over multiple datasets [
MARC uses the Weka's API3 as the core classi er. This API is used to convert
the input review's text into a Weka compatible le format (.arff). The lter
StringToWordVector is applied to generate the word x document matrix for
the input review to be classi ed. The classi er then uses term frequency (TF)
to assign weights to words. MARC uses a default training dataset of manually
classi ed reviews to train and test the underlying classi cation engine. This
dataset was compiled from di erent sources, including two datasets collected
from previous related research [
The performance of MARC is evaluated using reviews collected from 3 apps selected from the iOS App Store. These apps include: Adobe Acrobat (469337564), Chrome (535886823), and CNN (331786748). A total of 100 of most recent reviews of these apps were extracted. Each sentence was then manually classi ed as a bug report, a user requirement, or other. In total, 39 bug reports, 18 user requirements, and 43 other miscellaneous reviews were classi ed.
MARC is then used to automatically classify these reviews. The default dataset is used to train the classi er. For evaluation purposes, the reviews were classi ed using the BOW and BOF approaches. We further experimented with the text pre-processing features under the BOW approach. The results of the classi cation process is shown in Table 1. The performance of MARC is measured using precision and recall.
The results show that under the BOF approach, MARC managed to achieve
an average of 75% precision and 93% recall using SVM and an average of 57%
3 http://www.cs.waikato.ac.nz/ml/weka/
precision and 72% recall using NB. In comparison, under the BOW approach,
MARC managed to achieve an average of 32% precision and 55% recall using
SVM and an average of 55% precision and 72% recall using NB. On average,
the best results over our evaluation dataset was achieved using SVM under the
BOF approach. A thorough evaluation of the BOW and the BOF approaches is
available in [
The current release of MARC is intended for both practical and research applications. App creators can use MARC to quickly access and classify the most recent reviews of their apps. Researchers, on the other hand, can use MARC to prepare and classify large datasets under di erent classi cation settings. However, MARC still su ers from performance limitations that need to be addressed in our future releases. For instance, MARC currently takes around 200 seconds to train the classi cation model on a 3.30GHz CPU with 8.0GB of RAM. The current running time could be improved by implementing faster classi cation algorithms. Furthermore, classi cation results tend to be less accurate when classifying reviews from application domains that have never been classi ed before. Our expectation is that the classi cation accuracy could be signi cantly improved by implementing a feedback mechanism that keeps updating the training dataset with new instances. In this paper, we introduced MARC|a Mobile Application Review Classi er. MARC provides a set of features that enable users to extract reviews from the iOS App Store and classify them into actionable software maintenance requests, including bug reports and user requirements. MARC provides a set of text preprocessing features to allow users to classify input reviews under di erent con guration settings. The current release of MARC supports a Bag-of-Words (BOW) and a Bag-of-Frames (BOF) representations of the input text. Semantic frames are used to classify the input review sentences based on their context, or meaning, rather than relying on their individual words. MARC was evaluated using reviews collected from three sample applications. The results showed levels of accuracy that can be adequate for practical applications, with the BOF approach slightly outperforming the BOW approach. To enhance its practicality, our future releases of MARC will support more application stores (e.g. Google Play and Windows App Store), more classi cation algorithms, and other classication features (e.g., n-grams and POS tagging) to improve the accuracy and performance of the classi cation engine.
This work was supported by the Louisiana Board of Regents Research Competitiveness Subprogram, contract number: LEQSF(2015-18)-RD-A-07.