Usability and user experience (U&UX) as important components of software quality are now more critical than ever for mobile app store success. Usability experts use different protocols to evaluate the usability of mobile apps while app store user reviews also produce valuable related information. Our research study proposes a measurement design to compare user reviews and expertbased usability evaluation results that includes an exploratory analysis and topic modeling of user reviews. This design is structured to investigate whether mobile app usability features extracted from user reviews align with subject-matter expert usability evaluation results.
Usability evaluations have been performed traditionally by subject-matter experts and
end users, usually in laboratory and field contexts. However, such evaluations can cover
only a limited time-span of the applications and re-doing the same usability evaluation
for all available app versions would typically lead to large evaluation costs.
Furthermore, evaluating some usability dimensions, such as understandability or learnability,
requires more complex procedures and indicators [
With the advent of mobile ecosystems including mobile apps and related meta-data such as ratings and user reviews, app stores now contain a wealth of information about user experience and expectations. However, it is difficult to manually extract this information due to various factors such as the large quantity of reviews, their lack of structure and varying quality.
In this paper, we present a measurement design to compare the findings from subject-matter expert usability evaluations and corresponding app store user reviews. The measurement design proposes two different approaches: one is through an exploratory analysis and the other is through, first, a semi-supervised topic modeling to extract usability aspects from user reviews and next, comparing these findings with the results from a prior expert-based usability evaluation. To the best of our knowledge, our work is the first to propose topic modeling techniques to automatically extract usability and user experience (U&UX) information from app store user reviews and to compare usability evaluation results of experts and end-users. 2
Usability evaluation of mobile apps is an emerging research area that faces a variety of
challenges due to the limitations of mobile devices such as processing capacity, screen
size, connectivity, and a lack of a consensus on a usability evaluation methodology [
App stores are valuable repositories of app and user data where app users can give
feedback about different aspects of an app such as its functionality, design or value. A
previous study has reported that app store user reviews are valuable to understand user
experience and usability aspects [
However, we could not identify any related work comparing user reviews and expertbased usability evaluation results. 3
The first objective of the proposed measurement design is to extract usability related information from user reviews. The second objective is to compare expert-based usability results with user usability evaluation through reviews.
The overview of the proposed measurement design is presented in Figure 1 where Parts 1 and 2 address the first research objective and Part 3 the second research objective. Pre-defined usability keyword frequency analysis of a corpus of reviews was performed in Part 1, while topic modeling was performed in Part 2 to automatically extract usability related topics. The outputs obtained in Parts 1 and 2 and prior subject-matter expert usability evaluation findings are compared in Part 3. The details of the proposed measurement design are presented in the following sub-sections.
Preliminary Analysis of User
Reviews Compare Results
No
Reviews contain useful info?
Part 1 Expert Based Evaluation
Yes
Topic Modeling (LDA vs Guided LDA)
Topics Compare Results
Part 2
Part 3
In the first part of the research, a preliminary data analysis was performed to discover
usability related keyword frequencies in the review corpus through the following four
steps:
• Step 1. Selection of apps with the information available to build a review corpus. A
reference review corpus was identified and selected, which contained for the same
apps both the results of expert-based usability evaluation and user reviews.
• Step 2. Converting usability attributes to aspect words. After the review corpus was
populated for the selected versions of the apps, the usability attributes or heuristics
were converted into a bag-of-words (BOW). Step 3. Stemming. The Porter stemming
algorithm [
In part 2 of the measurement design, a topic modeling technique was used to help
identify individual topics in the document and understand the document corpora in an
automated manner. However, unsupervised topic models often lead to topics that are not
completely meaningful and/or topics discovered in an unsupervised way that may not
match the true topics in the data. To address this limitation, we leveraged the guided
latent Dirichlet allocation (LDA) topic model [
Step 1: For = 1 … … . : (a) Choose regular topic ø) *~ ℎ(_) (b) Choose seed topic ø) 8~ ℎ(8) Step 2: For each seed set = 1 … … . ,
(a) Choose group-topic distribution =~ ℎ() Step 3: For each document : (a) Choose a binary vector B⃗ of length S (b) Choose a document-group distribution =~ ℎ(B⃗) (c) Choose a group variable ~ (=) (d) Choose θ=~ ℎNOP (e) For each token = 1 … … . =, (1) (2) (3) (1) Select a topic S ~ ( θ=) (2) Select an indicator S ~ ( WS) (3) If S is 0
Select a word S ~ ( θWS*) topic (4) If S is 1
Select a word S ~ ( θWS 8) //choose from LDA style The generative process for a document collection D under the guided LDA model is as follows – see Figure 2: 1. First, the T topic-word distribution ø) and group-topic distribution =8 were generated. 2. Then for each document, a list of seed sets allowed for the document, represented as a binary vector B⃗, was generated, and then 3. B⃗ was populated based on the document words, and hence treated as an observed variable.
The review corpus generated in Part 1 was used in the topic modeling. To reduce the dimensionality of the document term matrix, certain data pre-processing and cleaning steps were carried out before proceeding with topic modeling. This pre-processing consisted of:
I. tokenization to segment the review corpus into its atomic elements using the
Natural Language Toolkit (NLTK) tokenize.regexp module; II. lower case conversion; and III.
stop-word (e.g. ‘the’, ‘and’, ‘or’, ‘a’), punctuation and non-alphabetic phrase removal.
The guided LDA library1 developed in Python was used in this study. The document term matrix that was generated in the pre-processing step was given as an input to the LDA model. The training step required the input parameters, such as seed topics, seed confidence, be set at 0.15 to bias the seeded words by 15% towards the seeded topic, the number of topics be set at 5, 10 and 20, chunksize at 2000, refresh at 20 and iterations at 100. 1 https://github.com/vi3k6i5/GuidedLDA
The guided LDA model performance was tested on a complete review corpus where (i) the number of topics K = 5, 10 and 15 and (ii) LDA model was taken as the baseline. The topic model was run on individual corpuses per app. 3.3
In this part of the research, usability aspects extracted from user reviews were compared with expert-based usability evaluation results for the most frequent and the less frequent usability attributes: • If the user reviews had more positive associations for the usability term, its user review evaluation rating were accepted as positive, corresponding to 4-5 stars given by usability experts in prior evaluations. • If the user reviews had more negative associations for the usability term, its user review evaluation rating was accepted as negative, corresponding to 1-2 stars given by usability experts in prior evaluations. • Equal numbers of positive and negative reviews were given a neutral evaluation rating, corresponding to 3 stars given by usability experts in prior evaluations. 4
In this proposed measurement design, usability aspects for expert-based usability evaluation questionnaires were first extracted and then converted into a BOWs in order to trace them back in user reviews. In addition, a guided LDA topic modeling was developed to automatically capture usability aspects intrinsic to the review texts. Expertbased usability evaluation results were compared with user evaluations expressed through reviews and the identified alignment and differences reported. We believe that this proposed measurement design is useful for supporting developers, U&UX designers and researchers to better understand user experience and opinion on mobile application usability aspects, which, finally, can lead to improved software quality.
In future work, we will explore the performance of our guided LDA topic model vs
LDA topic model as baseline. Our topic model will also be run on individual review
corpuses per app to find the percentage of clustered words that are directly related to
usability and usability aspects such as efficiency, errors/effectiveness, etc. Next, for the
top 10 and 10 lowest frequency terms identified with preliminary analysis in Part 1 and
topic modeling in Part 2, user and expert evaluation ratings will be compared to
determine possible alignments or differences between two different usability evaluation
methods. A reference review corpus has already been selected which contains for the
same apps both the results of expert-based usability evaluation as well as user reviews.
This is the app dataset from [