=Paper=
{{Paper
|id=Vol-2089/4_Adaji
|storemode=property
|title=Understanding Low Review Ratings in Online Communities: A Personality Based Approach
|pdfUrl=https://ceur-ws.org/Vol-2089/4_Adaji.pdf
|volume=Vol-2089
|authors=Ifeoma Adaji,Kiemute Oyibo,Julita Vassileva
|dblpUrl=https://dblp.org/rec/conf/persuasive/AdajiOV18a
}}
==Understanding Low Review Ratings in Online Communities: A Personality Based Approach==
https://ceur-ws.org/Vol-2089/4_Adaji.pdf
Understanding Low Review Ratings in Online
Communities: A Personality Based Approach
Ifeoma Adaji, Kiemute Oyibo, Julita Vassileva
MADMUC Lab, University of Saskatchewan, Saskatoon, Saskatchewan,
Canada
ifeoma.adaji@usask.ca,kiemute.oyibo@usask.ca,
julita.vassileva@usask.ca
Abstract. Online communities like Yelp thrive when users participate actively by
writing good and useful reviews. While useful reviews are needed to keep the
community active, understanding the users who post low rated and unhelpful re-
views is also important, so that developers can implement persuasive strategies
targeted at this group of users. In this paper, we identify those users who post low
rated, unhelpful reviews and their personality types in Yelp using the Linguistic
Inquiry and Word Count (LIWC) tool. The result of the analysis reveals that users
who post unhelpful reviews are mostly of the personality type neuroticism. Using
partial least squares structural equation modelling, we further explored the sus-
ceptibility of the different personality groups of users to rewards as a means of
influencing them to write more useful reviews. Our results show that only the
users that are high is extraversion who post unhelpful reviews are susceptible to
rewards. This result demonstrates that rewards might not be persuasive to most
of the Yelp users who post unhelpful reviews, hence the use of other persuasive
strategies should be explored to influence users to post helpful reviews. The result
of this study can be helpful to developers and stakeholders of online communities
in implementing personalized influence strategies that work.
Keywords: Online communities; reviews; rewards; personality
1 Introduction
Over the past decade, there has been an increase in online communities (such as
Yelp.com) that provide review and rating information about businesses that customers
have come to rely on [4]. The ratings and reviews provided on Yelp have been shown
to have a direct impact on the revenue of businesses, with an increase in Yelp ratings
resulting in an increase in revenue [11], [4]. The quality of reviews play a huge role on
the possible influence such reviews have on customers [16]. High quality positive or
negative reviews are more helpful to customers than low quality reviews, hence per-
suading customers to post quality reviews is important to businesses. Although com-
munities like Yelp include tips on how to write useful reviews, not all reviewers adhere
to these tips, thus some reviewers write reviews that are not helpful to other customers.
Copyright © 2018 held by the paper’s authors. Copying permitted for private and academic
purposes.
In: R. Orji, M. Kaptein, J. Ham, K. Oyibo, J. Nwokeji (eds.): Proceedings of the Personalization
in Persuasive Technology Workshop, Persuasive Technology 2018, Waterloo, Canada,
17-04-2018, published at http://ceur-ws.org
35 Understanding Low Review Ratings in Online Communities: A Personality Based
Approach
It is therefore important to identify who these users are that continuously post unhelpful
reviews and explore means through which they can be influenced to post helpful re-
views. Thus, this paper aims to 1) identify the personality type of uses who post un-
helpful reviews and 2) explore their susceptibility to rewards as a form of persuasive
strategy.
The use of a person’s personality as a means of influencing a target behavior has
been explored in various sectors such as health [13] and marketing [17]. A person’s
personality includes the notable features, characteristics or qualities that form their dis-
tinctive characteristic [12]. There are various models that classify people based on their
personality traits, with people in each group having a high tendency to behave in a
particular way under certain situations. One such model is the Big Five Model which
describes a person’s personality using five dimensions: openness to experience, con-
scientiousness, extraversion, agreeableness and neuroticism [5]. In this paper, we iden-
tify the personality of users who often post unhelpful reviews using the Big Five Model.
We chose this model because it has been studied extensively in several domains includ-
ing health [13], marketing [8]and social networks [1].
In order to identify users who often post reviews that are not useful to the Yelp com-
munity, we used the dataset available from the Yelp challenge of 2017 1. We selected
users that have posted at least two reviews with ratings less than 3 (out of 5) and which
other users have not complimented for being useful (useful vote = 0). We then applied
the Linguistic Inquiry and Word Count (LIWC) [14] tool which classifies users into the
five dimensions of the Big Five Model. We were able to identify the five personality
types of the Big Five Model.
To determine the susceptibility of the various personalities identified to rewards as
a means of influencing them to post helpful reviews, we developed and tested a struc-
tural model using Partial Least Squares-Structural Equation Modelling (PLS-SEM).
The result of this analysis suggests that of the five personality types in the dataset, only
the users who score high in extraversion are likely influenced by rewards. These users
form a small fraction of the community (based on the dataset we worked with), hence,
rewards might not influence users who write unhelpful reviews to write better reviews.
The results of this paper can provide useful insights in designing persuasive strate-
gies that work in online communities.
2 Related Work
2.1 Yelp
Yelp2 is an online community that helps people locate businesses such as restaurants
and hotels within a geographical location. Yelp thrives on user generated reviews and
ratings which are written by patrons of such businesses. Yelp rewards its members who
write good reviews and are active in the community with the Elite status. Members of
1 https://www.yelp.com/dataset_challenge
2 www.yelp.com
Understanding Low Review Ratings in Online Communities: A Personality Based 36
Approach
the community can rate reviews on a scale of 0 to 5 stars. In addition, they can comple-
ment reviews by voting them as being useful, funny or cool on a scale of 0 to 5. The
more stars and compliments a user gets for his/her reviews, the higher the chances
he/she has of being rewarded with the Elite status.
Reviews in online communities such as Yelp is currently an active research area.
Luca [11] in his study of Yelp explored the effect of consumer reviews on the revenue
of businesses. Their study suggests that an increase in a restaurant’s rating on Yelp
results in an increase in the business’ revenue. Luca concluded that online consumer
reviews in Yelp are an alternative to traditional reputation systems. Huang et al. [7]
studied customers’ reviews to determine what the customers expect from various busi-
nesses. They explored restaurant reviews to identify what hidden topics customers dis-
cuss that can be useful to restaurant owners in improving their ratings on Yelp.
Despite the ongoing research in online communities like Yelp, to the best of our
knowledge, there has not been any work done on identifying users that post low rated
reviews and the susceptibility of these users to rewards.
2.2 Personality Type; The Big Five Model
The Big Five model (also referred to as the Five-Factor Model) is a popular model that
describes a person’s personality using five dimensions: openness to experience, con-
scientiousness, extraversion, agreeableness and neuroticism [5]. These dimensions
were derived from the analysis of common natural language terms people use to de-
scribe individual differences in themselves and others [9]. People with extraversion
traits are talkative, energetic and assertive while those with agreeableness trait are co-
operative, good-natures and can be trusted. Conscientiousness trait describes people
that are dependable, responsible and orderly, while neuroticism characterizes people
that are calm and are not easily upset. People with personality trait, openness to expe-
rience, are known to be imaginative, independent-minded and intellectual [9]. We used
the Big Five Model because it has been used successfully in various domains including
health [13], e-commerce [8] and online communities [1].
2.3 Linguistic Inquiry and Word Count (LIWC)
The Linguistic Inquiry and Word Count (LIWC) [14] tool reads text and determines
what percentage of words in the text reflect personality, emotions, thinking styles and
social concerns of the writer. LIWC works by calculating the percentage of given words
that match its built-in dictionary of words. It assigns a percentage value for each per-
sonality type or trait such as reward bias or thinking style. The LIWC dictionary con-
sists of about 6,400 words, word stems and emoticons. LIWC has been used extensively
in analyzing users in social communities with success. Bazelli et al. [1] used the LIWC
tool in exploring the personality of users in a popular question and answer social media,
Stack Overflow. Their research suggests that top contributors in the community are
extroverts. Romero et al. [15] also used the LIWC tool in their study of social networks.
37 Understanding Low Review Ratings in Online Communities: A Personality Based
Approach
The authors explored how the personality traits and behavior of decision makers in a
large hedge fund change based on price shocks.
Based on the popularity and success of the LIWC tool as reported by other research-
ers, we chose to use it in this research.
3 Research Design and Methodology
This paper aims to 1) identify the personality type of uses who post unhelpful reviews
and 2) explore their susceptibility to rewards as a form of persuasive strategy.
For this study, we used the dataset from the 2017 Yelp challenge 1. In order to identify
reviews that are not useful to the community, we used the ratings and “useful” compli-
ments of the reviews. We selected users that wrote at least two reviews which met the
following criteria:
Reviews that were rated less than the median possible score, two out of five.
Reviews that had a useful compliment of zero.
We did this because we hypothesize that a review that has a rather low rating and
has not been voted or complimented as being useful by any member of the community
is likely not a helpful review.
Of all the reviews in the dataset, only 78,240 reviews met this criteria written by
6,448 people. In order to have a lot of written text by the reviewer (which enhances the
validity of the LIWC tool), we further excluded reviews that were shorter than 200
words. 6,249 reviews written by 2,154 users met this criteria. These reviews were ex-
plored in this paper.
4 Data Analysis And Result
To analyze our data, we used LIWC [14]. We used this tool because out of the existing
tools and models for analyzing the Big Five Model’s personality types, the LIWC tool
has been widely used in various domains with success [1], [15]. Using the criteria de-
scribed above in section 3, we extracted the user ids and text of the 6,249 reviews that
met our set criteria in a CSV file. This formed the input to the LIWC tool. The LIWC
tool reads text and determines what percentage of words in the text reflect personality,
emotions, thinking styles and social concerns of the writer. LIWC works by calculating
the percentage of given words that match its built-in dictionary of words. According
to the Big 5, everyone exhibits some traits of all personality types, however one trait is
more dominant than the others [5]. LIWC computes a value for each of the personality
types (openness, conscientiousness, extraversion, agreeableness and neuroticism) for
each user; the dominant trait for a user is the personality type with the highest value.
The LIWC tool also computes a value for other traits such as thinking style and reward
bias. We used the value for reward bias in evaluating users’ susceptibility to rewards.
Understanding Low Review Ratings in Online Communities: A Personality Based 38
Approach
4.1 Result of LIWC analysis
The result of the analysis shows that there were more people who scored high on neu-
roticism compared to the other personality traits. Figure 1 shows the boxplot of the
ranking of the five personality types.
In order to ensure that there are differences in the five personality types identified by
the LIWC tool, we carried out an ANOVA with repeated measures test with a Green-
house-Geisser correction. The result shows that the overall difference in the users based
on the five personality types was statistically significantly different at F(3.538,
22101.249) = 2914.798, p < 0.0005). To further identify how each personality type dif-
fered, we carried out a pair wise comparison between the five personalities. Post hoc
tests using the Bonferroni correction revealed that all but two of the personality types;
conscientiousness and openness differed significantly. This shows that there is signifi-
cant difference in users that possess personality types agreeableness, extraversion and
neuroticism.
The LIWC analysis on the data set also revealed two clusters of users who post re-
views that are not useful. These are shown in figure 2.
Figure 1. A boxplot showing the overall ranking of the personality types of users by LIWC. The
horizontal line indicates a median ranking of 50 out of 100
Cluster1:
This cluster formed 52.46% of the users and is shown in blue color in figure 2. Users
in this category can remain calm and cope with stressful or unpleasant situations. In
addition, they are easy-going and relaxed. They take problems as they come, they are
patient and slow to anger. They typically make others feel relaxed and comfortable. In
terms of family orientation, they are not close with or focused on family. They place
themselves over family and familial relationships.
39 Understanding Low Review Ratings in Online Communities: A Personality Based
Approach
Figure 2. Clusters of users identified by LIWC tool
Cluster 2:
This cluster formed 47.54% of the users and is shown in orange color in figure 2.
This set of users are closed-minded. They are generally conventional and may be per-
ceived by others as stubborn. They are not close with or focused on family and they
prioritize themselves and friends over their families. In addition, this group of users are
likely to have a hard time experiencing enjoyment; they ruminate and may often expe-
rience stress. Furthermore, these users are likely to have difficulties controlling nega-
tive moods and may seem distant.
4.2 Susceptibility of personality types to rewards
Having identified the personality types of users who post unhelpful reviews, we ex-
plored the effect of rewards on these different personalities. Rewards have been iden-
tified as one way through which users can be motivated to participate in online com-
munities [3], hence we explored the effect of rewards on each personality type. To do
this, we developed and tested a structural model using Partial Least Squares – Structural
Equation modelling (PLS-SEM) with the results of the personality types and suscepti-
bility to reward bias derived from the LIWC tool. We used PLS-SEM because we
wanted to measure the influence of the various personalities on reward as an influence
strategy.
Path coefficients (β) and path significance (p) are important criteria in measuring the
validity of relationships between variables in structural models [6]. While path coeffi-
cients measure how one variable influences another, path significance determines how
significant (or not) that influence is [6]. The result of our model is shown in figure 3
Understanding Low Review Ratings in Online Communities: A Personality Based 40
Approach
with the individual path coefficients (β) and their corresponding level of significance
(p) shown in brackets.
The result from the structural model reveals that the personalities of users in Yelp
who post unhelpful reviews influence the persuasiveness of rewards. Our results sug-
gest that rewards are likely to motivate behavior only from people who are high in
extraversion (β = 0.433, p<0.0001) compared to the other personality types. Although
the path significance between neuroticism and rewards is significant (p<0.0001), the
path coefficient is however very low (β =0.102) [18].
Figure 3. PLS-SEM model structure showing path significance and p-values shown in brackets.
AGREE=Agreeableness, CONSC= Conscientiousness, EXTRA= Extraversion, NEURO= Neuroticism,
OPEN=Openness to experience.
5 Discussion and Limitation
Because reviews are important in online communities like Yelp, it is important to en-
courage users to post useful reviews that are helpful to the community. This paper aims
to identify users who post unhelpful reviews and the possibility of influencing them to
post useful reviews using rewards. To do this, we identified the personality type of users
who post unhelpful reviews using the LIWC tool. Our analysis suggests that users who
post unhelpful reviews are those with the personality type neuroticism (figure 1). Peo-
ple who score high on this personality type are typically moody, usually anxious, de-
pressed and lonely [5]. We further investigated the effect of the various personality
types on rewards as a means of influencing change in users. The result of the structural
modelling we carried out suggests that only users with personality type extraversion
41 Understanding Low Review Ratings in Online Communities: A Personality Based
Approach
are significantly influenced by reward, but these users form only a small fraction of the
total users who post unhelpful reviews (see figure 1). Therefore, one can conclude that
the use of rewards does not influence participation of users that post unhelpful reviews
in Yelp. This is because most unhelpful users score high on neuroticism, however, re-
wards influence mostly those who score high on extraversion and these users are few.
While research shows that online communities like Yelp offer incentives and re-
wards to influence participation, the approach commonly used seems to be a one size
fits all method [3]. Our results suggest that because users have diverse personality types
and these different personality types are influenced by rewards differently, personaliz-
ing rewards to an individual’s personality type (instead of a one-size-fits-all method)
or user characteristics (as described in figure 2) might be a better approach to influenc-
ing users to write useful reviews in Yelp.
Personalization using personality types has been successful in several domains like
health [2] and e-commerce [10], hence online community developers and stakeholders
should consider personalization of influence strategies when persuading users to con-
tribute to the community.
Our research is limited in a few ways. The result of our personality test is based
solely on the LIWC tool. We are confident with the result of this tool because it has
been used extensively in various domains with success [1], [15]. We however plan to
compare the results of the LIWC tool to other existing tools that identify personality
through text. Another limitation is the dataset. We used existing data provided by Yelp
which might represent only a fraction of Yelp users. We however believe that the ap-
proach presented in this paper can be applied to any online community.
6 Conclusion
Online communities like Yelp depend on its users to actively participate in the network
by writing useful reviews. However, not all users do so. The aim of this paper is to
identify the personality type of users who post low rated and unhelpful reviews and
determine the susceptibility of the various personality types to rewards as an influence
strategy to encourage the posting of useful reviews. We identify the personality types
and reward bias of users using the Linguistic Inquiry and Word Count (LIWC) tool.
The result of our analysis reveals that users who post unhelpful reviews are mostly of
the personality type neuroticism. We further explored the susceptibility of the different
personality groups of users to rewards as a means of influencing them to write useful
reviews. Our results show that only the users that are high in extraversion who post
unhelpful reviews are susceptible to rewards. This result demonstrates that rewards
might not be persuasive to most of the Yelp users who post unhelpful reviews, hence
other persuasive strategies should be explored to influence users to post helpful re-
views. The result of this study could be helpful to developers and stakeholders of online
communities.
In the future, we plan on comparing the result of the personality types determined
by the LIWC to other existing tools to determine the validity of the LIWC. In addition,
Understanding Low Review Ratings in Online Communities: A Personality Based 42
Approach
we will also explore what persuasive strategies can be implemented in Yelp for the
users that scored high in neuroticism since they are not influenced by rewards.
7 References
[1] Bazelli, B. et al. 2013. On the personality traits of stackoverflow users. Software
Maintenance (ICSM),. (2013).
[2] Busch, M. et al. 2016. More than Sex: The Role of Femininity and Masculinity in the
Design of Personalized Persuasive Games. 219–229.
[3] Easley, D. and Ghosh, A. 2016. Incentives, Gamification, and Game Theory: An
Economic Approach to Badge Design. ACM Transactions on Economics and
Computation. 4, 3 (Jun. 2016), 1–26. DOI:https://doi.org/10.1145/2910575.
[4] Floyd, K. et al. 2014. How Online Product Reviews Affect Retail Sales: A Meta-
analysis. Journal of Retailing. 90, 2 (2014), 217–232.
DOI:https://doi.org/10.1016/j.jretai.2014.04.004.
[5] Goldberg, L.R. 1990. An alternative description of personality: the big-five factor
structure. Journal of personality and social psychology. 59, 6 (1990), 1216.
[6] Hair, J. et al. 2011. PLS-SEM: Indeed a silver bullet. Journal of Marketing theory and.
(2011).
[7] Huang, J. et al. 2014. Improving restaurants by extracting subtopics from yelp reviews.
iConference 2014 (Social Media Expo. (2014).
[8] Huang, J. and Yang, Y. 2010. The relationship between personality traits and online
shopping motivations. Social Behavior and Personality: an. (2010).
[9] John, O. and Srivastava, S. 1999. The Big Five trait taxonomy: History, measurement,
and theoretical perspectives. of personality: Theory and research. (1999).
[10] Li, Y. et al. 2013. How Can Personalized Online Services Affect Customer Loyalty: The
Relationship Building Perspective. Service Science and Innovation (. (2013).
[11] Luca, M. 2016. Reviews, reputation, and revenue: The case of Yelp. com. (2016).
[12] Lumsden, J. and MacKay, L. 2006. How does personality affect trust in B2C e-
commerce? Proceedings of the 8th International Conference on Electronic Commerce
(ICEC’2006) (2006).
[13] Orji, R. et al. 2017. Towards personality-driven persuasive health games and gamified
systems. Proceedings of SIGCHI Conference on. (2017).
[14] Pennebaker, J. Linguistic inquiry and word count: LIWC 2001.
downloads.liwc.net.s3.amazonaws. ….
[15] Romero, D.M. et al. 2016. Social Networks Under Stress. Proceedings of the 25th
International Conference on World Wide Web - WWW ’16 (New York, New York,
USA, 2016), 9–20.
[16] Tucker, T. 2011. Online word of mouth: characteristics of Yelp. com reviews. Elon
Journal of Undergraduate Research in. (2011).
[17] Wang, C. and Yang, H. 2008. Passion for online shopping: The influence of personality
and compulsive buying. Social Behavior and Personality: an. (2008).
[18] Wong, K. 2013. Partial least squares structural equation modeling (PLS-SEM)
techniques using SmartPLS. Marketing Bulletin. 24, (2013).