=Paper=
{{Paper
|id=Vol-1533/paper9
|storemode=property
|title=Decision Making through Polarized Summarization of User Reviews
|pdfUrl=https://ceur-ws.org/Vol-1533/paper9.pdf
|volume=Vol-1533
|authors=Paolo Cremonesi,Franca Garzotto,Matteo Guarnerio,Francesco Gusmeroli,Roberto Pagano
|dblpUrl=https://dblp.org/rec/conf/dmrs/CremonesiGGGP15
}}
==Decision Making through Polarized Summarization of User Reviews==
https://ceur-ws.org/Vol-1533/paper9.pdf
Decision Making through Polarized
Summarization of User Reviews
Paolo Cremonesi ,
Franca Garzotto ,
Matteo Guarnerio ,
Francesco Gusmeroli , and
Roberto Pagano
Politecnico di Milano, DEIB
P.zza Leonardo da Vinci 32, Milan, Italy
Abstract. When buying a mobile phone, booking an hotel, or watching
a movie, many people rely on the reviews available on the Web. However,
this huge amount of opinions make it difficult for users to have a compre-
hensive vision of the crowd judgments and to make an optimal decision.
In this work we provide evidence that automatic text summarization of
reviews can be used to design Web applications able to effectively reduce
the decision making effort in domains where decisions are based upon
the opinion of the crowd.
1 Introduction
People’s decision making is ever more influenced by the “voice of the crowd”.
When choosing an hotel, a restaurant or a movie many people rely on the reviews
available on the Web, trusting other users’ opinions more than the “official” ones
by critics, guidebooks, experts, or similar. At the same time, we are witnessing
a proliferation of user generated content. For instance, Amazon collects up to
16,000 new reviews per day1 , while TripAdvisor has recently passed the 150
million mark in terms of opinions posted to its site, now collecting more than 90
user contributions per minute2 .
This huge mass of web based user judgements encapsulate a potentially re-
liable “ground truth” that, in principle, can drive the decision making process.
Yet, too many pieces of information make it difficult for decision makers to get a
comprehensive vision of the crowd judgement and its global polarity. The goal of
our research is to help users overcome this problem. Bounded Rationality The-
ory [1] posits that decision-makers continuously try to find a balance between
the knowledge needed for the optimal decision and the effort required to pro-
cess such knowledge. When they lack the ability and resources to arrive at the
1
http://minimaxir.com/2014/06/reviewing-reviews/
2
http://ir.tripadvisor.com/releasedetail.cfm?ReleaseID=827994
2 Authors Suppressed Due to Excessive Length
optimal solution, they tend towards a suboptimal result after having greatly sim-
plified the choices available. In line with this theory, our approach is to decrease
user’s effort by reducing the amount of information to be cognitively processed,
offering users only a distilled vision of the voice of the crowd. More precisely
our approach is to present summaries of all available reviews that capture the
salient aspects of the crowd’s judgement and are appropriate to build at least
suboptimal solutions.
Summarization is emerging as a topic of growing importance. Summly3 , a
news summary service designed to help simplify the way users consume news
on mobile devices, was recently acquired by Yahoo for 30 million dollars. In
comparison with this service, as well as with other academic works [4,3]we have
a different motivation: supporting decision making. In addition, we employ a
different and novel technique that does not summarizes a single text but many
texts. Finally we provide two summaries, respectively summarizing positive and
negative judgements, in order to offer a potentially more balanced and trustable
vision of crowd’s opinion.
We have performed our studies in the domain of restaurant booking. We con-
ducted two separate user experiments: the first to chose the best summarization
technique for the target domain, the second to demonstrate the effectiveness of
the summarization technique as a tool to reduce the decision making effort.
The first research question we wish to answer is to assess which technique
among the proposed ones generates the summary which better represents the
entire set of reviews. To answer this question, we performed an online study
with 33 participants. For each participant: (i) a random restaurant is picked
among the entire dataset; (ii) the restaurant’s reviews are provided to the user
alongside with the 12 summaries (two for each of the six techniques, one for the
positive opinions and one for the negative ones); (iii) the user is asked to rank
the summaries based on his opinion on how well the summary summarizes the
reviews. The best technique according to this study is Edmunson [2] which uses
a statistical approach with a different weighting scheme for different classes of
features (cue words, keywords, title, location of words inside the corpus).
2 Summaries and Decision Making
In a second user experiment we studied if polarized summarization technique
provides benefits for the user decision making process. For this purpose we have
developed a web application that mimics the functionality of TripAdvisor, but
limited to the restaurants in New York City. Each user involved in the experiment
was asked to simulate the booking of a restaurant. Users were randomly split
into two experimental conditions: (i) with original TripAdvisor reviews (Figure
1a) and (ii) with positive and negative summaries only (Figure 1b). A total of
108 users participated to this study.
In order to estimate the decision making effort, we measured the the average
time each user spent in reading the description pages of restaurants. Users in
3
http://summly.com/
Decision Making through Polarized Summarization of User Reviews 3
the second experimental condition – with summaries of positive and negative
reviews for each restaurant – spent, on average, half of the time with respect to
the users in the first experimental condition – with the full set of reviews.
3 Discussion and Conclusions
The results from the second user experiment confirm that the summarization of
reviews effectively reduce the effort of users in the decision making process.
Although this is only a preliminary study, its internal validity is supported by
the accuracy of the research design. In terms of external validity, the applicability
of our results might not be confined to the specific domain of restaurant booking,
as in many decision making domains users rely their decision on the opinion of
the crowd.
In order to provide better evidence of the effectiveness of summarization as
a decision making tool we need to strengthen our study by (i) providing a better
engagement for users, (ii) by collecting data from a larger user base and (iii)
by collecting other metrics to measure the effectiveness of the decision making
process.
References
1. P. Cremonesi, F. Garzotto, and R. Turrin. Investigating the persuasion potential of
recommender systems from a quality perspective: An empirical study. ACM Trans.
Interact. Intell. Syst., 2(2):11:1–11:41, June 2012.
2. H. P. Edmundson. New methods in automatic extracting. Journal of the ACM
(JACM), 16(2):264–285, 1969.
3. L.-W. Ku, Y.-T. Liang, and H.-H. Chen. Opinion extraction, summarization and
tracking in news and blog corpora. In AAAI Spring Symposium, volume 100107,
2006.
4. C.-S. Lee, Z.-W. Jian, and L.-K. Huang. A fuzzy ontology and its application to
news summarization. Systems, Man, and Cybernetics, Part B: Cybernetics, IEEE
Transactions on, 35(5):859–880, 2005.
4 Authors Suppressed Due to Excessive Length
Romantic escape 5/5
My husband surprised me with a dinner out at Little Owl during our recent trip to NYC and it
is a date that I will not soon forget. The atmosphere is wonderful, with low, warm lighting
and the buzz of conversation.
Overpriced and overrated 2/5
Overpriced and overrated...Tryin to be like my spot Kitchen, but not as cool
don’t expect too much 3/5
although the little owl might get rave reviews, i find the food very average. the best thing
about it is that the service is down to earth and there is the cutest red door. not somewhere
i’d go out of the way for, but decent.
(a)
3,64 % of people agree with:
We went for a Sunday Brunch. The wait time was over 30 min. and when we were finally
seated, our table was assigned to a very rude waitress who looked rather surprised that it
was our first visit and we were clueless about their menu and hence was asking about
some of the dishes (the names are European, so we couldnt quite figure out the dish from
the names). Finally when the order came, the portion sizes were so tiny that we were thinking
of heading to another restaurant to complete our meal. And they don’t have a dessert menu
for brunch!! Super service&staff! Tryin to be like my spot Kitchen, but not as cool.
96,36 % of people agree with:
My husband surprised me with a dinner out at Little Owl during our recent trip to NYC and it
is a date that I will not soon forget. The place is super small so we were very lucky. A short
menu prepared from a cupboard sized kitchen means everything was super fresh. So nice to
experience such great service in a super busy city!. Nice cocktails and super service. The
servers are super friendly and welcoming in a casual sort of way, you almost feel as if you’ve
been invited over to their house to eat, its a good vibe all around. It was syrupy and super
vanilla driven, and was great with the cheese.
(b)
Fig. 1: Restaurant page (a) with original TripAdvisor reviews and (b) with
positive and negative summaries in place of reviews