Online reviews play a critical role in persuading or dissuading users when making purchase decisions. And yet very few users take the time to write helpful reviews. Encouragingly, recent advances in AI offer good potential to produce review-like natural language content. However, The main challenge is a lack of large, high-quality labeled data at both the aspect and sentiment level for training the automated review generators. Hence, we study the feasibility of a writing assistant framework in order to help users post online reviews and introduce a data-driven approach to label data required for training. We evaluate the effectiveness of our approach by launching a user study on how end-users perceive the quality of the labels and automated generated reviews.
________________________________________________________ Workshop proceedings Automation Experience across Domains In conjunction with CHI'20, April 26th, 2020, Honolulu, HI, USA Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Website: http://everyday-automation.tech-experience.at
Intelligibility; review generation; review writing assistant; user-based evaluation
There is a growing attention in creating new methods to
help users share their opinions on review platforms. For
example, Airbnb site require hosts and guests to write
mutual reviews [
Seed data [
Yelp Dataset. It is used to train the review generator containing about 4M reviews on 60k restaurants.
Since users express opinion on different aspects of the target in a single review, a review cannot be labeled in whole to state a specific aspect (see Figure 1). We first split a review into its topically coherent segments and propose to label the resulting segments.
The review segmentation algorithm traverses through each
review sentence by sentence in order to cluster coherent
sentences into one segment. At its core, review
segmentation is based on a sliding window technique with a window
size of two. Each sentence is compared with the right most
sentence in the previous segment and if their distance is
less than a specific threshold , then the sentence is added
to the segment, otherwise it forms a new segment. This
process continues until the end of the review in linear time.
We adopt the Word Mover’s Distance (WMD) [
The label assignment algorithm is based on a small seed set. The main intuition is to find semantically similar seeds to the unlabeled segments and use their labels to identify the aspect of the segments. For this purpose, we compare each sample in the seed set against the unlabeled segments using the WMD distance function. The unlabelled segment receives the label of closest seed sample.
Table 1 demonstrates examples of segments obtained from the segmentation algorithm and their labels assigned by the label assignment algorithm across different aspects and steak sandwich was delicious and the caesar salad had an absolutely delicious dressing with a perfect amount of dressing and distributed perfectly across each leaf . i know i m going on about the salad . today was my second visit to the place after having a good first experience but i am so disappointed with the quality of the food that i can say it has been my worst experience of food in months the sun dried tomatoes very absolutely stale to an extent that they tasted bitter the pizza base was so thick that it was uncooked and soggy the four cheese blend tasted completely different than the last time and so did the pesto sauce . no consistency with food quality . the ambiance is nice too . it s a bit dark but they have this nice light display above on the ceiling made with mason jars . there is a comfy seating area in the bar area that s nice too . however the one thing that surprised me was how dirty the restroom was in this restaurant . the floor was really dirty and toilet papers were unwell kept . the restaurant could at least have someone maintained the restroom in good shape and clean because this will reflect on how one maintains the cleanliness of the place . the price is very reasonable for a family of four with plenty of leftovers to take home . my wife i had a groupon for this place and for the price it was very poor value quality . i had a nice glass of california cabernet . the wine list while not expansive was good . the bartender i had seemed to have a nice knowledge of what was going on with the wine that encompassed it . i ordered a glass of Merlot that was delivered to me in a dirty glass . the waitress was very polite and went to get me a new glass of wine but i was still unimpressed at that point . highly recommend for lunch . even during lunch rush it was not super packed . this would be a good place for a lunch meeting . General (+) i am not sure why anyone would like this place . the only thing it has going is location and that is simply not enough not for me . sentiments.
We evaluate if automated labels are comparable with manual labeling. We set up a crowd-based user study to verify if the labels are assigned truthfully according to human readers. We post 100 surveys on Amazon Mechanical Turk (AMT) each including a guideline and a set of reviews for which we seek a label from Turkers. The guideline has two major points: (i) it shows a sample of reviews along with their labels from the seed set to provide a context on how reviews and labels are paired with each other, (ii) it asks Food (+) Food (-)
General (-)
Ambience (-)
Price (-)
Turkers to label the reviews through a series of multi-choice questions. We design 100 surveys each with 10 reviews to cover all the labels. Each unique survey is assigned to three workers, i.e., 3 HITs (Human Intelligence Task) per task, giving us a total of 300 surveys and 3,000 questions. To ensure the quality of responses, we insert a trivial question into each survey, which asks the Turker to check if a mathematical equation is False or True. It helps to manage the risk of blindly answered surveys. Furthermore, we only accept surveys from Turkers with approval rating of at least 95% and those who dwell on the survey for at least 7 minutes. We also restrict our tasks to workers located in the United States to guarantee English literacy.
Table 2 demonstrates the performance of automatic labeling against human judgment across various labels. The majority of the labels are found accurate by human evaluators with at least 80% accuracy. However, the accuracy for labels Price/negative and Drink/negative is relatively low and we can relate this to the fact that these labels do not have a significant representation in the seed set.
Here, we introduce a generative model based on language neural networks to automate user review generation. The main intuition is to propose a sequence to sequence model where the first sequence encodes the labels of the segments obtained from previous step into a context vector. This context vector along with review words are the input to the second sequence in order to train the review generator. Formally, Given the aspect A as input attribute, we aim to generate a review sequence R = (w1; ::; wjRj) constrained by the input to maximize the conditional probability p(RjA): jRj p(RjA) = Y p(wtjw<t; A) t=1 (1) where w<t refers to the tokens seen until time step t.
Similar to label assessment, we launch a crowd-based user study by posting surveys on AMT. We follow similar guidelines to ensure the quality of the answers. We design 100 surveys each with 10 generated reviews at various aspects. We assign 3 HITs per task, giving us a total of 300 surveys and 3000 questions. We ask Turkers to label the modelgenerated reviews through a multi-choice questions based on the aspect.
From Table 3, we observe that generated reviews stay with the desired aspect with higher than 90% accuracy for a majority of the labels. For example, 93% and 97% of reviews on Food (positive and negative) are perceived equally by the model and the human evaluators while this number is 34% for drink/negative. We can relate this to the fact that the label Food has a better representation in both seed set and our expanded dataset as it is the main topic of discussion when writing a review for a restaurant.
We have explored how to make use of automation to help users writing online reviews in particular when sufficient data required by neural networks are not available. We automate the review writing process at two steps: (i) build a ground truth of reviews at aspect and sentiment level and evaluate the effectiveness of the labels by human readers, (ii) propose a generative model that produces reviews conditioned on input aspects and evaluate the quality of the generated reviews through user study. In the next step, we aim to study how users are willing to use the proposed system and how we can incorporate their intention at design level.