In this paper we introduce a classification system devoted to predict the helpfulness of Italian online reviews. It is based on a wide set of features reflecting the different factors involved and tested on different categories of TripAdvisor reviews. For this purpose, we collected the first Italian corpus of online reviews enriched with metadata related to their helpfulness and we carried out an in-depth analysis of the most predictive features.1
Predicting and modeling relevant factors that
determine the helpfulness of online reviews have
been attracting a growing attention in the
Natural Language Processing (NLP) community. Both
practical applications and the interest to study
human variables underlying the assignment of
helpful/unhelpful votes are mainly involved. The
identification of product reviews which are useful to
customers can be important for several e-business
purposes (e.g. the development of product
recommendation systems) as well as to investigate
persuasive elements that make a review helpful for
a review reader
In this paper, we address these open issues and we present a study devoted to predict Italian review helpfulness with a specific focus on the role played by linguistic features in modelling the style of helpful reviews. Similarly to previous studies, we tackled the task as a text classification problem but with two main novelties. Firstly, we relied on different sets of predictors, considering both lexical (content) and structural features (i.e. morphosyntactic and syntactic) aimed at reconstructing the style of a text (the linguistic “form”). Secondly, we investigated which typology of features are the most effective to predict the helpfulness of online reviews and whether they remain the same across different review categories.
Our contribution. i) We collected a corpus of Italian online reviews enriched with metadata related to their helpfulness2. ii) We developed the first classification system devoted to predict the helpfulness of Italian online reviews, based on features modelling both lexical and linguistic factors involved, and tested it in two experimental scenarios, i.e. in- and out-domain with respect to the training category of reviews. iii) We identified and ranked the most predictive features, showing the key role played by linguistic features, especially to predict the helpfulness of reviews belonging to a category very different from the training one. 2
We collected a sample of almost 1 million
usergenerated reviews from the Italian section of
TripAdvisor, focusing on two travel-related
categories, restaurants and attractions (e.g. parks,
historical sites), and two geographical areas, Rome
2The corpus is available for research purposes at
http://www.italianlp.it/resources/
and Milan. We also gathered two types of
metadata associated with each review: review rating
and number of helpful votes. Firstly, we filtered
our data according to language (Italian) and length
(> 7 tokens), discarding 52.29% of the total
reviews. Then we empirically3 set a threshold at a
minimum of 3 votes in order to distinguish helpful
reviews (3+ votes) from unhelpful ones (0 votes).
Some examples of reviews that belong to the two
classes are reported in Table 2. In line with
studies carried out for the English language
Category Rome rest.
Milan rest.
Attractions
TOTAL
#Helpful #Unhelpful #Reviews
12,635 12,404 25,039
6,105 5,991 12,096
2,564 2,408 4,972
21,304 20,803 42,107
According to our research purposes, we
considered various categories of features aimed at
modeling both the content and the linguistic “form” of
online reviews. They can be grouped into three
main classes: lexical, linguistic and metadata
features. The first typology has already been tested
in the literature
In order to extract lexical and linguistic
predictors of helpfulness, the corpus was linguistically
annotated at different levels of analysis. In
particular, it was tagged by the PoS tagger described
in Dell’Orletta (2009) and dependency-parsed by
the DeSR parser
Lexical features. They include two types of
features: (i) the distribution of unigrams and bigrams
of characters, words and lemmas (hereafter NGR);
(ii) word embedding combinations (WE) obtained
by separately computing the average of the vector
representations of nouns, verbs and adjectives in
the review. The word embeddings were trained
on the ItWaC corpus
Linguistic features. They refer to four main
types, modelling diverse aspects of writing style:
raw text features, i.e. review, sentence and word
length, calculated in terms of sentences, tokens
and characters, respectively;
features related to lexical richness, which is
captured considering i) the internal composition of
the vocabulary of review with respect to the Basic
Italian Vocabulary and its usage repertories
Milan restaurants
Example (Italian) La prima regola di un buon ristorante che fa pizza no stop e`: Scegliere la pizza che preferisco. Qui non solo non si puo` scegliere la pizza ma capita spesso che escano le stesse pizze piu` volte cos`ı uno e` costretto a mangiare sempre la stessa!! Per non parlare dell’ambiente poi, un vero casino, capisco che l’area bambini e` la principale attrazione del ristorante, rivolto soprattutto alle famiglie, ma il casino che si crea non e` cmq giustificabile. La pizza e` di una qualita` davvero scadente, praticamente era cruda!!! La pizza con la Lonza....una semplice focaccia con un pezzo di prosciutto preso molto probabilmente al discount! Ragazzi, carina l’idea di prendersi cura dei pargoli, ma non prendiamoci in giro pero`.
Devo dire che trovandomi per caso in quella zona con i miei amici abbiamo provato il posto e` devo dire che´ e` molto accogliente e che la zona per mangiare nel cortile e` proprio intima e carina...Per quanto riguarda il mangiare posso dire di essere soddisfatto perche´ le portate erano nelle mie corde ed avendo preso il pesce ero soddisfatto di quanto cucinato dal cuoco. Bravi mica male.
Example (English) The first rule of a good restaurant that makes pizza no stop is: Choose the pizza I prefer. Here you can not only choose the pizza but it often happens that the same pizzas come out more times so one is forced to always eat the same one!!! Not to mention the environment then, a real mess, I understand that the children’s area is the main attraction of the restaurant, aimed above all at families, but the mess that is created is not justifiable anyway. The pizza is of a really poor quality, practically it was raw!!! Pizza with Lonza....a simple focaccia with a piece of ham most probably taken at the discount store! Guys, nice idea to take care of the little ones, but let’s not fool around.
I must say that finding myself by chance in that area with my friends we tried the place and I must say that it is very welcoming and that the area to eat in the courtyard is really intimate and pretty...
As for eating I can say I’m satisfied because the courses were on my ropes and having caught the fish I was satisfied with what the cook had cooked. ded prepositional chains modifying a noun; v) a influence the voting process outcome. The higher set of features aimed at modeling the behaviour of sentence length also has an expected effect on verbal predicates, i.e. the number of verbal roots, some syntactic features correlated to complexity. the average verbal arity and the distribution of Sentences occurring in helpful texts have deeper verbs by arity, the distribution of verbal predicates with elliptical subject; vi) the usage of subordisyntactic trees (Avg. max depth) and contain more subordinate clauses and embedded prepositional nation, calculated considering the ratio between chains.
However, they appear as simpler with principal and subordinate clauses, and the average respect to other features related for instance to depth and the distribution of embedded chains of canonicity effects. They show a more standard subordinate clauses; vii) a last set of features resyntactic structure, with a higher distribution of lated to the canonical construction of a sentence objects in post verbal position and subjects prein Italian, i.e. the relative ordering of subordinates ceding the main verb. Interestingly, helpfulness with respect to the main clause and of subject and is also positively correlated with a reader-focused object with respect to their verbal head.
The effectiveness of these features to predict helpful online reviews is confirmed by the fact that according to the Wilcoxon rank sum test, style, as shown by the greater use of pronouns and verbs in the first and second person.
Metadata feature. Review star rating (STR) is the rating score assigned by the reviewer, rang75% of the considered features (i.e. 160 out of ing from 1 to 5. Previous research reported in 212) turned out to vary in a statistically significant Diaz and Ng (2018) has shown that a connecway between helpful and unhelpful reviews. As tion exists between the rating of the review and shown in Table 3, helpful reviews are on average its helpfulness. In our dataset rating scores are 1-sentence longer than unhelpful ones and they unequally distributed across the different review also contain much longer sentences. The correlacategories. Restaurant reviews are more likely to tion between length and helpfulness is not surprishave an extreme rating, either low or high, rather ing since longer sentences are likely to be more inthan a neutral one, and helpful reviews follow the formative, thus offering more contents that might same pattern: e.g., in the Rome restaurant category 37.05% of the helpful reviews have a rating of 1 and 25.76% a rating of 5. On the contrary, attractions reviews tend to have higher ratings, with 56.12% of the helpful ones belonging to the highest-rated class. Only the attractions category seems to confirm the presence of the positivity bias that is discussed in Diaz and Ng (2018), according to which reviews with positive ratings are seen as more helpful. 4
We addressed the helpfulness prediction task as a
binary classification problem. In order to assess
the contribution of each set of features illustrated
in Section 3, we defined two experimental
scenarios differing at the level of review categories
chosen as test data and set-up (in terms of
feature configurations). We built a classifier based on
the LIBLINEAR implementation of Support
Vector Machines with a linear kernel
In the first experimental scenario we tested the
feature models generated by the SVM classifier
on a test set of 12,523 reviews that belong to the
same domain of the training data (i.e. the Rome
restaurants category) but were written for
restaurants different from the ones in the training set.
As shown in Table 4, we obtained a general
improvement over the baseline with all feature
configurations apart from the one that exploits only
the metadata feature (STR, the star rating of the
reviews). Nevertheless, this feature does improve
the accuracy score of all models by at least one
point, thus confirming its usefulness for
helpfulness prediction
Model STR NGR NGR+STR WE WE+STR NGR+WE NGR+WE+STR LING LING+STR ALL ALL+STR Baseline
Accuracy 49.6% 69.9% 71.13% 68.54% 69.96% 70.17% 71.14%
66% 70.81% 70.04% 71.05% 50.46%
In the out-domain scenario we tested the considered feature models on reviews that belong to the other two categories (Milan restaurants and attractions). As reported in Table 5, we observed that the performances of the classifier tested on the reviews of Milan restaurants, even if slightly worse, are very similar to the ones obtained on the test set of Rome restaurants. This result suggests that the system may perform consistently across different geographical areas, although further experiments should be carried out. For example, we might test our models on a greater number of cities or other types of geographical areas. As we expected, the accuracy decreases mainly in the domain more distant from the training one (i.e. the attractions category). This is especially the case of the lexical classification model, that has a drop of 10.5 points. The star rating feature is also shown to worsen the accuracy scores, probably because of the way the ratings are distributed in the attractions category with respect to the restaurant ones. It is interesting to note that the best performing model resulted to be the one exploiting the linguistic features (with a lower drop of 5.24%), thus showing the predictive power of sentence structure information in predicting review helpfulness.
Model NGR+WE NGR+WE+STR LING LING+STR ALL ALL+STR Baseline As discussed in the previous section, we found out that linguistic features allow achieving an accuracy almost in line with the one obtained using only lexical information. Interestingly enough, they are the most predictive ones in the out-ofdomain scenario. In order to gain insight into which of these features are the most effective in the task of automatic classification, we ranked them according to the absolute value of their weight in the linear SVM model generated with the linguistic feature configuration. Among the 50 top-ranked ones, besides the raw text features (whose role in predicting helpfulness has already been proven in the literature), we found morphosyntactic and syntactic features. They are typically related to a rich and articulated writing style. This is the case for example of features concerning nominal modification, in particular the number of prepositional chains (holding the 1st position in the ranking) and their average length but also the distribution of adjectives and determiners. Others involve verbal structures, e.g. the number of dependents instantiated by the verbal heads and the frequency of adverbs (especially negation ones). Features related to the usage of subordination, such as the number of subordinate structures and the average depth of parse trees, also appear among the top-ranked. Finally, another group of high-ranked features concerns a subjective writing style, as shown by the distribution of verbs in the first and second person. These types of features resulted to be discriminant in the comparison between helpful and unhelpful reviews (Section 3). This shows that the writing style of helpful reviews, informative but also personal and readerfocused, has an high predictive power.
The importance of the linguistic features is further confirmed by a second inspection in which the same ranking method was applied to the allfeature model. Also in this case, we found out that 59.6% of the whole set of 212 linguistic features we considered is in the 90th percentile of the ranking of the total 741,339 features. 6
In this paper, we have presented the first approach to the task of review helpfulness prediction for the Italian language. Two experimental scenarios have been tested in a corpus of TripAdvisor reviews belonging to different categories (restaurants and attractions). In line with previous findings obtained for the English language, we confirmed that lexical information plays a significant role in classifying helpful reviews. In addition, we proved for the first time the highly predictive power of linguistic features modeling the writing style independently from the content. This is particularly true in the two out-domain experiments: in the first case (same category, different geographical area), the classifier based on the linguistic features achieves the same accuracy of the model using lexical features and it even outperforms all the other configuration models when tested on the most distant review category (restaurants vs attractions).
Among the possible future issues that we would like to investigate, an interesting one concerns the role played by metadata features. In the reported results, we showed that star ratings are not relevant when considered alone, but they give a plus when combined with both lexical and linguistic features. Beyond this metadata, we would like to extend the analysis to further user information possibly related to review helpfulness.
This work was partially supported by the 2-year project PERFORMA (Personalizzazione di pERcorsi FORMativi Avanzati), co-funded by Regione Toscana (POR FSE 2014-2020).