In online book reviews readers often describe their reading experience and the impression that a book left. The great volume of online reviews makes these reviews a great source for investigating the impact books have on readers. Recently, a reading impact model was introduced that can be used to automatically identify expressions of reading impact in Dutch reviews and that is able categorise them according to emotional impact, aesthetic or narrative feeling, or feelings of reflection. This paper provides an analysis of the characteristics of the book review domain that afect how this computational model identifies impact. We look at features like the length of reviews, the nature of the website on which the review was published, the genre of book and the characteristics of the reviewer. The findings in this paper provide insight in how diferent selection criteria for reviews can be used to study various aspects of reading impact.
Online book reviews written by ordinary readers are an important feature of the ’Digital
Literary Sphere’ [
Recently, we introduced a reading impact model that can be used to automatically identify
expressions of reading impact in Dutch reviews and that is able to categorise them according
to emotional impact, aesthetic or narrative feeling, or remarks on reflection [
A pertinent question is how exactly tens of thousands of reviews can be harnessed to gauge
the impact of a novel on its audience. This cannot be a simple matter of tallying rule matches,
because some books have thousands of reviews while most others have none or only a few, and
some reviewers produce hundreds of reviews while many others write a few. Similarly, how
do we compare and aggregate a score of seemingly “calm and collected” reviews with a small
number of extremely passionately enthusiastic reviews that relate to the same work? Simple
tallying might justifiably lead to concerns about reductive handling of data, which is a concern
that is commonly raised as a problem with respect to quantified and computational approaches
in digital humanities [
This paper provides a first analysis of the characteristics of the book review domain that afect how this computational model identifies impact. We look at features like the length of reviews, the nature of the website on which the review was published, book genre and the characteristics of the reviewer. The operationalisation of our model prompts a number of research questions: • How can we translate matches of individual impact rules into an overall impact score for a review or a set of reviews for the same book? • How are impact rule matches related to other review characteristics, such as the length of the review or the website for which the review was written? • How are impact rule matches related to reviewers, reviewed books and book genres? We first discuss the background of analyzing reading impact and book reviews in Section 2, then describe the characteristics of the review dataset we use in Section 3. Then we analyse the relationship between review characteristics and impact matches and how to aggregate these into interpretable impact scores in Section 4. We close this paper with a discussion of our ifndings, their implications for future research and the limitations of our work in Section 5.
In this section we discuss related work on online book reviews before describing the Reading Impact Model in more detail.
The impact that reading fiction has on a reader has been studied for several decades. So far,
this has mostly been done through interviewing readers [
Online book reviews have also been used as a source to study literary reception, mostly through
close reading of a relatively small number of reviews [
Some potential issues with insincere reviews have been signaled. Authors and publishers
may game the system by writing positive reviews of their own books and negative reviews of
competitors’ books [
Reviews can also be written as a form of identity formation: reviewers are not just focusing
on their actual reading experience but care about how they are perceived by others and may
report a reading experience that is partly informed by their desired outcome [
Finally, diferences have been highlighted between reviews on book selling sites and those
on social book review sites, especially as to objectives and motivations of reviewers for writing
reviews [
The Reading Impact Model [
the phrases contain morphological variants (me|mijn) to compensate for such variation in the surface form of the sentence.
The model uses the four impact categories of Koopman and Hakemulder [
Several rules have the same impact term, like schitterend (English: ’beautiful’). If the sentence contains no specific book aspect, the expression will be categorized as ’emotional impact’, a general category with afective terms. However, if the sentence contains both schitterend and a story aspect like verhaal (’story’) or personage (’character’), the expression is labeled as ’narrative impact’. If schitterend co-occurs with a stylistic aspect like geschreven (’written’) or schrijfstijl (’writing style’), it is labeled as ’aesthetic impact’.
In this section we look at the characteristics of the reviews in the dataset and how these characteristics are related to matches from the Reading Impact model.
It is possible that certain kinds of reviews or certain kinds of reviewers have more matches
with the Reading Impact model than others, which might skew the overall picture we get for
the reviews of a certain book, author or genre. As is typical of web data [
Reviewers on book selling platforms like Amazon have additional aspects to review, such
as the acquisition process, and diferent motivations for writing the review, including to share
their experience with and opinion of the book seller [
The impact model comes with a matcher function that accepts sentences either as plain text strings or as syntactically parsed trees in the format created either by Alpino2 or spacy.io.3. Many rules are based on the lemma of a word instead of specific morphological variants. We
preprocessed all reviews using NLTK [
The collection of book reviews that we use consists of 472,810 reviews of fiction, written in
Dutch. The majority of reviews come from an earlier collection created by Boot [
• Lezers Tippen Lezers (LTL):9 a Flemish website where readers can find tips on what to read next and post reviews. • Wat Lees Jij Nu (WLJN):10 a small Dutch book review website that is no longer online.
The site places no restrictions on what members can review.
The reviews from the diferent platforms have some diferent characteristics, as shown in
Table 1. The majority of reviews come from Bol, which are shorter on average (85.8 words)
than those of other platforms. The Dutch Goodreads reviews have an average number of words
of 111.8 and an average number of sentences of 7.7, which is somewhat longer than reported
for English reviews from Goodreads by Dimitrov et al. [
The distribution of review length for the seven platforms is shown in Figure 1, using the number of words and sentences as units. The Y axis shows the probability of a review having a certain length. This normalisation to probabilities allows us to compare the reviews from diferent subsets of the collection. The plots use a logarithmic scale on the X axis. For the Y axis, the fraction of reviews are shown on a linear scale. The number of words (left) show that the review lengths for the seven platforms are roughly log-normally distributed.11 This means the diference between using 100 to 250 words is similar to the diference between using 10 to 25 words. Why is the word distribution log-normal? Probably because the review length is positive but open-ended for most platforms. Many reviews have at least a few dozen words to a 100 words (where the peaks of the distributions are). It is possible (though unlikely, as the plots show) to use thousands of words more than the median, but only a few dozens of words less than the median.
The word distributions show significant shifts between the distributions, indicating that there are platform-specific factors playing a role in how much text reviewers write. The reviews on Boekmeter, Hebban and Lezers Tippen Lezers (LTL) have fewer short reviews than the other platforms. Bol deviates strongly at the longer end of the distribution, where its distribution
10Originally at http://www.watleesjij.nu/, a short description can be found at https://mustreads.nl/watle esjij-nu/
11We confirmed this by fitting theoretical models on the data and computing the Residual Sum of Squares for the normal (RSS = 1.47e−5), log-normal (RSS = 2e−7) and exponential distributions (RSS = 1.3e−6). drops faster than the rest and stops at around 1000 words. This suggests an artificial limit, e.g. it looks like Bol restricts reviews to be at most 4000 characters. The longest review from Bol in our dataset is exactly 4000 characters, with another 178 reviews between 3950 and 4000 characters. Another deviation is seen in the Goodreads set, with many very short reviews of just two words. A manual check reveals that these are typically reviews saying e.g. ‘4 sterren’ (4 stars). Given that on Goodreads users also provide star-based ratings these reviews simply mimic the rating and provide no additional descriptive information. For computing reading impact it would be relatively easy to identify these reviews and, we argue, filter them out without negative consequences for the reading impact analysis. An additional advantage would be that the remaining Goodreads reviews have a length distribution that is more similar to those of the other platforms. Still, when we want to compare reading impact in reviews across platforms, we should take into account these diferences in length distribution.
The length distribution by number of sentences is shown on in the middle of Figure 1. There are few reviews longer than 40 sentences, but some are over 200 sentences. The longest is over 500 sentences and contains a very detailed summary of a book on finance. 12 Hebban has a higher proportion of reviews with more than 20 sentences than the other platforms.
Finally, the distribution of number of words per sentence (Figure 1) shows that on some platforms many reviews have some very short sentences, notably Goodreads and WLJN. But all platforms show a peak between 10 and 20 words and dropping of sharply after that, with virtually no sentences over 40 words. Also in terms of sentence length, the reviews from diferent platforms are comparable.
As well as by platforms, review lengths difer by other characteristics. In Figure 2 we show the average review length for nine genre groupings:13 fantasy, historical novel, literature, literary thriller, regional novel, romance, science fiction, suspense and youth (fiction for children of 13 years and older). As we note, reviews in the science fiction genre are clearly longer than in the other genres. This is mostly due to a larger number of sentences;14 the number of words per 12We filtered our reviews to be only on fiction books, but there are mistakes in the available genre information. 13The groups are combinations of on publisher-assigned genre codes (NUR). For instance, literature is a combination of NUR 301 (Dutch literary novel or novella) and 302 (translated literary novel or novella). The ifgures are based on the 242150 reviews for which we have the books’ NUR code.
14Post-hoc analysis using the Tukey-HSD test show SF difers significantly from the other genres for number of sentences and number of words. sentence vary only slightly.
In the next section (4) we will look at how these lengths afect impact. Here we look at how
genre afects subjectivity, which we define as the number of occurrences of first and second
person singular pronouns. We assume that sentences where the reviewers refer to themselves
(e.g. ’it made me laugh’) or to the reader of the review (’it will leave you speechless’) are
prime candidates to look for expressions of reading impact. We use LIWC to compute these
numbers [
The average review length is 7.1 sentences. But the average length of the reviews per book depends very much on the book. Figure 4 shows the distribution of the average number of reviews by book (left) and by reviewer (right). Some titles with on average very short reviews are Harry Potter and the Half Blood Prince, The Devil Wears Prada and Shopalicious! Titles with on average longer reviews include the thrillers I Am Pilgrim and Passenger 23. Similarly, reviewers vary widely in the amount of text that they produce.
If review length is related to the number of impact expressions, then it is important to understand how review length is related to other aspects of reviews. For instance, popular books have more reviews than unpopular or obscure books, but their reviews might also difer in length. Perhaps popular books get more short reviews than less popular books. The same applies to reviewers. Reviewers who write many reviews may write longer or shorter reviews than reviewers who write only a few reviews. We split the review set into three subsets with a low, medium and high number of reviews per book or per reviewer. Because the distribution is highly skewed, we use thresholds at diferent orders or magnitude, with low, medium and high respectively corresponding to 0 < x ≤ 10, 10 < x ≤ 100 and x ≥ 100 reviews per book or per reviewer. Figure 5 shows the distribution of review length for diferent subsets of reviews, for books (left side) and reviewers (right side). For the split in reviews per book, the Low, Mid 15Confirmed by post-hoc analysis. and High frequency sets contain 38%, 47% and 14% of all reviews respectively. For the split in reviews per reviewer, the sets contain 57%, 23% and 19% of the reviews.
On the left side of Figure 5, this frequency split is shown for the number of reviews per book. The diferent subsets have very similar distributions, with the reviews for high frequency reviewed books having a larger fraction of short reviews and a lower fraction of long reviews. The dotted vertical lines show the per-subset mean length of the logarithm of the number of words. The KL-divergence, which is a number to quantify the distance between two distributions, between the overall distribution and each of the three subsets is 0.01, 0.00 and 0.02 respectively. Although the diferences are statistically significant (a one-way ANOVA with Tukey post-hoc tests shows all paired diferences are significant with P < 0.001), they are very small. The take away message is that, as far as review length is concerned, popularity (in terms of number of reviews per book) causes no big distinction between books with diferent numbers of reviews. So even though individual reviews difer drastically in length, when aggregated at the book level, book popularity does not introduce a hurdle in comparing reading impact scores between books.
On the right side of Figure 5, the frequency split shows stronger diferences. The reviews by frequent reviewers tend to be longer than those of infrequent reviewers. The KL divergence between the overall set and each of the three splits is 0.05 for Low frequency, 0.04 for Mid and 0.13 for High frequency reviewers, and again all post-hoc tests are statistically significant with P-values well below 0.001. In other words, when comparing individual reviewers on reading impact scores, there is a review length efect that should be taken into account.
For each review we compute rule matches in diferent categories. But how do we get from the numbers of matches to numbers that help us compare the impact in individual reviews or the impact of a book, author, genre or reviewer, based on a set of reviews? More generally, we want to know to what extent we can generalize the findings from applying the Reading Impact model on (a subset of) the review dataset.
First of all we should note that counts of impact matches only make sense in comparison. To say that person x’s review of book y has narrative impact count z is only meaningful in comparison with another review. But there are other reasons why the raw numbers by themselves, even if averaged over book or genre (etc), are insufficient. The first is that, as we showed above, the length of reviews difers systematically by book or genre (etc). The second reason is that even within a single review there is no way to compare the impact numbers for diferent categories. We discuss these reasons in the next two subsections.
In the analyses in this section, we leave out the impact matches in the reflection category
as the impact rules in this category have not been validated with human judgements, whereas
emotional impact, aesthetic feeling and narrative feeling have [
The Reading Impact model uses word lemmas and longer phrases (continuous and discontinuous phrases where there can be words in between the impact phrase parts) to identify expressions of impact. Longer phrases cannot be matched with very short sentences. Very long sentences can match with many impact rules and as they tend to contain a larger number of distinct words, they also have a higher a priori probability of matching with impact rules. Therefore, there might be a sentence length efect on impact matches that has consequences for comparing impact scores across reviews, or subsets of reviews where one subset has longer sentences than another.
To gauge the extent of this efect we analyse the probability of matching impact for the various categories for sentences of diferent length (Figure 6, left). Of the 3.36 million sentences in our review dataset there are 866,541 sentences that match at least one impact rule (26%). So on average, one in four sentences contains an expression of reading impact. As expected, the probability of matching at least one rule of any of the four categories goes up with sentence length, though not indefinitely. For very long sentences of several hundreds words, the probability is lower than for sentences between 100 and 200 words. A manual inspection shows that many of these very long sentences are highly idiosyncratic. For instance, one such sentence has a few hundreds commas followed by a statement how these represent only a fraction of the superfluous commas in the book: „„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„
„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„ „„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„ „„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„„ „ Dit is een fractie van het aantal teveel gebruikte komma’s in dit boek.
Although the reviewer is conveying their dislike of the use of commas in the book, the rules do not consider this an expression of impact.
For emotional impact (blue line), sentences around 10 words have a lower probability of matching emotional impact rules than sentences of one or two words. This is probably due to many one or two word sentences being no more than an evaluative term like schitterend (’beautiful’), or aanrader (’recommended’). Without any further context, these are categorized as overall emotional impact. This has consequences for the distinction between popular and less popular books. As we saw in Figure 5, books with many reviews tend to have a somewhat higher proportion of very short reviews. If we would average impact scores for a book based on the fraction of reviews that express impact in a certain category, or on the density of impact expressions in the review, the many short reviews would boost the emotional impact score for popular books more than the scores for other impact categories. Put another way, popular books would have a tendency to score higher on emotional impact than non-popular books because of the short reviews. Whereas, if we were to compute scores by considering the amount of review text, these short reviews would have less influence than long reviews, because the longer reviews tend to have more overall matches in all of the categories. Although it is not entirely clear at this point how this insight should inform our choices for how to aggregate scores from reviews of the same book, at least this analysis has made us aware that the choice we make has consequences for how popular books compare to non-popular books.
As mentioned above, in longer reviews one would expect higher impact counts. It could be argued that, if e.g. genres have difering lengths of reviews, we should take that into account and look at impact counts per sentence or per word. But this is actually a tricky question. Do readers who write longer reviews also write more about the impact? If that is true, we have to correct for length to avoid giving more weight to the verbose reviewer. Or is a longer review longer because reviewers include a longer summary of the story, presumably more factual? In that case correction for length of review would be unfair to the verbose reviewer.
A total of 347,491 reviews in the dataset have at least one impact match (73% of all reviews). The probability that a review has at least one impact rule match is shown on the right in Figure 6. We see that, at the level of whole reviews, very short reviews have a very low probability of matching impact rules. One or two word reviews have 2% probability, with three, four or five words this goes up to 5%, 7% and 10% respectively. We draw two lessons from this observation. First, the low probability at the review level is in stark contrast with probability at the sentence level shown on the left of Figure 6, where very short (one or two word) sentences have an almost 20% probability of having an impact match. These very short reviews necessarily have very short sentences, but these are not ones with impact matches. Therefore, the very short sentences with impact matches must come from longer reviews. Second, the low probability at the review level means that in a set of reviews for e.g. the same book, a higher proportion of very short reviews results in a lower proportion of reviews with impact matches. In Figure 5 we saw that popular books have a somewhat higher proportion of very short reviews than less popular books. This finding suggests that, to compare popular and less popular books, the diferent proportions of short reviews needs to be taken into account. Or more generally, that in weighting the importance of finding an impact match in a review, we should take the review length into account.
Are there important diferences in how often reading impact expressions are found in reviews for books with diferent levels of popularity, or in reviews written by reviewers with diferent levels of reviewing frequency? These questions are important to understand whether such diferences in frequency are an underlying cause of any diferences observed when comparing reading impact across a specific set of books. If reviews of popular books would be much more likely to contain, for instance, expressions of aesthetic impact than less popular books, then observing this diference in comparing a Harry Potter book against a relatively unknown fantasy novel does not necessarily tell us much about how these specific books difer in terms of aesthetic impact. It is possible that popular books are more popular because their aesthetic impact is part of their appeal, but it is also possible that their popularity draws a reviewer’s attention to the writing style. The same goes for comparing two books, one of which having reviews by mostly frequent reviewers, the other having reviews by mostly infrequent reviewers. If frequent reviewers write longer reviews and have a checklist of aspects to include in their review, while infrequent reviewers write short reviews with only the first thing that comes to mind, then it is possibly more worthwhile to think about why diferent books draw diferent types of reviewers than looking at the impact they express. If we find no frequency efects, it is easier to interpret diferences between specific (sets of) books.
The impact of book and reviewer frequency in the collection on the mean number of impact
expressions per review of a certain impact type, is shown in Figure 7. We use the same
frequency levels as in Section 3.4. On the left the proportions are shown for books with a
low, medium and high number of reviews. In the following analysis, we measured statistical
significance of diferences using the Kruskal-Wallis test by ranks and Bonferroni-Holm post
hoc tests. These are non-parametric tests that do not assume normality of data distributions.
All diferences between impact types per book level are significant ( P < 0.001), between
book levels per impact type we find non-significant diferences for Emotional impact between
low and high (P = 0.74) and for Aesthetic feeling between medium and high (P = 0.49).
For emotional impact, the diferences between books with a low, medium or high number of
reviews (the blue bars) is small. For the two more specific impact types, the diferences are
more pronounced. Low popularity books tend to provoke more expressions of aesthetic feeling
than more popular books, but fewer expressions of narrative impact. This could mean that
more frequently reviewed books are more narrative-driven and draw the reader into the story
world, while less frequently reviewed books tend have more noticeable writing styles. It could
also mean that books with few reviews tend to be read and reviewed by reviewers who focus
more on style. On the right the proportions are shown for reviewers, and all diferences are
significant with P < 0.001. Here we see a diferent patterns. First, reviewers who write more
reviews tend to use more expressions of impact of all types. This is likely related to the fact that
they tend to write longer reviews. Second, reviewers who write few reviews use more generic
expressions of emotional impact than expressions of either aesthetic or narrative feeling, while
more prolific reviewers tend to use more expressions of narrative feeling than generic expressions
of emotional impact. Third, there is a upwards trend in the relative proportion of aesthetic
feeling to emotional impact expressions. As mentioned above, it is possible that reviewers
who write many reviews are more likely to go through a list of aspects they want to cover in
their reviews, e.g. plot and stylistics elements. This would make sense if reviews follow genre
conventions [
Obviously we would want to control for possible skewed relations between the number of impact matches of diferent kinds in a review. The question in this case is: is there some meaningful relation between e.g. the number of matches revealing narrative impact and those matches related to style? We can gauge this by pairwise plotting the number of matches of each impact category per review (Figure 8). The correlations of these pairwise scatter plots is 0.25 on average (cf. the column ”Correlation” in table 2). The relatively low correlations may mean that in further analysis it would be advisable to normalize the number of matches in categories Narrative ∼ Aesthetic Narrative ∼ Emotional Emotional ∼ Aesthetic when comparing across diferent impact categories.
We note that these correlations are not evenly distributed across reviewers. This can be shown by dividing reviews again in three categories depending on whether they are written by low, medium, or high frequency reviewers. For this we use the same procedure as in Section 3.4. The number of reviews per reviewer as a distribution is, as mentioned before, very heavily skewed. Reviewers writing only one review account for about 36 percent of the total amount of reviews while a long tail of reviewers produces hundreds of reviews per person with one reviewer topping out at 827 reviews. As can be gauged from table 2 the correlation between numbers of impact matches from diferent categories climbs as the number of reviews per reviewer increases. This trend becomes the more clear when we refine the procedure for dividing reviews according to reviewers’ frequencies of reviewing so we can produce a more continuous graph of correlations (cf. Figure 9). The trend may be an efect of reviewers that write reviews more frequently adopting a more regular structure for reviews, dedicating balanced space to diferent kinds of reader interests. This efect may thus be indicative of a developing genre convention.
A closer inspection of outliers having particular large and unbalanced impact matches (e.g. a review with 42 aesthetic feeling matches but only 10 narrative feeling matches) reveals that these reviews are almost without exception ”user generated data” artefacts that are rather atypical for online reviews. Mostly they are aggregate reviews constructed by compounding the findings of four or five readers into one review. Such reviews should of course not be ignored, but they do not seem to provide much useful additional information as to the point in question of determining how reviews can be made comparable for reader impact.
The previous sections have shown that, when comparing reviews or sets of reviews in terms of identified impact expressions, simple counts do not represent a meaningful impact score. There are diferent characteristics that afect how likely it is to find impact expressions in reviews, such as the length of a review. For very short reviews, it is much less likely than for reviews of a few hundred words. Therefore, to find an expression of impact in a very short review is more surprising – and we assume more significant – than finding one in a long review. This suggests we should weight impact rule matches diferently based on review length.
But how can we incorporate these diferent characteristics as part of the evidence for calculating an impact score?
Starting from intuition, we consider two assumptions. One is that a short review is a signal that not all impact has been expressed. Another is that the book had little impact. A simple solution to compensate for length is dividing the number of impact matches by length. But this takes into account only the first assumption. A single word review with a single impact match would score ten times higher than a 100 word review with 10 impact matches. To allow for the second assumption as well, length normalization should not be linear.
The probability curves for Emotional impact, Narrative feeling, and Aesthetic feeling on the right-hand side of Figure 6 show an almost linear trend for the logarithm of review length (in words). This suggest an alternative solution, namely, to divide by the log of the length. Although the curves drop after 900 words, this is possibly due to the size and composition of the review collection. A larger sample with more reviews from platforms that introduce no length constraints might show curves that flatten out above a certain length instead of drop down, as there is no reason why a longer review cannot have many expressions of reading impact. A generic way of compensating for review length would be to use a logarithm-weighted 1 normalization. That is, the number of matches I(ri) for a review ri is weighted by log(|ri|) , where |ri| is the length of the review.
We show the impact of length-weighted normalization on the reading impact scores in Figure 10 for narrative feeling (left) and aesthetic feeling (right). The bars show the relative average impact score16 for six popular books: The shadow of the wind by Carlos Ruiz Zafón, The Da Vinci code by Dan Brown, Fifty shades of grey by E.L. James, The girl on the train by Paula Hawkins, The girl with the dragon tattoo by Stieg Larsson and Sarah’s key by Tatiana de Rosnay. The blue bars show the relative average impact per review using the number of matches, while the orange bars show the weighted scores. The overall diferences between the distributions of absolute and normalized number of impact matches are significantly diferent (Kruskal-Wallis, P < 0.001). However, the weighting has almost no efect on the statistical significance of diferences between books compared to the original impact score. In most cases, what is significantly diferent using the impact counts is significantly diferent after normalization, and similar for what is not. The first thing to note is that by averaging over a large number of reviews (The girl on the train has the fewest reviews, with 504 reviews), including many very short reviews, the weighting has a relatively small impact on the relative scores. But there are subtle changes. It is not the case that weights compress all the scores so that the impact always becomes more similar across books. The highest narrative impact score–for Paula Hawkins’ The girl on the train–drops while for a few of the others the scores go up, but several lower scoring books always have a lower weighted relative score. The diferences in score between The girl on the train and the other books are all significant ( P < 0.001), apart from Sarah’s key by Tatiana de Rosnay. After weight normalization, E.L. James scores significantly diferent on narrative feeling than all the others. For aesthetic feeling (on the right) we see a similar pattern. The weighting does not necessarily reduce the diferences between books. Here, the scores for the books by Carlos Ruiz Zafón and Tatiana de Rosnay are significantly diferent from each other and all the other books (Conover-Iman, P < 0.001), the others are not diferent from each other ( P > 0.05).
However, if we compare the weighted and non-weighted relative average impact scores for an individual book across platforms, the typical pattern is that the diference between the platform with the highest average score (Hebban, which has reviews that tend to be longer 16The relative score of each book is the average impact score for that book divided by the sum of averages of all six books. By turning both the weighted and non-weighted scores into proportions, we can directly compare them. than those of other platforms) and the other platforms becomes smaller. Figure 11 shows this for Carlos Ruiz Zafón’s The shadow of the wind, but for the other five books, the trend is the same. This suggests that the weighting is efective in reducing diferences between review platforms and makes their reviews more comparable. However, these diferences across platform remain large, so further investigation is needed into the possibility that reviewers write diferent reviews for diferent platforms, either because platforms have diferent review writing conventions and reviewers modify their reviewing style to each platform, or because the diferent platforms attract diferent types of reviewers, who write diferent kinds of reviews or who experience books diferently.
This paper provides an in-depth data analysis of the characteristics of online book reviews, to
gain insight in how they are related to the reading impact that is expressed in them. Our aim
was to find an informed approach to translate impact expressions as identified by the reading
impact model [
Because collections of online reviews, like other user-generated content on the web, are skewed towards short reviews and popular books, we first analysed how the length of reviews is related to 1) the online platform on which the reviews were published, 2) the number of reviews that a reviewer has written, 3) the popularity of the reviewed books, and 4) book genre. We found that review lengths difer somewhat across platforms, either because of diferent length restrictions imposed by the platform or diferent motivations for writing a review on book selling platforms versus social cataloguing platforms, so reviews cannot be straightforwardly compared across platforms without taking these diferences into account. There is no substantial diference in review lengths between popular and non-popular books, indicating no underlying length biases when comparing sets of reviews across diferent books. However, review length is related to the number of reviews that a reviewer has written.
Next, we found that the probability that a review contains an expression of reading impact
grows close to log-linearly with the length of reviews, which suggest we should take this
relationship into account when comparing aggregate scores per review, book, author or genre.
We used this to derive a reasoned method for normalizing the number of impact matches by
review length. The impact of weighting is relatively small for books with many reviews, and
does not flatten all diferences between books, and in some cases makes them more pronounced.
However, for diferent review platforms with diferent communities of reviewers and diferent
motivations to write reviews, our findings suggest that weighting makes reviews more
comparable in terms of scoring impact, although there seem to be more aspects playing a role than
length alone. Length normalization reduces only a small part of the diferences across
platforms. Furthermore, we found that frequent reviewers write reviews that are more consistent
in length and in balancing impact expressions related to narrative and aesthetics. This might
a signal that frequent reviewers adopt or create genre conventions [
In future work, we want to investigate these frequent reviewers and genre conventions of online book reviews in more detail, as well how they relate to the platforms that the reviews are published on. Another aspect to look at is the types of books that reviewers read and review in terms of popularity and genre, and investigate how reading impact for a genre difers across types of readers. [31]