In the context of the Web 2.0, user-genrated reviews are becoming more and more prominent. The particular case of book reviews, often shared through digital social reading platforms such as Goodreads or Wattpad, is of particular interest, in that it ofers scholars data regarding literary reception of unprecedented size and diversity. In this paper, we test whether the evaluative criteria employed in Goodreads reviews can be included in the framework of traditional literary criticism, by combining literary theory and computational methods. Our model, based on the work of von Heydebrand and Winko, is ifrst tested through the practice of heuristic annotation. The generated dataset is then used to train a Tranformer-based classifier. Last, we compare the performance of the latter with that obtained by instructing a Large Language Model, namely GPT-4.
These authors contributed equally. £ simone.rebora@univr.i(tS. Rebora);gabriele.vezzani@univr.i(tG. Vezzani) ȉ 0000-0002-1501-3774 (S. Rebora)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1Although the specificities of the recommendation algorithms implemented by Amazon are not shared with the general public, at the following link it is possible to access an updated list of the algorithms (with relative publications) that obtained the best performance over the Amazon-Book dataset (a dataset containing information for over 3 million user generated book reviewhst)t:ps://paperswithcode.com/sota/recommendation-systems-onamazon-book as the ones that can be drawn from these platforms. Designing ways to analyze this data will allow us to shed a whole new light on the dynamics that regulate literary evaluation.
Literary studies have been preoccupied with evaluation since around the 1980s, when, thanks
to the influence of the emergent field of postcolonial studies [
According to these authors, society influences literary evaluation by setting up standards against which books are judged. This would imply the existence of criteria independent from any specific evaluative act, that manifest themselves across a multiplicity of such acts, thus ofering a means to systematize them. In fact, as von Hydebrand and Winko have shown, although they can vary greatly depending on the historical context, these “standards of value” can nonetheless be organized according to some general features: they can concern the aesthetic level of a book (its style, characters, plot, etc.), the way it relates to other works (thus being original or derivative, for instance), the impact it has on its reader, and so on.
In this paper, we build on von Heydebrand and Winko’s theory and interpret online reviews as acts of “linguistic evaluation”. What distinguishes such acts from more implicit forms of evaluation (even the simple purchase of a book could be interpreted as such) is that they “require a standard of value - as well as certain categorizing assumptions - in order to progress from the description of a text to its evaluation1”2[, p. 227]. Seen as evaluative acts of this kind, book reviews are not simple expressions of individual taste, but the result of a socially learned practice with specific schemes and regularities. Diferent kinds of evaluative criteria can then be taken as the axis along which to situate single reviews, thus quantifying their variability and reducing their complexity. Furthermore, knowing which criteria are implemented by an individual (or a group) could allow us to predict their future judgements or to reveal the societal influences operating on them.
To make the identification of evaluative criteria as objective as possible, we turned to the
practice of “interpretative markup”, a form of annotation “devoted to recording a scholar’s
or analyst’s observations and conjectures in an open-ended way2”0[, p. 202]. As argued by
Gius and Jake [
The paper is organized as follows. In sectio2n, we present the corpus that constituted the basis of our work. The discussion of the annotation process occupies Secti3o,nwhere we outline the tagset we developed3(.1), the workflow we followed and the main problems we encountered 3(.2). To automate the annotation process we tried two diferent approaches: finetuning a Bert-based classifier and instructing a Large Language Model (GPT 4). The results of these approaches are presented, respectively, in Section4s.1 and 4.2.
The dataset used to perform the annotation task was built on the same basis of the AbsORB
dataset: a selection of “approximately six million English language reviews of nine diferent
genres (i.e., fantasy, romance, thriller, horror, mystery, science fiction, historical fiction,
contemporary, classics)” 1[
A first partitioning of our corpus, composed by 11 reviews, was used as a toy dataset to train the annotators, and was not therefore retained for analysis. The remaining 89 reviews, which constituted our main corpus, had a mean length of 1155 tokens (SD = 449). They were written by 84 diferent reviewers in a timespan going from 2008 to 2018. The number of reviews per year is lightly skewed towards more recent dates reflecting the growth of the Goodreads platform. There are 84 diferent books reviewed in the corpus, spanning across 9 diferent genres (with most of the books belonging to more than one genre). The distribution of reviews per genre and per year can be seen, respectively, in Figur2esand 1.
All copyright and privacy implications in using such a dataset have already been discussed
[
The starting point for the creation of our tag set was von Heydebrand and Winko’s model, which systematizes literary evaluation based on the diferent kinds of criteria that can be employed for such a task. However, in order to be adapted to our material and goals, it needed to be simplified. First of all, as demonstrated by the high number of categories they devised, it is clear that von Heydebrand and Winko aim at maximizing the exhaustivity of their model. Our intent was slightly diferent. In fact, we needed to account for the highest possible number of scenarios with the fewest possible categories, in order to reach a number of examples per category high enough to train a classifier, without having to annotate an extreme number of reviews.
Furthermore, despite their intent to take into consideration the contributions of potentially any actor within the literary field, it is quite clear that the kind of evaluative acts that von Heydebrand and Winko had in mind while designing their model were quite diferent from what one can find today on digital social reading platforms (and quite understandably so, given that their essay predates the advent of the latter by almost ten years). Many of their categories reflect the judgment patterns of, if not professionals, at least cultivated readers, capable of assessing the salient stylistic features of a given book, as well as its positioning in the overall landscape of literary tradition. On the other hand, online reviews “are mostly expressions of consumer satisfaction or dissatisfaction”26[, p. 3]. In them, the individual dimension is magnified, with evaluations hinging around the reader’s personal relation with the book.
In our tag set, we devoted a special attention to criteria based on one’s own thoughts, feelings and personal experiences. However, not all can be traced back to the individual level. By contrast, we believe literary evaluation to be a complex phenomenon, involving not only a reader and a book, but also the whole community in which the interaction between the former two takes place. To account for such complexity, we have incorporated in our tag set two further dimensions alongside that of individual criteria: one for evaluations based on the book itself, its content-related or formal aspects, and one for considerations on its societal value, its impact on a given community of readers. Furthermore, we have included a label for evaluations that do not fall into any of the aforementioned categories. In AppendAix,we list the 7 labels of our tag set, with a short description for each. Figu3r,eon the other hand, shows the overall structure of the tag set.
After a brief training regarding the task they were going to perform, two annotators were asked to independently annotate all the reviews in our corpus, attributing one (and only one) label to each of their sentences.
The choice to take sentences as the basic unit of our analysis allowed us to organize the annotation in a very straightforward way, by providing annotators with tables where each row corresponded to one sentence, to which they had to associate a label in a dedicated column (columns containing the title of the reviewed book, the review and the sentence id numbers were also included). In the cases where an evaluation spanned across multiple sentences, annotators simply had to attribute the same label to each one of them.
A first partitioning of our corpus, composed by 11 reviews, was used as a toy dataset to train the annotators (annotations were therefore not retained for analysis). The actual annotation work was carried out on the remaining 89 reviews in 4 consecutive rounds, each followed by a meeting during which cases of disagreement were discussed, as well as possible shortcomings of our tagging system. Inter-annotator agreement was computed using Cohen’s Kappa. The scores for each round are reported in Tab1l.e
Considering the entirety of the tag set (‘all_labels’), the coefÏcients are in the range of a
moderate agreement 1[
The absence of agreement registered for the ‘social’ label in the first three rounds can be
explained by the extremely limited number of occurrences of this category, which made it
difÏcult to develop specific guidelines. To understand such scarcity, let’s remember that the
label was thought to account for references of socially established evaluative standards, such
as prestigious prizes and canonical works. In many cases, such standards are the manifestation
of an academic – as in the case of canonical works, see1[
More than giving rise to a stable and universal canon, on Goodreads the social component of literary evaluation seems to operate at the level of specific literary genres. Indeed, both annotators reported noticing that reviews tended to follow genre-specific evaluative patterns. One of these patterns, concerning the over-represented genre of young adult novels (in Fig2usrpelit between romance and fantasy) could be at the heart of the problems we encountered with the labels in the ‘individual’ category, reflected in the erratic trends of the corresponding kappa coefÏcients. Something that became apparent already from the first rounds of annotation, is that readers of this genre tend to develop very strong personal relationships with the characters in the books they read, either treating them as if they were real persons or completely identifying with them. This phenomenon gave rise to an ambiguity that caused a good part of the disagreement between annotators. Take, for instance, the sentence “there is a depth to his character that bit by bit when revealed you can’t help but fall that much harder for him because his character demands nothing less4”. Apart from the poor grammar, one notes right away that the reviewer is evaluating the way a character is built, which would make the sentence fall under the category ‘aesthetic’. However, far from being a neutral assessment of a stylistic feature, the sentence is highly emotionally charged, which would justify the label ‘ind_emotional’. Lastly, based on the patent involvement of the reader in what they are reading, one could argue for the attribution of this sentence to the ‘ind_pragmatic’ category. To face such ambiguity, we decided to tag as ‘aesthetic’ all the sentences that contained an explicit reference to literary art (e.g., plot, characters, style).
The last step after annotation has been a curation phase, during which we ‘settled’ the cases of disagreement between the annotators and attributed a definitive label to the respective sentences. This was necessary to allow for the subsequent phase, the training of a classifier, for which each sentence in our dataset needed to be assigned to one and only one label. In extremely rare cases (less than one percent of the sentences) we also intervened to correct attributions that, despite the annotators’ agreement, were blatantly wro5ng. 4https://www.goodreads.com/review/show/225468198 5The reason for the presence of such cases is not hard to find, given that, as we have said, several problems with
The following step in our project relied on the annotated dataset to train a classifier able to
recognize evaluative acts in large amounts of text. In recent years, two main approaches have
become dominant in machine learning for text classification. The first implies fine-tuning base
(or task-specific) Transformer models thanks to the knowledge stored in annotated datasets
[
Analyses have been carried out with a series of Python scripts, which can all be consulted in the project’s GitHub repositor6y.
For the first approach, the 6,014 annotated and curated sentences were used as ground truth to fine-tune multiple Transformer models of the Bert family. We decided to test the following three models: • google-bert/bert-base-uncased (from now on, referred to agsoogle-bert), as representative of a large, general-purpose model for English language; • LiYuan/amazon-review-sentiment-analysis (from now on,LiYuan), because it was finetuned on a similar kind of data (general product reviews) for a similar kind of task (sentiment analysis); • JoelVIU/bert-base-uncased-finetuned-amazon_reviews_books (from now on,JoelVIU ), because it was finetuned on an even more coherent dataset (Amazon book reviews). Given the already-discussed issues of unbalance and underrepresentation in the annotation labels, we decided to simplify them by adopting two diferent strategies: our tag set and guidelines were addressed during the course of the annotation itself, thanks to the insights we gathered at each new step. 6https://github.com/SimoneRebora/CHR2024 1. we reduced the labels to three classes: ’eval_individual’ (under which the three categories related to the impact on the individual reviewer were merged), ’eval_generic’ (under which all the evaluation categories were merged), and ’no_val’; 2. we further reduced the labels to two classes: ’eval’ (under which all the evaluation/impact labels were merged) and ’no_val’.
Testing was first performed via a 5-fold cross validation, to establish which one among the chosen models was the most performant. As shown by Tabl2e, performance depended on the task, with the JoelVIU model slightly outperforming the others for the three-label classification and google-bert producing the best results for the binary classification. However, it should be noted that the three-label setup produced low F1-macro scores because of the substantial failure in classifying the ’eval_individual’ sentences (F-1 scores for that label were never higher than 0.144 throughout models/folds), due probably to the very low number of available samples in the dataset (444 sentences, corresponding to 7.4% of the total). In light of these considerations, we decided to use thegoogle-bert model as a reference point for our subsequent analyses.
A second level of analysis concerned the fitness of the number of annotated sentences to efectively train the classifier. To address this question, we performed a series of 5-fold cross validations by fine-tuning the model with an increasing number of sentences (from 600 to 6,000, selected randomly from the dataset). Figur4eshows how, for both the binary and three-class classification, F1-macro scores reach a plateau at around 3,000 sentences, thus suggesting that the learning threshold for the classifier is substantially lower than the number of annotated sentences.
A third and final level of analysis was chosen in order to get efÏciency scores comparable to the ones obtained with Large Language Models (see section4.2). Here the selection strategy was changed slightly, by randomly choosing entire reviews instead of single sentences, without performing any k-fold cross validation. After testing diferent configurations, we selected one with 18 reviews in the test set (corresponding precisely to 20% of the sentences in the dataset, with comparable distributions of labels), which produced efÏciency scores similar to the ones obtained before (for an overview, see Tabl3eand 4).
Given the unsatisfactory performance reached on the three-labels dataset (the model simply ignored the under-represented ’eval_individual’ category) we decided to share only the model ifnetuned for the binary classification of evaluative and non-evaluative sentences, which can now be freely accessed on Hugging Face7.
For the second approach, the annotation guidelines were used as a starting point to develop system prompts for the GPT-4 Large Language Model. By situating itself in the emergent area of “prompt engineering” 4[], the work here became more exploratory, aiming at the identification 7https://huggingface.co/GVezzani/literary_evaluation_classifier of the best technique to instruct the model.
In the first phase of testing, the overall prompting strategy was a simple zero-shot (i.e., instructions plus input), with system prompts (i.e., the instructions) composed by adopting three diferent approaches: • The first, defined as complex, was a straightforward adaptation of the annotation guidelines in their latest stage. Note that guidelines were loosely structured, with occasional repetitions and multiple addenda; • The second, defined as simple, was a drastic simplification of the annotation guidelines, extracting only the most relevant information and giving it a simpler structure; • The third, defined as procedural, was inspired by the chart in Figur3e and by the “Tree of Thoughts” (ToT) prompting technique 4[], producing a more structured prompt, where the assignment of a label was the result of a set of nested choices.
These three approaches were then combined with three diferent tagsets: • The first, defined as full, adopted the 7 labels originally used by the annotators (note that this setup was not even tested with Bert models, because of the scarcity of annotations for some labels); • The second, defined as 3-class, reduced the labels to the first setup with Bert models: ’eval_individual’, ’eval_generic’, and ’no_val’; • The third, defined as binary, further reduced the labels to the second Bert setup: ’val’ and ’no_val’.
For each approach, the system prompt produced with thfeull tagset was then adapted into 3-class and binary by performing the minimum amount possible of modifications, so as to keep the core of the prompt intact. Final result was a set of 9 diferent system prompts, which can be consulted in AppendixB.
To profit from the ability of GPT-4 to process large amounts of text and to emulate as closely as possible the work of the annotators, we decided to give it as an input entire reviews split into sentences. The user prompt was therefore structured as a .csv file with three columns: ’book_title’ (containing the title of the reviewed book), ’sentence_id’ (with a numeric identifier of the sentence), and ’sentence’ (with the sentence text). At each trial, GPT-4 was prompted 18 times, with the 18 reviews identified with the criteria described in section4.1.
Requests were processed by using the “gpt-4o-2024-05-13” model (with temperature set to 0 to produce the most deterministic behavior) on the OpenAI API. The whole operation cost a total of 3$. Comparisons of the GPT-4 annotations with the ground truth are shown by Table 5. Note that the adaptation of thefull tagset to 3-class and binary, and of the 3-class tagset to binary could also be accomplished ex post (i.e., after having performed the annotation with GPT-4).
Overall, GPT-4 efÏciency is higher than fine-tuned Bert models for the 3-class condition (in fact, GPT-4 performs substantially better on the ’eval_individual’ label, overcoming the issue of its underrepresentation in the dataset), while it is lower for bthineary condition. When comparing the diferent system prompts, the complex approach performs slightly better than the procedural one, suggesting how a structured prompt may not be necessary to obtain the
3-class highest efÏciency. Finally, ex-post tagset adaptation produces mixed results (with the best efÏciency for the 3-class and binary tagsets obtained with and without adaptation, respectively), even if in the majority of the cases (6 out of 9) it worsens efÏciency. Tables6, 7, and 8 show more in detail the efÏciency of the best-performing setups.
In the second phase of testing, we adopted a “few-shot” strategy by using as a basis the best performing setup c(omplex approach withbinary tagset). The “few-shot” strategy implies providing not only instructions, but also examples for performing the task. Examples were extracted from the remaining 71 reviews, selecting the ones that showed the proportion of ’val’ vs. ’no_val’ labels closest to the overall mean in the dataset (i.e. the most “balanced” ones). These reviews were presented to the model together with the curated annotations, before asking it to annotate the new reviews. Three diferent tests were then performed, with an increasing number of sample reviews: • two reviews, corresponding to 172 sentences (the whole operation cost 0.66$); • four reviews, corresponding to 354 sentences (1.21$); • eight reviews, corresponding to 596 sentences (1.95$).
As prices increased substantially, we decided not to test bigger selections of sample reviews. Also, quite surprisingly, this prompting technique had a detrimental efect on the efÏciency of the model, with F1-macro scores decreasing to 0.780 (with two sample reviews), 0.754 (four reviews), and 0.724 (eight reviews).
We believe that our work can contribute to the field in several ways. First, the development of
a tag set to capture evaluative criteria in unstructured reviews enriches the current search for
possible ways to operationalize literary evaluation and study it from an empirical perspective
[
Furthermore, the application of the aforementioned tag set during the work of annotation
revealed some interesting features of the material analyzed. First, let’s note that many scholars
seem to interpret online book reviewing as animated by a distrust for (if not a full-fledged
opposition to) more traditional practices of literary criticism, as embodied by bourdeauian
“gatekeepers” [
The detection of genre-specific evaluative patterns is another interesting finding that
deserves to be further investigated in future works. Two interpretations for such result can be
hypothesized: first, that the diferent features of books of diferent genres [
Last, many researchers are searching for computational ways to analyze the evaluation of
books through user-generated online review3s0[
This research was developed in the context of the “Inclusive Humanities” Excellence Project at the Department of Foreign Languages and Literatures of the University of Verona. A. The Tag Set Here’s a list of the labels in our tag set, each followed by a brief description. • no_val: The sentence does not express an evaluation of the reviewed book • generic_val: Into this category fall all those evaluations related to the work as a whole (“beautiful book,” “highly recommended,” “an unbearable read,” and so on). Also, tag with this category all evaluative sentences that do not fall into any of the following categories. • aesthetic: Any evaluation concerning the specifics of literary language, both in its formal aspects (use of rhetorical figures, writing style, etc...) and content aspects (character or plot construction, narratological features, etc...). • social: This value concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book). Of particular interest here are all those judgments that seek to enact (or to reafÏrm) a canonization of the judged work, that is, to place it on the roster of ’important’ readings. • ind_cognitive: Evaluations regarding the cognitive impact of a book on a reader, the information that the latter extracted from the former or the intellectual stimulation they experienced while reading it. • ind_pragmatic: Evaluations regarding the impact of the book on the reader’s life or the existential “lessons” that the latter learned from the former • ind_emotional: Evaluations regarding the emotional impact of the book on the reader, the way the former made the latter feel.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines.
When the sentence does not express an evaluation of the reviewed book, assign the label ”no_val”.
When the sentence expresses an evaluation of the reviewed book, you will have to choose between six diferent labels.
1. ”aesthetic”: Any evaluation concerning the specifics of literary language, both in its formal aspects (use of rhetorical figures, writing style, etc...) and content aspects (character or plot construction, narratological features, etc...).
The following three labels do not refer to features present in the text (such as formal values), but rather to the impact it had on the reader. They are divided into:
2. ”ind_pragmatic”: Evaluations regarding the impact of the book on the reader’s life, the existential ”lessons” that the latter learned from the former.
3. ”ind_emotional”: Evaluation regarding the emotional impact of the book on the reader, the way the former made the latter feel. 4. ”ind_cognitive”: Evaluation regarding the cognitive impact of a book on a reader, the information that the latter extracted from the former or the intellectual stimulation they experienced while reading it.
The last two labels are: 5. ”social”: This value concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book). Of particular interest here are all those judgments that seek to enact (or to reafÏrm) a canonization of the judged work, that is, to place it on the roster of ’important’ readings.
6. ”generic_val”: Into this category fall all those evaluations related to the work as a whole (”beautiful book,” ”highly recommended,” ”an unbearable read,” and so on). Also, tag with this category all evaluative sentences that do not fall into any of the above categories.
In assigning one (and only one) of these labels to each sentence, please follow these generic guidelines:
- Work on individual sentences. ONLY in cases where one sentence is incomprehensible without the next, or expresses a concept that necessarily requires continuation in the next, treat the two as a single block and assign them the same label.
- If judgments related to several categories are made in a sentence, tag it with the category that seems the most important to you.
- Judgements regarding other books than the one reviewed, or judgments related to past readings of the same book must be tagged as ”no_val”.
- What does NOT constitute an evaluation: interpretations (”the author wishes to express...” and the like), personal anecdotes (”I first read the book when I was in college”), plot summaries.”ind_pragmatic” must contain an explicit reference to the reviewer’s real life or experience. - ”aesthetic” must have explicit reference to literary art (plot, characters, style). - An evaluation must specify, explicitly, its object. - An evaluation may have neutral, or ambiguous (’mixed feelings’) valence.
- Any comparison, resulting in the priority of one over the other, between the book in question and other cultural products is to be considered evaluative.
- All references to story, characters, style, or any other features that relate back to the writing are tagged as aesthetic regardless of the simplicity of the rating. Example: I like the story, I like the characters, etc.
- ”generic_val” consists of all those statements not accompanied by explanation, i.e., expressions of appreciation not related to specific aspects of the book or to specific efects it had on the reader.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines.
When the sentence does not express an evaluation of the reviewed book, assign the label ”no_val”.
When the sentence expresses an evaluation of the reviewed book, you will have to choose between two diferent labels.
Assign the label ”eval_generic” for any evaluation concerning the specifics of literary language, both in its formal aspects (use of rhetorical figures, writing style, etc...) and content aspects (character or plot construction, narratological features, etc...).
The other label, ”eval_individual” does not refer to features present in the text (such as formal values), but rather to the impact it had on the reader. It can be used for:
1. Evaluations regarding the impact of the book on the reader’s life, the existential ”lessons” that the latter learned from the former.
2. Evaluation regarding the emotional impact of the book on the reader, the way the former made the latter feel.
3. Evaluation regarding the cognitive impact of a book on a reader, the information that the latter extracted from the former or the intellectual stimulation they experienced while reading it.
The label ”eval_generic” can also be interpreted in two additional ways: 1. This label concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book). Of particular interest here are all those judgments that seek to enact (or to reafÏrm) a canonization of the judged work, that is, to place it on the roster of ’important’ readings.
2. Into this category fall all those evaluations related to the work as a whole (”beautiful book,” ”highly recommended,” ”an unbearable read,” and so on). Also, tag with this category all evaluative sentences that do not fall into any of the above cases.
In assigning one (and only one) of these labels to each sentence, please follow these generic guidelines:
- Work on individual sentences. ONLY in cases where one sentence is incomprehensible without the next, or expresses a concept that necessarily requires continuation in the next, treat the two as a single block and assign them the same label.
- If judgments related to several categories are made in a sentence, tag it with the category that seems the most important to you.
- Judgements regarding other books than the one reviewed, or judgments related to past readings of the same book must be tagged as ”no_val”.
- What does NOT constitute an evaluation: interpretations (”the author wishes to express...” and the like), personal anecdotes (”I first read the book when I was in college”), plot summaries. - ”eval_individual” must contain an explicit reference to the reviewer’s real life or experience. - An evaluation must specify, explicitly, its object. - An evaluation may have neutral, or ambiguous (’mixed feelings’) valence.
- Any comparison, resulting in the priority of one over the other, between the book in question and other cultural products is to be considered evaluative.
- All references to story, characters, style, or any other features that relate back to the writing are tagged as ”eval_generic” regardless of the simplicity of the rating. Example: I like the story, I like the characters, etc. - ”eval_generic” can consist of all those statements not accompanied by explanation, i.e., expressions of appreciation not related to specific aspects of the book or to specific efects it had on the reader.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines.
When the sentence does not express an evaluation of the reviewed book, assign the label ”no_val”.
When the sentence expresses an evaluation of the reviewed book, you will have to use the label ”val”.
Assign this label for any evaluation concerning the specifics of literary language, both in its formal aspects (use of rhetorical figures, writing style, etc...) and content aspects (character or plot construction, narratological features, etc...).
Assign the label ”val” also when the evaluation does not refer to features present in the text (such as formal values), but rather to the impact it had on the reader. It can be used for: 1. Evaluations regarding the impact of the book on the reader’s life, the existential ”lessons” that the latter learned from the former.
2. Evaluation regarding the emotional impact of the book on the reader, the way the former made the latter feel.
3. Evaluation regarding the cognitive impact of a book on a reader, the information that the latter extracted from the former or the intellectual stimulation they experienced while reading it.
The label ”val” can also be interpreted in two additional ways: 1. This label concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book). Of particular interest here are all those judgments that seek to enact (or to reafÏrm) a canonization of the judged work, that is, to place it on the roster of ’important’ readings.
2. Into this category fall all those evaluations related to the work as a whole (”beautiful book,” ”highly recommended,” ”an unbearable read,” and so on). Also, tag with this category all evaluative sentences that do not fall into any of the above cases.
In assigning one (and only one) of these labels to each sentence, please follow these generic guidelines:
- Work on individual sentences. ONLY in cases where one sentence is incomprehensible without the next, or expresses a concept that necessarily requires continuation in the next, treat the two as a single block and assign them the same label.
- If judgments related to several categories are made in a sentence, tag it with the category that seems the most important to you. - Judgements regarding other books than the one reviewed, or judgments related to past readings of the same book must be tagged as ”no_val”.
- What does NOT constitute an evaluation: interpretations (”the author wishes to express...” and the like), personal anecdotes (”I first read the book when I was in college”), plot summaries. - An evaluation must specify, explicitly, its object. - An evaluation may have neutral, or ambiguous (’mixed feelings’) valence.
- Any comparison, resulting in the priority of one over the other, between the book in question and other cultural products is to be considered evaluative.
- All references to story, characters, style, or any other features that relate back to the writing are tagged as ”val” regardless of the simplicity of the rating. Example: I like the story, I like the characters, etc.
- ”val” can consist of all those statements not accompanied by explanation, i.e., expressions of appreciation not related to specific aspects of the book or to specific efects it had on the reader.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines: - ”no_val”, when the sentence does not evaluate the reviewed book; - ”aesthetic”, any evaluation concerning the specifics of literary language, both in its formal and content aspects; - ”ind_pragmatic”, evaluations regarding the impact of the book on the reader’s life; - ”ind_emotional”: it is about the value the reader places on the work based on what it made him or her feel. It can range from aspects more related to the book itself, to more intimate and personal issues;
- ”ind_cognitive”: in this category fall all considerations of a book’s ability to teach the reader something or stimulate him intellectually;
- ”social”: this value concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book); - ”generic_val”: tag with this label all evaluative sentences that do not fall into any of the above categories.
sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines: - ”no_val”, when the sentence does not evaluate the reviewed book; - ”eval_individual”, evaluations regarding the impact of the book on the reader’s life; the value the reader places on the work based on what it made him or her feel (it can range from aspects more related to the book itself, to more intimate and personal issues); all considerations of a book’s ability to teach the reader something or stimulate him intellectually; - ”eval_generic”, any evaluation concerning the specifics of literary language, both in its formal and content aspects; it also concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book); tag with this label all evaluative sentences that do not fall into any of the above categories.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines: - ”no_val”, when the sentence does not evaluate the reviewed book; - ”val”, evaluations regarding the impact of the book on the reader’s life; the value the reader places on the work based on what it made him or her feel (it can range from aspects more related to the book itself, to more intimate and personal issues); all considerations of a book’s ability to teach the reader something or stimulate him intellectually; any evaluation concerning the specifics of literary language, both in its formal and content aspects; it also concerns the impact that a book has had not on a single reader, but on a community of readers (references to literary awards, to the popularity of the book); tag with this label all evaluative sentences that do not fall into any of the above categories.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines.
Guideline 1: You will have to treat each sentence as a single unit of meaning. Therefore, you will assign a label based on the sentence alone. Only in cases where one sentence is incomprehensible without the next, treat the two as a single block and assign them the same label.
Guideline 2: You will have to assign one (and only one) label to each sentence. If several labels can be assigned to one sentence, choose the label that fits best with the sentence.
Guideline 3: Possible labels are: ”no_val”, ”aesthetic”, ”ind_pragmatic”, ”ind_emotional”, ”ind_cognitive”, ”social”, ”generic_val”. Labels can be grouped into two main categories: nonevaluative sentences and evaluative sentences. When assigning a label to a sentence, you will have to (1) identify the best-fitting category and (2) choose the best-fitting label.
Possible labels are categorized and described here below.
Category 1: Non-evaluative sentences.
A non-evaluative sentence is a sentence that does not express an evaluation of the reviewed book, or that does not explicitly describe the impact it had on the reader.
Category 1, Label 1: ”no_val”
Use this label for all non-evaluative sentences. Use it also when the sentence expresses evaluations regarding other books than the one reviewed, or evaluations related to past readings of the reviewed book. Use it also in the case of: interpretations (”the author wishes to express...” and the like), personal anecdotes (”I first read the book when I was in college”), plot summaries.
Category 2: Evaluative sentences.
An evaluative sentence expresses an evaluation of the reviewed book, or it explicitly describes the impact it had on the reader. An evaluative sentence may have positive, negative, or ambiguous (’mixed feelings’) valence. An evaluative sentence must specify, explicitly, its object. Any comparison, resulting in the priority of one over the other, between the reviewed book and other cultural products is to be considered as an evaluative sentence.
Category 2, Label 1: ”aesthetic”
Use this label when the sentence explicitly expresses an evaluation concerning the specifics of literary language, both in its formal aspects (use of rhetorical figures, writing style, etc...) and content aspects (character or plot construction, narratological features, etc...). The evaluation must have explicit reference to literary art (e.g., plot, characters, style).
Category 2, Label 2: ”ind_pragmatic”
Use this label when the sentence explicitly describes the impact the reviewed book had on the reader. In particular, use it for sentences regarding the impact of the book on the reader’s life, the existential ’lessons’ that the reader learned from the book. The sentence must contain an explicit reference to the reviewer’s real life or experience.
Category 2, Label 3: ”ind_emotional”
Use this label when the sentence explicitly describes the impact the reviewed book had on the reader. In particular, use it for sentences regarding the emotional impact of the book on the reader, the way the former made the latter feel. Still, be careful in not assigning this label to sentences that use an emotional language in a generic and/or metaphorical way (e.g. ”I loved this book,” ”I enjoyed the reading”).
Category 2, Label 4: ”ind_cognitive”
Use this label when the sentence explicitly describes the impact the reviewed book had on the reader. In particular, use it for sentences regarding the importance of the information that the reader extracted from the book, or the intellectual stimulation the reader experienced while reading it.
Use this label when the sentence evaluates a book not based on the experience of a single reader, but on the experience of a community of readers (e.g., references to literary awards, to the popularity of the book). Of particular interest here are all those judgments that enact (or reafÏrm) a canonization of the judged work, placing it on the roster of ’important’ readings.
Category 2, Label 6: ”generic_val”
Use this label for all those statements not accompanied by explanation, i.e., expressions of appreciation not related to specific aspects of the book or to specific efects it had on the reader (e.g., ”beautiful book,” ”highly recommended,” ”an unbearable read”). Also, tag with this category all evaluative sentences that do not fall into any of the above categories.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines.
Guideline 1: You will have to treat each sentence as a single unit of meaning. Therefore, you will assign a label based on the sentence alone. Only in cases where one sentence is incomprehensible without the next, treat the two as a single block and assign them the same label.
Guideline 2: You will have to assign one (and only one) label to each sentence. If several labels can be assigned to one sentence, choose the label that fits best with the sentence.
Guideline 3: Possible labels are: ”no_val”, ”eval_generic”, and ”eval_individual”. They are described here below.
Label 1: ”no_val”
Use this label for all non-evaluative sentences. A non-evaluative sentence is a sentence that does not explicitly express an evaluation of the reviewed book, or that does not explicitly describe the impact it had on the reader. Use it also when the sentence expresses evaluations regarding other books than the one reviewed, or evaluations related to past readings of the reviewed book. Use it also in the case of: interpretations (”the author wishes to express...” and the like), personal anecdotes (”I first read the book when I was in college”), plot summaries.
Label 2: ”eval_individual”
Use this label when the sentence explicitly describes the impact the reviewed book had on the reader.
In particular, use it for sentences regarding the impact of the book on the reader’s life, the existential ’lessons’ that the reader learned from the book. The sentence must contain an explicit reference to the reviewer’s real life or experience.
Use it also for sentences regarding the emotional impact of the book on the reader, the way the former made the latter feel. Still, be careful in not assigning this label to sentences that use an emotional language in a generic and/or metaphorical way (e.g. ”I loved this book,” ”I enjoyed the reading”).
Finally, use it for sentences regarding the importance of the information that the reader extracted from the book, or the intellectual stimulation the reader experienced while reading it.
Use this label for all evaluative sentences. An evaluative sentence is a sentence that explicitly expresses an evaluation of the reviewed book. An evaluative sentence may have positive, negative, or ambiguous (’mixed feelings’) valence. An evaluative sentence must specify, explicitly, its object. Any comparison, resulting in the priority of one over the other, between the reviewed book and other cultural products is to be considered as an evaluative sentence.
Use this label when the sentence explicitly expresses an evaluation concerning the specifics of literary language, both in its formal aspects (use of rhetorical figures, writing style, etc...) and content aspects (character or plot construction, narratological features, etc...). The evaluation must have explicit reference to literary art (e.g., plot, characters, style).
In addition, use this label when the sentence evaluates a book not based on the experience of a single reader, but on the experience of a community of readers (e.g., references to literary awards, to the popularity of the book). Of particular interest here are all those judgments that enact (or reafÏrm) a canonization of the judged work, placing it on the roster of ’important’ readings.
Finally, use this label for all those statements not accompanied by explanation, i.e., expressions of appreciation not related to specific aspects of the book or to specific efects it had on the reader (e.g., ”beautiful book,” ”highly recommended,” ”an unbearable read”). Also, tag with this category all evaluative sentences that do not fall into any of the above cases.
book_title,sent_id,sentence The table includes the review of one book (identified by ”book_title”), split into sentences. You will have to produce as output another .csv table with the following structure: sent_id,label
You will assign the label to each sentence (even when the sentence is broken or incomplete) by following these guidelines.
Guideline 1: You will have to treat each sentence as a single unit of meaning. Therefore, you will assign a label based on the sentence alone. Only in cases where one sentence is incomprehensible without the next, treat the two as a single block and assign them the same label.
Guideline 2: You will have to assign one (and only one) label to each sentence. If several labels can be assigned to one sentence, choose the label that fits best with the sentence.
Guideline 3: Possible labels are: ”no_val” and ”val”. They are described here below. Label 1: ”no_val”
Use this label for all non-evaluative sentences. A non-evaluative sentence is a sentence that does not explicitly express an evaluation of the reviewed book, or that does not explicitly describe the impact it had on the reader. Use it also when the sentence expresses evaluations regarding other books than the one reviewed, or evaluations related to past readings of the reviewed book. Use it also in the case of: interpretations (”the author wishes to express...” and the like), personal anecdotes (”I first read the book when I was in college”), plot summaries.
Label 2: ”val”
Use this label for sentences expressing the impact of the reviewed book on the reader and for all evaluative sentences.
Use this label when the sentence explicitly describes the impact the reviewed book had on the reader. In particular, use it for sentences regarding the impact of the book on the reader’s life, the existential ’lessons’ that the reader learned from the book. The sentence must contain an explicit reference to the reviewer’s real life or experience.
Use it also for sentences regarding the emotional impact of the book on the reader, the way the former made the latter feel. Still, be careful in not assigning this label to sentences that use an emotional language in a generic and/or metaphorical way (e.g. ”I loved this book,” ”I enjoyed the reading”).
Finally, use it for sentences regarding the importance of the information that the reader extracted from the book, or the intellectual stimulation the reader experienced while reading it.