Digital transformation allows new forms of discussion on cultural and aesthetic practices. Several stakeholders are able to comment and discuss cultural artefacts (books, museums and their exhibitions). Today, a variety of platforms exists: from general platforms (e.g., Amazon, Tripadvisor) to specialized ones (e.g., LovelyBooks, Behance). So far, there has not been any analysis of reviews of cultural artefacts across platforms. Accordingly, this study identifies and classifies text components of reviews about cultural artefacts. Based on the coding paradigm in the sense of Grounded Theory, (1) the components are empirically identified and (2) structured, resulting in a first category system. After evaluating and modifying the system (3), a multi-layered category system resulted. Thereafter, a group of students applied the system to provide insights into the understandability of the system. By providing such a categorisation, we intend to contribute to further research in analysing the contents and text structure of online reviews.
Today, a variety of online platforms including
ecommerce, social media and specialized rating platforms,
allows consumers to rate and communicate their opinion
about products or services. According to industry research
reports, purchasing decisions of consumers are highly
influenced by online reviews
Furthermore, digital transformation supports new or
changed forms of collaboration
Earlier studies have tended to address heterogeneous and mostly isolated aspects concerning reviews. Most of them analysed reviews in English. For instance, based on product ratings, e-commerce platforms, especially Amazon (e.g.,
McAuley and Leskovec, 2013), analysed online consumer
reviews. A few studies analysed reviews in German
What kind of components are contained in reviews of cultural artefacts? Our contribution is a multi-layered category system for characterising components of reviews of cultural artefacts (e.g., contents and communicative acts). The category system contributes to our ongoing research on reviews in the digital world and aims at characterising and analysing reviews of cultural artefacts. Therefore, as a next step after building the category system, we hope to be able to find patterns of components within reviews. In a first step, we briefly outline the background of reviews, cultural artefacts and the digital world in which they are written (Section 2). Based on our research design (Section 3), we iteratively built the multi-layered category system (Section 4) and evaluated it, applying it several times (Section 5). We then discuss the results and future research directions (Section 6) and conclude with our main findings (Section 7). 2.
In this section, we specify the terms related to our research question. Therefore, we introduce our definition of a cultural artefact, the concept of a review and its components.
In this study, we define cultural artefacts as artistic artefacts (museums and their exhibitions) and books. This focus on the cultural field also defines the sort of review we are looking for. In this context the concept of a review is characterized by its strong relation to the tradition of art and literary criticism as a professional journalistic form of discussing and reviewing newly published cultural artefacts.
In general, scholars and practitioners do not agree on a universal definition of the term review and of its components. Therefore, a review could be analysed and understood from different point of views. Some of the most prominent positions include the following statements:
Review as a product of evaluation of a cultural
artefact: a review is considered as an article published
by a journalist, that describes, explains, interprets
and/or evaluates a cultural artefact. Most
characteristic parts of a review include
recommendation or dissuasion as well as statements
on the originality and entertaining qualities of an
artefact
Review as the central text type of literary criticism: a
review is understood as the critical discussion of a
new publication; the most common and most
important type of text in literary criticism
Review as an expert expression of opinion: a review
is the opinion-expressing form of literary and art
criticism. Book reviews, film criticism, a judgmental
report on a painting exhibition or an expert
journalistic expression are examples
Review means asking for terms of arts, their
functions and their origin. It further involves judging,
making oneself unpopular and not being afraid of
misunderstandings
As the statements indicate, a review may contain several components, addressing several aspects of a cultural artefact. For instance, descriptions, explanations, interpretations of an artefact as well as its critical discussion and expressions of the reviewer’s opinion are components of a review. Consequently, a review is characterised by several textual components and, therefore, can be analysed from different perspectives. Furthermore, reviews and their components might vary depending on the type of platform (e.g., general platforms like Amazon and Tripadvisor, specialized and community-based platforms like LovelyBooks or Behance) and on the addressed cultural artefact (e.g., books, museums and their exhibitions). However, existing research mostly tended to address isolated aspects of reviews. Therefore, there still seems to be a need to investigate multiple perspectives on reviews of cultural artefacts. Accordingly, in this study we aim to provide a multi-layered category system that covers several perspectives on a review, as a basis for further research in this field.
In order to identify components of reviews we conducted
an iterative three-stage research design, containing build
and evaluate activities that characterise the Design Science
Research in Information Systems
d il u B e t a u l a v E
Stage 1:
Identify components
Stage 2: Develop category system Stage 3: Evaluate category system
Inputs
Methods/Steps
Outputs • Sample reviews • 130 types of review components • Sample reviews • Multi-layered category system • Perform open coding iteratively • Perform axial coding iteratively • Apply multilayered category system iteratively • 130 types of review components • Multi-layered category system • Adapted multilayered category system As a compact guideline for annotators, we built a tabular overview that complements the hierarchically ordered, labelled components by operationalised explanations and examples (codebook).
Presentation of selected categories. As described above, each layer consists of several hierarchically ordered categories. In total, the category system contains 108 different categories. More than half of the categories are assigned to Layer 1. On the topmost level, Layer 1 contains eight categories (Figure 3). For reasons of space limitations, we will only present selected subcategories of category 1.1 in more detail (Figure 4).
Layer 1—Content 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8
have been independently analysed by three students.
Subsequently, we measured the nominal scale agreement
Following the research design (Section 3), we built a multilayered category system, containing multiple components of a review (Stage 1 and 2). In this section, we describe the category system in general and, for reasons of space limitations, we only present selected components of Layer 1 and 2 in more detail. 1 …
a … 1.1 1. Layer 1—Content Layer 2—General Criticism Layer 3—Style Layer 4—Further Information
The category system is used to annotate text strings in the reviews. Further, the categories are organized in a hierarchical structure, where each category has a numerical and a textual label used in the practical annotation work. For instance, Layer 1 contains category 1 that is divided into several subcategories from 1.1 to 1. . A subcategory can be divided into further subcategories and so on (Figure 2).
Layer 1—Category 1.1. This category contains various themes addressed by the reviewers that are directly related to the cultural artefact. First, the reviewer might discuss individual aspects (category 1.1.1) related to the artefact. For instance, he or she might quote some content, address the translation, the author, detailed content or the title of the artefact. Also physical properties such as the quality of paper or illustrations, the outer appearance of the reviewed book (e.g., cover or used condition) and the language style of the artefact can be addressed by the reviewer. Alternatively or in addition, not only individual aspects, but also the artefact as a whole can be discussed (Figure 4). Layer 2—Category 2. Addressing the categories of Layer 1, the reviewer always pursues some intention. Therefore, Layer 2 offers different communicative acts in the reviews for annotating these intentions (Figure 4).
5.
As a first test of the category system, 24 students were asked to apply it in the manual annotation of 430 randomly selected Amazon reviews (Stage 3 of the research design). We now present an example of an annotated text passage which illustrates how labels from our category system are attached to text components of a review. In addition, we present selected results of the evaluation of the category system.
Annotation example. Annotators of reviews might usefully read a given review several times, from different perspectives. As a result, each text passage of the review is annotated with categories of both, Layer 1 and 2. If it involves a particular language style or multimedia content, the categories of Layer 3 and 4 can be additionally annotated.
1.1.2, 2.10.1 1.1.2, 2.10.1 und unterhaltsames Buch and entertaining book ganz im typischen Stil 1.1.2, 2.10.1 1.1.1.8, 2.9
1.1.1.3, 2.9 1.2, 2.10.1 in the typical style of
1.1.1.8, 2.9 Ellen DeGeneres.
1.1.1.3, 2.9 mentioned by the reviewer (category 2.9). Third, the annotator should take the perspectives of Layer 3 and 4. However, as the sample text passage does not address these categories, no further annotation is required (Figure 5). Evaluation. Applying the category system and annotating 430 reviews, the students had to annotate sentences or indicator expressions (such as those underlined in Figure 5) with the labels from our category system. In sum, 14.235 text passages have been identified and annotated with categories by the students. Some categories have been more frequently used than others. For instance, the content of an artefact (category 1.1.1.4) is most commonly recognized by the students (2603 times). The summary of some content is the second most common category (category 2.1) that is recognized by the students (2403 times, Figure 6).
For analysing the agreement between three raters regarding each category, we reduced the data set to reviews that have been annotated threefold (Triple), resulting in 139 reviews and different overall frequency figures for the categories (Figure 7).
A calculation of the inter-annotator agreement on the task
is not trivial and yet provides only limited insight. On the
one hand, not all text passages have consistently been
annotated by all annotators at both levels (Layers 1 and 2);
this leads, for example, to low values of Fleiss’ kappa. On
the other hand, the category system contains a large number
of categories, and many of them have rarely been used, so
that the calculation does not provide adequately
interpretable results. As a consequence, we get rather low
kappa values
Relation of raters who agree and disagree (agree : disagree) No. 1.1.1.4 1.1.2
Category Content View of the Artefact as a Whole 3:0 48% 7%
As described in Section 5, the absolute observed agreement per category provides first insights into the comprehensibility of the categories. As a consequence, we are able to revise the category system. From the first annotation experiment we learn that positive evaluative statements are much more frequent in our sample than negative ones; many reviews make reference to the style or language of the reviewed book; and the physical appearance of the book is mentioned quite prominently, likely because of the special situation of Amazon delivering the book. Finally, we note that a considerable number of annotated text passages make reference to the emotions of the reviewers during the reception process. However, because of several circumstances (e.g., variety of incomplete annotations) the measurement of the nominal scale agreement (Fleiss kappa) could not provide sufficient results. As described above (Section 3), the category system is meant to be platform-independent and generic enough to be applicable to both, reviews of artistic artefacts and books. Thus, the described evaluation can only be seen as a partial evaluation activity in progress. The enhanced category system will be applied, in more controlled experiments, for reviews of artistic artefacts as well, resulting in new annotations of reviews and additional sample annotations to measure the agreement between raters.
Moreover, the category system represents an idealized
schema for annotating reviews. It serves as a starting point
to manually analyse the components of reviews. However,
as a next step, we want to identify the review components
automatically with methods of machine learning. For
instance, the use of support vector machines
Furthermore, in the medium term, we not only want to
identify components of reviews, but also patterns of and
relationships between components (e.g., relationship
between the view of the artefact as a whole and its
assessment). Therefore, the training data can be used as an
input for further machine learning algorithms, like cluster
analysis
As a result, this study provides a double contribution to the field of Digital Humanities: First, the category system and the first sample annotations constitute a starting point for a classification of textual components of reviews, and for its subsequent automation by means of machine learning methods. Second, this study and the ongoing research project which it is a part of go beyond the methodological questions and address—by examining digital practices of cultural participation through reviews—the understanding of digital culture and society on a more fundamental level which is, eventually, a central question of Digital Humanities.
7.
In order to identify components that are contained in
reviews of cultural artefacts, we used the coding paradigm
in the sense of Grounded Theory
Overall, our findings contribute to the ongoing research on reviews in the digital world and to the analysis and characterisation of reviews of cultural artefacts. Based on the category system and on sample annotations, researchers are further able to identify the review components automatically with methods of machine learning and to discover patterns of review components.
8.
This research was conducted in the scope of the research project “Rez@Kultur” (01JKD1703), which is funded by the German Federal Ministry of Education and Research (BMBF). We would like to thank them for their support.