=Paper=
{{Paper
|id=Vol-3232/paper20
|storemode=property
|title=The New Order of Criticism. Explorations of Book Reviews Between the Interpretative and Algorithmic
|pdfUrl=https://ceur-ws.org/Vol-3232/paper20.pdf
|volume=Vol-3232
|authors=Jonas Ingvarsson,Daniel Brodén,Lina Samuelsson,Victor Wåhlstrand Skärström,Niklas Zechner
|dblpUrl=https://dblp.org/rec/conf/dhn/IngvarssonBSSZ22
}}
==The New Order of Criticism. Explorations of Book Reviews Between the Interpretative and Algorithmic==
https://ceur-ws.org/Vol-3232/paper20.pdf
The New Order of Criticism. Explorations of Book Reviews
Between the Interpretative and Algorithmic
Jonas Ingvarsson1, Daniel Brodén1, Lina Samuelsson2, Victor Wåhlstrand Skärström1, and
Niklas Zechner3
1
Gothenburg University, The Centre for Digital Humanities (CDH), Department of Literature, History of Ideas,
and Religion (LIR), Box 100, SE-405 30 Gothenburg, Sweden
2
Mälardalen University, School of Education, Culture and Communication, Division of Language and Literature,
Box 883, SE-721 23 Västerås, Sweden
3
Gothenburg University, Språkbanken Text, Department of Swedish, Multilingualism, Language Technology, Box
200, SE-405 30 Gothenburg, Sweden
Abstract
The New Order of Criticism (2020–2024) is a mixed-methods project combining algorithmic
and interpretative approaches to the study of literary criticism. The project expands on a prior
study of Swedish book reviews from the years 1906, 1956 and 2006 (‘The Order of Criticism’,
Samuelsson 2013), re-examining and re-evaluating the original results through the use of
computational tools, language technology and big data. The aim of the present paper is to
discuss early experiences and results from the interdisciplinary approach utilized by the current
project, a collaborative process where interpreter and programmer are in dialogue, and where
methodologies, and their instantiation in tools, are reflexively discussed from an
epistemological point of view. In our analysis we ask: How can insights from working with
digital methodologies and tools inform traditional scholarship on literary criticism? How can
interpretative approaches and results inform digital methods?
Keywords
Literary criticism, machine learning, discourse analysis, big data, network analysis, close
reading
1. Introduction
The project ‘The New Order of Criticism: A Mixed-Methods Study of 150 Years of Book Reviews’
(2020–2024, funded by Riksbankens Jubileumsfond) compares and evaluates the results of
interpretative and computational methodological approaches to the analysis of literary criticism in
Swedish newspapers and periodicals.
The project takes as a starting point Lina Samuelsson’s study ‘The Order of Criticism: A Study of
Book Reviews in Sweden in 1906, 1956, 2006’ (Kritikens ordning: Svenska bokrecensioner 1906, 1956,
2006) from 2013 [1]. We compare the results of Samuelsson’s comparative literature (and ‘manual’)
analysis with the results emanating from language technology (LT) analysis and other data-driven
methods. The material in focus consists of transcripts of the reviews originally studied by her but also
the totality of book reviews in the newspaper database of the National Library of Sweden (Kungliga
biblioteket, KB) as well as additional digitised periodicals.
By combining analogue (or ‘traditional’) and computational methods, ‘The New Order of Criticism’
seeks to develop a mixed-methods approach to large-scale studies of literary criticism and text mining
________________________________
The 6th Digital Humanities in the Nordic and Baltic Countries Conference (DHNB 2022), Uppsala, Sweden, March 15-18, 2022.
EMAIL: jonas.ingvarsson@lir.gu.se (J Ingvarsson); daniel.broden@lir.gu.se (D Brodén); lina.samuelsson@mdu.se (L Samuelsson);
victor.wahlstrand.skarstrom@lir.gu.se (V Wåhlstrand Skärström); niklas.zechner@gu.se (N Zechner)
ORCID: 0000-0002-4600-1178 (J Ingvarsson); 0000-0002-5914-1516 (D Brodén); 0000-0002-0904-3408 (L Samuelsson); 0000-0001-6569-
120X (V Wåhlstrand Skärström); 0000-0001-6669-0698 (N Zechner)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR Workshop Proceedings (CEUR-WS.org)
228
analysis of argumentative, critical text corpora. On a more abstract level, one project goal is to
explore the possible synthesis between what may be perceived as two epistemological stands
(traditional and digital text analysis). This ambition echoes the positions taken by, among
others, Liu [2] and Ingvarsson [3], who claim that the role of digitisation within the humanities is not
only a matter of tools or gadgets, databases and digital objects, but also a concern for the
organisation, production and dissemination of knowledge [4, 5].
Albeit most projects that engage in distant reading practises owe some debt to the pioneering works
of Franco Moretti and Matthew Jockers [6, 7], the epistemological claims and mixed-methods approach
utilised here, leans more to the positions taken by for example Katherine Bode [8, 9], who, on the one
hand, affirms the methodological achievements of distant reading, while, on the other, subscribes to the
criticism that Moretti and Jockers transform the complex notion of ‘literature’ to merely ‘data.’
However, as Bode concludes, ‘distant reading and close reading are not opposites’ [9]. Another mixed-
methods stance is taken by Paul Saint-Amour [10] who advocates a ‘meso-analysis’ strategy,
deliberately chosing the very ‘middle road’ that Moretti rejects as ‘not leading to Rome.’
The aim of the present paper is to discuss the project and its core focus points, drawing on some
initial findings. The overall goals and ambitions of the project could be stated as follows:
• A re-examination of the manual transcriptions in Samuelsson’s prior study (approx. 700
book reviews), through a mixed-methods application of language technology and other
research tools.
• An exploration of all available literary criticism in KB’s newspaper database from the years
1906, 1956, 2006 (the same years as in Samuelsson’s study) in order to compare the
analytical results with the original study, but also to train algorithms for identifying literary
criticism in large text corpora (this part of the project will be further elaborated into a LT-
study of all years from 1870–2020).
• An epistemological reflection on differences, similarities and synergies between
interpretative and computational methods.
• An overview of literary criticism from 2016, in the vein of Samuelsson’s approach,
drawing on the digital tools the project has utilised and generated, in order to
further study and evaluate the possible synergies between traditional interpretation,
network analysis, and LT-driven approaches.
In the following, we will focus on the first of these four challenges and also break down the project’s
interdisciplinary and comparative approach to discuss our methodological and technical points of
departure. We introduce early results from the collaborative process, where digital and
traditional methods and approches to both digitised and printed material are set in a continuing dialogue,
and where methodologies and their utilisation in tools are reflexively discussed [11]. The presentation
follows the goals stated above and discusses some of the tasks and challenges that we are confronted
with.
2. Data – transcripts of book reviews
Samuelsson’s original study [1] investigates the field of literary criticism in Sweden from three years
(1906, 1956 and 2006), each year studied from two distinct perspectives. The first perspective
focuses on the sociology of the landscape of literary criticism – the publications and critics involved,
including, for example, the role of gender and ideology. The second perspective concerns the
discourse of book reviews, where the texts in focus are read as a collective voice of the critics at the
time [12]. This approach includes the study of the norms and generic traits of literary criticism by
identifying key attributes of the texts in the form of recurring themes, criteria of evaluation as well as
the critics’ approaches towards the literary works and the writers. The combination of discourse analysis
and sociology of literature also serves as a guiding framework for the different tasks carried out with the
tools of language technology and network analysis in the project.
229
The material in the prior study consists of book reviews from what could be regarded as the most
culturally significant publications of each period (as well as from radio in 1956 and 2006, and television
in 2006). The printed material for each year was assembled from 12 newspapers and 2–3 periodicals, in
total 200–300 book reviews per year. The reviews were manually identified using microfilm copies of
newspapers and transcribed by hand into a digital format. For each year the material was sorted out in a
database and categorised with metadata, for instance the name of the critic and the author, their gender,
the title of the book reviewed, the source, article length (word count), the date of publication, and literary
genre. This material constitutes Samuelsson’s original corpus.
3. Preliminary Results
In our initial exploration of the book reviews previously studied by Samuelsson, we have pursued the
actual difference in the language of criticism in her corpora for the years 1906, 1956 and 2006. We have
made preliminary computational studies in terms of word frequencies, document embeddings and visual
analysis, comparing the results of Samuelsson’s study [1] (the words and phrases that were part of the
discourse analysis) with the actual text data in her transcribed material.
3.1. The Order of Criticism – some topics and words from the original study
In the discourse analysis sections of her study, Samuelsson [1] noted that for each year examined, a few
topics and themes emerged from the material. Together, these words and topics formed a tentative
mapping of the critical discourse 1906, 1956, and 2006. For the case of this paper, we present a sample
of the topics in the original study which form a point for departure for the computational analyses
described further below.
1906:
Utveckling; mognad; äkta; levande; själ; författare; skald; diktare; personlig; personlighet;
känsla; land; nation; fosterland; talang; konst; konstnär; nordisk
[Progress; maturity; authentic; living; soul; author; poet; poet; personal; personality; feeling;
country; nation; motherland; talent; art; artist; Nordic]
1956:
Tolkning; obegriplig; läsaren; verk; verket; klar; klart; klarhet; genre; enkelhet; tydlighet;
enkelt; människa; mänsklig; mänskligt; psykologi
[Interpretation; incomprehensible; the reader; (literary) work; the (literary) work; clear;
clearly; clarity; genre; simplicity; clarity; human (noun); human (adj); humanly; psychology]
2006:
Själv; mig; originalitet; författare; bok; boken; komplexitet; kön; kropp; språk; äkthet; politisk;
politiskt
[Self; me; originality; author; book; the book; complexity; sex (gender); body; language;
authenticity; political; politically]
3.2 Statistical analysis of topics using word frequencies
To test these prior results computationally, we looked for each of these words (from all three years) in
Samuelsson’s material, which resulted in some interesting observations. Some of the findings may at
first glance in part challenge the original study, but we must keep in mind that the discourse analysis of
Samuelsson’s corpus is not quantitative but qualitative – that is, the traditional analysis focus on the
value or importance the text gives to the different topics. Nevertheless, our findings call for further
230
analysis, in the form of a re-reading of Samuelsson’s study, but also of the ‘totality’ of the material
from the individual years when we access the larger corpora.
For each period, we combine all the available data. For each word, we then find the relative frequency
for each period. As an example, the word ‘äkta’ makes up approximately 300 per million words in 1906,
but only 150 per million in 1956, and 50 per million in 2006. By considering the change in frequency,
we can get a picture of the changes in discourse over time.
In 1906, the topic of national identity is strong in the manual analysis, and we verify it here: the
words ‘land’, ‘fosterland’, and ‘nordisk’ are all prominent. Words related to the development of the
author, such as ‘mognad’ and ‘utveckling’ are partly verified, but less clearly so.
Another hypothesis about 1906 was that we would find words qualitatively related to the author, such
as ‘personlighet’, ‘personlig’, ‘levande’, and ‘diktare’. These are not found to be common in 1906, but
instead seem to have increased in 1956. ‘Skald’ is more common in 1906, but it may be that the word
has become obsolete by 1956.
In 1956, we expected to find the topic of humanity, as evidenced by ‘människa’, ‘mänsklig’ and
‘mänskligt’, which is verified by the statistical analysis. The topic of clarity was also expected in 1956,
but the result is not clear; we find that while ‘klarhet’ is at its most common in 1906, ‘klar’ reaches its
peak in 1956, and ‘klart’ in 2006. For the related concept of simplicity, exemplified by for example
‘tydlighet’ and ‘enkelhet’, we do not find sufficient data to draw any conclusions.
Words about the relationship between book and reader, such as ‘läsaren’ and ‘genre’, which we
hypothesised to find in 1956, are in fact more prominent in 2006. In 2006, we also expect a greater focus
on the reviewer, which is partly verified, with ‘själv’ but not ‘mig’ increasing. Mentions of the book in
the form of ‘verk’ or ‘verket’ were predicted in 1956, but seem to be relatively stable over time, whereas
‘bok’ and ‘boken’ increase over time.
Moreover, topics of gender and identity were expected to be prominent in 2006, which is partly
verified. We see an increase in ‘kön’ and ‘språk’, and tentatively in ‘kropp’, whereas ‘politisk’ and
‘politiskt’ are inconclusive.
The manual analysis revealed that the number of female reviewers and authors varies over time,
showing a decrease in 1956 compared to 1906 and 2006. In the computational analysis we also find
that ‘hon’ (she) actually decreases in 1956, but increases in 2006, while ‘han’ (he) shows the opposite
pattern. Following the hypothesis of increased focus on individualism and subjectivity, we might
expect ‘jag’ (I) to increase overall, but it follows the same pattern as ‘hon’, decreasing in 1956 and
increasing in 2006. This is in line with the original study, where Samuelsson argues that this may be an
effect of a trend in New Criticism to focus not on the self, but rather on ‘the reader’ or ‘one’s
perception.’
3.3 Embedding and visualisation
As previously mentioned, Samuelsson’s corpus is richly annotated with metadata. Besides the date of
publication, the metadata includes details on the gender of the author and critic, the publication, or
forum, of the review as well as the literary genre of the work. For Samuelsson, this data contained
valuable information regarding the sociological aspects of the original study.
In order to create an effective and approachable visualisation of the individual reviews, the texts are
projected into a numerical embedding space. An embedding is an information extraction technique
which characterises a set of data in terms of only a few features, e.g. as coordinates on a plane. Part of
this process is the reduction of noise in the data. In our case, the text variability is reduced by extracting
only the uninflected word lemmas of the text, as well as significant filtering. As suggested by for
example Martin and Johnson [13], an important part of this filtering is the extraction of only certain
parts-of-speech, e.g. nouns. The resulting embedding can be seen in Figure 1, where each review is a
point on the plane, in total 694 texts. The points are here scaled in terms of the size of their vocabulary
and labelled by year of publication, indicating clear language divides across the years. Interestingly, this
visual method of analysis seems to replicate the findings of the distance measures, namely a clear
difference in language between the years.
The raw text from the 694 reviews is extracted manually and preprocessed by means of the
Språkbanken Sparv tool [14], with lemmatisation and part-of-speech categories. The reviews are filtered
231
with respect to nouns, verbs, adjectives and, importantly, also the words in title of the reviewed text.
The filtered corpus is transformed into a documentation-term matrix from term frequency-inverse
document frequencies, a word frequency-based method which penalises words that exist in all
reviews, and emphasises unique terms. Finally, the data is visualized in a 2D chart by means of
dimensionality reduction. Since the objective of this procedure is visualization, the choice of
algorithm is based on qualitative measures. However, care should be taken to use non-linear
transformations, which aim to preserve the highly non-linear relationships between terms. Figure 1
and Figure 2 were generated using the UMAP algorithm (using 50 neighbours, a minimal distance
of 0.1, and 10,000 epochs), which usually outperforms alternatives in terms of speed and qualitative
performance on textual data [15].
Figure 1: Reviews from Samuelsson’s corpus, embedded by means of UMAP and labelled by year of
publication, indicating clear divides between years.
Figure 2: Reviews from Samuelsson’s corpus, embedded by means of UMAP and labelled by genre of
the reviewed work. Although prose makes up the largest and most diverse set of reviews, there are
clusters of varying coherence corresponding to for example poetry, drama and children.
The method reveals other relationships as well: labelling the reviews by literary genre also indicates
a separation and constellations of reviews, see Figure 2. While prose is the largest and most diverse of
the genres in the set of reviews, both poetry and drama form distinct clusters, indicating a degree of
232
genre-based commonality. Interestingly, even though the work titles are filtered from the texts, to stop
certain unique words from dominating, closer inspection reveals that many smaller clusters consist of
either a) reviews of the same work of literature, or b) reviews by the same critic – for example, Örnsjö-
tjuren (Axel Klinckowström, 1906), Gravidchock! Reporter erkänner (Belinda Olsson, 2006), and
Gästen (Niklas Rådström, 2006), each forming clusters of no less than 7 reviews. The driving force
behind these clusters is an overlap in language use, but a close reading does not reveal that they are
immediately similar.
For example, Gästen is a novel about a meeting between the two authors H. C. Andersen and Charles
Dickens, but since their personal names are not part of the digital analysis, this fact can not account for
the numerical similarity of the reviews. From a traditional close reading perspective, the reviews are not
particularly similar, nor do they use the same quotes, and the critics have fairly different angles in their
readings. However, the digital tool calls on us to invest this more deeply, pointing at a special discourse
being at work within the reception of each title.
Our mixed-methods approach allows digitally-oriented and traditional scholars to collaborate on the
analysis of the material. From a close-reading perspective, an interactive version of this embedding
visualisation allows selection and inspection of individial reviews, and their relative position in the
projected space. In practice, not only the clusters are interesting features of the embedding, but also the
outliers, for example some 1956 reviews firmly positioned in 2006 space. Although this visualisation is
a powerful illustration, it also serves as a critical tool for the researcher to question established themes
in a large material.
4. Concluding reflections
The use of tools for text mining in many digital humanities contexts, have in a way created a new version
of C.P. Snow’s ‘two cultures’ [16], this time as a divide within the humanities itself, between, on the
one hand, those who rely on traditional forms of text analysis (close reading, historical
contextualisation, etc.) and, on the other hand, those who through big data and distant reading practices
claim to have found a way to finally make humanities research quantifiable and ‘scientific’ [3, 9, 10].
Many arguments have been made that ideas of scientific validity cannot easily be adapted for the
humanities, not least since the practice of interpretation runs the risk of being reduced to what can be
modelled. At the same time, strong arguments are made that quantitative practices nevertheless can
enrich and contribute to the methodological development in the field of literary study. [5] To insist on a
dialogical relation between the digital tools (including machine learning and text mining) and the more
traditional approaches (in the present project mainly represented by discourse analysis), thus seems to
be of critical interest for further methodological and epistemological discussions within the humanities.
While the symbiosis of Foucauldian discourse analysis and text mining is not well explored, there is
a growing interest in the merging of digital tools for text mining and traditional text analysis, including
its methodological implications, and, recently, there have been studies where questions traditionally
applied to comparative literature have been productively approached using distant reading tools and
practices [17, 18].
Beside subscribing to positions taken by Bode and Saint-Amour above [8, 9, 10], our project follows
what Rockwell and Sinclair [11] call a ’dialogical practice’ resulting in a so-called Agile Hermeneutics
(AH) by which they mean a collaborative process where data scientists and humanities scholars are in a
dialogical relationship: ’Whereas traditional programmers aim to have complete specifications, and
scholars attempt to theorise their work largely beforehand, AH is pragmatic. Small experiments generate
hermeneutical theories as the products of interpretation: texts and tools. Code is quickly hacked to test
an idea. Methods, and their instantiation in tools, are discussed reflexively throughout the experiment.
Above all, […] AH is about talking with another person who has complementary skills and summarising
those conversations in various ways’ [11]. As our project digs deeper into the intersections of
computational and interpretative analysis, we will further pursue an epistemological interest in the
dialectical and dialogical tensions of this very process.
233
5. References
[1] L. Samuelsson, Kritikens ordning: Svenska bokrecensioner 1906, 1956, 2006 (diss.), Bild, text
och form, Karlstad, 2013. Open access here:
http://kau.diva-portal.org/smash/get/diva2:621542/FULLTEXT01.pdf (accessed February
2022).
[2] A. Liu, Theses on the Epistemology of the Digital: Advice for the Cambridge Centre for Digital
Knowledge, 14 August 2014. URL: http://liu.english.ucsb.edu/theses-on-the-epistemology-of-
the-digital-page/ (accessed February 2022).
[3] J. Ingvarsson, Bomber, virus, kuriosakabinett: texter om digital epistemologi, Rojal förlag,
Göteborg 2018, and: Towards a Digital Epistemology: Aesthetics and Modes of Thought in
Early Modernity and the Present Age, 2nd. ed., Palgrave Macmillan, Cham, Switzerland, 2021.
[4] D.M. Berry, A. Fagerjord, Digital Humanities: Knowledge and Critique in a Digital Age, Polity
Press, Cambridge, 2017.
[5] J. E. Dobson, Critical Digital Humanities: The Search for a Methodology, University of Illinois
Press, Chicago, 2019.
[6] F. Moretti, Distant Reading, Verso, London, 2013.
[7] M. L. Jockers, Macroanalysis: Digital Methods and Literary History, University of Illinois
Press, Champaign, 2013.
[8] K. Bode, Reading by Numbers: Recalibrating the Literary Field, Anthem Press, London, 2012.
[9] K. Bode, A World of Fiction: Digital Collections and the Future of Literary History, University
of Michigan Press, Ann Arbor, 2018. See especially Chapter 1.
[10] P. Saint-Amour, The Medial Humanities: Toward a Manifesto for Meso-Analysis, in
modernism/modernity, Vol. 3, Cycle 4, 2019. URL: https://doi.org/10.26597/mod.0092
(accessed May 2022).
[11] G. Rockwell, S. Sinclair, Hermeneutica: Computer-assisted Interpretation in the Humanities.
MIT Press, Boston, 2016.
[12] M. Foucault, The Order of Things, in: R. Young (Ed.), Untying the Text. A Post-Structuralist
Reader, Boston, Routledge & Kegan Paul, London & Henley, 1981 (1970).
[13] F. Martin, M. Johnson, More efficient topic modelling through a noun only approach, in
Proceedings of the Australasian Language Technology Association Workshop 2015, pp. 111–
115. URL: https://aclanthology.org/U15-1 (accessed February 2022).
[14] L. Borin, M. Forsberg, M. Hammarstedt, D. Rosén, R. Schäfer, A. Schumacher, Sparv:
Språkbanken’s corpus annotation pipeline infrastructure, in: The Sixth Swedish Language
Technology Conference (SLTC), Umeå University, 2016, pp. 17–18.
[15] L. McInnes, J. Healy, N. Saul, L. Grossberger, UMAP: Unifrom Manifold Approximation and
Projection, The Journal of Open Source Software, vol. 3, number 29, p. 891, 2018.
[16] C. P. Snow, The Two Cultures and the Scientific Revolution, Martino Fine Books, Eastford,
2013 (1959).
[17] A. Piper, Enumerations: Data and Literary Study, The Univeristy of Chicago Press, Chicago,
2018.
[18] T. Underwood, Distant Horizons: Digital Evidence and Literary Change, The Univeristy of
Chicago Press, Chicago, 2019.
234