Predicting spatial representations in literature is a challenging task that requires advanced machine learning methods and manual annotations. In this paper, we present a study that leverages manual annotations and a BERT language model to automatically detect and recognise non-named spatial entities in a historical corpus of Swiss novels. The annotated data, consisting of Swiss narrative texts in German from the period of 1840 to 1950, was used to train the machine learning model and fine-tune a deep learning model specifically for literary German. The annotation process, facilitated by the use of Prodigy, enabled iterative improvement of the model's predictions by selecting informative instances from the unlabelled data. Our evaluation metrics (F1 score) demonstrate the model's ability to predict various categories of spatial entities in our corpus. This new method enables researchers to explore spatial representations in literary text, contributing both to digital humanities and literary studies. While our study shows promising results, we acknowledge challenges such as representativeness of the annotated data, biases in manual annotations, and domain-specific language. By addressing these limitations and discussing the implications of our findings, we provide a foundation for future research in sentiment and spatial analysis in literature. Our findings not only contribute to the understanding of literary narratives but also demonstrate the potential of automated spatial analysis in historical and literary research.
Building on previous work examining fictional space and sentiment in Swiss-German narrative
[
In recent criticism across disciplines there has been an increased emphasis towards
considering place and space as crucial factors in understanding social, cultural, and historical
phenomena. This perspective on spatiality is generally referred to as the ‘spatial tu9r,n1’0[
In this paper, we present a case study on the prediction of what we call ‘non-named spatial entities’ (NNSE) in a historical corpus of Swiss-German novels using a deep learning model in conjunction with BERT and Prodigy. By combining manual annotations and advanced machine learning methods, we aim to automatically detect and recognise NNSE within the literary narratives via a similar approach to named entity recognition (NER).
NER techniques are used to identify and categorise text segments that refer to entities such as people, places, or companies, and that ‘constitute proper names1’1[]. The latest NER techniques rely on manually annotated text corpora, which are automatically analysed to build models that capture language use and grammar. These models can then identify and classify entities in new, unprocessed documents. State-of-the-art NER systems come with pre-built models trained on extensive collections of annotated documents, like news articles. These models typically perform well and are ideal for specific applications such as analysing customer feedback or extracting locations and characters. However, when applied to documents with linguistic features not well-represented in the training data, such as literary texts, NER performance can decline, increasing the likelihood of errors, also increased for most languages other than English.
Within literary studies, various scholars have used NER, particularly to identify fictional
characters [
A similar perspective has been considered by Schumacher, Flüh, and Nantk1e4][, who used conditional random fields to automatically annotate among other things non-named places, which is conceptually similar to NNSEs. However, the operationalization of space in their research is focused on places that can be found on a map, leaving therefore the broader concept of NNSEs unexplored. Also, the popular BookNLP toolkit by Bamman is able to work to identify what they call ’locations’ (for natural entities) and facilities’ (for man-made structures) in English-language texts with an accuracy of up to 902%. While BookNLP is able to diferentiate NNSEs somewhat according to our own needs, we propose a distinction for ’facilities’ into more discrete classes shown in Sectio2n.1. This paper sets out to fill this gap, training a model that will help us identify spatial elements in narrative. 2As is shown on their ofÏcial GitHub page: https://github.com/booknlp/booknlp?tab=readme-ov-file
In order to train our model to recognise literary space, we decided to train it not only to be able
to recognise non-named spatial entities (NNSEs), but also to distinguish among four diferent
types of spatial environments. We decided to base our categorisation on the research by Grisot
and Herrmann [
To produce the training set, two annotators were provided with written guidelines and trained in person to understand the diference between the various NNSE categories. They were then instructed to use the platform Prodigy12[], which allowed them to read sentences from the training set in random order, and to annotate NNSEs directly on the interface by adding labels to individual tokens. For the annotation process, six novels were sampled from the complete corpus of Swiss-German novels8[].3 The novels were then split into sentences (N=9,062), which were manually annotated by the annotators. The annotators featured a high interannotator agreement (Cohen’s Kappa) for the NNSE types ‘interior’, ‘natural’, and ‘rural’, and medium agreement for ‘urban4’, as well as a high agreement for the distinction of NNSEs against a not-NNSE token (Cohen’s Kappa = 0.898).
The values for annotators’ agreement in relation to individual types are shown in T2a.ble Also shown in Table2 are the number of sentences in our dataset in which the various classes of spatial entities were identified by manual annotation. Type O shows the number of sentences where no NNSEs were identified in the annotation process. These amount to over 86% of the 3These are: Der Wetterwart (1905) by Jacob Christoph Heer,Heimatscholle (1914) by Maria Goswina von Berlepsch, Berge und Menschen (1911) by Heinrich FedererH,eidis Lehr- und Wanderjahre (1880) by Johanna Spyri,Friedli, der Kolderi (1891) by Carl Spitteler, andMartin Salander (1886) by Gottfried Keller. 4The low score for ’urban’ is explainable by the low number of occurrencesof NNSE of this type in the dataset. dataset, making the distribution of the five categories in our dataset unbalanced. However, the NNSE categories are much closer together, ranging between 2.2% (urban) and 4.5% (interior) of all sentences.
After the annotation process, sentences were randomly assigned to either the training
dataset (80%, N=7,249) or the test dataset (20%, N=1,813). This was done according to
common best practices for training a machine learning mode6l].[ To train the classifier, PyTorch
version 2.1.1 was used as the deep learning framework13[]. In conjunction with PyTorch, the
popular hugging face library (version 4.35.2) was used to load and interact with the language
model [
The modelgbert-large by deepset was utilised as input layer for the token classifier, since it outperformed all other language mode4ls]. [ The model classifies each token of a given text, and attempts to predict whether the token under consideration can be classified as a NNSE (one of the four types mentioned above) or whether it can be considered not a spatial term (O). It was tested if the model performs better with the complete, unbalanced training dataset (N=7,249 sentences) or with a more balanced, downsampled training dataset (N=2,004 sentences). The downsampled training dataset was composed of sentences including at least one NNSE (N=1,002) and a random selection of sentences with no NNSE of equal size. The downsampled train dataset was then split again into a final training dataset (N=1,603) and a validation set (N=401).
The training was repeated for 17 epochs with a learning rate of 5e-6 and a dropout of 0.1. These parameters are determined by extensive hyperparameter testing. Since the maximum length of all sentences is 40 words or 61 tokens, the max length for the BERT model was set to 64 tokens.
With the annotation and training process described above, we produced a classifier able to 1) identify NNSEs in a given sentence, and 2) classify the identified NNSE as belonging to one of the four discrete classes: ‘rural’, ‘urban’, ‘natural’ or ‘inter5ioTro’.find the best parameters, 5The results reported here should be understood as a report of an ongoing process, rather than as a final product. For example, a theoretically more sound model may understand ‘interior’ not as a category of its own, but allow the F1 score for each class was calculated for validation after each epoch. While the F1-score for the O class was very high for every epoch, the scores for the discrete categories of NNSE lfuctuated quite strongly.
Figure 1 illustrates the performance evaluation of the classifier on the validation dataset across diferent epochs. Class O, which represents tokens not classified as NNSE according to our guidelines, consistently achieves an almost perfect F1 score of 1. However, after 17 epochs, there is a significant decline in performance. Among the other classes, the ’interior’ class performs best with an F1 score of 0.792, while the ’urban’ class performs worst with an F1 score of 0.632 on the validation dataset. For a more detailed analysis, Fig2urperesents the average F1 scores for the ’interior’, ’urban’, ’rural’, and ’natural’ classes on the validation dataset, excluding class O. The black dotted line indicates the highest overall F1 score of 0.743.
Table3 below presents the final scores on the test dataset. Class O, with an F1 score of 0.99, significantly outperforms all other classes, as is expected. The ’interior’ class follows with an F1 score of 0.60. The remaining classes have F1 scores ranging between 0.64 and 0.53. for categorization into interior-rural or interior-urban. We are planning to run a set of annotations on the interior items to add this type of information and explore the diferences.
In addition to the general F1 scores for each class, we analysed false classifications across all five classes. The separation between NNSEs and class O was highly efective, with tokens belonging to class O rarely being incorrectly classified as NNSE. Conversely, the most common error for all types of spatial entities was their misclassification as class O. Notably, the diferentiation between various spatial entities was generally accurate, except for the ’urban’ class, which was occasionally misclassified as ’rural’.
In this work, we have developed a tool that already in its present state facilitates valuable
quantitative spatial research on 19th and early 20th century German-language literary
corpora. While out-of-the-box solutions typically only provide Named Entity Recognition (NER)
models, to the best of the authors’ knowledge, a classification of non-named spatial entities as
conducted here, classifying each entity into diferent types, has never been published before for
German. Schumacher, Flüh, and Nantke1[
The high performance in distinguishing spatial entities from non-spatial tokens is unsurprising, as this was the least contentious aspect during the evaluation of the annotation process. The high error rate of ’rural’ being misclassified as ’urban’ but not vice versa can be explained by the prevalence of ‘rural’ space in the training data. Additionally, the boundary between ’rural’ and ’urban’, as described in the guidelines, is more ’fuzzy’ compared to the respective distinctions to ’interior’ and ’natural’. This fuzziness may be aggravated by the inherent ambiguity of using sentences as training units.
This classifier is considered a work in progress, as it has currently been exclusively trained on Swiss-German texts from the late 19th to early 20th century. Potential improvements include gathering more training data and adapting thgebert-large model to Swiss-German literary texts from the long 19th century and beyond, as well as remodelling the categories to include interiorurban and interior-rural.
We plan to utilise this classifier to explore the remainder of the Swiss-German novel corpus built by Herrmann and Grisot8][, qualitatively examining patterns in the representation of space, with a particular focus on interior items. Subsequently, we intend to extend our research to a broader corpus of German literature. Methodologically, we plan to evaluate the use of synthetic training data provided by generative AI to enhance our model. One key aspect of space that will be analysed in-depth in future research is the relationship between afect and space in literature, building upon the previous work of Grisot and Herrma7n].n [
A. Géron.Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems. Second edition. Covid-19 Collection.
Beijing Boston Farnham Sebastopol Tokyo: O’Reilly, 2019. 819 pp.