Building on previous work examining fictional space and sentiment in Swiss-German narrative [7], this paper reports on the development and evaluation of a novel machine learning model for the analysis of fictional space in literary texts. 1 In recent criticism across disciplines there has been an increased emphasis towards considering place and space as crucial factors in understanding social, cultural, and historical phenomena. This perspective on spatiality is generally referred to as the 'spatial turn' [9,10,19], and, in literary studies, it highlights the integral components of space representation in how we under-stand and contextualise narrative and fictional texts. The exploration of spatial representations in literary works offers valuable insights into the landscapes, the constructed environments and their social implications within narratives, as well as into the cultural and socio-political constructs surrounding certain images. while there are several valuable proposals to a quantitative approach to spatial research [24,2], including for example a differentiation of space as background and place more specifically as locus of events [23,15], we however set a different focus.
In this paper, we present a case study on the prediction of what we call 'non-named spatial entities' (NNSE) in a historical corpus of Swiss-German novels using a deep learning model in conjunction with BERT and Prodigy. By combining manual annotations and advanced machine learning methods, we aim to automatically detect and recognise NNSE within the literary narratives via a similar approach to named entity recognition (NER).
NER techniques are used to identify and categorise text segments that refer to entities such as people, places, or companies, and that 'constitute proper names' [11]. The latest NER techniques rely on manually annotated text corpora, which are automatically analysed to build models that capture language use and grammar. These models can then identify and classify entities in new, unprocessed documents. State-of-the-art NER systems come with pre-built models trained on extensive collections of annotated documents, like news articles. These models typically perform well and are ideal for specific applications such as analysing customer feedback or extracting locations and characters. However, when applied to documents with linguistic features not well-represented in the training data, such as literary texts, NER performance can decline, increasing the likelihood of errors, also increased for most languages other than English.
Within literary studies, various scholars have used NER, particularly to identify fictional characters [20] , build social networks [5], identify geographical locations [3], to assign headings to novels [16], or to analyse relationships between literary works [21,22,17]. Only few researchers until recently however focused on the identification of NNSE, i.e. those terms, or elements of space representation, which are not necessarily named geographical locations, like Berlin, London, or Zurich, and that therefore typically cannot be located on a map. It is this kind of entity that generally contributes most to effectively to create the so-called 'storyworld' [18]: simple terms or phrases such as 'mountain', 'bridge', 'beach' or 'cave', as well as objects and architectural parts that make them tangible, such as 'window', 'table', or 'wall'.
A similar perspective has been considered by Schumacher, Flüh, and Nantke [14], who used conditional random fields to automatically annotate among other things non-named places, which is conceptually similar to NNSEs. However, the operationalization of space in their research is focused on places that can be found on a map, leaving therefore the broader concept of NNSEs unexplored. Also, the popular BookNLP toolkit by Bamman is able to work to identify what they call 'locations' (for natural entities) and facilities' (for man-made structures) in English-language texts with an accuracy of up to 90%2 . While BookNLP is able to differentiate NNSEs somewhat according to our own needs, we propose a distinction for 'facilities' into more discrete classes shown in Section 2.1. This paper sets out to fill this gap, training a model that will help us identify spatial elements in narrative.
In order to train our model to recognise literary space, we decided to train it not only to be able to recognise non-named spatial entities (NNSEs), but also to distinguish among four different types of spatial environments. We decided to base our categorisation on the research by Grisot and Herrmann [7], who looked at the sentiment encoded in narrative texts in relation to both named and non-named entities. They used a dictionary based approach, collecting spatial terms for geographical locations as well as for non-named entities, distinguishing in particular the categories 'rural', 'natural' and 'urban'. While these three categories offered a promising base, we felt that for a more comprehensive perspective on the spaces and places rendered in fictional texts we also needed to include spatial elements describing the interiors/indoor space of buildings and rooms. We therefore created an additional category of NNSE, 'interior'. Some examples for each category are shown in Table 1 above.
To produce the training set, two annotators were provided with written guidelines and trained in person to understand the difference between the various NNSE categories. They were then instructed to use the platform Prodigy [12], which allowed them to read sentences from the training set in random order, and to annotate NNSEs directly on the interface by adding labels to individual tokens. For the annotation process, six novels were sampled from the complete corpus of Swiss-German novels [8]. 3 The novels were then split into sentences (N=9,062), which were manually annotated by the annotators. The annotators featured a high interannotator agreement (Cohen's Kappa) for the NNSE types 'interior', 'natural', and 'rural', and medium agreement for 'urban' 4 , as well as a high agreement for the distinction of NNSEs against a not-NNSE token (Cohen's Kappa = 0.898).
The values for annotators' agreement in relation to individual types are shown in Table 2. Also shown in Table 2 are the number of sentences in our dataset in which the various classes of spatial entities were identified by manual annotation. Type O shows the number of sentences where no NNSEs were identified in the annotation process. These amount to over 86% of the dataset, making the distribution of the five categories in our dataset unbalanced. However, the NNSE categories are much closer together, ranging between 2.2% (urban) and 4.5% (interior) of all sentences.
After the annotation process, sentences were randomly assigned to either the training dataset (80%, N=7,249) or the test dataset (20%, N=1,813). This was done according to common best practices for training a machine learning model [6]. To train the classifier, PyTorch version 2.1.1 was used as the deep learning framework [13]. In conjunction with PyTorch, the popular hugging face library (version 4.35.2) was used to load and interact with the language model [25].
The model gbert-large by deepset was utilised as input layer for the token classifier, since it outperformed all other language models [4]. The model classifies each token of a given text, and attempts to predict whether the token under consideration can be classified as a NNSE (one of the four types mentioned above) or whether it can be considered not a spatial term (O). It was tested if the model performs better with the complete, unbalanced training dataset (N=7,249 sentences) or with a more balanced, downsampled training dataset (N=2,004 sentences). The downsampled training dataset was composed of sentences including at least one NNSE (N=1,002) and a random selection of sentences with no NNSE of equal size. The downsampled train dataset was then split again into a final training dataset (N=1,603) and a validation set (N=401).
The training was repeated for 17 epochs with a learning rate of 5e-6 and a dropout of 0.1. These parameters are determined by extensive hyperparameter testing. Since the maximum length of all sentences is 40 words or 61 tokens, the max length for the BERT model was set to 64 tokens.
With the annotation and training process described above, we produced a classifier able to 1) identify NNSEs in a given sentence, and 2) classify the identified NNSE as belonging to one of the four discrete classes: 'rural', 'urban', 'natural' or 'interior'. 5 To find the best parameters, Figure 1 illustrates the performance evaluation of the classifier on the validation dataset across different epochs. Class O, which represents tokens not classified as NNSE according to our guidelines, consistently achieves an almost perfect F1 score of 1. However, after 17 epochs, there is a significant decline in performance. Among the other classes, the 'interior' class performs best with an F1 score of 0.792, while the 'urban' class performs worst with an F1 score of 0.632 on the validation dataset. For a more detailed analysis, Figure 2 presents the average F1 scores for the 'interior', 'urban', 'rural', and 'natural' classes on the validation dataset, excluding class O. The black dotted line indicates the highest overall F1 score of 0.743.
Table 3 below presents the final scores on the test dataset. Class O, with an F1 score of 0.99, significantly outperforms all other classes, as is expected. The 'interior' class follows with an F1 score of 0.60. The remaining classes have F1 scores ranging between 0.64 and 0.53.
for categorization into interior-rural or interior-urban. We are planning to run a set of annotations on the interior items to add this type of information and explore the differences. In addition to the general F1 scores for each class, we analysed false classifications across all five classes. The separation between NNSEs and class O was highly effective, with tokens belonging to class O rarely being incorrectly classified as NNSE. Conversely, the most common error for all types of spatial entities was their misclassification as class O. Notably, the differentiation between various spatial entities was generally accurate, except for the 'urban' class, which was occasionally misclassified as 'rural'. The highest error rate for all four NNSE types involves their misclassification as class O, ranging from 6.9% for 'interior' to 14% for 'rural'. The biggest error after that is by misclassifing 'rural' as 'urban' 13% error rate for being classified as 'rural'.
In this work, we have developed a tool that already in its present state facilitates valuable quantitative spatial research on 19th and early 20th century German-language literary corpora. While out-of-the-box solutions typically only provide Named Entity Recognition (NER) models, to the best of the authors' knowledge, a classification of non-named spatial entities as conducted here, classifying each entity into different types, has never been published before for German. Schumacher, Flüh, and Nantke [14] developed a classifier based on Conditional Random Fields (CRF), which includes several more categories than our classifier, but it can only detect places, not their types. Bamman's bookNLP is able to differentiate between locations, which alignes with our 'natural' type, and facilities, which covers 'urban' and 'rural' spatial entities. For comparison, we also tested the large language model Llama3 7b [1] with a fewshot prompt for recognising unnamed spatial entities (NNSE). The automatic classification on the test dataset with Llama3 resulted in only a 5.6% partial match with the manual annotation, and a 0.7% perfect match. The main issue was the model's tendency to hallucinate new NNSEs when attempting to continue a sentence, contrary to instructions.
The high performance in distinguishing spatial entities from non-spatial tokens is unsurprising, as this was the least contentious aspect during the evaluation of the annotation process. The high error rate of 'rural' being misclassified as 'urban' but not vice versa can be explained by the prevalence of 'rural' space in the training data. Additionally, the boundary between 'rural' and 'urban', as described in the guidelines, is more 'fuzzy' compared to the respective distinctions to 'interior' and 'natural'. This fuzziness may be aggravated by the inherent ambiguity of using sentences as training units.
This classifier is considered a work in progress, as it has currently been exclusively trained on Swiss-German texts from the late 19th to early 20th century. Potential improvements include gathering more training data and adapting the gbert-large model to Swiss-German literary texts from the long 19th century and beyond, as well as remodelling the categories to include interiorurban and interior-rural.
We plan to utilise this classifier to explore the remainder of the Swiss-German novel corpus built by Herrmann and Grisot [8], qualitatively examining patterns in the representation of space, with a particular focus on interior items. Subsequently, we intend to extend our research to a broader corpus of German literature. Methodologically, we plan to evaluate the use of synthetic training data provided by generative AI to enhance our model. One key aspect of space that will be analysed in-depth in future research is the relationship between affect and space in literature, building upon the previous work of Grisot and Herrmann [7].