The classification of historical and contemporary novels is a nuanced task that has traditionally relied on expert literary analysis. This paper introduces a novel dataset comprising Danish and Norwegian novels from the last 30 years of the 1th9century, annotated by literary scholars to distinguish between historical and contemporary works. While this manual classification is time-consuming and subjective, our approach leverages pre-trained language models to streamline and potentially standardize this process. We evaluate their efectiveness in automating this classification by examining their performance on titles and the first few sentences of each novel. After fine-tuning, the models show good performance but fail to fully capture the nuanced understanding exhibited by literary scholars. This research underscores the potential and limitations of NLP in literary genre classification and suggests avenues for further improvement, such as incorporating more sophisticated model architectures or hybrid methods that blend machine learning with expert knowledge. Our findings contribute to the broader field of computational humanities by highlighting the challenges and opportunities in automating literary analysis.
a late 18th-century setting. In contrast, Albert GnudtzmanR’sidder Thorvald. En lille københavnsk Roman (Knight Thorvald. A small Copenhagen novel, 1899) initially misleads with the historical-sounding keyword “knight” in the title, but the opening scene set in a modern urban café distinctly establishes it as a contemporary novel.
The question of whether a novel is set in modern days or historical times (contemporary or
historical novel?) was by no means uncontroversial in the time of the so-called Modern
Breakthrough in Scandinavian literature circa 1870-19006][. On the contrary, it was a question
of taste (good vis-à-vis bad) and, accordingly, a detection and quantification of the historical
novel give insight into the cultural divides of the period. Modern realist aesthetics ostracized
the historical novel and insisted that literature should be situated in the present and address
current problems. In 1871, famously and characteristically, the influential Danish critic Georg
Brandes, referring to Scott’s Waverley novels from the early 1800s, rejected the historical novel
as “an unfortunate and now abandoned genre, imported from Scotland and invented by a
pureblooded Tory, which originated in a state of mind similar to ours, one with all its ideals in the
past” [
We introduce a dataset of Danish and Norwegian novels from the last 30 years of the 1800s, annotated by literary scholars according to whether they are historical or contemporary. The novels are taken from the MeMo (Measuring Modernity) corpu5s],[comprising 859 novels. We assess the ability of language models to generalize this generic distinction as expressed in their titles and first few sentences, by training them on a portion of the dataset and evaluating them on unseen novels. While fine-tuned Danish language models show good performance in the task, error analysis reveals they still lack sensitivity to salient cues that literary scholars observe.1
Text classification is a pivotal task in natural language processing (NLP) that entails
categorizing text into predefined labels or classes. It has a broad spectrum of applications, including
sentiment analysis [1], word sense disambiguation 2[
When dealing with literature, in academic as well as everyday contexts, taxonomic thinking
and practices seem both habitual and inevitable. Literary genre studies got underway with
the Ancient Greeks, from which the division of poetic literature in three main genres: lyric,
epic and drama, often ascribed to Aristotle and hisPoetics, has proliferated1[
Within computational literary studies of recent years, genre has been an important touch
point for NLP approaches and literary theory and historiography. Text genre classification is a
crucial area of research that aids in systematically categorizing vast and diverse collections of
literary works. This task involves distinguishing between various genres such as fiction and
non-fiction [
There is a long and intensive research tradition that has been interested in the political and
social-historical implications of the historical novel of thteh 1c9entury and the decline of the
genre in the latter part of the century as a reflex of a new aesthetic positions with a primacy
of immediate perception and contemporaneity2[
In this paper, we construct a dataset using the genre classification of the novels of the MeMo corpus performed by Bjerring-Hansen and Ørto6ft[], which has followed such pragmatic and intuitive understanding of the genre as something that can be decoded immediately by a quick inspection of the temporal coordinates of the individual texts. To carry out an analogous quantification of genre trends and proportions between historical and contemporary novels, the authors performed close readings of both (certain) paratexts and (particular) parts of each novel. More specifically, the annotation was carried out on the following premises: 1. Many historical novels “reveal” themselves already in the title (as is the case wDirtohnning Caroline Mathilde af Danmark = Queen Mathilde of Denmark by the pseudonym Caja from 1889) or in the subtitle (as is the case with the anonymously publishCeadroline. Bøhmens frygtelige Svøbe eller et Gammel Bjergslots Hemmelighed: Historisk-romantisk Fortaelling = Caroline. Böhmen’s terrible scourge or the secret of an old mountain castle; Historical-romantic tale), or more redundantly both the one and the other (cf. H.F.
Ewald’sGrifenfeld. Historisk Roman = Grifenfeld. Historical Novel from 1888). 2. If the titles do not contain clear paratextual signals, genre afÏliation is often indicated on the first few pages of the novel (as, for example, in the case ofIndianerpigen fra Cape Breton = The Indian Girl from Cape Breton by “L.M”, which, although the title page does not indicate that we are dealing with a historical novel, immediately sets the temporal scene with the opening sentence: “It was in the year 1780 [...]”).
So, generally, it is striking to what extent the historical novels of theth1c9entury clearly
and actively give away their generic afÏliation. As several literary scholars have pointed out,
this is probably because the historical novel’s foremost characteristic—and selling point—is
its historical setting, which, then, producers and distributors clearly wants to mark for the
intended readership and therefore already on the title page or the first pages “come clea4n7”,[
This transparent literary communication, or consumer information, which is of course less obvious in the few and often canonized instances of experimental novels that ”play” with genre ifction such as the historical novel, can be said to be a general feature of popular literature, including also romances and detective stories etc., and the genre-fiction system, developing
Total novels 859 Total sentences 3,282,643 Total words 53,588,381 Average sentences per novel 3,821 Average words per novel 62,385
Average words per sentence 16.3
in the latter part of the 1t9h century [
To address this question, we treat the problem from a machine learning perspective. We introduce an annotated corpus and fine-tune pre-trained transformer language models on it, evaluating their performance on a held-out test set.
We rely on the MeMo corpus5[], comprising 859 Danish and Norwegian novels spanning the
last 30 years of the 19th century, with more than 64 million tokens. The corpus is a rich and
diverse collection of texts that provides valuable insights into the classification of novels as
historical and contemporary during the period under investigation. Ta1bslheows statistical
information about the corpus. We obtain the annotated dataset of novels from Bjerring-Hansen
and Ørtoft [
We use the dataset for training and evaluating transformer-based language models. Specifically, inspired by the observations made by Bjerring-Hansen and Ørtoft, we consider three settings: 1. Providing the title and sub-title of the novel as input to the model, 2. Providing the first 15 sentences of the novel as input to the model, 3. Concatenating the title, sub-title and first 15 sentences and providing them to the model. In all cases, we train the model to classify the novel according to the binary label obtained from the annotated dataset. Subsequently, we evaluate the models on a test set of held-out novels to assess their ability to generalize the ability to identify the cues learned during training.
We experiment with four pre-trained language models and three types of provided context, comparing their performance and using them as the basis for an elaborate analysis of errors and indicative features.
The models evaluated in our novel classification experiments had been pre-trained on text
corpora including Danish and Norwegian text. We train them on the task using supervised
finetuning. Importantly, all models are selected based on their performance evaluated on Danish
and Norwegian literary benchmark datasets22[], the Scandinavian Embedding Benchmark3
and ScandEval4, [
DanskBERT. DanskBERT,5 a top-performing Danish language model noted for its success
on the ScandEval benchmark 4[1], is based on the XLM-RoBERTa architecture and trained
on the Danish Gigaword Corpus4[
based on the BERT architecture, featuring 24 layers, 16 attention heads, and a hidden size of 1024. It incorporates a dropout rate of 0.1 for attention probabilities and hidden states, using GELU activation and supporting up to 512 position embeddings. With a vocabulary size of 50,000 tokens, this model, referred to as DFM (Large), excels in tasks such as Danish sentiment analysis and named entity recognition6.
MeMo-BERT-03. Developed by continuing the pre-training of the pre-trained Transformer
language model DanskBERT 2[
NB-BERT-base. A general-purpose BERT-base model was developed using the extensive digital collection at the National Library of Norw2a0y].[8 It follows the architecture of the BERT Cased multilingual model and has been trained on a diverse range of Norwegian texts, encompassing both Bokmål and Nynorsk from the past 200 years. This comprehensive training allows the NB-BERT-base to efectively handle a wide array of NLP tasks in Norwegian. The model achieved the second-highest performance ranking in the Norwegian Named Entity Recognition task compared to other models listed on the ScandEval benchmark for Norwegian natural language understanding.
Our experiments involve fine-tuning the pre-trained language models on the annotated novels
from our corpus. To enable testing of generalization in the face of temporal sh26ift ][, the last
130 novels according to publication yea≈r1(5%) are used as a testing set, while the remaining
novels were randomly divided into training and validation with 70% and 15% respectively. The
experiments involve fine-tuning the models on the dataset using a batch size of 32, training
for 20 epochs with the AdamW optimizer2[
In this experiment, we concatenate the title and subtitle of each novel and perform classification by fine-tuning the aforementioned pre-trained language models with the novel labels, using the cross-entropy objective. Table2 (left) presents the fine-tuning results of the selected models. DFM (Large) achieved the highest performance on the validation set with an F1-score of 92%, while the MeMo-BERT-03 model excelled on the testing set with an F1-score of 91%.
We use the Danish pipeline in spaCy 1[
The results of fine-tuning the models is shown in Table2 (middle). DFM (Large) and MeMoBERT-03 achieved the highest performance on the validation set with an F1-score of 81%, while DFM (Large) excelled on the testing set with an F1-score of 86%. Notably, for all models, using the first 15 sentences as input performs worse than using the title and sub-title.
In this experiment, we combine both the titles & sub-titles and the first 15 sentences of each novel in the corpus: technically, we repeat the same setup as using the first 15 sentences, but additionally prepend the concatenated title and sub-title as if they were another sentence. The ifne-tuning results of the four models of this experiments are shown in Tab2le(right). DFM (Large), MeMo-BERT-03 and NB-BERT-Base achieve equal performance on the validation set with an F1-score of 82%, while DanskBERT and NB-BERT-Base perform best on the testing set with an F1-score of 84%. Overall, performance in this setting is similar to just using the first 15 sentences, but the best performance on the test set is in fact obtained when just using titles and sub-titles, and ignoring the first 15 sentences.
While all models are highly accurate after fine-tuning, surprisingly, we observe that the best predictions are obtained by just using the title and sub-title as input, disregarding the first 15 sentences of the novel. This suggests either that the genre information is less salient in the first 15 sentences, or that the models are not as capable of extracting it from them. To analyze this further, we investigate model confidence on mislabeled predictions, and perform a fine-grained error analysis.
When reading the text opening, introspection from expert annotation reveals that genre
identification often hinges on specific key sentences (e.g., mentioning specific entities, events, or
years) rather than the entire opening passage. While most sentences in the opening text do
not clearly suggest one genre or another, these ”giveaways” are sparse but salient. This nuance
may be lost by the majority voting procedure over sentences, leading to misclassifications when
the first 15 sentences are used as input. Therefore, we are interested in the model’s confidence
and whether it is calibrated to match the experts’ uncertainty or disagreement about the labels
[
To evaluate model calibration, we use Expected Calibration Error (ECE), a metric that measures how well the model’s predicted probabilities reflect the true accura9c]y: [ | | ∑ =1
= | ( ) − ( )| bin , ( ) is the accuracy in bin , and (
) is the average confidence in bin . where = 10 is the number of bins (confidence intervals),| | is the number of samples in When using titles and sub-titles as input, the best-performing model (MeMo-BERT-03) achieved a relatively low ECE of 0.062, as shown in Tab3le,indicating that it was reasonably well-calibrated when relying on paratextual information. Titles and sub-titles often contain clear genre markers that allow the model to make high-confidence, mostly accurate predictions. However, the model still made misclassifications, particularly when historical-sounding titles misled the model. Despite this, the model’s overall confidence generally matched its performance in this setting, and it exhibited the lowest calibration error compared to other settings. This result underscores the strength of using paratextual clues, though it also reveals that misleading terms in the title can cause overconfidence in wrong predictions.
In contrast, when the models used the first 15 sentences of the novels as input, calibration worsened across all models. For example, the ECE for DFM (Large) increased to 0.085, and other models similarly struggled with higher calibration errors (see 3T)a.blTehis is likely because, as noted in expert analyses, genre signals are not uniformly distributed across the opening sentences. Instead, they tend to appear in specific key sentences that reveal important genre-relevant details. In many cases, the first 15 sentences are ambiguous, lacking explicit time markers or character descriptions, which increases the model’s uncertainty. However, rather than reflecting this uncertainty in their confidence scores, the models often exhibited overconfidence, resulting in higher calibration errors. This overconfidence indicates that the models are not adequately capturing the uncertainty present in the text openings, a discrepancy that reflects the challenge of extracting nuanced genre information from longer inputs.
Combining both titles, sub-titles, and the first 15 sentences of the novels did not uniformly improve calibration. For MeMo-BERT-03, the ECE increased to 0.094, suggesting that integrating both sources of information did not lead to better confidence alignment. Although the models had access to more context, they struggled to efectively weigh the paratext against the more ambiguous textual cues from the opening sentences. In some cases, conflicting signals between the title and the text may have caused the models to oscillate between genres, ultimately leading to poorer calibration. DanskBERT, however, exhibited slightly better calibration in this setting, indicating that it was more adept at integrating the two types of input compared to other models. This slight improvement over single-input settings suggests that certain models can benefit from additional context, though the integration process remains challenging for most.
We discuss the errors encountered during the classification of historical and contemporary novels, focusing on prediction errors made by the best-performing models, as well as annotation errors identified by expert inspection of the mislabeled predictions.
A notable pattern in the prediction errors is the tendency of the models to misclassify contemporary novels as historical based on titles (in 12 out of 17 cases where at least one model misclassified the label in this setting) and based on text openings (first 15 sentences) or the combination of titles and text openings (11 out of 11 cases). An illustrative example of the first type of error is Albert Gnudtzman’s urban novReildder Thorvald. En lille københavnsk Roman (Knight Thorvald. A small Copenhagen novel, 1899). The title’s keyword “knight” leads the models to misinterpret it as a historical romance. However, the opening scene set in a lively urban café correctly classifies it as a contemporary novel, highlighting the discrepancy between title-based and content-based classification.
When models misinterpret text openings (first 15 sentences), a common issue is their failure to recognize historical settings established through character introductions rather than explicit time clues. For instance, in Marie HenckelL’solotte. En Roman fra den Gustavianske Tid (Lolotte. A Novel from the Gustavian Period, 1898), while the subtitle clearly indicates a late 18th-century setting, the models fail to date the characters like Prince Gustaf and Sofie Magdalene, leading to incorrect classifications.
Machine readings of genre clues also shed light on borderline cases, such as novels set in the near past relative to the modern breakthrough period (1870-99). Examples include novels set during the Danish-German wars of 1848-50 and 1864, like Chr. ChristensenK’aesrlighedens Mysterier. En Historie fra 1848-50 (The Mysteries of Love. A Tale from 1848-50, 1899) and P.A. Worm’sForbrydelsernes Konge eller Den skalperede Praest (The King of Crime and the Scalped Priest, 1899). An intriguing case involves a novel beginning in the narrator’s present with modern elements like electric light and a telephone but transitioning to a historical analepsis: U. Ravn’s Interioerer fra vores Bedsteforaeldres Tid (Interiors from the Time of our Grandparents, 1899).
After in-depth expert analysis, eight of the models’ “errors” in the test set turned out to be Bjerring-Hansen and Ørtoft’s annotation errors, including four plain mistakes and four tricky in-between novels where further inspection validated the models’ predictions. These erroneous annotations were evenly distributed between misclassified historical and contemporary novels.
An interesting case is the novelHvorfor hun blev Nonne. En Fortaelling om fransk Klosterliv (Why she became a nun. A story about French monastic life, 1899) by the pseudonym “Herdis”. Both models and annotators were misled by the title into thinking it was a medieval story. However, a close reading of the text’s opening pages revealed it to be set in modern times, aligning with the neo-romantic current of the 1890s that revived the historical novel and Catholic themes.
In this study, we presented a dataset of Danish and Norwegian novels from the lateth1c9entury, classified as historical or contemporary by literary scholars. We investigated the performance of several pre-trained language models in distinguishing between these two genres based on titles and the first few sentences. While the models demonstrated commendable accuracy, the error analysis revealed limitations in capturing the nuanced cues recognized by human experts. These findings underscore the complexity of literary text classification and suggest that while NLP models can significantly aid in the categorization process, they still require further refinement to match human interpretative abilities fully. In our approach, we chose to limit the textual input to only the titles and the first 15 sentences of each novel. This decision was informed by the pretraining of the models, which predominantly focused on non-literary and contemporary content, meaning that historical figures, settings, and subtleties were likely underrepresented in the training data. As a result, we hypothesized that the opening framing of the novels, which often serves to establish the genre, would be more efective for detection than deeper content. This was evident in our findings, where the titles and subtitles emerged as the most straightforward indicators of the genre distinction. While expanding the analysis to include more content from the novels could potentially capture stronger genre signals, the results indicated that the models performed best on the titles and subtitles; in fact, the first 15 sentences did not surpass the efectiveness of the titles alone. This suggests that the model, like the historical readers, efectively identifies key genre signals from surface-level clues, such as titles and subtitles. While our method is aligned with historical signals by focusing on initial clues that reflect the genre distinctions recognized during the period, it also highlights the need for deeper content analysis, which would require models pretrained on a substantial amount of historical material extending well before the end of thteh 1c9entury. To address this, future work could explore more sophisticated model architectures or hybrid approaches that combine machine learning with expert knowledge to enhance the accuracy and depth of genre classification in literary studies.
Our study shows that the historical novel was by no means an extinct genre in DanishNorwegian literature at the end of the 1th9century, as the prevailing modern aesthetic would have it, and furthermore that this was a rather obvious fact, since genre decoding–historical novel – yes or no? – is a relatively trivial afair. It can be determined with great certainty, both by people and models, by reading the paratext and/or the first lines of the novels. Of course, this generic stability cannot be taken for granted or universalized if the fine-tuned models from our study are applied to other textual sources, such tahsc2e0ntury novels, where genre innovations are increasing and where the historical novel is also exposed to modernist experiments (an early Danish example of this is Nobel Prize winner Johannes V. Jensen’s novel Kongens Fald (The fall of the king, 1900-01), which represents both a historical depiction of the dramatic events leading to the fall of the Kalmar Union and a modeflârnneur novel. In future literary studies, we will be able to test the stability of the genre distinctions created during the 19th century, when the literary field and the formation of taste were established, but for accuracy in prediction, we will most likely have to adjust our methods to consider the content of novels on a broader scale.
The comparison between qualitative annotations and machine predictions enhanced our understanding of the quantitative arguments applicable to the period’s literature. It highlighted the coexistence and interaction of old and new forms and meanings within an aesthetic timespan. This approach aligns with a broader perspective on genre classification that leverages predictive modeling. As Ted Underwood suggests, our objective shifts from defining a genre to developing a model that can replicate the judgments made by specific historical observer4s4][. This paradigm not only advances our technical capabilities but also deepens our literary and historical understanding, bridging the gap between computational methods and humanistic inquiry. [1] A. Allaith, K. Degn, A. Conroy, B. Pedersen, J. Bjerring-Hansen, and D. Hershcovich. “Sentiment Classification of Historical Danish and Norwegian Literary Texts”. InP:roceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa). Ed. by that work on artificial languages (programming languages, logics, formal systems) that does not explicitly address natural-language issues broadly construed (natural-language processing, computational linguistics, speech, text retrieval, etc.) is not appropriate for this area.'].