This paper addresses methodological issues in diachronic data analysis for historical research. We apply two families of topic models (LDA and DTM) on a relatively large set of historical newspapers, with the aim of capturing and understanding discourse dynamics. Our case study focuses on newspapers and periodicals published in Finland between 1854 and 1917, but our method can easily be transposed to any diachronic data. Our main contributions are a) a combined sampling, training and inference procedure for applying topic models to huge and imbalanced diachronic text collections; b) a discussion on the differences between two topic models for this type of data; c) quantifying topic prominence for a period and thus a generalization of document-wise topic assignment to a discourse level; and d) a discussion of the role of humanistic interpretation with regard to analysing discourse dynamics through topic models.
This paper reports our experience on studying discursive change in Finnish newspapers from the second half of the nineteenth century. We are interested in grasping broad societal topics, discourses that cannot be reduced to mere words, isolated events or particular people. Our long-lasting goal is to investigate a global change in the presence of such topics and especially finding discourses that have disappeared or declined and thus could easily slip away in modern research. We believe that these research questions are better approached in a data-driven way without deciding what we are looking for beforehand, though the choice of the most suitable techniques for such research is still an open problem.
In this paper we focus on developing methodology. Choosing available algorithms for analysis guides possible outcomes as they are designed to be operationalised in ‡SH was affiliated with the University of Helsinki for most of this work. *Equal contribution.
Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). certain ways. Approaching our goal with mere word counts is counterproductive due to the sparseness of the language and the variety of discourse realisations in a given text. Further, word counts are unreliable with historical data due to never ending language change, spelling variations and text recognition errors.
Thus, as many other papers in the area of digital humanities, we utilize topic modelling as a proxy to discourses. In particular, we apply the “standard” Latent Dirichlet Allocation model [3, LDA] and its extension the Dynamic Topic Model [2, DTM], which is developed specifically to tackle temporal dynamics in data. However, any model has its limitations and tends to exaggerate certain phenomena while missing other ones. We focus on the difference between models and try to reveal their limitations in historical data analysis from the point of view that is relevant for historical scholarship.
Our main contributions are the following: – We propose a combined sampling, training and inference procedure for applying topic models to large and imbalanced diachronic text collections. – We discuss differences between two topic models, paying special attention to how they can be used to trace discourse dynamics. – We propose a method to quantify topic prominence for a period and thus to generalize document-wise topic assignment to a discourse level. – We acknowledge and discuss the drawbacks of topic stretching, which is typical for DTM. It is commonly known that DTM sometimes represents topics beyond the time period, but thus far there is no discussion in how researchers should tackle this for humanities questions.
In order to illustrate the appropriateness of the proposed methodology we discuss two use cases, one relating to discourses on church and religion and one that relates to education. The role of religion and education has been studied extensively in historical scholarship but there are no studies that deal with these topics through text mining of large-scale historical data. These two topics were chosen due to the the fact that the former was in general a discourse in decline relating to the process of secularization in Finnish society, whereas the latter increased in the second half of the nineteenth century and relates to the modernization of Finnish society and the inclusion of a larger share of the population in the sphere of basic education. In addition to these two interlinked discursive trends, we also use other examples to illustrate the strengths and weaknesses of LDA and DTM for this type of historical research. 2
Our dataset is from the digitised newspaper collection of the National Library of
Finland (NLF). This dataset contains articles from all newspapers and most periodicals that
have been published in Finland from 1771 to 1917. Several studies have used parts of
this dataset to investigate such issues as the development of the public sphere in
Finland, the evolution of ideological terms in nineteenth-century Finland and the changing
vocabulary of Finnish newspapers [
The full collection includes articles in Finnish, Swedish, Russian, and German. In
this work we focus only on the Finnish portion starting from 1854 because this is the
point where we determined we have sufficient yearly data to train topic models. The
resulting subset has over 3.6 million articles and is composed of over 2.2 billion tokens.
Figure 1a shows that the number of tokens published per year in Finnish-language
papers increased steadily. The average article has 526 tokens but article length varies
widely from year to year, as seen in Figures 1b and 1c which show the average article
length and the number of articles per year. As made clear by these figures, there is a
noticeable difference in the number of articles and average article length after 1910.
This shift does not reflect the actual articles in the newspapers, but is the result of a
change of OCR engine used to digitise the collection [
Still, even if the article segmentation differs in the latter period, Fig. 1a shows that
there is steady increase in the vocabulary used in the Finnish-language newspapers
published in the second half of the nineteenth century. They also covered more themes
and regions. This entailed a process of diversification and modernization of the Finnish
press, which has been widely discussed in historiography. As a collection, the
newspapers vary a lot in style and focus. Some larger newspapers mainly contain political
content, whereas others are rather specialised, and yet others thrived by giving a voice
to the local public [
Given the size of the data and its inherent nature, notoriously the OCR quality and the unbalanced data from different time slices, we performed a series of pre-processing steps on the data.1
Despite prior work (albeit on English), showing that stemming has no real
advantage for likelihood and topic coherence and can actually degrade topic stability [
(c) No. of articles
modelling [
LDA
Topic modelling is an unsupervised method to extract topics from a collection of
documents. Typically, a topic is a probability-weighted list of words that together express
a theme or idea of what the topic is about. One of the most popular topic modelling
methods currently in use is Latent Dirichlet Allocation (LDA), which is “a generative
probabilistic model for collections of discrete data such as text corpora” [
There are different ways to overcome this limitation. One possibility is to split the data into time slices and train LDA separately on each slice. However, in this case LDA models for each slice would be independent of each other and there is no straightforward approach of matching topics from independent models trained on disjoint data. Another possibility, which we explore in this paper, is to train a single model for a subset of the whole data set over the entire time period and then use topic prominence as proxy for the dynamics of discourses over time.
To do this, we compute the prominence of a topic in a given year by summing up the topic contribution for each document in that year and then normalise this number by the sum of all topic contributions from all topics for that year, as in Equation 1. (1) where y is a year in the dataset, k is a topic index, Dy is the number of documents in year y, dj is the jth document in year y and T is the number of topics in the model.
The large size of the collection and its unbalanced nature is a problem for training topic models. It is computationally expensive to train a model with millions of articles and the resulting model would be heavily biased towards the latter years of newspaper collection because it has far more data. To overcome these issues, we sampled the collection such that we have a roughly similar data size for each year of the collection and as a result, we also get a vastly reduced dataset. However, to have a model of discourse dynamics that reflects the collection more closely, we compute topic prominence using the entire collection and not just the sampled portion. We do this by inferring the topic proportions of all the documents in the collection and using these inferred distributions to compute topic prominence. 3.2
DTM As mentioned above, there are topic models that explicitly take into account the temporal dynamics of the data. One such model is the dynamic topic model (DTM). DTM is an extension of LDA that is designed to capture dynamic co-occurence patterns in diachronic data. In this model, the document collection is divided into discrete time slices and the model learns topics in each time slice with a contribution from the previous time slice. This results in topics that evolve slightly–words changing in saliency in relation to a topic–from one time step to the next.
However, DTM also has its own limitations. It is based on an assumption that each topic should be to some extent present in each time slice, which is not always the case with real-world data such as news archives where events and themes can sometimes disappear and then re-appear at some point in the future.
Perhaps more importantly for historical research, a weakness of DTM lies in its
design: to accomplish alignment across time the topic model is fit across the whole
vocabulary and thus smoothing between time slices is applied. As a result, events end
up being “spread out” before and after they are known to happen. This problem only
becomes evident after a thorough analysis: similar models in different fields such as
lexical semantic change present the same issue – the dynamic topic model SCAN [
For both the LDA and DTM models, we use the Gensim implementation [
Topic models are widely used in the digital humanities and social sciences to draw
insights from large-scale collections [
In [
Topic modelling has also been used in discourse analysis of newspaper data. In [
LDA was not designed for capturing trends in diachronic data and so several
methods have been developed to address this, such as DTM, Topics over Time [39, TOT],
and the more recent Dynamic Embedded Topic Model [6, DETM], an extension of
DTM that incorporates information from word embeddings during training. As far as
we are aware, DTM and TOT have not been used for historical discourse analysis or
applied to large-scale data collections. In the original papers presenting these methods,
DTM was applied to 30,000 articles from the journal Science covering 120 years and
TOT was applied to 208 State of the Union Presidential addresses covering more than
200 years. This was to demonstrate the evolution of scientific trends for the former and
the localisation of significant historical events for the latter. Recently DETM was
applied on a dataset of modern news articles about the COVID-19 pandemic where the
authors observed differences between countries in how the pandemic and the reactions
to it were framed [
In the mentioned cases researchers tackle the interpretative part of using topic
models for humanistic research in different ways. Like Pa¨a¨kko¨nen and Ylikoski [
What a discourse is, has been heavily theorised within the different strands of discourse
analysis [
To get at the tension between topics and discourses, we approached the material
without a predefined idea about which topics we wanted to study in order to keep the
study as data-driven as possible. Our interest was to use topic modelling to capture
topics that could in a meaningful way be related to societal discourses, that is themes
that cannot be narrowed down to individual words, but still are reasonably coherent
and form at least loose topics. To this end, we trained topic models with k 2 f30; 50g,
inferred topic distributions for the whole collection and inspected models by carefully
going through the top words in each topic and using PyLDAVis2 [
To discuss the benefits of LDA and DTM, we chose to focus on two specific themes, the discourse relating to religion and religious offices, and education. They are both rather neatly identifiable in the data, but display different trends. The former is in decline over the period of interest, whereas the latter increases in topic prominence. They can also be related to large scale processes in Finland, religious discourse to the secularization of society and education to the modernization of civic engagement. 5.1
The two topic modelling methods perform in somewhat different ways. As mentioned, DTM is designed to incorporate temporal change in the topics, which means it includes a stronger sense of continuity in its representations of data. Whether or not this is desirable, depends on the research question, but our contention is that for studies interested in discursive change, this is either a problem or at least it is something that needs to be factored in making the historical interpretation. If we want to understand when certain discourses became dominant, declined, or even disappeared, this type of stretching cannot be allowed.
An exceptionally illustrative example of stretching among our fifty topics, is an introduction of the Finnish mark as a currency (Fig. 2a). With top words such as “mark”, “penny”, “price”, “thousand”, “pay” etc. the topic comes across as one with high internal coherence. We also see that the topic grows in prominence over time, from being relatively modest in the 1850s to gradually increased prominence after 1860. This makes sense, as the mark was adopted as currency in the year 1860 and after that self-evidently figured in public discourse. However, when we look at a heatmap visualization of the topic (Fig. 2b), we see how the topic stretches from the period 1854–1859 to the period 1860–1917, that is, from the period before the introduction of the mark to the period it was in use. After 1860 the words “mark” and “penny” are by far the most dominant terms in the topic, but for the period before 1860, the dominant terms are “price” and “thousand.” It is clear that “mark”, “penny”, “price”, and “thousand” are words that can belong to the same topic, but the heatmap representation clearly shows that the focus in the topic shifts. It is almost as if two related topics are merged as to represent one topic over the whole time period. In a situation where a historical interpretation highlights a change in past discourse, DTM produces continuity.
While there is obviously no right answer as to when one topic is stretched a bit or
when different topics are simply merged together to provide a temporally continuous
topic, it seems that DTM is especially problematic if one wants to study discourses that
emerge or disappear in the middle of a time period studied. This means that any
historical analysis using DTM requires a component of historical interpretation of not only
topic coherence, but also topic coherence over time. Here, relying on word embeddings
like in [
The speed of topic evolution can be controlled by a parameter in the DTM model.
However, the ‘ideal’ amount of stretching is difficult to assess. For analysing discourse, this
might in some cases be productive as it can point at links between nearby discourses,
but is largely problematic as it hides discontinuities in the data. It becomes even
problematic when dealing with material factors, like the introduction of the Finnish mark,
as the stretching effect is likely to produce anachronistic representations, that is,
placing something in the wrong period of time. Dealing with anachronism can perhaps be
seen as one of the cornerstones of the historian’s profession, which makes DTM as an
anachronism prone method a poor match for historical study. Avoiding anachronisms
completely is impossible, most historians would agree, but knowing when to avoid them
and how to communicate about anachronistic elements in historical interpretation is key
to history as a discipline [
In order to inspect the discourse dynamics of religious topics, we have combined several topics that related to religious themes in the LDA model, whereas in the latter, DTM model, we only chose one topic to be represented.3
To our knowledge, topic models have not been used to study discursive change
regarding secularization. However, in line with some earlier qualitative assessments [
Keeping in mind the issue of stretching with DTM, we can look into the shifting saliency of words within the topic of religious offices and notice a shifting focus over time (Fig. 3c). In the early 1900s terms relating to “holding an office” a nd n ames of particular congregations become more dominant in the topic. This, again, suggests that DTM as a method does some stretching. There is a downside and an upside to this. On the one hand, the stretching distorts the topic prominence a bit by making it look like there is more continuity than in the LDA visualization. However, this may not be that crucial as the declining trends in Fig. 3a and Fig. 3b are rather similar. On the other hand, the stretching may be good for detecting conceptual links between different groups of words. In this particular case the stronger link between religious offices and some towns like Kerava and Porvoo, is probably indicative of a move of religious discourse from an overarching question to something that is more likely dealt with in conjunction to matters at local parishes. That is, religious offices were more often than before dealt with in connection to local congregations. This is in line with our abovementioned assumption about religious discourse becoming more distinct. 5.3
While we expected religious themes to decline and become less central, we assumed
there would be some themes that partly overlap with religion, but also would show an
increasing trend. One example of this is the topic of education, which has historically
been heavily interwoven with the church, but at the same time when basic education
became available for a higher amount of people, it also became central in questioning the
role of the church and religion. Education in nineteenth-century Finland was both
central for ensuring conformity of the Lutheran faith, but paradoxically also was a vehicle
of secularization. [
As in the case of religious discourse, alignment between DTM and LDA can only be made through human interpretation. It seems, that in this case DTM captures one topic 3We also experimented with more data-driven methods to cluster topics, including for example methods based on Jensen-Shannon Divergence. They unfortunately did not lead to clusters that our domain experts would make sense of. Nonetheless, despite this, we still believe this is an interesting avenue to pursue which could help answer the common ‘number of topics’ question often brought up within the field. (a) Topics related to religion on (b) Development of religious (c) Heatmap of terms linked to decline (LDA) topic (chaplain, priest and of- office of religion topic.
fice) over time that is fairly coherent, revolves around education and schooling, and is on the rise in the research period (Fig. 4b). For LDA, this is not the case, as an PyLDAVis inspection of most salient words across all fifty topics show that words like “school” and “folk school” appear mostly in three topics of which two are in decline and one heavily on the rise (Fig. 4a).
Interestingly, LDA and DTM seem to be pointing at a similar historical development. The two declining LDA topics are based on their most salient terms and are more focused on schools as buildings and institutions as well as teaching as a profession, whereas the topic on the rise includes salient vocabulary relating to, not only schools, but also meetings, civic engagements, and decision making. The DTM topic at hand shows a similar development which can be inspected in a heatmap of most salient terms over time. The terms “school”, “child”, and “teacher” dominate early in the period. By the end of the period the topic becomes broader, and terms like “municipality” and “meeting” have become more salient than the vocabulary relating to schools. Here the stretching of DTM creates the links that are also visible in the three LDA topics, and it shows a transformation in which educational issues are present in the whole topic, but focus shifts from concrete schools to civic engagement. 6
Our focus in this text has been on discourses that cannot be reduced to mere words, isolated events or particular people, but concern broader societal topics that either declined or gained in prominence. The interpretation of these topics and their contextualisation to nineteenth-century Finnish newspapers revealed clear topical cores that can be interpreted as an encouraging point of departure for further explorations based on topic models when aiming to understand Finnish public discourse through historical newspapers. (a) Development of education topic over (b) Development of education topic over time (LDA) time (DTM) In this paper, we have learned that although it is difficult to pinpoint exactly where a discourse or topic ends, LDA and DTM can fairly reliably grasp many semi-coherent themes in past discourse and help us study the dynamics of discourses. However, our comparison of LDA and DTM as methods for getting at past discourse also shows that both methods require a very strong interpretative element in analysing historical discourses. DTM is much more prone to stretch or even merge topics, which requires an interpretative assessment of whether the stretching highlights interesting historical continuities or if it hides historical discontinuities that would require attention. We found that producing heatmaps of term saliency over time for each topic is a very useful way of doing this type of assessment. For LDA, stretching is not so much a problem, but often it seems interpretation is needed in seeing which topics logically relate to one another. While historical discourse analysis is traditionally tied strongly to a tradition of hermeneutic interpretation, the use of topic models to grasp discourse dynamics does not remove that need even if they allow for a quantification of discourse dynamics over time.
While we regard stretching in DTM as a predominantly negative feature, in some
cases it can be useful. In the topics relating to education discussed above, the stretching
in DTM actually points out links in discourses and is quite productive for the
interpretative process of trying to figure out discourse dynamics. However, also in this case,
the relevance of historical interpretation should be highlighted because it is very hard
to tell whether the stretching of topics is an accurate reflection of the data or a
shortcoming of the model. This can be addressed only by relating visualisations of topics to
existing historical research and reading source texts. Humanities scholars are in general
very good at making such interpretations, but it also needs to be noted that when we
move further into the domain interpretative scholarship, we also lose some of the
benefits of working with quantifying models. While it would be foolish to claim that a topic
model represents data in a way that it provides simple facts about historical
development, our use cases show that if we seek to find more reliable quantification LDA may
provide better results than DTM. Further, using LDA moves the interpretative stage
further down in the research process, as it is likely to be about evaluating the connections
between different topics over time. In DTM, the interpretation is likely moved forward
to an evaluation of how well the algorithm did this merging topics. On this sense, our
take on topic models harmonises with [
This work has been supported by the European Union Horizon 2020 research and innovation programme under grants 770299 (NewsEye) and 825153 (EMBEDDIA). SH is funded by the project Towards Computational Lexical Semantic Change Detection supported by the Swedish Research Council (2019–2022; dnr 2018-01184).