The enhanced access to ever-expanding digital data collections and open computational methods have led to the emergence of new research lines within the humanities and social sciences, bringing in new quantitative evidence and insights. Any data interpretation depends critically on understanding of the scope and limitations in data collection, as well as on reliable downstream analysis. Quantitative analysis can complement qualitative research by providing access to overlooked information that is accessible only through systematic discovery and analysis of latent patterns underlying the available data collections. Probabilistic programming is an expanding paradigm in machine learning that provides new statistical tools for intuitive interpretation of complex data sets. This new paradigm stems from Bayesian analysis and emphasizes explicit modeling of the data generating processes and associated uncertainties. Despite its remarkable application potential, probabilistic programming has so far received little attention in computational humanities. We use a brief case study in computational history to demonstrate how probabilistic programming can be incorporated in reproducible data science workflows in order to detect and quantify bias in a widely studied historical text collection, the Eighteenth Century Collections Online.
Research questions in computational humanities often deal with very similar quantitative
challenges than the natural or social sciences, which have a long tradition in data-driven research.
Techniques from other fields can be often readily borrowed in new application fields with small
adaptations. This is enabling the translation of well-established methodological paradigms
from other disciplines, such as ecology, econometrics, or physics into the emerging field of
computational humanities. Research in computational humanities benefits from a rich
mixture of data science techniques that range from database management to data harmonization,
computational modeling and visualization. Reproducible data science workflows that unify the
complementary steps of data analysis have become a standard tool to facilitate collaborative
research [
Understanding biases and uncertainty is fundamental to research. Even the most perfectly
harmonized and clean research data sets contain subtle selection and other biases. Such
biases can, however, be potentially detected and treated through explicit formal analysis. In
addition to questions of raw data quality, one needs to critically assess and understand data
representativeness and consider the limitations of the chosen methodology. Whereas much of
the recent technical eforts that have been aimed at ensuring data quality have focused on the
development of automated procedures for data processing and harmonization, data curation is
only the stepping stone towards critical statistical analysis and modeling [
Here, we consider probabilistic programming as a flexible tool that can be used to detect
and characterize bias and uncertainties in research data collections that are relevant to
computational humanities. Probabilistic programming is a statistical paradigm that has increased
in popularity in various fields of applied computational science, leading to the emergence of
a variety of alternative probabilistic programming languages such as Stan [
We provide a brief overview of probabilistic analysis and demonstrate its application on
detecting specific biases in the composition of Eighteenth Century Collections Online (ECCO).
This full text collection of British literature has become one of the most central digital resources
for studying printed eighteenth-century texts in English language. However, the contents of
ECCO are fully derived from the information contained in the English Short Title Catalogue
(ESTC) [
2.1. Data
Our case study is based on the systematic and ongoing data collection and harmonization
that has already been carried out and reported previously [
The Eighteenth Century Collections Online (ECCO) is a digitized full text collection of printed eighteenth-century texts in English language1. The ECCO collection of full texts has been 1https://www.gale.com/primary-sources/eighteenth-century-collections-online constructed based on the ESTC library catalogue (see below) and, subsequently, analysis of ECCO is often based on the implicit assumption that it has a comprehensive and unbiased coverage of the English titles catalogued in the ESTC. In this report, we assess the validity of this assumption by comparing the coverage of the ESTC titles in ECCO and formally quantifying specific key factors that potentially influence the coverage. We have obtained access to ECCO confidentially for research purposes upon a separate agreement. We do not analyze the full text contents of the ECCO, and hence the identification of the matching records between the two collections is sufficient for our current research purpose and could be readily done based on shared document identifiers. By design, all titles in ECCO are being covered in ESTC, whereas some titles catalogued in ESTC remain missing from ECCO.
The English Short-Title Catalogue (ESTC) focuses on British early modern publishing and
includes bibliographic information on mainly English documents from the eighteenth century.
The cataloguing work has been coordinated by the British Library [
We have previously implemented the background data harmonization as a collection of
reproducible bibliographic data science workflows [
Whereas the ECCO and ESTC data collections have been obtained from external memory
organizations based on a separate agreement and they are thus not openly available, our own
data harmonization and analysis workflows are open source and available online 2. The overall
data processing workflow consists of a collection of custom Python scripts, R packages, and
other components [
Our task is to analyse how often titles listed in ESTC library catalogue are covered by the collection of full texts included in ECCO. This well-defined task for formal analysis is motivated by the need to understand the correspondence between these two sources. In particular, we analyse how specific background factors influence ESTC coverage in ECCO and contribute potential bias to the subsequent interpretation of this full text collection.
The coverage of the ESTC titles in ECCO can be formally modeled as a repeated cointossing experiment, where each title in the ESTC have a certain probability of being covered in ECCO. This probability may be then influenced by a number of factors, such as author gender or publication decade. Joint analysis of a large number of documents can reveal systematic diferences in coverage associated with such factors. A common approach to modeling such probabilities is based on logistic regression coefficients in a standard binomial model: k ∼ Bin(n, p) p ∼ logit−1(β0 + ΣiJ=1βixi), (1)
Here, n and k are the numbers of titles included in the ESTC and ECCO, respectively, and p is the probability that an ESTC title with the given properties will be included in ECCO. Here, we use J = 3 covariates that include author gender (x1), publication type (x2), and publishing time (x3). These covariates are encoded as binary indicator variables that indicate female authors (x1 = 1), pamphlets (x2 = 1), and publication time (x3 = 1 for publications after the year 1750). We have excluded authors with an unknown gender (5.8% of unique authors in our harmonized version of ESTC). For publication time, we use a binary encoding into early and late century (i.e. later than 1750) since our preliminary experiments indicated that the main temporal variation is associated with this split, rather than publication year or decade. The covariates influence the log-odds of the binomial probability through the regression coefficients β; where β0 represents the standard intercept term. This will allow us to assess the efect of each covariate on the estimated probability p through the logit link. Here, the k, n and the covariate information encoded in x are observed variables, whereas the regression parameters β are to be inferred from the data and can be used to retrieve posterior estimates of the latent binomial probability p. In the probabilistic formulation, we can additionally set priors for the model parameters. We have used a Gaussian prior β0 ∼ N (0, 5) for the intercept term and βi ∼ N (0, 2.5) for the regression terms i ∈ {1, 2}. The prior distributions that we have chosen to use here are relatively uninformative; they cover the expected range of plausible parameter values, with a negligible efect on the posterior distributions that are essentially determined by the data in our experiments.
The standard binomial regression model in Equation 1 could be fitted based on classical statistical techniques as well as through probabilistic programming. We have chosen this relatively standard model for our case study in order to draw analogies and facilitate further comparison between the two approaches. In standard statistical analysis, one would find the maximumlikelihood solution for the equation 1 by identifying a single optimal point estimate for β. Probabilistic programming allows us to not only implement this standard model but additionally allows us to incorporate prior information and quantify uncertainty in the inferred parameters in the form of a posterior distribution, and compare the evidence that the data is providing between alternative models.
Probabilistic programming is an emerging paradigm that bridges the gap between Bayesian
probabilistic analysis and its practical application. Our case study has been implemented in
Stan, which is one of the several available options and has an active user community and a
dedicated collection of tutorials [
Probabilistic analysis difers from classical statistical analysis in certain important ways.
First, it allows a flexible construction of complex models that do not need to be analytically
tractable, in contrast to common classical alternatives. Second, probabilistic models emphasize
the need to assess the uncertainty in the inferred model and model parameters. Third, the
framework allows the incorporation of prior information whenever this is available to support
the modeling. Whereas probabilistic programs provide access to an increasingly large family of
statistical models, a reliable inference is often slower compared to classical statistical analysis.
For more details, we refer to the original publications on Stan [
We implemented the probabilistic version of the binomial regression model of Equation 1
with the rstanarm R package [
The first step in understanding potential biases in the ECCO full text collection with respect to the ESTC library catalogue is to quantify and compare the coverage frequencies between diferent groups of published titles.
Altogether, 62.6% of the unique titles catalogued in our harmonized version of ESTC are covered in ECCO. We excluded titles where author gender is unknown (5.8% of all harmonized ESTC titles). Of the remaining ESTC titles 4.8% have been authored by a woman, and we also see that the ECCO coverage is in general somewhat lower for women (Table 1). However, the lower coverage of female authors might be partially explained by other, confounding diferences. For instance, a closer view reveals that women also author pamphlets considerably more often than men do (54.2% versus 43.6% of all titles for women and men, respectively). Since pamphlets have a lower coverage in general (Table 2), the lower coverage of female authors in ECCO might be readily explained by their diferent publishing patterns, rather than gender. Also other factors, such as the publishing time, may play a role as we demonstrate in the next section. Such general comparisons of publication frequencies indicate that certain groups of publications are over- or underrepresented in ECCO with respect to ESTC.
Such preliminary observations of publication frequencies or other broad patterns have obvious limitations. Mere quantification of the coverage frequencies can tell us little about the uncertainties about these observations, and distinguishing between the overlapping efects such as author gender, genre, and time is not always straightforward. Whereas the overall frequencies shown in Figure 1 can be readily estimated from data, a a full statistical analysis is necessary when we are willing to draw conclusions on the overall influence of gender or other factors on ECCO coverage and quantify the uncertainty associated with such factors.
The log-odds for the model parameters - the intercept and the regression coefficients associated with genre (pamphlet), gender (female), and time period (latter half of the eighteenth century) were estimated by probabilistic programming (Figure 2).
Negative log-odds indicate a negative impact on ECCO coverage. For instance, according to
our results a female author will reduce the probability that a title is covered in ECCO. When
we include the gender efect as the only explanatory factor in the binomial model, female
gender is associated with the log-odds of -0.16. This corresponds to the diference in the
ECCO coverage probabilities between female and male authors that we can also see in Table 1
(58.9% vs. 62.8%, respectively). However, when we take genre (books versus pamphlets) into
account in the model, being a female author reduces the log-odds of ECCO coverage by only
-0.08. Thus, the gender efect is less striking when we also model diferences in the typical
publishing patterns between male and female authors. Moreover, the publishing time period
shows additional major influence; in general, the ECCO drops over time from 67% in 1700’s to
58% in 1790’s. This drop is observed specifically at the latter half of the century. According
to our binomial model, a publication date after 1750 has a larger negative impact on ECCO
coverage than author gender (Fig. 2). The evidence for a negative gender efect remains also
after controlling this time factor, however (β1 = −0.06) with a high certainty indicated by
the 99% posterior intervals being below zero. Thus, our quantitative analysis reveals
genderspecific diferences and supports the earlier arguments that the publication activity of female
authors may have been underestimated due to their biased coverage in common research data
collections [
The analysis is also allowing us to compare the relative efects of the diferent document and author properties on ECCO coverage. The efect of genre is notably larger than the efect of gender. Pamphlets have the estimated log-odds of -0.80, which roughly translates into the 45% odds of ECCO coverage for pamphlets as compared to books (Figure 2). The odds of ECCO coverage for titles published during the latter half of the century is 73% compared to the first half of the century. The odds for female authors in the full model including both genre and publication time is 94%.
Female author Published after 1750
Pamphlet Intercept
Gender
Female
Male Books
Pamphlets
The improved availability of digital resources has led to the emergence of new research lines
that complement classical humanities and social sciences. Quantitative analysis can provide
access to insights that are only obtained through systematic large-scale analysis. When applied
thoughtfully and critically, this has the potential to enrich and complement qualitative research.
Interpreting the available data depends critically on understanding of the inherent limitations
in data collection and analysis, as well as on the availability of reliable quantitative methods.
Such considerations are at the core of statistical practice [
We have previously demonstrated the benefits of large-scale data harmonization in
computational history, and in particular shown how data curation can be scaled up by semi-automated
means in order to analyse metadata collections covering millions of titles [
We have now demonstrated how probabilistic programming can be incorporated into reproducible data science workflows in computational humanities. Whereas similar results could be obtained with classical statistical techniques for our specific case study, the probabilistic framework can open up access into a wider set of extensions and the use of prior information. Limited data availability leads to uncertainty in the parameter estimates, which are naturally captured by the posterior distributions of the probabilistic models. Also our case study highlighted some remaining uncertainty in the estimated gender efect. As such, probabilistic analysis can facilitate the detection of biases and uncertainty that influence data interpretation.
We provided supporting evidence for the hypothesis that the Eighteenth Century Collections Online (ECCO) may contain previously unreported biases, and that specific aspects of literature are being underrepresented in the data collection. In particular, female authors, pamphlets, and later publication dates led to reduced odds of ECCO coverage. The negative impact of later publishing might be partially explained by diferences in the frequencies in first editions and reprints but this would warrant a more careful analysis. Such biases have obvious implications for any conclusions that would be potentially drawn from the data. Whereas in this technically oriented work we have highlighted probabilistic programming as a new method in computational humanities, we are now carrying out complementary work that will yield a more comprehensive critical historical analysis and interpretation. A more thorough benchmarking of the alternative bias detection methods and a wider set of case studies could further help to understand the overall range of applications and pragmatic limitations in such modeling tasks.
Bayesian analysis and probabilistic programming have obvious limitations that are
slowing down their wider adoption. In addition to slower model inference compared to classical
alternatives, efective construction, use and interpretation of probabilistic models requires a
robust understanding of modern statistics as well as statistical programming. The emergence
of probabilistic programming languages and standardization of common modeling tasks is now
lowering these barriers, however. A wider adoption and open sharing of these workflows can
help to standardize the analysis of biases as well as other specific approaches that are common
in computational humanities [
Limitations on data availability are common in the humanities. Our case study is based on a joint analysis of the ECCO and ESTC data collections that we have obtained for research purposes. We hope that this can inspire related research on such data collections, and help to demonstrate the benefits for wider research collaborations that could be obtained if the open data science workflows could be combined with fully open historical data resources.
The currently emerging ecosystems of computational programming languages continues to facilitate the translation of methodology from other data-intensive fields of research such as ecology, bioinformatics, or econometrics. Probabilistic programming places emphasis on formal analysis of uncertainty in inference and models, and on continuous model criticism. As such, it can be extended far beyond our brief case study in order to construct more comprehensive hierarchical models of the data-generating processes that characterize various aspects of spatial, temporal, seasonal, and other variation in the quantitative aspects of social sciences and humanities.
The work has been supported by Academy of Finland (grants 293316, 295741, 333716).
M. Tolonen et al. “Data Visualization in Enlightenment Literature and Culture”. In: Palgrave Macmillan, 2020. Chap. Examining the Early Modern Canon: The English Short Title Catalogue and Large-Scale Patterns of Cultural Production, In press.