Bibliographic data science is an emerging research paradigm in digital humanities. It aims at systematic quanti cation of the trends in knowledge production based on large-scale analysis of bibliographic metadata collections and the methods of modern data science. Compared to the earlier related attempts in book history and sociology of literature, advances in data processing and quality control are now making it possible for the rst time to scale up the analysis to millions of print products while at the same time paying attention to data quality, representativity and completeness. This provides a new quantitative method that can support the analysis of classical research questions in intellectual history. Here, we discuss the methodological challenges that we have encountered in such studies and how to scale up the solutions based on collaborative research e orts.
Bibliographic data collections contain a vast array of data on publishing
activities, and they have been traditionally used as a tool for information retrieval.
A systematic analysis of bibliographic catalogues can generate rich information
on historical patterns in knowledge production, and their research potential has
been debated for at least half centuries now [
Bibliographic data science (BDS) [
Harmonization of the original records is the rst step in facilitating reliable research use. Bibliographic metadata is often manually entered in the databases, and seldom su ciently standardized and readily amenable to quantitative analysis. Biases, inaccuracies, gaps, and varying standards and languages pose challenges for data integration both within and across catalogues. The heterogeneity of the data elds, spanning from time intervals to persons, physical dimensions, or geographical locations is also a notable challenge. The scale of these issues greatly exceeds our ability to manually verify and correct the entries. Essential for this undertaking is to move towards automated analysis work ows that enable standardized treatment of the data and e cient implementation of similar work ows across multiple catalogues and iterative data corrections.
The integration of data across catalogues enables the analysis of publishing
activity beyond what is accessible by the use of individual national bibliographies
alone, as we have recently suggested in [
Standardization and reproducibility can bene t from the design of dedicated software packages and unit tests. Hence, we have implemented systematic algorithmic approaches that help to assess and improve data reliability in a semi-automated fashion. These work ows combine various procedures to remove spelling errors, disambiguate and standardize terms, augment missing values, and incorporate manually curated information such as known pseudonymes or synonymous entries. Some steps are straightforward, including removal of extra periods or spaces; in other cases, custom algorithms have to be developed. One example is the page count information, which follows a speci c MARC notation4 that separates paginations in di erent parts of the document, such as preface, contents, gures and tables, and in separate volumes. Speci cally designed algorithms are necessary for scalable interpretation of such information.
Remarkable portions of information is typically missing in library catalogues but can be readily augmented based on information that is already contained elsewhere in the catalogue, or based on external information sources. For instance, missing country information can be lled in when the publication place is available and uniquely mappable to a country; and exact physical dimensions of a document can be often inferred with a good accuracy based on the gatherings information that is more frequently available. In such cases, information is however retained whether the value is based on the original entries, or later estimated based on other information. We have also complemented the original records with entirely new derived elds, such as print area, which quanti es the paper consumption per unique physical copy of a document, or title. The combined print area across all unique titles then re ects the overall breadth of printing activities per time period, genre, or geographic location, rather than the mere volume of print products. Hence, it complements total paper consumption which quanti es the overall volume of print products but does not consider the uniqueness or diversity across titles. Estimating the overall paper consumption is more di cult as it requires information on print run sizes and this is often not readily available. The estimated print run size of one thousand copies is often used for early-modern books but notable variations over time and geography have been reported. The print area and paper consumption can hence provide complementary ways to assess trends in knowledge production.
Monitoring the quality of the harmonization process is an important part of such e orts. We are routinely generating conversion tables that show how the original raw entries have been converted into the nal harmonized versions, list of common entries in each eld (e.g. authors, publishers, languages), and statistical summaries such as average document dimensions by format, histograms of publication years and author life spans, or changes in gender distribution
for the full description of the MARC21 format. over time. Such overviews have been invaluable in spotting unexpected events that may in turn facilitate the detection of remaining inaccuracies or biases in the harmonized data sets. The quality monitoring has been greatly facilitated by such reproducible overviews, and the most up-to-date summaries for each bibliographic catalogue that we are working on can be accessed via Helsinki Computational History Group website5. By openly sharing the summaries and algorithms, we are aiming to improve the transparency, reliability, and overall quality of our work by providing the broader audience with the means to detect and report potential inaccuracies.
Advances in machine learning and arti cial intelligence are providing further means to scale up the analysis. We are routinely utilizing methods from natural language processing, feature selection, clustering, and classi cation to facilitate duplicate identi cation and quality monitoring. Information in the publisher eld, for instance, often contains lengthy verbal explanations that have to be subjected to named entity recognition. Moreover, detecting alternative spellings for author or publisher names can be greatly accelerated by string distance matching algorithms that can rank potential duplicates and facilitate semi-automated data curation. Page number estimation is a prominent example of the potential of this approach, as the original text entries can be converted into a single wellde ned number, which is the nal page count estimate. The overall accuracy of such fully automated estimates can be quanti ed to a great extent by examining the most common conversions and representative sets of random examples. Future developments could take increasing advantage of such statistical techniques in order to reduce the need for human input, thus improving the overall scalability of data harmonization. The already curated data sets provide ample training material for supervised machine learning techniques.
Finally, the research community can bene t from open sharing of the data and algorithms. The lack of open data availability is forming a major bottleneck for collaborative development of bibliographic data science but this might be gradually changing. The National Library of Finland, for instance, recently released the complete MARC entries of the FNB6 under an open data license that allows the modi cation, reuse, and sharing of derivative versions. We have made the key algorithms for harmonization and analysis openly available in the bibliographica R package7, and our harmonized versions of the FNB data set can be accessed via Helsinki Computational History Group website. The harmonized data sets can be further veri ed, investigated, and enriched by others, and they could be integrated into Linked Open Data and other popular formats in order to utilize the vast pool of existing software tools. Combining such largescale harmonization with existing data management infrastructures could open up new doors for research on national bibliographies.
Our recent analyses based on four large bibliographies provides examples of the
research potential of this approach [
We have conceptualized a new approach, bibliographic data science, to expand the research potential of bibliographic records. This derives from the already established eld of data science and associates this general paradigm speci cally with quantitative analysis of bibliographic metadata and related information sources. While having a speci c scope, BDS is opening up pragmatically oriented and substantial new research opportunities in the digital humanities.
Drawing valid conclusions critically depends on data quality, representativity and completeness. Automation and quality control are essential when the data collections may contain information on millions of documents, and the overall data science ecosystem integrates a number of distinct work ows that are dedicated to harmonizing speci c subsets of the data. We have indicated how speci cally tailored open data analytical ecosystems can help to address this challenge. Our approach has potential for wider implementation in related studies, and provides guidelines for more extensive integration of national collections.
Our future work envisions continued harmonization and data integration for
the HPBD as well as further, related data resources such as the Universal Short
Title Catalogue.8, in order to expand the study to cover public communication
more broadly. We have incorporated best practices and tools from data science,
such as unit tests, tidy data [
potential for wider implementation in related studies in the digital humanities.
As we have extracted and harmonized publisher information from imprints from
ESTC and FNB [