Digital humanities (DH) scholars have been increasingly interested in using BERT for document representation in computational text analysis. However, most word embeddings, including BERT embeddings, have been developed using “clean” corpora, while DH research is usually based on digitized texts with optical character recognition (OCR) errors. Will these errors introduced by the digitization process reduce BERT's performance and distort the research findings? To shed light on the impact of OCR quality on BERT models, we conducted an empirical study on the resilience of BERT embeddings (pre-trained and fine-tuned) to OCR errors by measuring BERT's ability to enable classification of book excerpts by subject domain. We developed specialized parallel corpora for this task consisting of matching pairs of OCR'd text (19,049 volumes) and “clean” re-keyed text (4,660 volumes) from English-language books in six domains published from 1780 to 1993. This study is the first to systematically quantify OCR impact on contextualized word embedding techniques with a use case of OCR'd book datasets curated by digital libraries (DL). Experimental results show that pre-trained BERT is less robust when used on OCR'd texts; however, fine-tuning pre-trained BERT on OCR'd texts significantly improves its resilience to OCR noise in classification tasks according to the changes of classifier performance. These findings should assist DH scholars who are interested in using BERT for scholarly purposes.
The accessibility of ever-growing digitized textual curations in digital libraries (DL) and the
rapid development of natural language processing (NLP) techniques have opened up a variety
of new research opportunities to humanities scholars for computational text analysis [
Digital humanities (DH) scholars working with computational analysis have been
increasingly interested in using this technique for their research on digitized texts. However, a majority
of large DL text curations and other historical text collections are machine-transcribed and
include varying degrees of optical character recognition (OCR) noise. Such noise might
decrease the generally impressive performance of BERT because it was originally developed on
born-digital texts without OCR errors [
Therefore, we believe BERT’s performance on OCR’d texts is an important problem to look
into. This study aims to empirically investigate this problem with three research questions:
(1) Would the original BERT model [
To shed light on the interaction between OCR’d texts and BERT, we focused on measuring
the ability of BERT to encode digitized texts’ semantics and comparing the performance of
BERT encoding on clean (i.e., re-keyed) versus OCR’d texts. The texts we used were book
excerpts generated from ∼4,000 pairs of book volumes selected from a parallel corpus of digital
English-language books, with 4,660 human-proofread “lean” volumes from Project Gutenberg
(Gutenberg) and their matching pairs of 19,049 OCR’d volumes from HathiTrust Digital
Library (HathiTrust) [12]. Books in this corpus cover six subject domains published from 1780
to 1993. We chose subject domain classification as the application downstream from BERT
in order to quantify its encoding performance, because document classification in general is a
popular application for digital humanists studying subject, genre, authorship, and many other
features of their texts. [
The remainder of this paper is organized as follows. In section 2, we review related work on BERT and OCR’d texts. In section 3, we provide detailed information about the parallel book dataset that we created and leveraged, and how we built the book excerpt corpora needed for our experiments. In section 4, we describe our research design and workflow. We also give explanations for the specific decisions made and methods adopted. In section 5, we present our experimental results and findings. Finally in section 6, we discuss our conclusions and future
Fiction Social_Science Agriculture World_War_History Medicine Business Total
BERT used in existing work for digital history and literary studies generally plays a text
preprocessing role by encoding text information into vectors for further computation. Popular
research topics in this field mainly focus on the diachronic analysis of literary texts [
Based on a literature review on OCR noise analysis, common error types include character
misidentification (e.g., “inse rted”→“insorted”), broken words (e.g., un-rejoined hyphenated
words “talking”→“talk- ing”), incorrectly joined words (e.g., “the
belief”→“thebelief”), and meaningless symbols (e.g., OCR attempts to recognize hand-written marginalia)
[
Prior work on the impact of uncorrected OCR’d texts on other NLP tasks can be divided
into two groups: (1) those quantifying impact by measuring the performance diferences of
a set of popular NLP techniques applied on a parallel corpus consisting of OCR’d and clean
texts [
The source data for this study is a parallel corpus of English monographs [12] collected from two real-world digital libraries: (1) Gutenberg for a human-proofread “clean” corpus; and, (2) HathiTrust for an OCR’d “noisy” corpus. This corpus has a total of 4,660 Gutenberg volumes in 6 domains (i.e., fiction, social science, agriculture, medicine, business, world war history), each of which is matched with several diferent copies (4 on average) of the same work held in HathiTrust.
Since classification is a supervised learning task, we started by preparing three parallel data splits from the raw corpus for training, validation, and testing, respectively. Considering the many-to-one matching relationship between Hathitrust and Gutenberg volumes, in order to make the clean and OCR’d version of each data split, aligned by volume, and to avoid volume duplication in splits with clean data, we first split Gutenberg data by randomly selecting 10% of 4,660 Gutenberg volumes for validation (465 volumes), 10% for testing (467 volumes), and the rest for training. Then we randomly picked one paired HathiTrust copy of each Gutenberg volume to build corresponding training, validation and testing splits of OCR’d texts.
Following [
There are two main challenges in the encoding of book content by BERT. First,
booklength texts and the computational cost of BERT models make it expensive to encode each
volume’s full text. Moreover, BERT models are restricted to processing at most 512 tokens
at a time, which limits their encoding abilities on long sentences. To address these issues,
we followed prior work [
To make each classifier’s predictions on clean versus OCR’d test set comparable, the sampled
text chunks from each pair of test volumes were aligned by an existing text alignment algorithm
[
Book Excerpt Domain Classification
Text Encoding
Classifier
Construction token vectors t1 t2 … tn BERT
Mean Pooling d1
Model-based Measurement ➔ BERT embedding types ➔ Sampling strategy of training corpora ➔ Source of training / testing data Content-based Measurement ➔ Book characteristics (e.g., genre, topics)
Outcomes The primary goal of this study is to analyze the performance of BERT embeddings in encoding book excerpts into n D-dimensional (D=768) token vectors for book domain classification based on the parallel clean and OCR’d texts. We measured and compared BERT embeddings’ encoding ability in diferent classifiers using macro-averaged precision (P), recall (R), and F1 score (F1). Considering the potential influence of experimental settings on BERT embeddings’ performance, we analyzed the classification outcomes based on the model settings and data characteristics respectively. Figure 1 visualizes the overall workflow of this study, which includes two stages: (1) building classifiers based on text representations ofered by BERT embeddings on book excerpts; and (2) quantifying BERT embeddings’ performance in diferent classification settings to analyze BERT embeddings’ resilience to OCR noise.
With the encoded BERT token representations per excerpt, we first generate a single
chunklevel feature vector by averaging token vectors, one of standard practices popularly used in
prior work [
The detailed implementation of model training is as follows. We used the Adam optimizer
[
Based on the classification results of 12 generated classifiers on our parallel testing corpus, we analyzed the relations among BERT embedding types (i.e., pre-trained or fine-tuned BERT), the source of training and testing data, and the sampling strategy of training corpora by pairwise comparison of any two of three variables. Our goals were: (1) finding the optimal BERT embedding with the highest resilience against OCR errors; and (2) identifying the optimal sampling strategy for building the training corpus that most significantly improves the BERT embedding performance.
Given that the above analysis primarily focused on the comparison of BERT-based classifiers’ overall performance, we further proposed a fine-grained investigation of BERT embeddings’ resilience to OCR errors regarding the amount of noise. To conduct this investigation, we first prepared three subsets of OCR’d testing data containing diferent amounts of OCR errors. The level of OCR noise was measured by the character-level error rate (CER) based on the comparison of each OCR’d book excerpt with its paired clean text. After sorting the OCR’d excerpts by their CER in an ascending order, from this ranked excerpt list, we separately sampled 1500 excerpts at the top, middle, and the bottom position as the low-, medium, and high-noisy testing subsets. Figure2 displays the distribution of CER in each testing subset, where the average CER per subset is around 0.40, 0.54, and 0.65, respectively. We then evaluated each classifier’s predictability on each subset. Note that, in this analysis, we only considered those classifiers trained on the corpus with the identified optimal sampling strategy. To further look into the resilience of BERT embeddings with respect to the change of the downstream classification’s training corpus source, rather than exploring each individual classifier’s results, we measured the divergence of classification results between the classifier trained on the clean versus the OCR’d texts for each type of BERT embedding.
Although each book in the raw parallel corpus was assigned to a single subject domain tag, given the diversity of content-based characteristics (e.g., topics, genres, narrative styles) inherent in a book-sized text and its randomly sampled excerpts, it is possible that the input 1The number of epochs was optimized empirically by trying a set of values (i.e., 15, 20, 30, 50).
Low
Medium
High data itself might bring challenges for a BERT-based classifier to identify its annotated domain tag. Moreover, whether and how such challenges occur with OCR’d texts vary from those occurring with clean texts is uncertain. For instance, if all BERT-based classifiers fail to classify either clean or OCR’d excerpts of the same book correctly, one potential reason for this result could be that the original book includes more than one subject. In contrast, if all classification models work well on the clean texts only, it is likely that OCR noise is resulting in diferent predictions. To address these concerns, we started by exploring semantic associations among misclassified domains by visualizing the confusion matrix of each classiifer. To further capture book excerpts’ individual features for understanding their influence on classification, we then grouped the predictions made per classifier on individual excerpts by book, to measure the consistency of classifiers’ prediction accuracy at the book level. This measurement is based on calculating the number of testing excerpts of the same book that were assigned to the same correct domain across diferent classifiers on average. Given the quantitative outcomes, we sampled some cases with poor prediction accuracy, and explored potential reasons for misclassification by close reading of the book content.
Table 2 provides an overview of the classification results grouped by (1) source of training and testing data (Gutenberg or HathiTrust); (2) sampling strategy of parallel training corpus (small-balanced, small-unbalanced and large-unbalanced); and (3) type of BERT embedding (pre-trained or fine-tuned). Overall, we observe that classifiers built with fine-tuned BERT outperformed those built with pre-trained BERT by 20% (F1 score) based on the balanced training corpora and 10% (F1 score) based on the unbalanced training corpora. This result indicates that the fine-tuning process, intended to adapt the generic pre-trained BERT embedding space to fit into a specific text corpus (either clean or OCR’d), will substantially improve the encoding ability of BERT for digitized literary texts even with the distortion of OCR noise.
Regarding the influence of training sampling strategies to BERT encoding, in general, unbalanced corpora were more helpful in training classifiers than balanced corpora, which suggests that excessive artifact intervention of training data distribution indeed could hurt BERT’s encoding ability. Table 3 further shows the paired t-test scores of the statistical diference of performance between any two comparable classifiers that difer only in either size or data distribution of training corpus. It is to be noted that diferences between any two compared classifiers’ performances over 20 testing folds follow an approximately normal distribution based on the Shapiro-Wilks Test. According to the results, pre-trained BERT-based classifiers are all sensitive to both size and data distribution in the training corpus (p-value < 0.05 at least). However, the increase in size of the OCR’d training corpus has no significant impact on fine-tuned BERT embedding. This observation may be understood as a positive signal to humanities scholars that a small training corpus is enough to achieve optimal performance of ifne-tuned BERT when working with OCR’d texts. Comparatively, training corpus size (t-test score from -0.71 to 3.32 where p-value < 0.01 at most) is less influential on BERT embeddings’ performance than is training data distribution (t-test score from 2.05 to 15.54 where the majority of p-values < 0.001).
Similar to the analysis of training sampling strategies, we compared classifiers’ performance with respect to the source of training data. Table 4 shows the paired t-test results. Pretrained BERT-based classifiers were significantly more sensitive to their training data source when these classifiers were built on unbalanced training corpora (p-value tends to be < 0.001). In particular, the growth of training corpus size increased such sensitivity (t-test score increased from 4.09*** to 5.85 when testing on the clean corpus, and from 3.49** to 4.31*** when testing on the OCR’d corpus). Meanwhile, for fine-tuned BERT, classifiers only showed their sensitivity to the source of training data in small unbalanced training corpora (t-test score was -2.86** when testing on the clean corpus, and -2.10* when testing on the OCR’d corpus). According to the F1 score of these classifiers’ prediction results shown in Table 2, we found that, compared with fine-tuning on clean texts, fine-tuning on OCR’d texts improved BERT-based classifiers’ performance by ∼2%, which suggests that potential OCR noise in the small-unbalanced corpus for BERT fine-tuning can boost the resulting embedding’s encoding performance.
Given three testing sample sets with diferent levels of OCR noise (see details of data preparation in section 4.2.1), Table 5 shows the divergence of F1 score between classifiers built with either pre-trained or fine-tuned BERT embeddings on each sample set. This divergence was calculated by the subtraction of classification results using OCR’d texts for training from the one using clean texts for training.
Overall, we found that classifiers obtained greater benefit from clean training data compared with OCR’d data except in the case of fine-tuned BERT-based classifiers making predictions on the low-noise testing data. Regarding the classification divergence across the three testing sample sets, we observed a gradual decrease in diference on testing samples with low (4.88%), medium (3.96%), and high (0.70%) level of OCR noise when classifiers employed pre-trained BERT for text encoding, while the pattern was the opposite in classifiers built with fine-tuned BERT (i.e., -1.96% for low noisy group, 1.43% for medium noisy group, and 3.79% for high noisy group). We further compared the absolute diferences of classification results between two classifiers per embedding type, and found that testing samples with lower-level OCR noise were more sensitive to the training data source than those with higher-level noise in pre-trained BERT-based classifiers. On the contrary, for the classifiers built with the fine-tuned BERT, the largest performance diference was found in the testing set with a high amount of dirty OCR size.
Here are three major conclusions. First, the consistency of text quality in an embedding’s pre-training corpus, downstream training, and downstream testing corpus is helpful in improving pre-trained BERT’s applicability for literary text classification. Second, the heterogeneous nature of OCR noise can improve the generalization ability of fine-tuned embeddings to process texts with comparatively low levels of OCR noise. Finally, fine-tuned BERT-based classifiers are more stable with regard to changes in the source of training corpus than pre-trained BERTbased classifiers, which further confirms that fine-tuned BERT outperforms pre-trained BERT in its resilience to OCR errors.
represent the ratio of correct predictions, while the other values indicate the ratio of misclassifications (actual VS predicted). The higher the value is, the darker its corresponding cell color. For example, in the first matrix (fine-tuned, G →G), the value “0.45%” in the cell at the upper left corner indicates that 0.45% of ”world war history” excerpts were misclassified as ”agriculture” by the fine-tuned BERT-based classifier, which was trained and tested on Gutenberg texts. For both pre-trained and fine-tuned BERT-based classifications, we found that book excerpts in the business domain were more likely to be misclassified as fiction ( 25.4% on average) and social science (19.8% on average), while book excerpts in the medicine domain were more likely to be mistakenly classified as social science, especially with fine-tuned BERT-based classifiers trained on the OCR’d texts (32.86% misclassifications in H →G classification and 27.86% misclassifications in H →H). By looking more closely at social-science instances, we observed that the pattern of misclassifications was diferent in the classifier built with pre-trained BERT compared with that built with fine-tuned BERT. Specifically, in the classifications using pre-trained BERT for text encoding, prediction errors mainly concentrated in the domains of business (10% on average), medicine (8.5% on average), and fiction ( 7.5% on average). Meanwhile, for fine-tuned BERT-based classification, fiction ( 17% on average) and medicine (11% on average) were the top two misclassifications for social-science excerpts.
Comparing prediction errors with respect to the source of data for training and testing, we found that the pattern of misclassification in fine-tuned BERT-based classifications tended to be similar among all four types of classification. However, the ratio of errors per domain in pre-trained BERT-based classifications was likely to be diferent depending on the classifiers’ training corpus source. For example, business instances tended to be misclassified as fiction (25%-28%) when the training corpus is clean, but as social science (23%-27%) when using OCR’d texts for training. Similarly, medicine instances have an markedly higher ratio of misclassification as social science (27.89%-32.86%) in the OCR’d training corpus compared with the clean one (11.43%-16.43%). These observations reaffirm that fine-tunbed BERT is more robust for processing OCR’d texts compared with pre-trained BERT.
We further looked into the prediction consistency of all BERT-based classifiers on each book in both clean and OCR’d versions. Given two aligned lists (i.e., clean and OCR’d) of book-level average prediction accuracy across diferent classifiers, we found that there was a large overlap of books with comparatively low accuracy in clean versus OCR’d corpus, which suggests that content-based characteristics of these particular books may be the main cause of recurring prediction mistakes. We verified this hypothesis by manually checking the books with the lowest prediction scores, and confirmed that these books had heterogeneous genre-related features which were confusing even for human readers. For instance, the book The Story of My Life by Helen Keller is generally considered a classic “social science” work because of its main subject and its many non-fiction features. However, this is a classic autobiography composed of touching stories of a great woman struggling with severe disability, first published in 1903. Pre-trained
A
B F M S W
Predicted
Therefore, it is less surprising and even understandable for the models to label its instances as “medicine” or “fiction” based on their learning of the training data.
We have investigated the resilience of pre-trained and fine-tuned BERT embeddings for encoding OCR’d texts through a case study of classifying book excerpts into subject domains. To the best of our knowledge, this is the first empirical study to systematically quantify the influence of OCR quality on BERT. By changing BERT embedding types and classification model settings, we built 12 BERT-based classifiers using book excerpt corpora extracted from a large parallel book corpus of aligned clean and OCR’d volumes sourced from two well-known digital libraries. Our analysis shows that the original BERT embedding pre-trained on born-digital texts is not resilient to OCR noise, at least according to its classification accuracy. However, ifne-tuning the pre-trained BERT on OCR’d texts will significantly improve BERT’s resilience to OCR noise, and hence will benefit downstream applications. Besides, fine-tuned BERT outperforms the pre-trained one in its encoding stability with regards to changes in training corpus size and training data source. For both types of BERT embedding, unbalanced training corpora benefit embeddings’ resilience to OCR noise in downstream classifications. Our findings suggest that DH scholars should consider employing fine-tuned BERT for digitized-text-based scholarly research, particularly when their research involves document classification.
While our experiments yield significantly positive evidence for fine-tuned BERT embeddings’ resilience to OCR noise in the use-case of document classification, the impact of OCR noise on BERT for other downstream tasks remain under-investigated. For example, it is possible that BERT could react to OCR noise diferently at more fine-grained levels, such as sentence-level tasks (e.g., next sentence prediction, sentence-based sentiment analysis, etc.) and word-level (e.g., part-of-speech tagging, etc.). Therefore, future work focusing on BERT’s performance on OCR’d texts both at diferent text granularities and for diferent downstream NLP tasks would be useful to deepen our understanding of how OCR impacts this contextualized embedding technology. Furthermore, since our corpora consist exclusively of English-language books from the 18th and 19th centuries, expanding this study to curated datasets from other historical periods, languages, and publication types would be a very worthwhile future exercise. [13]