The use of OCR software to convert printed characters to digital text is a fundamental tool within diachronic approaches to Corpusassisted discourse Studies because allow researchers to expand their interest by making many texts available and analysable through a computer. However, OCR software are not totally accurate, and the resulting error rate compromises their effectiveness. This paper proposes a mixed qualitative-quantitative approach to OCR error detection and correction in order to develop a methodology for compiling historical corpora. The proposed approach consists of three main steps: corpus creation, OCR detection and correction, and application of the automatic rules. The rules are implemented in R using a “tidyverse” approach for a better reproducibility of the experiments.
In Corpus-assisted Discourse Studies (CADS) [
Corpus creation Selection of corpus
Keyword selection
Document extraction
OCR detection and definition of rules OCR Error Detection
OCR error correction
Definition of rules
OCR correction and quality analysis Application of rules
Analysis
not always achieve satisfactory results because of several aspects that affects
the quality of the original scan: the quality of the camera through which the
image has been taken, the image compression algorithm, the quality of the
paper especially when working with ancient or easily perishable texts such as old
newspapers. These errors may affect in a crucial way the results of a search for
documents which may compromise the compilation of a corpus in CADS [
In this paper, we propose a procedure for collecting and creating corpora for discourse analysis from old paper documents and we present a semi-automatic method for detection and correction of OCR errors. The outcome of this work consists in a set of rules which are, eventually, valid for different context and applicable to different datasets. The proposed procedure, in terms of computational readability, is aimed at making more readable and searchable the vast array of historical text corpora which are, at the moment, only partially usable given the high error rate introduced by an OCR software.
In Figure 1, we show an overview of the proposed approach which consists of three main steps: corpus creation, OCR detection and correction, and application of the automatic rules. The details of each step are described in the following sections.
The remainder of the paper is organized as follows: in Section 2, we describe our case study and the choices of the corpora; in Section 3, we present a brief overview of the state-of-the art in OCR correction, we describe our proposal for the error detection and correction, and we analyze the preliminary results; in Section 4, we give our final remarks and discuss our future work. 2
Case Study: Searching for Metaphors to Represent Immigrants Our case study focuses on the analysis of the metaphors used in the newspapers to represent migration to/from the United States of America and Italy from a diachronic perspective between the beginning of the XX century and the beginning of the XXI century. Given the vast amount of documents available, we needed to define a criterion in order to select a representative sample of documents that allows the comparison for the type of discourse analysis which is the object of our work. 2.1
In order to reduce the amount of data to collect, we selected two moments in
history which represent two sampling points in time which have a significant
value in relation to migratory movement: 1900-14 and 2000-2014. The decision
to focus on the aforementioned time periods lies in the fact that these represent
important moments for migratory movements, for both USA and Italy:
– As for USA:
• 1900-1914: intense migration movements to USA particularly from
Europe [
For USA, we selected the New York Herald4, for the time period 1900-1914, and the New York Times5, for the time period 2000-2014, because they are both examples of quality press published in New York.6 Even though the analysis of 4 http://chroniclingamerica.loc.gov 5 http://www.lexisnexis.com 6 By quality press we mean the more accurate newspapers which give detailed accounts of events, as well as reports on business, culture, and society, which contrasts with tabloid newspapers which are more devoted in giving sensational news. the same newspaper was preferable for a matter of homogeneity and integrity, we could not find an American newspaper available for both time period.
Regarding Italy, we selected La Stampa,7 a newspaper belonging to the category of quality press and which is published in Turin, a crossroads for many migration routes, both internal and from foreign countries. Fortunately, La Stampa provides an archive concerning all of its daily editions in digital format from 1867 to nowadays. 2.2
Having chosen the newspapers and the historical period, we needed to select the
keywords to filter the articles useful for our study. We decided to use
searchterms to sort and collect the newspaper articles. On the one hand, not using
search terms would have provided for more versatile corpora, that could be used
for other research purposes. On the other hand, a corpus collected by narrowing
down the amount of texts retrieved using search terms is more manageable. In
addition, as shown by [
When compiling a specialised corpus using keywords, there is a trade-off between precision and recall. That is, there is a tension between, on the one hand, creating a corpus in which all the texts are relevant, but which does not contain all relevant texts available in the database, and, on the other, creating a corpus which does contain all available relevant texts, albeit at the expense of irrelevant texts also being included. Seen from a different perspective, the tradeoff is between a corpus that can be deemed incomplete, and one which contains noise (i.e. irrelevant texts). Therefore, considering our research purposes which essentially consisted in identifying metaphors of migration, we decided to define a set of search terms to retrieve texts, in order to create a specialised corpus.
In particular, given the task of identifying metaphor of migration, we needed
to identify a set of search terms which would fit into both time periods and
which would be comparable and should denote the same meaning [
Italian keywords As for Italian, we needed to select a set of comparable
search terms between English and Italian [
After the identification of the two sets of query terms, we compiled four datasets.
In Table 1, we show a summary of the statistics for each corpus. The tokens and
types values represent the total number of occurrences versus the number of
unique words, respectively. We report the type/token ratio (TTR) which serves
as indicator of lexical diversity [
A careful analysis of a sample of texts showed that in both the old corpora there were a lot of misspellings or non-meaningful words caused by the OCR software which produced those documents. For example, there are many occurrences of the sequence tbe instead of the in the English corpus, as well as many occurrences of cho in the Italian corpus instead of che (that). In the following section, we describe the semi-automatic procedure for the detection and correction of these OCR errors. 3 3.1
As argued by [
The main approaches to OCR post-processing error correction are
1. a dictionary-based approach which aims at the correction of isolated errors
without considering the context [
The former approach is not able to capture all the real-word errors. For example,
the English expression a flood of irritants is not recognized as an error because all
the words are part of the dictionary. However, analyzing the context, it should be
corrected in a flood of immigrants. The latter approach intends to overcome the
problems of the dictionary-based, however it requires more effort in terms of time
and energy invested and is characterized by a lower level of efficiency in terms
of automation. Moreover, the procedures which are generally adopted to
overcome OCR errors [
In this paper, we propose a semi-automatic mixed approach to OCR detection which brings together the dictionary-based and the context-based approaches. The first problem in our case study concerns the fact that we did not have the corresponding ground truth version of the corpora. Therefore, we decided to use the contemporary corpora where the text was digital since the beginning. The error detection correction task consisted in a three-step procedure: 1. Definition of a list of plausible error candidates by comparing the list of words of the old corpus with the new corpus. The words that do not appear in the latter, or that have a statistically significant difference in frequency, compose a list of plausible error candidates. For example, the previously mentioned expressions tbe or te, in the English dataset, and cho and olla in the Italian dataset. Successively, each error candidate has been qualitatively analysed by being manually observed through concordance lines within its context of occurrence in order to verify if it was an error or not. Lastly, a list of detected errors has been produced. 2. Analysis and categorization of the error in the list of candidates. Each error is categorised according to three categories: i) Standard Mapping: the error contains the same number of characters than the respective correct form. For example: ‘hear’ (correct) vs ‘jear’ (error); ii) Non-standard Mapping: the error contains a higher or a smaller number of characters than the correct form. For example: ‘main’ (correct) vs ‘rnain’ (error); iii) Split errors: the word is interpreted by the OCR as two distinct words. This is a very common error when digitalizing newspapers because of the shape of the column in which articles are written. For example: ‘department’ vs ‘depart’ and ‘ment’. 3. Define the error correction rule as a regular expression to match the pattern of the error (i.e. jear) and substitute it with the (supposedly) correct form (i.e. hear)18. 18 Ambiguous and dubious cases where two or more plausible corrections were available, were not inserted in the list to avoid compromising the validity of the corpora.
Before OCR correction After OCR correction Difference Corpus Tokens Types Tokens Types ∆ Tokens ∆ Types NY Herald 1900-1914 64,061,101 3,085,080 64,246,208 3,082,880 +0.29% -0.04% La Stampa 1900-1914 19,396,796 899,688 19,396,558 899,676 ∼-0.0% ∼-0.0% The implementation of these procedure follows the principles described by [18] where the idea is to efficiently and effectively mine textual information from large text collections by means of pipelines in order to allow for a sequential process of text analysis. For our experiments, we used the R programming language which has a set of packages, named ‘tidyverse’ 19, that implements this idea of pipelined in a clear way. For space reasons, we are not going to describe in detail the code and we will make the source code used in our experiments available online.20 3.4
A total of 476 errors for English and 80 errors for Italian have been manually individuated and, respectively, as many correcting rules have been written for each language.21 The automatic application of the rules produced 722,371 substitutions for English and 99,255 substitutions for Italian. As the Table 3 shows, for both the American and the Italian corpora, the number of Tokens and Types have been moderately changed. As a general comment, it is not easy to predict in what way OCR correction will work. On one side, an increase in the number of tokens might happen because many errors, such as p/r or th/ were not 19 https://www.tidyverse.org 20 https://github.com/gmdn 21 In this analysis, we excluded the split errors since this type of error require a longer evaluation procedure given the amount of false positives errors. previously recognized as valid tokens. On the other side, the number of types are in general reduced for both corpora since different errors are mapped to the same type. For example, the English article the has been differently misspelled in many ways: tne, tha, tbe, tna. These four words are counted as four different types. Then, by correcting substituting all these words with the, the number of types is reduced of three units. Similarly, the Italian female article la has been misspelled as jd, ja, ln. These three words are counted as three different types. Then, by correcting all these words with the, the number of types is reduced of three units. The correction task has been repeated four times for English corpus and two times for Italian. Any ambiguous and dubious case, such as ii and ih in Italian which could be corrected were not corrected to not compromise the validity of the corpora. 4
In this paper, we described a procedure for collecting and creating corpora for discourse analysis from old paper documents and we presented a semi-automatic method for detection and correction of OCR errors. The semi-automatic approach for correcting OCR errors developed for this project has proved to be effective. Despite the fact that the rules produced for the corrections may be less useful with other corpora, the methodology itself is applicable to different contexts.
We are currently investigating how to evaluate the set of rules which have
been developed for the actual project to other corpora in order to verify if it
is successfully applicable to different contexts. Secondly, we would replicate the
same methodology for other set of documents and produce a different set of rules
which would be compared with the ones developed for the actual work. Our aim
is to create rules which can be generally reused from everyone for correcting their
own OCR processed documents. Given the number of substitutions (for English
we were close to a million of substitutions), it is important to understand the
number false positives introduced. In this sense, we will explore how to evaluate
the rules in a semi-automatic way and produce a ground truth. Recent papers
have explored advanced automatic corrections based on edit distances, n-grams
and neural models [
There are still open questions that we will investigate in this line of work: for
example, how many documents have we missed during the compilation of the
corpus given that a search keyword may be subject to OCR correction as well.
How these types of keyword search error can affect a CADS analysis? For this
reason, following [