Digitized newspapers as a source of data bring many opportunities, however, also many challenges and pitfalls. In the JobAds project (FWF P35783), we investigate the evolution of the Austrian labor market through historical job advertisements from digitized newspapers between 1850-1950. Through their analysis, we aim to get insights into the regional and temporal trends in positions offered and sought for, the skills and qualifications required and offered, media strategies, but also social aspects such as gender-specific job offers.

Our project aims to cover as much as possible of the Austrian labor market in the defined time period. In this paper, we discuss the difÏculties which typically arise when one tries to access data from digitized newspapers. We refer to these difÏculties as 'biases', as they have a great potential to skew results and their interpretation.

We used 29 periodics (see Appendix) with the largest number of editions within our time span from the ANNO corpus [16]. From each newspaper, we preselected pages containing job advertisements and converted them to machine-readable text through page segmentation, optical character recognition (OCR), and post-correction.

Each processing step introduces potential biases: selection of periodicals may not be representative, a classifier may systematically misclassify pages containing certain types of ads, segmentation quality may vary with layout complexity or scan quality, OCR may be affected by font use or style particularities such as white letters on a black background, and post-correction effectiveness may vary between newspapers or years because of different typical mistakes.

Biases also arise from the historical context of newspapers, which were only one of several channels realizing matches between job seekers and vacancies [25]. Some jobs were advertised through newspapers more often than others, as in a short radius, it was harder to find highly specialized employees than e.g. unqualified workers who could be placed through a bourse system [13].

This paper addresses how we confront a research question with the technical and historical reality, from corpus selection to OCRed text post-processing. Section 2 describes related work, section 3 provides details about dataset creation. Sections 4 and 5 discuss biases arising from historical context and processing steps respectively, and section 6 outlines future work and concludes the paper.

Oberbichler and Pfanzelter [17] discuss a large number of 'biases that come along with the processing and datafication of historical newspapers' (p.127) and illustrate them in a case study about return migration. In a keyword-based search, the first challenge is to select the right terms, which is hindered by word flexions and spelling variants, but also by semantic uncertainty, e.g. false positive hits if the term is too broad or not finding all occurrences if the term is too narrow. They show how missing data and varying OCR skew absolute frequencies, and use therefore relative frequencies instead. They also propose improving OCR quality with tools like Transkribus [18] and argue for adding metadata, contextualization of the source documents, providing information about the limitations of the collection, and additional tools in the interfaces.

Wijfjes [28] discusses the relationship between traditional humanities and new digital methods. The most prominent obstacle in using digitized newspapers as a research source is the incompleteness of digitized collections, caused by factors such as costs, time or copyright issues. Relying on the available collections regardless of their broader context can result in working with a 'convenience sample' (p.16), closely related to 'digital laziness' (p.10) which arises from an overreliance on easily accessible digital information. Author also mentions unreliability in OCR and the need for 'complete and uniform data' (p.21).

The errors in Optical Layout Recognition (OLR) and Optical Character Recognition (OCR) are crucial problems in machine-readable text creation that have been addressed by several scholars ( [23]; [27]; [8]; [6]; [21]). Noisy OCR or its varying quality across the corpus poses problems not only in a keyword search, but also in subsequent tasks, such as Part-of-Speech (POS) tagging, dependency parsing, Named Entity Recognition (NER) or topic modeling. Therefore, it is necessary to assess the influence of the OCR quality on the outcomes of NLP tasks, as discussed e.g. by ([22]; [20]; [26]).

Cordell [4] largely discusses digitized collection in a broader context of the process of digitalization and decisions leading to which material shall be digitized. He distinguishes between printed and digital editions and argues for taking digitized text 'seriously within its own medium' (p.217). On the example of the Raven by E. A. Poe, he shows how results of keyword search are dependent on the quality of the OCR output.

We are aware of all these and handle the challenges where they occur. For example, working with relative frequencies of job titles can help to capture the changing demands and offers of a specific position instead of unintentional identification of changing trends of advertising jobs in newspapers. Considering the previous research and our own experience, we search for strategies for mitigating the bias in our data.

Our dataset comprises 29 newspaper titles from the ANNO corpus, a collection of digitized newspapers provided by the Austrian National Library, from 1850-1950. The 29 newspapers were selected based on the largest number of editions, with a minimal issue period of 10 years. These newspapers are predominantly in German, containing minor numbers of ads in French, English, Czech, Hungarian, Italian and other languages. Because of the time needed for processing the pages, we initially preselected pages containing job advertisements manually, based on observed patterns in the advertisement sections appearance, and later refined the preselection automatically using a transformer-based preselection.

From the preselected pages, we randomly sampled one page per year for each newspaper available for that year, resulting in 3 300 pages. On these pages, we manually annotated all job advertisements using doccano software [15], resulting in 14 985 annotated job ads. These annotations serve us as ground-truth data. The annotated ads were OCRed with the frak2021 model [10] and manually corrected using Transkribus.

In this section, we explore biases arising from the historical context and the nature of newspapers as a medium. As these biases are beyond direct control, we have to gather comprehensive information to adjust our research questions based on available data.

The bias in our research does not start with the selection of the 29 newspaper titles; it starts with the decision to use newspapers as our primary data source to understand labor market development. The matches between the job seekers and vacancies were realized through several channels [5], with different proportions for various types of jobs.

To address this bias, it is crucial to quantify the extent to which job seekers and vacancies were matched through newspaper ads compared to other channels and to understand which jobs were underrepresented or missing in them. If sufÏcient facts concerning these aspects cannot be found, it is necessary to redefine the research goals. Instead of aiming to describe the entire labor market, we can narrow our scope to explore specific job categories, time periods, or regions.

For our period, approximately 30% of the matches between job seekers and employers were realized through job advertisements in newspapers [13]. Other channels to find matches were personal contacts, asking around, bourse system, municipal placement services and commercial brokers, which came increasingly under administrative control to reduce abuse and were finally replaced by the public job services. Search via newspaper ads was dominating among white-collar jobs [5]. Blue-collar workers were underrepresented in this channel [13]. Based on these facts, we can aim to compare e.g. qualifications offered with requirements presented in job searches and job offers for white-collar workers.

The focus of newspapers shall also be considered. To gain a comprehensive understanding of our data, we should address the following questions regarding the newspapers:

• What is the political orientation of the newspapers? • Which geographical area do they cover? • What is their temporal coverage? • What is their social focus? • Who are the intended readers?

Access to such meta-information about newspapers provides crucial context and facts. While no single newspaper can fully represent the labor market, a selection of heterogeneous titles covering various aspects can collectively offer a broader perspective. Our selected newspapers were issued in several geographical regions (e.g. Arbeiterwille in Graz, Arbeiter Zeitung in Wien, Linzer Volksblatt in Linz, Salzburger Chronik für Stadt und Land in Salzburg), covering longer periods (see Fig. 1), most of them daily newspaper but also with focus on travels (Fremden-Blatt) or workers (Arbeiterwille). We included newspapers with various orientations, such as social-democratic (e.g. Arbeiterwille, Arbeiterzeitung), liberal-democratic (Prager Tagblatt), or nationalistic (Salzburger Volksblatt: die unabhängige Tageszeitung für Stadt und Land Salzburg). However, achieving an ideal representation still remains extremely challenging due to realworld complexities.

To draw unbiased conclusions from job advertisements about the entire labor market, the underlying assumption would be that people offering and searching through job ads (or the job ads that we analyze) have the same characteristics of interest as job offerers or seekers who do not use newspaper ads. Since we cannot often know that, we need to ask who actually advertises in newspapers.

For instance, in a small town with only one factory, there may be no need to advertise for unqualified workers in newspapers. Similarly, workers in such towns may not need to advertise either, as they can rely on local announcements or neighbors or family. Consequently, these types of ads may be missing or underrepresented in our corpus. On the contrary, if the factory seeks highly specialized personnel, they may expand their outreach by utilizing the newspapers.

A similar scenario applies to a city baker. If they seek someone locally, they might rely on the people in their surrounding. However, if they require an apprentice from the countryside, such as e.g. in (Fig. 2), they would more likely advertise in newspapers. This highlights that in our corpus, advertisements might be overrepresented that involve a greater distance, e.g. geographical distance (a baker from a city seeks an apprentice from the countryside) or social distance (higher class person looking for a servant). In other cases, despite a short geographical distance in large cities, complexity of an urban society and anonymity plays a role. Also, the ads in our corpus may be extreme from some point of view, e.g. seeking a highly qualified person, a factory with bad working conditions, high demand for workers for seasonal work, a person who struggles to find a job for a longer time. On the other hand, some ads may be missing, e.g. from people who could not afford to pay for an ad in newspapers.

This section describes biases that data processing introduces, and presents strategies to mitigate them. Our processing pipeline contains steps from corpus selection to the cleaned data, which can be used for meaningful economical analysis.

The initial step involves building a corpus. While in the ideal case, researchers would work with the entire population of data, practical limitations in terms of time and computation power or missing newspapers in digitized collections make this only hardly feasible.

We come as close as we can to the entire population given our resource constraints by selecting the newspapers with the most editions that were issued for at least 10 years, which allows for a comparison in time. In a corpus selection, attention needs to be paid to include heterogeneous newspapers that will cover a wide scope. For further information, see subsection 4.1.

Initially, we manually preselected pages likely to contain job advertisements by examining sampled issues from all newspapers and identifying patterns in the job ad sections. In a later stage, we refined this process using a transformer-based model for the same task, by fine-tuning the microsoft/dit-large-finetuned-rvlcdip model [9] on our data. The fine-tuned model reached a f1 score of 0.88 and recall of 0.89 on the testing data.

Preselection can be a dangerous process that can lead to excluding relevant data and bias the results. Our strategy against this pitfall was aiming for the highest recall possible, favoring the inclusion of non-relevant pages over the exclusion of relevant ones. Our model's results, which indicated that only about 34% of the preselected 4,000,000 pages actually contained job advertisements, give us confidence that we have captured most relevant pages.

Page segmentation is a step which has a direct impact on the OCR quality ( [11]; [12]; [1]; [3]). However, segmentation quality is often assessed through subjective visual control [7], which does not offer complex insights into segmented data quality.

To address this task, we adopt a methodology for segmentation evaluation from [24]. This method is not based only on the area of intersection between the annotated and predicted region, as this can be skewed if graphical elements or large blank spaces are present in data. Instead, it also benefits from information about the presence of the text in the non-intersecting parts of the predicted region and its ground-truth.

We manually annotated nearly 15 000 job ads from our corpus to create ground truth data. We make the annotated data publicly available in the GitHub repository: https://github.com/JobAds-FWFProject/Ground-Truth-CHR2024. This allows us to identify newspapers with lower segmentation accuracy and generate additional ground-truth data to fine-tune our segmentation algorithm effectively. The results of the ongoing work on the evaluation of the segmentation quality will be published in another publication.

OCR quality is the most prominent source of bias, as highlighted in prior research ( [17]; [23]; [27]). Variations in OCR accuracy can lead to discrepancies in e.g. keyword searches and affect data reliability.

The first step to mitigate this bias is quantifying the OCR quality by e.g. a character error rate (see Fig. 3). Although an approximation of the OCR quality can be obtained by checking words against a dictionary [21], we decided to manually check and correct a sample of the advertisements. This cleaned data serves us to (1) quantify the quality of the OCR, (2) provide us with high-quality data for text-mining experiments and (3) give us information about the most common mistakes in recognition which we can use for automatic post-corrections. A pure dictionary-based approach would introduce a bias by words missing in the dictionary, e.g. abbreviations or names, which both are often in our datasets. Based on our manually transcribed ground-truth, our OCR reaches a SacreBLEU score of 67.5%, word error rate of 30.6% and character error rate of 12.2%. However, apart from this overall evaluation, it is crucial to compare the OCR quality across newspapers and years.

Post-correction is the last step in our pipeline that affects data quality. Starting with a varying OCR quality across newspaper titles or years, the post-correction can both reduce or amplify the discrepancies in data quality. To quantify this problem, we measure the error rate after different post-correction steps on the samples that were manually checked and corrected. As this sample contains ads across all newspaper titles and years, we will be able to compare text quality before and after the post-correction step.

For the post-correction process, we fine-tune the hmbyt5-preliminary model1 on the IDCAR2019-POCR dataset for OCR correction [19], which significantly improves text accuracy. On the IDCAR data, we reach the SacreBLEU score of 72.25% compared to only 10.83% achieved by the original OCRed text.

While the model significantly improves the quality of a very bad OCR, our OCR already reaches a higher score (67.5%) before the post-correction step. As there is a large discrepancy between the OCR quality of the training data and our OCR data, the post-correction model itself sometimes introduces new errors (see Tab 1). Some mistakes concern only interpunction, some change letters leading to words with no meaning, but some of them introduce a semantically different result, such as painters (Malerinnen) from ironers (Büglerinnen). We also need to consider that not all characters can be corrected through post-processing. For example, for incorrectly recognized numbers, the original information is lost and cannot be gained back by post-processing (see Fig. 4). This potentially affects our ability to analyze details about e.g. offered salary where available.

To illustrate how technical biases can affect historical data interpretation, we examine the demand for five different positions by comparing the frequency of job ads mentioning them between 1850-1900 and 1901-1950. We use a sample of 2779 job ads with both raw OCR and manually corrected transcriptions. These ads are divided into two data sets: one for 1850-1900 and another for 1901-1950.

First, we compare the absolute frequencies of the positions in both OCRed text, with the results in Tab. 2. E.g., the 'Tapezierer' (paperhanger) appears 2, resp. 4 times in the data sets. For the example of the paperhanger, the OCR data suggests a low frequency of ads, with a slight increase in the 20th century. This might lead to the conclusion that paperhanger jobs were rare and that their demand doubled in the first half of the 20th century. However, when we look at the manually corrected versions of the text, we obtain the following results (Tab. 3):

Manually corrected data also indicates an increase in demand over time. However, absolute frequencies can more reflect the amount of available data rather then trends of employment. To account for this, we divide absolute frequencies by the number of ads available for these time periods in our sample, and we get another information (Tab. 4): When adjusted for the total number of ads, the relative frequency of paperhanger ads shows that the demand remained almost constant. The apparent increase in absolute numbers is due to the larger volume of data available for the later period, not an actual rise in demand. The example of the paperhanger, and similarly of the other positions, demonstrate the need for cautious interpretation of historical job ads, and that the OCR errors do not impact all positions in the same way. Although similar tests need to be done on a larger scale, this example illustrates how easily we can run into a false interpretation.

In this paper, we addressed the biases encountered in our research of the labor market based on job advertisements from digitized newspapers. Our investigation focused on two sources of bias: those arising from the historical context of the newspapers and those introduced during data processing. The first set of biases stems from the nature of newspapers as a data source. This includes the selection of newspaper titles, their political and social orientations, geographic reach, and the profiles of advertisers and readers. These factors shape the type of job ads available, leading to over-or underrepresentation of certain job types and demographic groups. We emphasize that understanding these contextual elements is crucial. Researchers must adjust their research questions to align with the data's actual scope and representation rather than assuming a comprehensive view of the labor market.

The second set of biases arises from the various stages of data processing. This encompasses corpus creation, pre-selection of relevant pages, page segmentation, OCR quality, and post-correction processes. Each stage presents potential pitfalls that can skew results. We highlighted the importance of high recall in data preselection, evaluating segmentation and OCR accuracy, and manual corrections and error analysis. Through a practical example, we demonstrated how technical biases can skew historical job market analyses.

Our ongoing work includes gathering meta-information about each newspaper title and the labor market to better understand the matching of job applicants with vacancies. Also, we continue with further manual corrections of OCRed text to ensure consistent data quality across our corpus. We plan to deeper investigate the post-processing aspects, such as the model's skills transferability in cases, when there is a great difference between the OCR quality of the training and testing data. Additionally, we plan to implement strategies like data augmentation for underrepresented job categories.