Digitized newspapers are a source of unique and rich historical data but pose significant challenges in the interpretation of results obtained through their mining. The JobAds project (FWF P35783) explores the evolution of the labor market through job advertisements from digitized newspapers between 1850-1950, aiming to reveal regional and temporal trends in job ofers, required skills, media strategies, and social aspects such as gender-specific ads. Using the ANNO corpus, we selected 29 newspapers with the most editions. Their processing involved job ads pages preselection, layout segmentation, optical character recognition (OCR), and post-correction, each introducing potential biases due to varying efÏciency of these processes. Additionally, the inherent bias of newspapers as historical sources must be considered, as they reflect only a subset of the job market dynamics of their time. This paper identifies these biases, quantifies their impact, and proposes solutions for steps from corpus selection to data preparation for subsequent text-mining and analysis. We discuss and exemplify the implications of these biases on research outcomes and suggest methodological adjustments to mitigate their efects, ensuring more reliable insights into the historical labor market. Also, we make a dataset of 15 000 manually annotated ground-truth data available as part of this paper.
Digitized newspapers as a source of data bring many opportunities, however, also many challenges and pitfalls. In the JobAds project (FWF P35783), we investigate the evolution of the Austrian labor market through historical job advertisements from digitized newspapers between 1850-1950. Through their analysis, we aim to get insights into the regional and temporal trends in positions ofered and sought for, the skills and qualifications required and ofered, media strategies, but also social aspects such as gender-specific job ofers.
Our project aims to cover as much as possible of the Austrian labor market in the defined time period. In this paper, we discuss the difÏculties which typically arise when one tries to access data from digitized newspapers. We refer to these difÏculties as ‘biases’, as they have a great potential to skew results and their interpretation.
We used 29 periodics (see Appendix) with the largest number of editions within our time
span from the ANNO corpus 1[
Each processing step introduces potential biases: selection of periodicals may not be representative, a classifier may systematically misclassify pages containing certain types of ads, segmentation quality may vary with layout complexity or scan quality, OCR may be afected by font use or style particularities such as white letters on a black background, and post-correction efectiveness may vary between newspapers or years because of diferent typical mistakes.
Biases also arise from the historical context of newspapers, which were only one of several
channels realizing matches between job seekers and vacancies25[]. Some jobs were advertised
through newspapers more often than others, as in a short radius, it was harder to find highly
specialized employees than e.g. unqualified workers who could be placed through a bourse
system [
This paper addresses how we confront a research question with the technical and historical reality, from corpus selection to OCRed text post-processing. Section 2 describes related work, section 3 provides details about dataset creation. Sections 4 and 5 discuss biases arising from historical context and processing steps respectively, and section 6 outlines future work and concludes the paper.
Oberbichler and Pfanzelter17[] discuss a large number of ‘biases that come along with the processing and datafication of historical newspapers’ (p.127) and illustrate them in a case study about return migration. In a keyword-based search, the first challenge is to select the right terms, which is hindered by word flexions and spelling variants, but also by semantic uncertainty, e.g. false positive hits if the term is too broad or not finding all occurrences if the term is too narrow. They show how missing data and varying OCR skew absolute frequencies, and use therefore relative frequencies instead. They also propose improving OCR quality with tools like Transkribus18[] and argue for adding metadata, contextualization of the source documents, providing information about the limitations of the collection, and additional tools in the interfaces.
Wijfjes [
The errors in Optical Layout Recognition (OLR) and Optical Character Recognition (OCR)
are crucial problems in machine-readable text creation that have been addressed by several
scholars (2[
Cordell4[] largely discusses digitized collection in a broader context of the process of digitalization and decisions leading to which material shall be digitized. He distinguishes between printed and digital editions and argues for taking digitized text ‘seriously within its own medium’ (p.217). On the example of theRaven by E. A. Poe, he shows how results of keyword search are dependent on the quality of the OCR output.
We are aware of all these and handle the challenges where they occur. For example, working with relative frequencies of job titles can help to capture the changing demands and ofers of a specific position instead of unintentional identification of changing trends of advertising jobs in newspapers. Considering the previous research and our own experience, we search for strategies for mitigating the bias in our data.
Our dataset comprises 29 newspaper titles from the ANNO corpus, a collection of digitized newspapers provided by the Austrian National Library, from 1850-1950. The 29 newspapers were selected based on the largest number of editions, with a minimal issue period of 10 years. These newspapers are predominantly in German, containing minor numbers of ads in French, English, Czech, Hungarian, Italian and other languages. Because of the time needed for processing the pages, we initially preselected pages containing job advertisements manually, based on observed patterns in the advertisement sections appearance, and later refined the preselection automatically using a transformer-based preselection.
From the preselected pages, we randomly sampled one page per year for each newspaper
available for that year, resulting in 3 300 pages. On these pages, we manually annotated all
job advertisements using doccano software [
In this section, we explore biases arising from the historical context and the nature of newspapers as a medium. As these biases are beyond direct control, we have to gather comprehensive information to adjust our research questions based on available data.
The bias in our research does not start with the selection of the 29 newspaper titles; it starts
with the decision to use newspapers as our primary data source to understand labor market
development. The matches between the job seekers and vacancies were realized through several
channels [
To address this bias, it is crucial to quantify the extent to which job seekers and vacancies were matched through newspaper ads compared to other channels and to understand which jobs were underrepresented or missing in them. If sufÏcient facts concerning these aspects cannot be found, it is necessary to redefine the research goals. Instead of aiming to describe the entire labor market, we can narrow our scope to explore specific job categories, time periods, or regions.
For our period, approximately 30% of the matches between job seekers and employers were realized through job advertisements in newspaper1s3[]. Other channels to find matches were personal contacts, asking around, bourse system, municipal placement services and commercial brokers, which came increasingly under administrative control to reduce abuse and were ifnally replaced by the public job services. Search via newspaper ads was dominating among white-collar jobs5[]. Blue-collar workers were underrepresented in this chann13e]l. [Based on these facts, we can aim to compare e.g. qualifications ofered with requirements presented in job searches and job ofers for white-collar workers.
The focus of newspapers shall also be considered. To gain a comprehensive understanding of our data, we should address the following questions regarding the newspapers: • What is the political orientation of the newspapers? • Which geographical area do they cover? • What is their temporal coverage? • What is their social focus? • Who are the intended readers?
Access to such meta-information about newspapers provides crucial context and facts. While no single newspaper can fully represent the labor market, a selection of heterogeneous titles covering various aspects can collectively ofer a broader perspective. Our selected newspapers were issued in several geographical regions (eA.gr.beiterwille in Graz,Arbeiter Zeitung in Wien, Linzer Volksblatt in Linz, Salzburger Chronik für Stadt und Land in Salzburg), covering longer periods (see Fig. 1), most of them daily newspaper but also with focus on traveFlrse m(denBlatt) or workersA(rbeiterwille). We included newspapers with various orientations, such as social-democratic (e.g.Arbeiterwille, Arbeiterzeitung), liberal-democraticPr(ager Tagblatt), or nationalistic S(alzburger Volksblatt: die unabhängige Tageszeitung für Stadt und Land Salzburg). However, achieving an ideal representation still remains extremely challenging due to realworld complexities.
To draw unbiased conclusions from job advertisements about the entire labor market, the underlying assumption would be that people ofering and searching through job ads (or the job ads that we analyze) have the same characteristics of interest as job oferers or seekers who do not use newspaper ads. Since we cannot often know that, we need to ask who actually advertises in newspapers.
For instance, in a small town with only one factory, there may be no need to advertise for unqualified workers in newspapers. Similarly, workers in such towns may not need to advertise either, as they can rely on local announcements or neighbors or family. Consequently, these types of ads may be missing or underrepresented in our corpus. On the contrary, if the factory seeks highly specialized personnel, they may expand their outreach by utilizing the newspapers.
A similar scenario applies to a city baker. If they seek someone locally, they might rely on the people in their surrounding. However, if they require an apprentice from the countryside, such as e.g. in (Fig. 2), they would more likely advertise in newspapers.
This highlights that in our corpus, advertisements might be overrepresented that involve a greater distance, e.g. geographical distance (a baker from a city seeks an apprentice from the countryside) or social distance (higher class person looking for a servant). In other cases, despite a short geographical distance in large cities, complexity of an urban society and anonymity plays a role. Also, the ads in our corpus may be extreme from some point of view, e.g. seeking a highly qualified person, a factory with bad working conditions, high demand for workers for seasonal work, a person who struggles to find a job for a longer time. On the other hand, some ads may be missing, e.g. from people who could not aford to pay for an ad in newspapers.
This section describes biases that data processing introduces, and presents strategies to mitigate them. Our processing pipeline contains steps from corpus selection to the cleaned data, which can be used for meaningful economical analysis.
The initial step involves building a corpus. While in the ideal case, researchers would work with the entire population of data, practical limitations in terms of time and computation power or missing newspapers in digitized collections make this only hardly feasible.
We come as close as we can to the entire population given our resource constraints by selecting the newspapers with the most editions that were issued for at least 10 years, which allows for a comparison in time. In a corpus selection, attention needs to be paid to include heterogeneous newspapers that will cover a wide scope. For further information, see subsection 4.1.
Initially, we manually preselected pages likely to contain job advertisements by examining
sampled issues from all newspapers and identifying patterns in the job ad sections. In a later
stage, we refined this process using a transformer-based model for the same task, by fine-tuning
the microsoft/dit-large-finetuned-rvlcdip model [
Preselection can be a dangerous process that can lead to excluding relevant data and bias the results. Our strategy against this pitfall was aiming for the highest recall possible, favoring the inclusion of non-relevant pages over the exclusion of relevant ones. Our model’s results, which indicated that only about 34% of the preselected 4,000,000 pages actually contained job advertisements, give us confidence that we have captured most relevant pages.
Page segmentation is a step which has a direct impact on the OCR quality 1([
To address this task, we adopt a methodology for segmentation evaluation fro2m4].[ This method is not based only on the area of intersection between the annotated and predicted region, as this can be skewed if graphical elements or large blank spaces are present in data. Instead, it also benefits from information about the presence of the text in the non-intersecting parts of the predicted region and its ground-truth.
We manually annotated nearly 15 000 job ads from our corpus to create ground truth data. We make the annotated data publicly available in the GitHub repository: https://github.com/JobAds-FWFProject/Ground-Truth-CHR2024. This allows us to identify newspapers with lower segmentation accuracy and generate additional ground-truth data to ifne-tune our segmentation algorithm efectively. The results of the ongoing work on the evaluation of the segmentation quality will be published in another publication.
OCR quality is the most prominent source of bias, as highlighted in prior resear1c7h](;[[
The first step to mitigate this bias is quantifying the OCR quality by e.g. a character error
rate (see Fig. 3). Although an approximation of the OCR quality can be obtained by checking
words against a dictionary2[
Post-correction is the last step in our pipeline that afects data quality. Starting with a varying OCR quality across newspaper titles or years, the post-correction can both reduce or amplify the discrepancies in data quality. To quantify this problem, we measure the error rate after diferent post-correction steps on the samples that were manually checked and corrected. As this sample contains ads across all newspaper titles and years, we will be able to compare text quality before and after the post-correction step.
For the post-correction process, we fine-tune the hmbyt5-preliminary mod1elon the
IDCAR2019-POCR dataset for OCR correction1[
While the model significantly improves the quality of a very bad OCR, our OCR already reaches a higher score (67.5%) before the post-correction step. As there is a large discrepancy between the OCR quality of the training data and our OCR data, the post-correction model itself sometimes introduces new errors (see Tab 1). Some mistakes concern only interpunction, some change letters leading to words with no meaning, but some of them introduce a semantically diferent result, such as painters (Malerinnen) from ironers (Büglerinnen). 1hmbyt5-preliminary model [18.6.2024] We also need to consider that not all characters can be corrected through post-processing. For example, for incorrectly recognized numbers, the original information is lost and cannot be gained back by post-processing (see Fig. 4). This potentially afects our ability to analyze details about e.g. ofered salary where available.
To illustrate how technical biases can afect historical data interpretation, we examine the demand for five diferent positions by comparing the frequency of job ads mentioning them between 1850-1900 and 1901-1950. We use a sample of 2779 job ads with both raw OCR and manually corrected transcriptions. These ads are divided into two data sets: one for 1850-1900 and another for 1901-1950.
First, we compare the absolute frequencies of the positions in both OCRed text, with the results in Tab. 2. E.g., the ‘Tapezierer’ (paperhanger) appears 2, resp. 4 times in the data sets.
For the example of the paperhanger, the OCR data suggests a low frequency of ads, with a slight increase in the 20th century. This might lead to the conclusion that paperhanger jobs were rare and that their demand doubled in the first half of the 20th century. However, when we look at the manually corrected versions of the text, we obtain the following results (Tab. 3):
Manually corrected data also indicates an increase in demand over time. However, absolute frequencies can more reflect the amount of available data rather then trends of employment. To account for this, we divide absolute frequencies by the number of ads available for these time periods in our sample, and we get another information (Tab. 4):
When adjusted for the total number of ads, the relative frequency of paperhanger ads shows that the demand remained almost constant. The apparent increase in absolute numbers is due to the larger volume of data available for the later period, not an actual rise in demand. The example of the paperhanger, and similarly of the other positions, demonstrate the need for cautious interpretation of historical job ads, and that the OCR errors do not impact all positions in the same way. Although similar tests need to be done on a larger scale, this example illustrates how easily we can run into a false interpretation.
In this paper, we addressed the biases encountered in our research of the labor market based on job advertisements from digitized newspapers. Our investigation focused on two sources of bias: those arising from the historical context of the newspapers and those introduced during data processing. The first set of biases stems from the nature of newspapers as a data source. This includes the selection of newspaper titles, their political and social orientations, geographic reach, and the profiles of advertisers and readers. These factors shape the type of job ads available, leading to over- or underrepresentation of certain job types and demographic groups. We emphasize that understanding these contextual elements is crucial. Researchers must adjust their research questions to align with the data’s actual scope and representation rather than assuming a comprehensive view of the labor market.
The second set of biases arises from the various stages of data processing. This encompasses corpus creation, pre-selection of relevant pages, page segmentation, OCR quality, and post-correction processes. Each stage presents potential pitfalls that can skew results. We highlighted the importance of high recall in data preselection, evaluating segmentation and OCR accuracy, and manual corrections and error analysis. Through a practical example, we demonstrated how technical biases can skew historical job market analyses.
Our ongoing work includes gathering meta-information about each newspaper title and the labor market to better understand the matching of job applicants with vacancies. Also, we continue with further manual corrections of OCRed text to ensure consistent data quality across our corpus. We plan to deeper investigate the post-processing aspects, such as the model’s skills transferability in cases, when there is a great diference between the OCR quality of the training and testing data. Additionally, we plan to implement strategies like data augmentation for underrepresented job categories.
The annotated ground truth data is publicly available at: https://github.com/JobAds-FWFProject/Ground-Truth-CHR2024.
We thank the Austrian National Library (ÖNB) for providing the data. We would also like to thank Meike Linnewedel, Clara Hochreiter and Melanie Frauendorfer for their eforts in correcting and annotating the ground truth. This work was supported by the FWF under grant number P 35783.
A r b e i t e r w i l l e A r b e i t e r − Z e i t u n g B r e g e n z e r T a g b l a t t / V o r a r l b e r g e r T a g b l a t t Das V a t e r l a n d D e u t s c h e s V o l k s b l a t t Die P r e s s e F r e i e Stimmen Fremden − B l a t t G r a z e r T a g b l a t t G r a z e r V o l k s b l a t t I l l u s t r i e r t e Kronen Z e i t u n g I n n s b r u c k e r N a c h r i c h t e n L i n z e r Tages − P o s t L i n z e r V o l k s b l a t t Morgen − P o s t Neue F r e i e P r e s s e Neues Wiener J o u r n a l : u n p a r t e i i s c h e s T a g b l a t t Neues Wiener T a g b l a t t ( T a g e s a u s g a b e ) N e u i g k e i t s −Welt − B l a t t P e s t e r L l o y d P i l s n e r T a g b l a t t P r a g e r A b e n d b l a t t P r a g e r T a g b l a t t R e i c h s p o s t S a l z b u r g e r C h r o n i k f u e r S t a d t und Land S a l z b u r g e r V o l k s b l a t t : d i e u n a b h a e n g i g e T a g e s z e i t u n g f ure S t a d t und Land S a l z b u r g V o r a r l b e r g e r Landes − Z e i t u n g V o r a r l b e r g e r Vo l k s − B l a t t Wiener Z e i t u n g