Newspaper archives are some of the richest historical document collections. Their study is, however, very tedious: one needs to physically visit the archives, search through reams of old, very fragile paper, and manually assemble cross-references. We present Shax, a visual newspaper-archive exploration tool that takes large, historical archives as an input and allows interested parties to browse the information included in a chronological or geographic manner so as to re-discover history. We used Shax on a selection of the Neue Zurcher Zeitung (NZZ)|the longest continuously published German newspaper in Switzerland with archives going back to 1780. Speci cally, we took the highly noisy OCRed text segments, extracted pertinent entities, geolocation, as well as temporal information, linked them with the Linked Open Data cloud, and built a browser-based exploration platform. This platform enables users to interactively browse the 111906 newspaper pages published from 1910 to 1920 and containing historic events such as World War I (WWI) and the Russian Revolution. Note that Shax is neither limited to this newspaper nor to this time-period or language but exempli es the power in combining semantic technologies with an exceptional dataset.
During the past decade, many newspapers (most notably the New York Times 3
but see also [
The NZZ kindly provided us with a part of the archive covering the issues published from 1910 to 1920. This period covers historic events such as WWI and the Russian Revolution. The scanning, digitizing, and OCRing of the NZZ archive was conducted by the Fraunhofer Institute4. The dataset we use consists of 354 GB scanned PDFs and 111906 OCRed pages in XML format (one XML le per newspaper page).
One of the biggest problems when processing the data is noise. The OCR struggled with the Gothic font, which is used during the longest period in the archive, including the one under discussion. Additionally, during wartime, when printing resources were scarce, ink and paper quality decreased: some pages are simply not readable and others were printed on thin paper, causing the text of the backside to shimmer through the front side in the scans. The recognized text also contains unavoidable errors, such as di erent word-spelling due to language change. However, the names of high-frequency entities remain the same during this time period. These errors cannot a ect our results considerably.
As a result only a part of the text was correctly recognized. Using a spellchecker, we found that only 64% of the words were correctly recognized. As we assume a random distribution of the errors, our results contain insigni cant biases. 3
OCRed results
NER
Geo entities
Data Enrichment
Geo-entities with context
Data Analysis
Visualization Existing ways of dealing with historical corpora rely on Information Retrieval methods requiring a substantial amount of human e ort. Based on the assumption that the locations of important historical events are explicitly mentioned in the newspaper, we develop a purely data-driven approach that leverages Semantic Web technologies to analyze the noisy dataset (Fig. 1).
Speci cally, we rst perform the Named Entity Recognition (NER) on the
dataset with DBpedia Spotlight [
In this section, we brie y introduce the functions of our exploration platform
which is available at https:// les.i .uzh.ch/ddis/nzz-demo/WebContent/.
Function 1: Country Mentions over time A choropleth-map of Europe
was generated for each year based on the country counts. As shown in Fig. 2(a)
and 2(b), the color intensity of a country is in proportion to its counts (i.e. the
darker the color, the more the counts). By navigating through the years, the
way the colors change provides an overview of the popularity of each country.
For example, the Balkan countries are mentioned more often at the beginning
of WWI. In order to avoid biases, such as countries being mentioned extensively
due to higher geographical proximity to Switzerland or due to larger population,
the annual counts of each country were also normalized by relative distance ([
(a) 1910
(b) 1915
Function 2: Linking countries, issues and historical events By
constructing an inverse index that links the countries to the issues where they are
mentioned, users can further explore the reasons behind the change in the colors.
By clicking on a country, they can see the historical events it was involved in
Fig. 3 as well as the relevant newspaper's PDFs. The historical events presented
are systematically extracted from DBpedia [
Function 3: Entity Mentions over time A more detailed analysis of entity mentions is visualized using the word cloud (Fig. 4(a)) and trend line (Fig. 4(b)) of all geographic entities per year. It is possible to directly observe the changes in the popularity of each entity over time, as well as correlations among them. Additionally, we plot each entity as a bubble on the map based on its coordinates and number of mentions (Fig. 4(c)). Thus, users could discover the actual location of a historical event.
(a) Word Cloud 4
(b) Trend Line
(c) Bubble Map Shax is a browser-based exploration platform, which highlights the major role of Semantic Web tools in extracting entities mentioned in highly noisy newspaper archives. Moreover, it shows that interactive visualization is necessary not only in presenting the information within the newspaper in a user-friendly way, but also in discovering implicit knowledge from the corpus. In the future, we plan to apply our method on the whole 250 years of NZZ archives and try to extend it to explore other kinds of entities such as notable people.
We would like to thank the Neue Zurcher Zeitung, Thilo Haas, Thomas Scharrenbach, and Daniel Spicar.
References