This paper presents an extended overview of the rst edition of HIPE (Identifying Historical People, Places and other Entities), a pioneering shared task dedicated to the evaluation of named entity processing on historical newspapers in French, German and English. Since its introduction some twenty years ago, named entity (NE) processing has become an essential component of virtually any text mining application and has undergone major changes. Recently, two main trends characterise its developments: the adoption of deep learning architectures and the consideration of textual material originating from historical and cultural heritage collections. While the former opens up new opportunities, the latter introduces new challenges with heterogeneous, historical and noisy inputs. In this context, the objective of HIPE, run as part of the CLEF 2020 conference, is threefold: strengthening the robustness of existing approaches on non-standard inputs, enabling performance comparison of NE processing on historical texts, and, in the long run, fostering e cient semantic indexing of historical documents. Tasks, corpora, and results of 13 participating teams are presented. Compared to the condensed overview [31], this paper includes further details about data generation and statistics, additional information on participating systems, and the presentation of complementary results.
Recognition and identi cation of real-world entities is at the core of virtually any text mining application. As a matter of fact, referential units such as names of persons, locations and organizations underlie the semantics of texts and guide their interpretation. Around since the seminal Message Understanding Conference (MUC) evaluation cycle in the 1990s [44], named entity-related tasks have undergone major evolutions until now, from entity recognition and classi cation to entity disambiguation and linking [71, 87].
Context. Recently, two main trends characterise developments in NE
processing. First, at the technical level, the adoption of deep learning architectures and
the usage of embedded language representations greatly reshapes the eld and
opens up new research directions [
Motivation. Admittedly, NE processing tools are increasingly being used in the context of historical documents. Research activities in this domain target texts of di erent nature (e.g., museum records, state-related documents, genealogical data, historical newspapers) and di erent tasks (NE recognition and classi cation, entity linking, or both). Experiments involve di erent time periods, focus on di erent domains, and use di erent typologies. This great diversity demonstrates how many and varied the needs|and the challenges|are, but also makes performance comparison di cult, if not impossible.
Furthermore, it appears that historical texts pose new challenges to the application of NE processing [25, 83], as they do for language technologies in general [101]. First, inputs can be extremely noisy, with errors which do not resemble tweet misspellings or speech transcription hesitations, for which adapted approaches have already been devised [65, 17, 100]. Second, the language under study is mostly of earlier stage(s), which renders usual external and internal evidences less e ective (e.g., the usage of di erent naming conventions and presence of historical spelling variations) [12, 11]. Further, beside historical VIPs, texts from the past contain rare entities which have undergone signi cant changes (esp. locations) or do no longer exist, and for which adequate linguistic resources and knowledge bases are missing [48]. Finally, archives and texts from the past are not as anglophone as in today's information society, making multilingual resources and processing capacities even more essential [26, 72].
Overall, and as demonstrated by Vilain et al. [109], the transfer of NE tools from one domain to another is not straightforward, and the performance of NE tools initially developed for homogeneous texts of the immediate past are affected when applied on historical materials [104]. This echoes the proposition of Plank [85], according to whom what is considered as standard data (i.e. contemporary news genre) is more a historical coincidence than a reality: in NLP non-canonical, heterogeneous, biased and noisy data is rather the norm than the exception.
Objectives. In this context of new needs and materials emerging from the humanities, the HIPE shared task3 puts forward for the rst time the systematic evaluation of NE recognition and linking on diachronic historical newspaper material in French, German and English. In addition to the release of a multilingual, historical NE-annotated corpus, the objective of this shared task is threefold: 1. strengthening the robustness of existing approaches on non-standard inputs; 2. enabling performance comparison of NE processing on historical texts; 3. fostering e cient semantic indexing of historical documents in order to support scholarship on digitized cultural heritage collections.
The remainder of this paper is organized as follows. Section 2 brie y presents previous work on NE processing, particularly for cultural heritage domains. Sections 3 and 4 present the tasks and the material used for the evaluation. Section 5 details the evaluation metrics and the organisation of system submissions. Section 6 introduces the 13 participating systems while Section 7 presents and discusses their results. Finally, Section 8 summarizes the bene ts of the task and concludes. 2
This section brie y summarizes the evolution of NE processing and the adaptation of main approaches to named entity recognition and classi cation (NERC) and entity linking (EL) to the cultural heritage domain.
NE processing overview. Since the seminal Message Understanding
Conference series where NE recognition and classi cation was de ned for the rst
time [45], numerous research work and evaluation campaigns subsequently
developed. They re ect the complexi cation and diversi cation of NE-related tasks,
as well as the evolution of information extraction from a document-oriented to
a more entity-centric perspective. First, NER setting itself evolved, with the
extension of typologies [97, 38], the enlargement of the scope of linguistic units to
take into account (i.e. not only proper names) [
languages other than English, with e.g. CoNLL, ESTER, HAREM, Evalita and
Germeval [
NERC. Approaches to NERC over historical documents have grown with the
evolution of techniques, from symbolic systems to traditional machine learning
and, more recently, deep neural network-based approaches. Early approaches
include the crafting of rule-based systems based of nite-state grammars and
gazetteers, applied on e.g. American and Finish newspapers [52, 54], Swedish
literary classics [12], or British parliamentary proceedings [46]. Main reported
di culties relate to OCR noise, often tackled via normalization rules based on
string similarity. Then, following the (relative) greater availability of raw and
annotated historical texts, research moved away in favor of machine learning
approaches. Experiments rst consisted in applying existing models (the most
widely-spread being the Conditional Random Field-based Stanford NER
system [34]) in order to assess their potential and compare their performances
on OCRized vs. corrected texts [90], or on a diachronic basis [25]. Thereafter,
work focused on training new (CRF) models on custom material, such as
Australian or European newspapers [56, 73], or medieval charters [
Memory.
Entity Linking. Appeared most recently, the task of linking entity mentions to their corresponding referents in a KB has received much attention since the pioneering experiments of [15] and [20] with English Wikipedia. Given its many applications in e.g. information retrieval, content analysis, data integration and knowledge base population, numerous works on EL were published during the last decade, and we refer the reader to [87] and [99] for an overview and analysis of main approaches to EL up to the neural wave, and to [98] for an overview of neural approaches. Main challenges in EL are name variation (several surface forms for one entity), name ambiguity (one surface form for several entities), and absence of the entity in the KB (NIL). Up to the apparition of neural approaches, EL methods traditionally belonged to two families: text similaritybased approaches (computing the similarity between the context of the mention to link and the entity candidate description in the KB), and graph-based approaches (computing the closeness between the mention and the candidate in a graph representing information on these objects). In both cases, main EL steps include: mention detection, candidate selection, and candidate ranking. In the digital humanities context, rst experiments made use of existing, `o -the-shelf' EL systems, such as [48] working on Dutch museum documents, or [93] on Italian WWII partisans' memoirs. [35] built their own graph-based system (reden) for the disambiguation of authors' mentions in a corpus of French literary criticism, and demonstrated the gain of complementing a generic KB (here DBpedia) with a domain-speci c one (authority les from the French national library). Recent neural approaches in `mainstream' (i.e. non-historical) EL outperformed stateof-the-art results, highlighting the role and importance of contextual word and entity embeddings, together with neural similarity function [58]. To date, and to the best of our knowledge, neural-based approaches were not yet applied on historical texts and, if much remains to be done, the HIPE shared task started paving the way in this direction.
Overall, experiments with NERC and EL on historical material were so far carried out on di erent types of documents, following diverse guidelines, and evaluated in isolation. The HIPE shared task allows, for the rst time on such material, to compare performances and approaches in a systematic way. 3
The HIPE shared task includes two NE processing tasks with sub-tasks of increasing level of di culty.
{ Subtask 1.1 - NERC coarse-grained (NERC-Coarse): this task includes the recognition and classi cation of entity mentions according to high-level entity types. { Subtask 1.2 - NERC ne-grained (NERC-Fine): this task includes the recognition and classi cation of mentions according to ner-grained entity pers org prod date loc
Sub-types pers.ind pers.coll org.ent org.adm prod.media prod.doctr time.date.abs loc.adm loc.phys loc.oro loc.fac loc.add pers.ind.articleauthor org.ent.pressagency loc.adm.town loc.adm.reg loc.adm.nat loc.adm.sup loc.geo loc.hydro loc.astro loc.add.phys loc.add.elec types, as well as of nested entities and entity mention components (e.g. function, title, name).
Task 2: Named Entity Linking (EL). This task requires the linking of named entity mentions to a unique referent in a knowledge base { here Wikidata { or to a NIL node if the mention's referent is not present in the base. The entity linking task applies to non-nested mentions only and includes two settings: without and with prior knowledge of mention types and boundaries, referred to as end-to-end EL and EL only respectively. 4 4.1
The shared task corpus is composed of digitized and OCRized articles originating from Swiss, Luxembourgish and American historical newspaper collections and selected on a diachronic basis.5 5 From the Swiss National Library, the Luxembourgish National Library, and the Library of Congress (Chronicling America project), respectively. Original collections correspond to 4 Swiss and Luxembourgish titles, and a dozen for English. More details on original sources can be found in [28].
Corpus selection. The corpus was compiled based on systematic and purposive sampling. For each newspaper and language, articles were randomly sampled among articles that a) belong to the rst years of a set of prede ned decades covering the life-span of the newspaper (longest duration spans ca. 200 years), and b) have a title, have more than 50 characters, and belong to any page. For each decade, the set of selected articles was additionally manually triaged in order to keep journalistic content only. Items corresponding to feuilleton, tabular data, cross-words, weather forecasts, time-schedules, obituaries, and those with contents that a human could not even read because of extreme OCR noise were therefore removed. Di erent OCR versions of same texts are not provided, and the OCR quality of the corpus therefore corresponds to real-life setting, with variations according to digitization time and preservation state of original documents. Figure 1 hereafter shows an example of a newspaper page facsimile, a selected article thereof, and its corresponding OCR. The corpus features an overall time span of ca. 200 years, from 1798 to 2018.
Named entity tagset and guidelines. The corpus was manually annotated according to the HIPE annotation guidelines [30]. Those guidelines were derived from the Quaero annotation guide, originally designed for the annotation of named entities in French speech transcriptions and already used on historical press corpora [92, 91]. HIPE slightly recast and simpli ed this guide, considering only a subset of entity types and components, as well as of linguistic units eligible as named entities. HIPE guidelines were iteratively consolidated via the annotation of a `mini-reference' corpus|consisting of 10 content items per language|where annotation decisions were tested and di cult cases discussed6. Despite these adaptations, the HIPE corpus mostly remains compatible with Quaero-annotated data, as well as with the NewsEye project's NE data sets7, annotated with guidelines derived from HIPE.
Table 1 presents the entity types and sub-types used for annotation, which participant systems had to recognize for NERC-Coarse (types) and NERC-Fine (most ne-grained sub-types). Named entity components, annotated for the type Person only, correspond to name, title, function, qualifier and demonym. Nested entities were annotated for Person, Organization and Location (a depth of 1 was considered during the evaluation), as well as metonymic senses, producing double tags for those entities referring to something intimately associated (metonymic sense) to the concept usually associated with their name (literal sense). As per entity linking, links correspond to Wikidata QIDs8. 6 The mini-reference corpus was released during the initial phase of the shared task as sample data and is available at https://github.com/impresso/CLEF-HIPE-2020/ tree/master/data/sample-v1.0. 7 https://www.newseye.eu/ 8 The November 2019 dump used for annotation is available at https://files.ifi.
uzh.ch/cl/impresso/clef-hipe.
Fig. 1: Illustration of HIPE data, from scanned image to NE annotation and CoNLL-U-liked released material. Example taken from the Gazette de Lausanne 1908.07.01, page 2.
Annotation framework. We used INCEpTION, a web-based platform for text annotation and knowledge management [57]. Segment (d) of Figure 1 o ers two screenshots of the annotation platform, with the annotation of a person mention and its function (left), and of location mentions, with their metonymic senses (right). The functionalities of INCEpTION that proved particularly useful during the annotation campaign were: a) the support for querying against very large knowledge bases (e.g. Wikidata) with regards to EL annotation; b) the possibility of enabling the use of recommenders, which can considerably speed up the annotation process (e.g. when the very same named entity occurs multiple times within the same document); and c) the provision of an Abstract Programming Interface (API) that allows for automating certain operations, such as the bulk import/export of annotated documents. Moreover, since the shared task was one of the o cial use cases of the INCEpTION project,9 some of the annotation platform's features were developed to accommodate speci c needs of the HIPE annotation campaign, most notably the ability of displaying image segments alongside OCR transcriptions. Nevertheless, some aspects of our annotation process did not perfectly t the generic work ows implemented in INCEpTION. First, the annotation by multiple annotators of di erent layers (i.e. mentions and entity links) within the same document and, second, the validation of annotated data so as to verify, for example, that every annotated mention has either a Wikidata link or the NIL ag. Both limitations were overcome by means of scripts based on the API.
Overall, INCEpTION proved to be a mature, stable and highly con gurable annotation platform, able to support the complex work ows required by a collaborative annotation campaign such as the one undertaken for HIPE, as well as to deal with the speci ties of historical newspaper data.
Annotation di culties. The annotation campaign was carried out by the task organizers with the contribution of trilingual collaborators. Before starting annotating, each annotator was rst trained on the mini-reference corpus in order to ensure a good understanding of the guidelines. This work ow proved to be valuable in resolving instruction's imprecisions and annotator's doubts, however some unclear points persisted and new di culties appeared throughout the annotation campaign. As per NERC, major complications included, among others: a) the determination of entity boundaries in case of long functions or titles in apposition (e.g. M. Curtoys d'Anduaga, doyen du corps diplojtelfsue espagnol, et ministre plenipotentiaire pendant 50 ans )10; b) the determination of what is to be considered (or not) as an Organization: despite clear speci cation, the de nition of this class is not clear-cut and there are always groupings of some sort which prompt an interpretation as Organization, while they are not11 (e.g. Commission imperiale, les gouvernements de l'Entente, Die franzosische Regierung ); c) the quali cation of a location name as being of a region (loc.adm.reg) or of a nation (loc.adm.nat), particularly in a historical context (e.g. Savoie, Moldavia); d) the entanglement of entities, some of which have to be identi ed as nested or as components (e.g. the mix of Person, Function and Organisation in Chez Manguet et Cherbuliez imprimaires-libraires a Geneve ); e) the harmoniza
10 In these cases, we found that the annotation of components was greatly supporting the de nition of entities' scope. 11 According to our guidelines. tion of rules across languages, e.g. with German compounds (e.g. Zurichputsch, Baslerpropaganda); f) the attempt to avoid country-related biases, such as the importance and role of canton councils in Switzerland vs. in other countries; and f) the annotation of metonymy, whose interpretation is rather subjective and may di er between annotators. We had no di culties related to unreadable OCR since extremely noisy articles were ltered out beforehand, and since annotators could see original facsimiles while annotating.
With respect to Entity Linking, di culties naturally related to the historical nature of the material. If it is highly preferable to have some historical background knowledge related to the collection|which most HIPE annotators had|, it appears that this is not in itself any guarantee of a swift resolution of mention referents. As a matter of fact, most person mentions in newspapers correspond to people who enjoyed a certain popularity at a speci c time, but who are now medium- or little-known, except for experts in this or that Spanish dynasty, Swiss mountain tunnel, or local football club. As a result, historical background knowledge was mainly helpful for cases involving VIPs (e.g. Wilhelm II, Jean Jaures), and the linking of person mentions often proved to be comparable to detective work where one has to rst understand who could be the person (by cross-referencing clues), before nding its ID in Wikidata (which, to our surprise, existed quite often). As a lesson learned, curiosity, persistence and investigation skills are as important as historical knowledge. Besides the mere identi cation of who's who, another di culty was the choice of the relevant Wikidata ID for `changing' entities, often locations, whose geographical and/or administrative realities evolved through time. Here the main issue turned out to be the unequal `tracking' in the KB of the various historical statuses an entity could have had: while some countries might have an entry for each of their geopolitical phase (e.g. all French political regimes or German-related states throughout 19 and 20C), others have only a generic entry. This posed the problem of the coherency of annotation granularity, which, despite consistency checks, is not fully guaranteed in our data set, since none of the annotation|speci c or generic|is entirely wrong, and annotators did take di erent decisions in the heat of annotation. The fuzzy setting in EL evaluation mitigates this aspect (see Section 5).
Overall, besides being time consuming, the annotation of multilingual historical texts proved to be rather challenging compared to our experience on contemporary data. As future improvements we took note of detailing further some points of the guidelines, speci cally with respect to metonymy annotation and to entity linking.
Annotators' agreement. The inter-annotator agreement rates between two annotators were computed on a selection of documents (test set) using Krippendorf's [59], as provided by INCEpTION version 0.15.2. Scores correspond to, for Fr, De and En respectively: .81, .79 and .80 for NERC, .73, .69 and .78 for linking towards a QID, and .95, .94 and .90 for linking towards NIL. NERC and linking towards NIL show a good agreement between annotators. The lower scores on entity linking con rm the di culty of the task, especially in the conLg. Docs Tokens Mentions
Nested Comp.
Train Total Dev Total Test Total All Total text of historical documents. The low score observed on German (.69) is due to annotation discrepancies with respect to the linking of metonymic entities. Corpus characteristics. For each task and language|with the exception of English|the HIPE corpus was divided into training, dev and test data sets (70/15/15). English was included later in the shared task and only dev and test sets were released for this language. The overall corpus consists of 563 annotated documents, for a total of 444,596 tokens and 18,962 (linked) mentions (see Table 2 for detailed overview statistics12). With 10,923 and 6,584 mentions, French and German corpora are larger than the English one (1,455). Despite our e orts to devise a balanced sampling strategy, the diachronic distribution of mentions is not entirely uniform across languages (see Fig. 2). This is mainly due to the following factors: the temporal boundaries of data to sample from (the German corpus stops at 1950, and the English one shortly afterwards); the varying content of newspaper articles; and, nally, the di culty of sampling enough materials for certain decades due to OCR noise, such is the case with years 1850-1879 in the English corpus. 12 These statistics are slightly di erent than those presented in [31] but, after thorough double checked, are to be considered as the reference ones.
An important aspect of the HIPE corpus, and of historical newspaper data in general, is the noise generated by OCR. Annotators were asked to transcribe the surface forms of noisy mentions so as to enable studying the impact of noisy mentions on NERC and EL tasks. In the test set|where we manually veri ed the consistency of annotators' transcriptions|about 10% of all mentions contain OCR mistakes.
Together with OCR, the limited coverage of knowledge bases such as Wikidata tends to have an impact on historical NE processing, and especially on linking. In our corpus, 25.72% of all literal mentions could not be linked to a Wikidata entry (NIL entities). Interestingly, and contrary to our initial assumption, NIL entities are uniformly distributed across time periods (see Fig 3). The NIL ratio is higher for Person, Media and Organisation entities, whereas for geographic places (Location) Wikidata shows a substantial coverage (see Table 3). Date mentions were not linked as per HIPE annotation guidelines.
Corpus release. Data sets were released in IOB format with hierarchical information, in a similar fashion to CoNLL-U13, and consist of UTF-8, tab-separatedvalues les containing the necessary information for all tasks (NERC-Coarse, NERC-Fine, and EL).
Given the noisy quality of the material at hand, we chose not to apply sentence splitting nor sophisticated tokenization but, instead, to provide all necessary information to rebuild the OCR text. The tokenization applied to produce the IOB les is based on simple white space splitting, leaving all punctuation signs (including apostrophes) as separate tokens.14 Participants could choose to apply their own sentence splitting and tokenization. Alongside each article, metadata (journal, date, title, page number, image region coordinates) and IIIF links 13 https://universaldependencies.org/format.html 14 The ag `NoSpaceAfter' provides information about how to reconstruct the text. Location Total Person Total Total Date Total Media Total Organization fr de en to original page images are additionally provided when available [29]. Segment (e) of Figure 1 corresponds to an excerpt of the IOB HIPE data.
The HIPE corpus, comprising several versions of each data set for the 3
languages, is released under a CC BY-NC 4.0 license15 and is available on Zenodo16
as well as on the HIPE GitHub repository17.
In order to support participants in their system design and experiments, we
provided auxiliary resources in the form of `in-domain' word and character-level
15 https://creativecommons.org/licenses/by-nc/4.0/legalcode
16 https://zenodo.org/deposit/3706857
17 https://github.com/impresso/CLEF-HIPE-2020/tree/master/data
embeddings acquired from the same impresso newspapers titles and time
periods from which HIPE training and development sets were extracted. Those
embeddings correspond to fastText word embeddings [10] and air
contextualized string embeddings [
More speci cally, fastText embeddings came in two versions, with subword 36 character n-grams and without, and were computed after a basic pre-processing (i.e., lower-casing, replacement of digits by 0 and deletion of all tokens and punctuation signs of length 1) that also tried to imitate the tokenization of the shared task data. Flair character embeddings were computed using air 0.4.518 with a context of 250 characters, a batch size of 400-600 (depending on the GPU's memory), 1 hidden layer (size 2048), and a dropout of 0.1. Input was normalized with lower-casing, replacement of digits by 0, and of newlines by spaces; everything else was kept as in the original text (e.g. tokens of length 1). It is to be noted that the amount of training material greatly di ered between languages (20G for French and 8.5G for German taken from Swiss and Luxembourgish newspapers; 1.1G for English taken from Chronicling America material).
These embeddings are released under a CC BY-SA 4.0 license19 and are available for download.20 Contextualized character embeddings were also integrated into the air framework21. 5 5.1
NERC and EL tasks are evaluated in terms of Precision, Recall and F-measure (F1) [67]. Evaluation is done at entity level according to two metrics: micro average, with the consideration of all TP, FP, and FN22 over all documents, and macro average, with the average of document's micro gures. Our de nition of macro di ers from the usual one: averaging is done at document-level and not across entity-types, and allows to account for (historical) variance in document length and entity distribution within documents instead of overall class imbalances.
Both NERC and EL bene t from strict and fuzzy evaluation regimes. For NERC (Coarse and Fine), the strict regime corresponds to exact boundary matching and the fuzzy to overlapping boundaries. It is to be noted that in the strict regime, predicting wrong boundaries leads to a `double' punishment of one false negative (entity present in the gold standard but not predicted by the system) and one false positive (entity predicted by the system but not present in the gold standard). Although it punishes harshly, we keep this metric to be in line with CoNLL and refer to the fuzzy regime when boundaries are of less importance. 18 https://github.com/flairNLP/flair 19 https://creativecommons.org/licenses/by-sa/4.0/legalcode 20 https://files.ifi.uzh.ch/cl/siclemat/impresso/clef-hipe-2020/flair/ 21 https://github.com/flairNLP/flair 22 True positive, False positive, False negative.
The de nition of strict and fuzzy regimes di ers for entity linking. In terms of boundaries, EL is always evaluated according to overlapping boundaries in both regimes (what is of interest is the capacity to provide the correct link rather than the correct boundaries). EL strict regime considers only the system's top link prediction (NIL or QID), while the fuzzy regime expands system predictions with a set of historically related entity QIDs. For example, \Germany" QID is complemented with the QID of the more speci c \Confederation of the Rhine" entity and both are considered as valid answers. The resource allowing for such historical normalization was compiled by the task organizers for the entities of the test data sets, and is released as part of the HIPE scorer. For this regime, participants were invited to submit more than one link, and F-measure is additionally computed with cut-o s @3 and @5.
The HIPE scorer23 was provided to the participants early on and is published under the MIT license. After the evaluation phase, a complete HIPE evaluation toolkit was also released, including the data used for evaluation (HIPE corpus v1.3), the anonymized system runs submitted by participating teams, and all the recipes and resources (e.g. historical mappings) required to replicate the present evaluation24. 5.2
In order to allow the greatest exibility to participating teams as to which tasks to compete for while keeping a manageable evaluation frame, we introduced a system of task bundles o ering di erent task combinations (see Table 4). Teams were allowed to choose only one bundle per language and to submit up to 3 runs per language. Only Bundle 5 (EL only) could be selected in addition to another one; this exception was motivated by the intrinsic di erence between end-to-end linking and linking of already extracted entity mentions. Detailed information on system submission can be found in the HIPE Participation Guidelines [29].
Bundle
Tasks # teams # runs 23 https://github.com/impresso/CLEF-HIPE-2020-scorer 24 https://github.com/impresso/CLEF-HIPE-2020-eval
In this rst HIPE edition, 13 participating teams submitted a total of 75 system runs. All teams participated to NERC-Coarse, 3 to NERC-Fine, and 5 to end-to-end EL and EL only. The distribution of runs per language re ects the data, with 35 runs for French (42%), 26 for German (31%), and 22 for English (26%). Besides, six teams worked on all 3 languages. For NERC, all but 2 teams applied neural approaches, and most of them also worked with contextualized embeddings, in particular with BERT embeddings [108]. As a baseline for NERC-Coarse, we trained a traditional CRF sequence classi er [77] using basic spelling features such as a token's character pre x and su x, the casing of the initial character, and whether it is a digit. The model, released to participating teams as part of the HIPE scorer, dismisses the segmentation structure and treats any document as a single, long sentence. No baseline is provided for the NERC-Fine sub-task.
The baseline for entity linking (end-to-end EL and EL only) corresponds to AIDA-light [74], which implements the collective mapping algorithm by [47]. The wikimapper25 tool was used to map Wikipedia URLs onto Wikidata QIDs, and the end-to-end EL baseline run relied on the CRF-based NERC baseline. Given the multilingual nature of the HIPE shared task, it is worth noting that AIDA-light was trained on a 2014 dump of the English Wikipedia, therefore accounting for a generous baseline. The following system descriptions are compiled from information provided by the participants. More accurate implementation details for most of the systems are available in the participants' papers [16]. As preliminary remarks, it can be noted that for NERC many teams experimented with di erent input embeddings, often testing character, sub-word and word-level representations trained on contemporary or historical material, and often combining classical type-level word embeddings (fastText) with contextualized embeddings (BERT, Flair, ELMo). Several teams also tried to improve the (newspaper) line-based input format by reconstructing linguistically motivated sentences and uniting hyphenated words. This preprocessing step turned out to be helpful.
Cisteria, a collaboration of the Ludwig-Maximilians Universitat and the
Bayerische Staatsbibliothek Munchen from Germany, focused on NERC-coarse
for German [96]. They experimented with external and HIPE character and word
embeddings as well as several transformer-based BERT-style language models
(e.g., German Europeana BERT26), all integrated by the neural air NER
tagging framework [
Ehrmama, a liated with the University of Amsterdam, tackled coarse and ne-grained NERC for all languages [107]. They build on the bidirectional LSTMCRF architecture of [63] and introduce a multi-task approach by splitting the top layers for each entity type. Their general embedding layer combines a multitude of embeddings, on the level of characters, sub-words and words; some newly trained by the team, as well as pre-trained BERT and HIPE's in-domain fastText embeddings. They also vary the segmentation of the input: line segmentation, document segmentation as well as sub-document segmentation for long documents. Their results suggest that splitting the top layers for each entity type is not bene cial. However, the addition of various embeddings improves the performance. Using (sub-)document segmentation clearly improved results when compared to the line segmentation found in newspapers. No additional NER training material was used for German and French; for English, the Groningen Meaning Bank [14] was adapted for training.
Ertim, a liated with Inalco, Paris, applied their legacy (2010-13) NER system mXS27 [75] for contemporary texts on the historical French HIPE data without any adaptation or training [76]28. The system uses pattern mining and nonneural machine learning for NERC and their model is based on the QUAERO standard [92], which is the basis for the HIPE annotation guidelines. For EL, only the type Person was considered. The resolution is done in two steps, rst an approximate string match retrieves French Wikipedia pages, second the Wikidata item is selected whose Wikipedia article has the highest cosine similarity with the HIPE newspaper article containing the mention.
Inria, by the ALMAnaCH project team a liated at Inria, Paris, used DeLFT (Deep Learning Framework for Text)29 for NERC tagging of English and French [60]. For English, the pre-trained Ontonotes 5.0 CoNLL-2012 model was used with a BiLSTM-CRF architecture. For EL-only, the o -the-shelf named entity recognition and linking system entity- shing30 was run on the HIPE data for predicting links for the literal meaning. For English, it achieved the best performance overall, for French, it ranked second best in F1 score. This Wikipediabased system speci cally stands out with its high recall.
Irisa, by a team from IRISA, Rennes, France, focused on French NERC and EL [33]. For NERC, they improved the non-neural CRF baseline system with additional features such as context tokens, date regex match, ASCII normalization of the focus token, and the 100 most similar words from the HIPE fastText word embeddings provided by the organizers. For EL, a knowledge-base driven approach was applied to disambiguate and link the mentions of their NERC 27 https://github.com/eldams/mXS 28 The nal paper contains post-submission experiments. 29 https://github.com/kermitt2/delft 30 https://github.com/kermitt2/entity-fishing systems and the gold oracle NERC mentions [32]. Their experiments with the HIPE data revealed that collective entity linking is also bene cial for this type of texts|in contrast to linking mentions separately.
L3i, by the L3i laboratory team a liated with La Rochelle University, France, tackled all prediction tasks of HIPE for all languages and achieved almost everywhere the best results [13]. For NERC, they used a hierarchical transformerbased model [108] built upon BERT [22] in a multi-task learning setting. On top of the pre-trained BERT blocks (multilingual BERT for all languages, additionally Europeana BERT for German31 and CamemBERT for French [69]), two task-speci c transformer layers were optionally added to alleviate data sparsity issues, for instance out-of-vocabulary words, spelling variations, or OCR errors in the HIPE dataset. A state-of-the-art CRF layer was added on top in order to model the context dependencies between entity tags. For ne-tuning, relatively small batch sizes were used: 4 for German and English, 2 for French. For base BERT with a limited context of 512 sub-tokens, documents are too long and newspaper lines are too short for proper contextualization. Therefore, an important pre-processing step consisted in the reconstruction of hyphenated words and in sentence segmentation with Freeling [80]. The team submitted several runs based on di erent con gurations of their model and resources. For the two languages with in-domain training data (French and German), the results of run 1 on literal NERC-coarse without the two transformer layers were slightly lower (roughly 1 percentage point in F score) than run 2 with transformer layers. For English without in-domain training data, two options for ne-tuning were tested: a) training on monolingual CoNLL 2003 data, and b) transfer learning by training on the French and German HIPE data. Both options worked better without transformer layers, (a) was slightly better on strict boundary evaluation, and (b) on fuzzy boundary evaluation. For their EL approach, based on [58], the team built a Wikipedia/Wikidata knowledge base per language and trained entity embeddings for the most frequent entries [40]. Based on Wikipedia co-occurrence counts, a probabilistic mapping table was computed for linking mentions with entities|taking several mention variations (e.g. lowercase, Levenshtein distance) into account to improve the matching. The candidates were ltered using DBpedia and Wikidata by prioritizing those that corresponded to the named entity type. For persons, they analysed the date of birth to discard anachronistic entities. Finally, the ve best matching candidates were predicted.
Limsi, a liated with LIMSI, CNRS, Paris, France, focused on NERC-coarse for French and achieved second best results there [41]. They submitted runs from 3 model variations: a) A model based on CamemBERT [69] that jointly predicts the literal and metonymic entities by feeding into two di erent softmax layers. This model performed best on the dev set for metonymic entities. b) The model (a) with a CRF layer on top, which achieved their best results on literal tags (F1=.814 strict). c) A standard CamemBERT model that predicts concatenated literal and metonymic labels directly as a combined tag (resulting 31 https://github.com/stefan-it/europeana-bert in a larger prediction tagset). This model performed best (within Limsi's runs) on the test set for metonymic entities (F1=.667 strict).
Nlp-uqam, a liated with Universite du Quebec, Montreal, Canada, focused
on coarse NERC for French [21]. Their architecture involves a BiLSTM layer for
word-level feature extraction with a CRF layer on top for capturing label
dependencies [63], and an attention layer in between for relating di erent positions of
a sequence [108]. For their rich word representation, they integrate a
characterbased CNN approach [18] and contextualized character-based air embeddings
[
Sbb, a liated with the Berlin State Library, Berlin, focused on NERC-coarse and EL for all languages [61]. For NERC, they applied a model based on multilingual BERT embeddings, which were additionally pre-trained on OCRed historical German documents from the SBB collection and subsequently ne-tuned on various multilingual NER data sets [62]. For EL, they constructed a multilingual knowledge base from Wikipedia (WP) articles roughly resembling the categories Person, Location, and Organization. The title words of these pages were embedded by BERT and stored in a nearest neighbor lookup index. A lookup applied to a mention returns a set of linked entity candidates. The historical text segment containing the mention and sentences from WP containing a candidate are then scored by a BERT sentence comparison model. This model was trained to predict for arbitrary WP sentence pairs whether they talk about the same entity or not. A random forest classi er nally ranks the candidates based on their BERT sentence comparison scores.
SinNer, a liated with INRIA and Paris-Sorbonne University, Paris, France, focused on literal NERC-Coarse for French and German [78] and ranked third on both languages with their best neural run. The team preprocessed the linebased format into sentence-split segments. They provided two runs based on a BiLSTM ELMo architecture [82]. Run 1 is based on the classical ELMo architecture (without a CRF layer), combining type-level CNN word representations with a contextualized two-layer ELMo representation. For run 2, which performs better than their run 1 and is the one reported here, they combined modern Common Crawl-based fastText [43, 10] and pre-trained contextualized ELMo embeddings32 in a modern BiLSTM-CRF architecture [103]. They optimized hyperparameters by training each variant three times and by selecting on F1 score performance on the dev set. For run 3, they retrained SEM33 with the ofcial HIPE data sets and applied entity propagation on the document level. For German, they augmented SEM's gazetteers with location lexicons crawled from Wikipedia. The considerably lower performance of run 3 illustrates the advantage of embedding-based neural NER tagging. Ablation experiments on sentence splitting showed an improvement of 3.5 F1 percentage points on French data for their neural system of run 1. 32 [79] for French, [70] for German. 33 SEM [24] is a CRF-based tool using Wapiti [64] as its linear-chain CRF implementation.
Upb, a liated with the Politehnica University of Bucharest, Bucarest, Bulgaria, focused on literal NERC-coarse for all languages. Their BERT-based model centers around the ideas of transfer and multi-task learning as well as multilingual word embeddings. Their best performing runs combine multilingual BERT embeddings with a BiLSTM layer followed by a dense layer with local SoftMax predictions or alternatively, by adding a CRF layer on top of the BiLSTM.
Uva-ilps, a liated with the University of Amsterdam and Radboud University, The Netherlands, worked on NERC-coarse and end-to-end EL for literal senses in all languages, as well as on literal and metonymic EL-only for English [86]. They ne-tuned BERT models for token-level NERC prediction using Huggingface's transformer framework [110], using the cased multilingual BERT base model for French and German and the cased monolingual BERT base model for English. For training their English model, they used the CoNLL-03 data [106]. Their end-to-end EL approach was implemented by searching for each entity mention in the English Wikidata dump indexed by ElasticSearch34, an approach that outperformed the baseline system. The main problem there was the lack of German and French entities, although person names still could be found. For run 1 and 2 of EL-only on English, they improved the candidate entity ranking by calculating cosine similarities between the contextual embeddings of a sentence containing the target entity mention and a modi ed sentence where the mention was replaced with a candidate entity description from Wikidata. The semantic similarity scores were multiplied by relative Levenshtein similarity scores between target mention and candidate labels to prefer precise characterlevel matches. Run 2 added historical spelling variations, however, this resulted in more false positives. Run 3 used REL [51], a completely di erent neural NERC and EL system. Candidate selection in REL is twofold, 4 candidates are selected by a probabilistic model predicting entities given a mention, and 3 candidates are proposed by a model predicting entities given the context of the mention. Candidate disambiguation combines local compatibility (prior importance, contextual similarity) and global coherence with other document-level entity linking decisions. Their REL-based run 3 outperformed their runs 1 and 2 clearly.
Webis, by the Webis group a liated with the Bauhaus University Weimar,
Germany, focused on NERC-coarse for all languages. For each language, they
trained a air NERC sequence tagger [
We report results for the best run of each team and consider micro Precision, Recall and F1 scores exclusively. Results for NERC-Coarse and NERC-Fine for the three languages, both evaluation regimes and the literal and metonymic senses are presented in Table 5 and 6 respectively, while results for nested entities and entity components are presented in Table 7. Table 8 reports performances for end-to-end EL and EL only, with a cut-o @1 and Table 9 for EL only with cut-o s @3 and @5.
General observations. Neural systems with strong embedding resources clearly prevailed in HIPE NERC, beating symbolic CRF or pattern-matching based approaches by a large margin (e.g., compare baseline performance in Table 5). However, we also notice performance di erences between neural systems that rely on BiLSTMs or BERT, the latter generally performing better.
In general and not unexpectedly, we observe that the amount of available training and development data correlates with system performances. French with the largest amount of training data has better results than German, and English is worse than German (see median numbers in Table 5). The one exception is EL only where English, as a well-resourced language, seems to have the necessary tooling to also excel on non-standard, historical text material (cf. Inria results). NERC-Coarse performances show a great diversity but top results are better than expected, speci cally for French where they are almost on a par with performances on contemporary texts. Here, six teams have fuzzy F1 scores higher than .8, suggesting good prospects for entity extraction systems on historical texts, when trained with appropriate and su cient data. Fine-grained NERC with more than 12 classes is obviously more di cult than predicting only 5 categories. However, the performance drop of the best performing system L3i is relatively mild for French, 6.5 percentage points on fuzzy F1, and a little stronger for German (10.7).
The recognition of entity components shows reasonable performances and suggests that knowledge base population and/or biography reconstruction from historical texts is feasible. The same cannot be said of nested entities.
Finally, EL performances are, as expected, lower than for NERC (best F1 score in the range of .58 to .63 for EL only strict across languages), and systems' performances are as diverse (cf. Table 8). The propagation of NERC mistakes in the end-to-end setting induces lower performances, however the provision of mention boundaries does not drastically improve results (e.g. 4 percentage points for the best system on French), suggesting that being able to deal with OCR noise (provided mentions are not OCR-corrected) and NIL entities is as important as exact mention recognition. When given the possibility to provide a list of F
P F
P
P
F .657 .043 .657 .329 .367
P .801 .074 .773 .448 .357 results and not only the top one, performances of all systems increase by about 1 (cut-o @3) to 2 (cut-o @5) points, showing the importance of candidate ranking (cf. Table 9).
System-based observations. With L3i, the HIPE 2020 campaign has a clear overall winner on NERC coarse and ne, literal and metonymic entities, components, as well as EL. The one exception is EL only for English, where Inria's entity- shing system outperforms L3i. L3i is particularly convincing in terms of F1, as it consistently keeps precision and recall in good balance (even trending toward recall many times). Other systems, e.g. Inria, Ehrmama, or the baseline, typically su er from a bias towards precision. It seems that actively tackling the problems of OCR noise, word hyphenation and sentence segmentation helps to achieve better recall.
Time-based observations. In order to gauge the impact of the article's publication date on system performances, we analyze the variation of F1 scores as a function of time (see Fig. 4). The initial hypothesis here was that the older the article, the more di cult it is to extract and link the mentions it contains. In general, there does not seem to be a strong correlation between the article's publication date and F1 scores. In the speci c case of EL, this nding is in line with the uniform distribution of NIL entities across time (see Section 4).
(a) NERC-Coarse.
(b) End-to-end EL with the relaxed evaluation regime and a cuto @3. Fig. 5: Impact of OCR noise: distribution of performances across systems on entities with di erent noise level severity for NERC (a) and end-to-end EL (b). End-to-end EL F
P F
P F
P .206 .342 .257 .257 .358 .270 .173 .187 .180 .188 .203 .195 .220 .263 .239 .220 .263 .239 .023 .295 .043 .041 .527 .076 - - - - - - - - - - - .236 .402 .297 .366 .625 .462 .324 .508 .396 .384 .602 .469 .172 .200 .185 .172 .200 .185 .002 .027 .004 .008 .098 .015 .025 .136 .042 .026 .144 .044 .004 .040 .007 .004 .040 .007 P .502 .495 .498 .516 .508 .512 .420 .416 .418 .440 .435 .437 .506 .506 .506 .506 .506 .506 .025 .357 .047 .041 .580 .076 - - - - - - - - - - - .303 .446 .361 .461 .679 .549 .443 .627 .519 .515 .729 .604 .286 .480 .358 .286 .480 .358 - - - - - - - - - - - - .031 .058 .031 .031 .058 .031 .213 .312 .254 .323 .473 .384 .265 .373 .310 .331 .466 .387 .219 .280 .246 .219 .280 .246
French
R
F
P
German
R
F
P
English
R
F Irisa .530 .463 .494 - - - - - L3i .676 .686 .681 .621 .630 .626 .627 .649 .638 Sbb .624 .325 .428 .590 .332 .425 .299 .112 .163 Uva.ilps .393 .218 .281 .300 .332 .315 .285 .429 .343 @5 P R F P R F P R F Irisa .554 .497 .524 - - - - - L3i .695 .705 .700 .627 .636 .632 .651 .674 .662 Sbb .629 .328 .431 .601 .338 .432 .299 .112 .163
Uva.ilps .397 .220 .283 .311 .345 .327 .304 .458 .366 Impact of OCR noise. To assess the impact of noisy entities on the task of NERC and EL, we evaluated systems' performances on various noise levels (see Fig. 5). The level of noise is de ned as the length-normalized Levenshtein distance between the surface form of an entity and its manual transcription. There is a remarkable di erence between the performances for noisy and nonnoisy mentions on both NERC and EL. Already as little noise as 0.1 severely hurts systems' abilities to predict an entity and may cut their performance by half. Interestingly, EL also su ers badly from little noise (norm. lev. dist. > 0.0 and < 0.1), even when provided with gold NERC annotations (EL only, not shown in the plot). Slightly and medium noisy mentions (norm. lev. dist. > 0.0 and < 0.3) show a similar impact, while for highly noisy mentions, the performance deteriorates further. We can observe the greatest variations between systems at the medium noise level, suggesting that the most robust systems get their competitive advantage when dealing with medium noisiness. On the e ect of OCR noise on NERC, [104] claim that OCR errors impact more geo-political (GPE) mentions than persons or dates; in our breakdown of OCR noise impact by type, we can con rm that claim for little noise only (norm. lev. dist. > 0.0 and < 0.1), while this trend turns into the opposite for highly noisy entities. 8
From the perspective of natural language processing, the HIPE evaluation lab provided the opportunity to test the robustness of NERC and EL approaches against challenging historical material and to gain new insights with respect to domain and language adaptation. With regard to NERC, results show that it is possible to design systems capable of dealing with historical and noisy inputs, whose performances compete with those obtained on contemporary texts. Entity linking, as well as the processing of metonymy and nested entities remain challenging aspects of historical NE processing (the latter two probably due to the limited amount of annotated material). The results across the three languages present in the HIPE 2020 campaign suggest that performances mainly depend on the amount of the available in-domain training material. The evaluation study on in uence of OCR noisy on performance con rmed the expectation of degraded quality for NERC and EL if more OCR errors are present. More surprising is the fact that neither NERC nor EL performance seem to correlate with the date of publication.
From the perspective of digital humanities, the lab's outcomes will help DH practitioners in mapping state-of-the-art solutions for NE processing on historical texts, and in getting a better understanding of what is already possible as opposed to what is still challenging. Most importantly, digital scholars are in need of support to explore the large quantities of digitized text they currently have at hand, and NE processing is high on the agenda. Such processing can support research questions in various domains (e.g. history, political science, literature, historical linguistics) and knowing about their performance is crucial in order to make an informed use of the processed data.
Overall, HIPE has contributed to advance the state of the art in semantic indexing of historical newspapers and, more generally, of historical material. As future work, we intend to explore several directions for a potential second edition of HIPE: expanding the language spectrum, strengthening the already covered languages by providing more training data, considering other types of historical documents, and exploring to what extent the improvements shown in HIPE can be transferred to similar tasks in other domains, or to linking problems that require knowledge bases other than Wikidata.
This HIPE evaluation lab would not have been possible without the interest and commitment of many. We express our warmest thanks to: the Swiss newspapers NZZ and Le Temps, and the Swiss and Luxembourg national libraries for sharing part of their data in the frame of the impresso project; Camille Watter, Gerold Schneider, Emmanuel Decker and Ilaria Comes for their commitment and hard work with the construction of the data set; the inception project team for its valuable and e cient support with the annotation tool; Richard Eckart de Castillo, Clemens Neudecker, Sophie Rosset and David Smith for their encouragement and guidance as part of the HIPE advisory board; and, nally, the 13 teams who embarked in this rst HIPE edition, for their patience and scienti c involvement. HIPE is part of the research activities of the project \impresso { Media Monitoring of the Past", for which we also gratefully acknowledge the nancial support of the Swiss National Science Foundation under grant number CR-SII5 173719. [10] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5, 135{146 (2017), https://www.aclweb.org/ anthology/Q17-1010 [11] Bollmann, M.: A Large-Scale Comparison of Historical Text Normalization Systems. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 3885{ 3898. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-1389 [12] Borin, L., Kokkinakis, D., Olsson, L.J.: Naming the past: Named entity and animacy recognition in 19th century Swedish literature. In: Proceedings of the Workshop on Language Technology for Cultural Heritage Data (LaTeCH 2007). pp. 1{8 (2007) [13] Boros, E., Linhares Pontes, E., Cabrera-Diego, L.A., Hamdi, A., Moreno, J.G., Sidere, N., Doucet, A.: Robust Named Entity Recognition and Linking on Historical Multilingual Documents. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [14] Bos, J., Basile, V., Evang, K., Venhuizen, N.J., Bjerva, J.: The Groningen meaning bank. In: Handbook of Linguistic Annotation, pp. 463{496.
Springer Netherlands (2017) [15] Bunescu, R., Pasca, M.: Using encyclopedic knowledge for named entity disambiguation. In: 11th Conference of the European Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, Trento, Italy (Apr 2006), https://www.aclweb.org/anthology/ E06-1002 [16] Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.): CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR Workshop Proceedings (2020) [17] Chiron, G., Doucet, A., Coustaty, M., Visani, M., Moreux, J.P.: Impact of OCR Errors on the Use of Digital Libraries: Towards a Better Access to Information. In: Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries. pp. 249{252. JCDL '17, IEEE Press, Piscataway, NJ, USA (2017), http://dl.acm.org/citation.cfm?id=3200334.3200364 [18] Chiu, J.P., Nichols, E.: Named entity recognition with bidirectional LSTMCNNs. Transactions of the Association for Computational Linguistics 4, 357{370 (2016). https://doi.org/10.1162/tacl a 00104 [19] Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug), 2493{2537 (2011) [20] Cucerzan, S.: Large-scale named entity disambiguation based on Wikipedia data. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). pp. 708{716. Association for Computational Linguistics, Prague, Czech Republic (Jun 2007), https://www.aclweb. org/anthology/D07-1074 [21] Dekhili, G., Sadat, F.: Hybrid Statistical and Attentive Deep Neural Approach for Named Entity Recognition in Historical Newspapers. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [22] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018), http://arxiv.org/abs/1810.04805 [23] Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., Weischedel, R.: The automatic content extraction (ACE) program, tasks, data, and evaluation. In: Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC-2004). European Language Resources Association (ELRA), Lisbon, Portugal (May 2004), http://www.lrec-conf.org/proceedings/lrec2004/pdf/5.pdf [24] Dupont, Y., Dinarelli, M., Tellier, I., Lautier, C.: Structured Named Entity Recognition by Cascading CRFs. In: Intelligent Text Processing and Computational Linguistics (CICling) (2017) [25] Ehrmann, M., Colavizza, G., Rochat, Y., Kaplan, F.: Diachronic Evaluation of NER Systems on Old Newspapers. In: Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016)). pp. 97{107. Bochumer Linguistische Arbeitsberichte (2016), https://infoscience. epfl.ch/record/221391?ln=en [26] Ehrmann, M., Nouvel, D., Rosset, S.: Named Entity Resources - Overview and Outlook. In: Calzolari, N., Choukri, K., Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA), Paris, France (may 2016) [27] Ehrmann, M., Romanello, M., Bircher, S., Clematide, S.: Introducing the CLEF 2020 HIPE Shared Task: Named Entity Recognition and Linking on Historical Newspapers. In: Jose, J.M., Yilmaz, E., Magalh~aes, J., Castells, P., Ferro, N., Silva, M.J., Martins, F. (eds.) Advances in Information Retrieval. pp. 524{532. Lecture Notes in Computer Science, Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-454425 68 [28] Ehrmann, M., Romanello, M., Clematide, S., Strobel, P.B., Barman, R.: Language Resources for Historical Newspapers: the Impresso Collection. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 958{968. European Language Resources Association, Marseille, France (May 2020), https://www.aclweb.org/anthology/2020. lrec-1.121 [29] Ehrmann, M., Romanello, M., Fluckiger, A., Clematide, S.: HIPE - Shared Task Participation Guidelines (v1.1) (2020). https://doi.org/10.5281/zenodo.3677171 [30] Ehrmann, M., Romanello, M., Fluckiger, A., Clematide, S.: Impresso Named Entity Annotation Guidelines (Jan 2020). https://doi.org/10.5281/zenodo.3604227 [31] Ehrmann, M., Romanello, M., Fluckiger, A., Clematide, S.: Overview of CLEF HIPE 2020: Named Entity Recognition and Linking on Historical Newspapers. In: Arampatzis, A., Kanoulas, E., Tsikrika, T., Vrochidis, S., Joho, H., Lioma, C., Eickho , C., Neveol, A., Cappellato, L., Ferro, N. (eds.) Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the 11th International Conference of the CLEF Association (CLEF 2020). Lecture Notes in Computer Science (LNCS), vol. 12260. Springer (2020) [32] El Vaigh, C.B., Goasdoue, F., Gravier, G., Sebillot, P.: Using Knowledge Base Semantics in Context-Aware Entity Linking. In: Proceedings of the ACM Symposium on Document Engineering 2019. pp. 1{10. DocEng '19, Association for Computing Machinery, Berlin, Germany (Sep 2019). https://doi.org/10.1145/3342558.3345393 [33] El Vaigh, C.B., Le Noe-Bienvenu, G., Gravier, G., Sebillot, P.: IRISA System for Entity Detection and Linking at HIPE'20. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum.
CEUR-WS (2020) [34] Finkel, J.R., Grenager, T., Manning, C.: Incorporating non-local information into information extraction systems by Gibbs sampling. In: Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05). pp. 363{370. Association for Computational Linguistics, Ann Arbor, Michigan (Jun 2005). https://doi.org/10.3115/1219840.1219885, https://www.aclweb. org/anthology/P05-1045 [35] Frontini, F., Brando, C., Ganascia, J.G.: Semantic Web Based Named Entity Linking for Digital Humanities and Heritage Texts. In: Zucker, A., Draelants, I., Zucker, C.F., Monnin, A. (eds.) First International Workshop Semantic Web for Scienti c Heritage at the 12th ESWC 2015 Conference. Portoroz, Slovenia (2015), https://hal.archives-ouvertes.fr/ hal-01203358 [36] Galibert, O., Rosset, S., Grouin, C., Zweigenbaum, P., Quintard, L.: Extended Named Entity Annotation on OCRed Documents : From Corpus Constitution to Evaluation Campaign. In: Proceedings of the Eighth conference on International Language Resources and Evaluation. pp. 3126{ 3131. Istanbul, Turkey (2012) [37] Galibert, O., Leixa, J., Adda, G., Choukri, K., Gravier, G.: The ETAPE speech processing evaluation. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). pp. 3995{ 3999. European Language Resources Association (ELRA), Reykjavik, Iceland (May 2014), http://www.lrec-conf.org/proceedings/lrec2014/ pdf/1027_Paper.pdf [38] Galibert, O., Rosset, S., Grouin, C., Zweigenbaum, P., Quintard, L.: Structured and extended named entity evaluation in automatic speech transcriptions. In: IJCNLP. pp. 518{526 (2011) [39] Galliano, S., Geo rois, E., Mostefa, M., Choukri, K., Bonastre, J.f., Gravier, G.: The ESTER phase II evaluation campaign for the rich transcription of French broadcast news. In: Proceedings of the 9th European Conference on Speech Communication and Technology (INTERSPEECH'05. pp. 1149{1152 (2005) [40] Ganea, O.E., Hofmann, T.: Deep Joint Entity Disambiguation with Local Neural Attention. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 2619{2629 (2017) [41] Ghannay, S., Grouin, C., Lavergne, T.: Experiments from LIMSI at the French Named Entity Recognition Coarse-grained task. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [42] Goulart, R.R.V., Strube de Lima, V.L., Xavier, C.C.: A systematic review of named entity recognition in biomedical texts. Journal of the Brazilian Computer Society 17(2), 103{116 (Jun 2011). https://doi.org/10.1007/s13173-011-0031-9, https://doi.org/10.1007/ s13173-011-0031-9 [43] Grave, E., Bojanowski, P., Gupta, P., Joulin, A., Mikolov, T.: Learning word vectors for 157 languages. In: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA), Miyazaki, Japan (May 2018), https://www.aclweb.org/anthology/L18-1550 [44] Grishman, R., Sundheim, B.: Design of the MUC-6 evaluation. In: Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland (1995) [45] Grishman, R., Sundheim, B.: Message Understanding Conference - 6: A Brief History. In: Proceedings of the 16th International Conference on Computational Linguistics - Volume 1. pp. 466{471. COLING'96, Association for Computational Linguistics, Stroudsburg, PA, USA (1996). https://doi.org/10.3115/992628.992709, event-place: Copenhagen, Denmark [46] Grover, C., Givon, S., Tobin, R., Ball, J.: Named Entity Recognition for Digitised Historical Texts. In: Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08). European Language Resources Association (ELRA), Marrakech, Morocco (May 2008), http://www.lrec-conf.org/proceedings/lrec2008/pdf/ 342_paper.pdf [47] Ho art, J., Yosef, M.A., Bordino, I., Furstenau, H., Pinkal, M., Spaniol, M., Taneva, B., Thater, S., Weikum, G.: Robust disambiguation of named entities in text. In: EMNLP (2011) [48] Hooland, S.V., De Wilde, M., Verborgh, R., Steiner, T., Van de Walle, R.: Exploring entity recognition and disambiguation for cultural heritage collections. Digital Scholarship in the Humanities 30(2), 262{279 (2015). https://doi.org/10.1093/llc/fqt067 [49] Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. CoRR abs/1508.01991 (2015), http://arxiv.org/abs/1508. 01991 [50] Hubkova, H.: Named-entity recognition in Czech historical texts: Using a
CNN-BiLSTM neural network model. Ph.D. thesis (2019) [51] van Hulst, J.M., Hasibi, F., Dercksen, K., Balog, K., de Vries, A.P.: REL: An entity linker standing on the shoulders of giants. In: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. SIGIR '20, ACM (2020) [52] Jones, A., Crane, G.: The challenge of virginia banks: an evaluation of named entity analysis in a 19th-century newspaper collection. In: Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL'06). pp. 31{40. IEEE (2006) [53] Kaplan, F., di Lenardo, I.: Big Data of the Past. Frontiers in Digital Humanities 4 (2017). https://doi.org/10.3389/fdigh.2017.00012 [54] Kettunen, K., Ruokolainen, T.: Names, Right or Wrong: Named Entities in an OCRed Historical Finnish Newspaper Collection. In: Proceedings of the 2nd International Conference on Digital Access to Textual Cultural Heritage - DATeCH2017. pp. 181{186. ACM Press (2017). https://doi.org/10.1145/3078081.3078084 [55] Kim, J.D., Ohta, T., Tateisi, Y., Tsujii, J.: Genia corpus|a semantically annotated corpus for bio-textmining. Bioinformatics 19(suppl 1), i180{i182 (2003) [56] Kim, S.M., Cassidy, S.: Finding Names in Trove: Named Entity Recognition for Australian Historical Newspapers. In: Proceedings of the Australasian Language Technology Association Workshop 2015. pp. 57{65.
Parramatta, Australia (Dec 2015) [57] Klie, J.C., Bugert, M., Boullosa, B., de Castilho, R.E., Gurevych, I.: The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In: Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations. pp. 5{9 (2018) [58] Kolitsas, N., Ganea, O.E., Hofmann, T.: End-to-End Neural Entity Linking. In: Proceedings of the 22nd Conference on Computational Natural Language Learning. pp. 519{529. Association for Computational Linguistics, Brussels, Belgium (Oct 2018). https://doi.org/10.18653/v1/K18-1050 [59] Krippendor , K.: Content analysis: An introduction to its methodology.
Sage publications (1980) [60] Kristanti, T., Romary, L.: DeLFT and entity- shing: Tools for CLEF HIPE 2020 Shared Task. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [61] Labusch, K., Neudecker, C.: Named Entity Disambiguation and Linking Historic Newspaper OCR with BERT. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [62] Labusch, K., Neudecker, C., Zellhofer, D.: BERT for Named Entity Recognition in Contemporary and Historic German. In: Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers. pp. 1{9. German Society for Computational Linguistics & Language Technology, Erlangen, Germany (2019) [63] Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural Architectures for Named Entity Recognition. arXiv:1603.01360 [cs] (Mar 2016), http://arxiv.org/abs/1603.01360 [64] Lavergne, T., Cappe, O., Yvon, F.: Practical very large scale CRFs. In: Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. pp. 504{513. Association for Computational Linguistics (2010) [65] Linhares Pontes, E., Hamdi, A., Sidere, N., Doucet, A.: Impact of OCR Quality on Named Entity Linking. In: Jatowt, A., Maeda, A., Syn, S.Y. (eds.) Digital Libraries at the Crossroads of Digital Information for the Future. pp. 102{115. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-03034058-2 11 [66] Magnini, B., Cappelli, A., Tamburini, F., Bosco, C., et al., A.M.: Evaluation of natural language tools for italian: Evalita 2007. In: Proc. of the 6th International Conference on Language Resources and Evaluation (LREC'08). Marrakech, Morocco (2008) [67] Makhoul, J., Kubala, F., Schwartz, R., Weischedel, R.: Performance measures for information extraction. In: In Proceedings of DARPA Broadcast News Workshop. pp. 249{252 (1999) [68] Markert, K., Nissim, M.: SemEval-2007 task 08: Metonymy resolution at SemEval-2007. In: Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007). pp. 36{41. Association for Computational Linguistics, Prague, Czech Republic (Jun 2007), https://www. aclweb.org/anthology/S07-1007 [69] Martin, L., Muller, B., Ortiz Suarez, P.J., Dupont, Y., Romary, L., de la Clergerie, E., Seddah, D., Sagot, B.: CamemBERT: a tasty French language model. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 7203{7219. Association for Computational Linguistics, Online (Jul 2020), https://www.aclweb.org/ anthology/2020.acl-main.645 [70] May, P.: German ELMo Model (2019), https://github.com/ t-systems-on-site-services-gmbh/german-elmo-model [71] Nadeau, D., Sekine, S.: A survey of named entity recognition and classi cation. Lingvisticae Investigationes 30(1), 3{26 (2007) [72] Neudecker, C., Antonacopoulos, A.: Making Europe's Historical Newspapers Searchable. In: 2016 12th IAPR Workshop on Document Analysis Systems (DAS). pp. 405{410. IEEE, Santorini, Greece (Apr 2016). https://doi.org/10.1109/DAS.2016.83 [73] Neudecker, C., Wilms, L., Faber, W.J., van Veen, T.: Large-scale re nement of digital historic newspapers with named entity recognition. In: Proc IFLA Newspapers/GENLOC Pre-Conference Satellite Meeting (2014) [74] Nguyen, D.B., Ho art, J., Theobald, M., Weikum, G.: Aida-light: Highthroughput named-entity disambiguation. In: LDOW (2014) [75] Nouvel, D., Antoine, J.Y., Friburger, N.: Pattern Mining for Named Entity Recognition. LNCS/LNAI Series 8387i (post-proceedings LTC 2011) (2014) [76] Nouvel, D., Zagabe Seruti, J.C.: Adapting a pre-neural ML NER System to Historical Data. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [77] Okazaki, N.: CRFsuite: a fast implementation of Conditional Random
Fields (CRFs) (2007), http://www.chokkan.org/software/crfsuite/ [78] Ortiz Suarez, P.J., Dupont, Y., Lejeune, G., Tian, T.: SinNer@CLEFHIPE2020: Sinful adaptation of SotA models for Named Entity Recognition in historical French and German newspapers. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum.
CEUR-WS (2020) [79] Ortiz Suarez, P.J., Dupont, Y., Muller, B., Romary, L., Sagot, B.: Establishing a new state-of-the-art for French named entity recognition. In: Proceedings of The 12th Language Resources and Evaluation Conference. pp. 4631{4638. European Language Resources Association, Marseille, France (May 2020), https://www.aclweb.org/anthology/2020.lrec-1.569 [80] Padro, L., Stanilovsky, E.: FreeLing 3.0: Towards wider multilinguality.
In: Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12). pp. 2473{2479. European Language Resources Association (ELRA), Istanbul, Turkey (May 2012), http:// www.lrec-conf.org/proceedings/lrec2012/pdf/430_Paper.pdf [81] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: EMNLP. vol. 14, pp. 1532{43 (2014) [82] Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep Contextualized Word Representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 2227{2237. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-1202 [83] Piotrowski, M.: Natural language processing for historical texts. Synthesis
Lectures on Human Language Technologies 5(2), 1{157 (2012) [84] Piskorski, J., Ehrmann, M.: On Named Entity Recognition in Targeted Twitter Streams in Polish. In: Proceedings of the 4th Biennial International Workshop on Balto-Slavic Natural Language Processing, co-located with ACL (BSNLP 2013). pp. 84{93. So a, Bulgaria (2013) [85] Plank, B.: What to do about non-standard (or non-canonical) language in NLP. In: Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016)). Bochumer Linguistische Arbeitsberichte (2016) [86] Provatorova, V., Vakulenko, S., Kanoulas, E., Dercksen, K., van Hulst, J.M.: Named Entity Recognition and Linking on Historical Newspapers: UvA.ILPS & REL at CLEF HIPE 2020. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum. CEUR-WS (2020) [87] Rao, D., McNamee, P., Dredze, M.: Entity linking: Finding extracted entities in a knowledge base. In: Poibeau, T., Saggion, H., Piskorski, J., Yangarber, R. (eds.) Multi-source, multilingual information extraction and summarization, pp. 93{115. Springer Berlin Heidelberg, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642-28569-1, https://doi. org/10.1007/978-3-642-28569-1 [88] Riedl, M., Pado, S.: A named entity recognition shootout for german. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 120{125 (2018) [89] Ritter, A., Clark, S., Etzioni, O., et al.: Named entity recognition in tweets: an experimental study. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1524{1534 (2011) [90] Rodriquez, K.J., Bryant, M., Blanke, T., Luszczynska, M.: Comparison of named entity recognition tools for raw OCR text. In: Jancsary, J. (ed.) 11th Conference on Natural Language Processing, KONVENS 2012, Empirical Methods in Natural Language Processing, Vienna, Austria, September 19-21, 2012. Scienti c series of the O GAI, vol. 5, pp. 410{414. O GAI, Wien, O sterreich (2012), http://www.oegai.at/konvens2012/ proceedings/60_rodriquez12w/ [91] Rosset, S., Grouin, C., Fort, K., Galibert, O., Kahn, J., Zweigenbaum, P.: Structured named entities in two distinct press corpora: Contemporary broadcast news and old newspapers. In: Proceedings of the 6th Linguistic Annotation Workshop. pp. 40{48. Association for Computational Linguistics (2012) [92] Rosset, Sophie, Grouin, Cyril, Zweigenbaum, Pierre: Entites nommees structurees : guide d'annotation Quaero. NOTES et DOCUMENTS 201104, LIMSI-CNRS (2011) [93] Rovera, M., Nanni, F., Ponzetto, S.P., Goy, A.: Domain-speci c named entity disambiguation in historical memoirs. In: CEUR Workshop Proceedings. vol. 2006. RWTH (2017) [94] Santos, D., Seco, N., Cardoso, N., Vilela, R.: HAREM: An Advanced NER Evaluation Contest for Portuguese. In: Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC'06). pp. 1640{ 1643. Genoa (2006) [95] Schweter, S., Baiter, J.: Towards robust named entity recognition for historic german. arXiv preprint arXiv:1906.07592 (2019) [96] Schweter, S., Marz, L.: Triple E - E ective ensembling of embeddings and language models for NER of historical German. In: Cappellato, L., Eickho , C., Ferro, N., Neveol, A. (eds.) CLEF 2020 Working Notes. Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum.
CEUR-WS (2020) [97] Sekine, S., Sudo, K., Nobata, C.: Extended Named Entity Hierarchy. In: Proceedings of The Third International Conference on Language Resources and Evaluation (LREC). Iles Canaries , Espagne (2002) [98] Sevgili, O., Shelmanov, A., Arkhipov, M., Panchenko, A., Biemann, C.: Neural Entity Linking: A Survey of Models based on Deep Learning. arXiv:2006.00575 [cs] (May 2020), http://arxiv.org/abs/2006.00575, arXiv: 2006.00575 [99] Shen, W., Wang, J., Han, J.: Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions. IEEE Transactions on Knowledge and Data Engineering 27(2), 443{460 (Feb 2015). https://doi.org/10.1109/TKDE.2014.2327028, http://ieeexplore. ieee.org/document/6823700/ [100] Smith, D.A., Cordell, R.: A Research Agenda for Historical and Multilingual Optical Character Recognition. Tech. rep. (2018), http://hdl. handle.net/2047/D20297452 [101] Sporleder, C.: Natural Language Processing for Cultural Heritage Domains. Language and Linguistics Compass 4(9), 750{768 (2010). https://doi.org/10.1111/j.1749-818X.2010.00230.x [102] Sprugnoli, R.: Arretium or Arezzo? A Neural Approach to the Identi cation of Place Names in Historical Texts. In: Cabrio, E., Mazzei, A., Tamburini, F. (eds.) Proceedings of the Fifth Italian Conference on Computational Linguistics (CLiC-it 2018), Torino, Italy, December 1012, 2018. CEUR Workshop Proceedings, vol. 2253. CEUR-WS.org (2018), http://ceur-ws.org/Vol-2253/paper26.pdf [103] Strakova, J., Straka, M., Hajic, J.: Neural architectures for nested NER through linearization. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 5326{5331. Association for Computational Linguistics, Florence, Italy (Jul 2019), https: //www.aclweb.org/anthology/P19-1527 [104] van Strien, D., Beelen, K., Ardanuy, M., Hosseini, K., McGillivray, B., Colavizza, G.: Assessing the Impact of OCR Quality on Downstream NLP Tasks. In: Proceedings of the 12th International Conference on Agents and Arti cial Intelligence. pp. 484{496. SCITEPRESS - Science and Technology Publications, Valletta, Malta (2020). https://doi.org/10.5220/0009169004840496 [105] Terras, M.: The Rise of Digitization. In: Rikowski, R. (ed.) Digitisation Perspectives, pp. 3{20. SensePublishers, Rotterdam (2011). https://doi.org/10.1007/978-94-6091-299-3 1 [106] Tjong Kim Sang, E.F., De Meulder, F.: Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In: Proceedings of the seventh conference on Natural language learning at HLT