This paper describes our submission to the CLEF HIPE 2020 shared task on identifying named entities in multi-lingual historical newspapers in French, German and English. The subtasks we addressed in our submission include coarse-grained named entity recognition, entity mention detection and entity linking. For the task of named entity recognition we used an ensemble of ne-tuned BERT models; entity linking was approached by three di erent methods: (1) a simple method relying on ElasticSearch retrieval scores, (2) an approach based on contextualised text embeddings, and (3) REL, a modular entity linking system based on several state-of-the-art components.
Named entity identi cation is an important task in information extraction.
Detecting, classifying and linking named entities helps to enable semantic search,
which can be used for di erent domain applications, such as digital
humanities [
Our submission targeted three of the subtasks in HIPE: (1) coarse-grained named entity recognition (NERC), (2) end-to-end named entity linking (NEL) using a modi ed NERC task for entity mention detection, and (3) named entity linking using mention spans provided by the organisers (NEL-only). Entity mention detection in this case was a supplementary task: it was not evaluated directly within the system submissions, but served as a preparation step for NEL in the setting of bundle 2, where entity mention boundaries were not given in the test data. In all the subtasks, we only considered the literal sense of the entities.
For the rst phase of the shared task, we designed solutions for English,
German and French languages within bundle 2, which included identifying,
classifying and linking coarse-grained entities. For the second phase, bundle 5, we
focused on one language only (English) and compared our results to the
out-ofthe-box tool, Radboud Entity Linker (REL) [
Datasets and resources. The dataset provided by the CLEF HIPE organisers consists of diachronically organised digitised historical newspaper articles in English, German and French. The data is annotated using the standard inside{outside{beginning (IOB) format and presented as tab-separated values, where each row corresponds to a single token.
While validation datasets are provided for all of the three languages, training
data are only available for German and French. To provide the token classi
cation model with a su cient amount of training data for English, we used
CoNLL-03 [
Approach. We consider both NERC and entity mention detection tasks as
instances of the sequence classi cation task. For the NERC task, 5 entity types
(org, pers, prod, loc, and time) form 11 classes when annotated in the IOB
format: each of the types has its "B-" and "I-" labels corresponding to the tokens
at the beginning and inside of an entity (e.g., "B-pers" and "I-pers"), while the
"O" label marks the remaining tokens which are outside of named entities. For
mention detection, 3 classes are considered: \B-entity", \I-entity", and \O". To
perform sequence classi cation, we ne-tuned two pretrained BERT models [
To perform entity linking, we used ElasticSearch [
The submissions were evaluated with the HIPE scorer, which is provided by the shared task organisers and available on github.3 The scores achieved by our submissions on the NERC task are presented in Table 1.
The baseline provided by the HIPE organisers for the NERC-coarse task
uses a traditional CRF sequence classi cation method. The top solution for
all languages is developed by the L3i team, with extra layers added on top of
several pre-trained BERT models and trained in a multi-task learning setting to
minimize the impact of OCR-generated noise, historical spelling variations and
other challenges speci c to the data [
For the end-to-end NEL task, the HIPE baseline is AIDA-light trained on English Wikipedia. The best solution was submitted by the L3i team using entity embeddings trained on Wikipedia and Wikidata, combined with probabilistic mapping. The results achieved by our submissions are presented in Table 2 and compared with these two approaches.
For English and German, our submission scores above the baseline but far below the top solution, which is not surprising given the simplicity of our approach. For French the recall values of our submission are below the baseline. We assume that the main reason for this performance drop is due to the fact that most of the French entities could not be found in the English-only Wikidata index used in our system. We conclude that the bottleneck of our approach is the entity retrieval rather than entity mention detection. 3
3.1
Datasets and resources. Our system runs were prepared using the same HIPE
corpora as in bundle 2, with no extra training data. The algorithm designed for
the rst two runs used pre-trained contextualised Flair string embeddings [
Methods. For the rst two runs, candidate entity retrieval was done the same way as in bundle 2. To perform candidate entity ranking, we calculated cosine similarity between contextual embeddings of a sentence containing the target entity mention and a modi ed sentence, where the target entity mention is replaced with candidate entity description extracted from Wikidata. For example, if the target sentence is "We went to London for a weekend" and a candidate entity is Q84 with the label London and the description "capital and largest city of the United Kingdom", then the modi ed sentence would be "We went to capital and largest city of the United Kingdom for a weekend".
The idea behind our approach resides upon two basic assumptions: (1)
Wikidata entity descriptions are semantically similar to the corresponding entity
labels, and (2) contextualised string embeddings capture similarity between
entity descriptions and entity labels. After calculating the cosine similarity score,
it is multiplied by the Levenshtein similarity ratio between target and candidate
entity labels to prefer precise matching where possible. In the example above,
if one of the candidates is Q23306: Greater London then its score would be
multiplied by sim('London', 'Greater London') = 0.6, while the score for Q84:
London would remain the same, as sim('London', 'London') = 1. The similarity
ratio was calculated using the FuzzyWuzzy string matching library [
After using the resulting score to rank the list of candidate entities, a NIL
value is inserted to the list before the rst candidate that has a score below
threshold. We chose the threshold value of 0.7 after tuning this parameter on the
development set. For submission 2 only, we added historical spelling variations
to the step of candidate retrieval using Natas library that performs historical
normalisation via neural machine translation [
The third run was prepared using REL [
Entity disambiguation decisions are made by combining local compatibility (which includes prior importance and contextual similarity) and coherence with the other entity linking decisions in a document (global context). 3.2
Run 1: Baseline. While the results @1 are below the HIPE baseline (Table 3),
the performance @3 and @5 is better (Table 4). Similar results were achieved on
the development set: while the correct entity would often make it to the top-5 or
top-3 of the ranked candidate list, it was rarely selected by the algorithm as the
most relevant answer, and the di erence between candidate scores was usually
small. The algorithm was not directly optimised for top-1 candidate selection.
Another obstacle for the algorithm was NIL detection: as 30% of the mentions
were not linkable [
@3 @5
F P R F P R Run #1 Baseline .463 .467 .465 .552 .557 .555
Run #2 Historical .451 .463 .457 .540 .555 .548
Run 2: Historical normalisation. Adding extra candidate entities by means
of historical normalisation in the second submission has resulted in more false
positives and slightly decreased overall performance in comparison to the rst
submission. A likely explanation is that the normalisation algorithm was focusing
on infrequent historical spellings [
Run 3: REL. REL performs very well and takes the second place in the scoring table, which is rather remarkable for an out-of-the box linking system. We showed that REL provides a strong baseline for the NEL task on historical documents, demonstrating the state-of-the-art performance that can be reached without accounting for additional properties, such as OCR errors and language change. 4
Our contributions within the CLEF HIPE shared task approached coarse-grained named entity recognition (NERC) and two settings of entity linking: end-to-end and NEL-only. The results for NERC show that although ne-tuning BERT models for sequence classi cation is enough to outperform the baselines for all three languages, achieving top performance requires extra modi cations in order to deal with the challenges speci c to historical data. The NEL results show that, while using an embedding-based approach that takes historical spelling variations into account is better than relying solely on ElasticSearch retrieval scores, this approach is clearly outperformed by REL, as well as by many other solutions { mostly due to its poor performance on NIL prediction and an overwhelming number of false positives on the candidate selection step. REL, in its turn, proves very e cient in the setting of the shared task, even without speci cally addressing the challenges of the historical data.
There are several possible directions for future work considering all the
subtasks that we approached in the context of the shared task:
Entity recognition and classi cation. Some examples of the ways to achieve
improvements over the state-of-the-art sequence classi cation methods within
the given task setup include (i) performing a more extensive parameter search
for the Transformer models; (ii) ne-tuning more advanced pre-trained models
(such as RoBERTa [
Entity linking. Since the task of entity linking consists of several steps, including candidate generation and entity disambiguation, we see further opportunities for improvement on each of these steps. Firstly, candidate generation can be improved to increase recall. One of the ways to achieve this goal is to use OCR correction as a pre-processing step in the algorithm. Secondly, entity disambiguation should be improved upon in order to increase precision by decreasing the number of false positives. We consider graph-based disambiguation methods as a promising research direction. Thirdly, using entity types as features instead of only relying on mention boundaries could also improve entity disambiguation in the end-to-end setting.
This research was supported by the NWO Innovational Research Incentives Scheme Vidi (016.Vidi.189.039), the NWO Smart Culture - Big Data / Digital Humanities (314-99-301), the H2020-EU.3.4. - SOCIETAL CHALLENGES Smart, Green And Integrated Transport (814961) the Google Faculty Research Awards program. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.