This paper summarizes the participation of the L3i laboratory of the University of La Rochelle in the Identifying Historical People, Places, and other Entities (HIPE) evaluation campaign of CLEF 2020. Our participation relies on two neural models, one for named entity recognition and classification (NERC) and another one for entity linking (EL). We carefully pre-processed inputs to mitigate its flaws, notably in terms of segmentation. Our submitted runs cover all languages (English, French, and German) and sub-tasks proposed in the lab: NERC, endto-end EL, and EL-only. Our submissions obtained top performance in 50 out of the 52 scoreboards proposed by the lab organizers. In further detail, out of 70 runs submitted by 13 participants, our approaches obtained the best score for all metrics in all three languages both for NERC and for end-to-end EL. It also obtained the best score for all metrics in French and German for EL-only.
EnIdentifying historical people, places and other entities is a key task in the
automatic understanding of historical newspapers. However, the use of electronic
Copyright c 2020 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0). CLEF 2020, 22-25
September 2020, Thessaloniki, Greece.
? This work has been supported by the European Union’s Horizon 2020 research and
innovation program under grants 770299 (NewsEye) and 825153 (Embeddia).
formats for storing text content is relatively new in comparison to the origins
of newspapers. For instance, in Europe, the first newspapers appeared at the
beginning of the 17th century [25]. Electronic text files started to be widely
used since the adoption of operating systems such as MS-DOS in the 1980s [
The remaining of the paper is organized as follows: Section 2 presents the task and the used corpus. Section 3 presents the global architecture of our participation, Section 4.1 presents the pre-processing strategy, while Sections 4 and 5 present individually our NERC and EL systems respectively. 2
The HIPE corpus [
The corpus uses the IOB format with hierarchical information and, provides
training, development, and test datasets for each language, except for English.
In the case of the latter, the organizers provided only partitions for development
and test. In Table 1, we present the statistics regarding the number of named
entities found in each dataset. See [
Regarding the HIPE evaluation, it consists in assessing both tasks, NERC
and EL, in terms of Precision (P), Recall (R), and F-measure (F1) at macro
and micro levels [
CAMEMBERT-LARGE Transformerlayer Transformerlayer
CRF Multitask
Transformerlayer Transformerlayer
CRF Multitask
ENGLISH BERT-Multilingual
CRF Multitask
Format Reconstruction
Wikipedia Extractpages andlinks eTmrabinedednitnitgys Copm(ep|mu)te
EL GERMAN
FRENCH
ENGLISH OCRcorrectionof mentions ELmodel Kolitsasetal.
In CLEF HIPE 2020, the NERC task consists in the recognition and
classification of entities, such as people and locations, within historical multilingual
newspapers. According to the organizers [
Due to the complexity and characteristics of both coarse-grained and
finegrained NERC sub-tasks, we propose the use of a hierarchical, multitask learning
approach consisting in a fine-tuned encoder based on Bidirectional Encoder
Representations from Transformers (BERT) [
CAMEMBERT-LARGE Transformerlayer Transformerlayer
CRF Multitask
Transformerlayer Transformerlayer
CRF Multitask
ENGLISH BERT-Multilingual
CRF Multitask
Format Reconstruction
Positional Encoding
F R C
Fine-tuned Language Model
Transformer Layer The First Lady . .
Multi-Head Attention
Add & Normalize Feed Forward Feed Forward Feed Forward
Add & Normalize O B-PERS I-PERS (a) NERC architecture for all the languages,(b) Detailed model proposed for each lanincluding the pre-processing step. guage.
We decided to use BERT not only because it is easy to fine-tune, but it has
also proved to be one of the most performing technologies in multiple NLP tasks
[
The added layers consist in a stack of Transformer blocks (Transformer encoders). As proposed in [24], this model is a deep learning architecture based on multi-head attention mechanisms with sinusoidal position embeddings 5. It is composed of a stack of identical layers. Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. A residual connection is around each of the two sub-layers, followed by layer normalization. All sub-layers in the model, as well as the embedding layers, produce outputs of dimension 512. 4.1
The HIPE dataset has three different levels of segmentation: article-level, linelevel, and newspaper-level. Figure 3 shows an example of the segmentation proposed in the HIPE dataset.
Since BERT is able to consume only a limited context of tokens (512) and a line-level context would have been too short to grasp, we segment the articles at sentence level. We reconstructed the original text, including hyphenated words, using the miscellaneous annotated column that indicates if a word is split into two or more text lines. Then, the reconstructed text was passed through Freeling 4.1 [18] which determined the boundaries of each sentence.6 4.2
For the German NERC, we chose as a pre-trained model the
bert-base-germaneuropeana. This BERT model was trained using the open-source corpus
Euro4 To increase the chances for misspelled, non-canonical, or new words to be recognized,
we enrich the vocabulary of the tokenizer with these tokens, while allowing not only
the BERT encoder but also the added Transformer layers to learn them from scratch.
5 In our implementation, we used learned absolute positional embeddings [
For the French NERC, we relied on a pre-trained CamemBERT [15] model,
specifically on the large version, camembert-large. Unlike BERT, this French
version makes use of a whole-word masking and SentencePiece tokenization [
For the English NERC, since no training data was provided, we tackled the task with two approaches. The first one was to train the NERC using the English CoNLL 2003 dataset and the bert-large-cased model. The second approach was to use the German and French training data and the pre-trained multilingual BERT model, bert-base-multilingual-cased.
We denote the number of layers (i.e., Transformer blocks) as L, the hidden size as H, and the number of self-attention heads as A. bert-base has L=12, H=768, A=12, bert-large and camembert-large, L=24, H=1024, A=16. In all the cases, the top Transformer blocks have L=1 for 1 Transf and L=2 for 2 Transf, H=128, A=12, chosen empirically.
The BERT-based encoders are fine-tuned on the task during training. For
training, we followed the selection of parameters presented in [
The experiments consider two configurations of our previously described model. The first one consists in using only the BERT encoder along with the CRF layers. The second configuration adds the Transformer blocks to the BERT encoder and the CRF layers. In Table 2, we present these experiments per language. – RUN1: for German, French, and English, the models consist in only the finetuning BERT and the CRF layers, with the difference that, for English, we use the CoNLL dataset, and the fine-tuned BERT encoder is the English bert-large-cased – RUN2: for German and French, the models consist in only the fine-tuning BERT, two stacked Transformer blocks, and the CRF layers, while for English, the model is bert-large-cased – RUN3: for English, the model is the one used in RUN1, with the difference that the training data consists of the French and German training data, and the fine-tuned BERT encoder is bert-base-multilingual-cased 7 http://www.europeana-newspapers.eu/ RUN1 RUN2 RUN3 From the results in Table 2, we can see the evidence that the BERT-based models with n Transf achieve, for both German and French languages, higher fuzzy and strict performance values than the stand-alone BERT model.
For a more qualitative analysis, we examine the number of unrecognized words by the pre-trained BERT-based models that were added to the specific tokenizers (WordPiece for BERT and SentencePiece for CamemBERT). Following this observation, we notice that there is a tendency of performance increase of around 1 percentage F1 points for the n Transf models (RUN2 for German and French). In Table 3, the highest values for all the coarse and fine metrics are presented.
In the case of English, when comparing RUN1 and RUN2, where the CoNLL 2003 dataset was used for training, with RUN3, where only HIPE German and French datasets were used, we notice that the F1 values are usually degraded by the use of modern datasets in the training process.
In summary, the methods that performed the best for the NERC task were the BERT-based models with n stacked Transformers for German and French. For English, the transfer learning from these two languages was clearly better than the models trained on modern English data. 5
Regarding EL, in CLEF HIPE 2020, the task consists in the disambiguation of named entities using two settings: – End-to-end EL: We do not have prior knowledge of the named entities. Thus, we rely on the information obtained from the NERC system. – EL-only: We have access to the ground-truth regarding named entities, i.e.
types and boundaries.
In both settings, it is necessary to take into account literal and metonymic senses. Furthermore, all the disambiguated named entities have to be linked to the Wikidata knowledge base (KB).
Our EL system is the composition and improvement of two EL approaches
(Figure 4). First, we make use of the methodology proposed by [
More precisely, our approach consists of the following four steps which will
be elaborated in the subsequent sections:
1. Building resources: the setup of a knowledge base per language.
2. Entity embeddings: the creation of entity feature representations based
on the model proposed by [
FRENCH
ENGLISH OCR correction of mentions EL model Kolitsas et al. We build a KB for English, French, and German, in order to have a richer KB following these steps: – Retrieve the last language version of the Wikipedia dump. – Extract titles and ids of Wikipedia pages. – Extract list of disambiguation pages and redirection pages. – Calculate the probability entity-map p(ejm) that analyzes how an entity e is related to a mention m based on the number of times that mention refers to that entity. 5.2
We also build a dataset to train entity embeddings for each language, in which
case, we use the methodology proposed by [
For the entity disambiguation, our model is based on Kolitsas et al.’s work [
First, for recognizing all entity mentions in a document, Kolitsas et al. proposed an empirical probabilistic entity-map8 p(ejm) to analyze each span m and select top entities e that might be referred by this mention in p(ejm).
The end-to-end EL model starts by encoding every token in the text input by
concatenating word and character embeddings that are fed into a Bidirectional
Long Short Term Memory (BiLSTM) network. This representation is used to
project mentions of this document into a shared dimensional space with the
same size as the entity embeddings. These embeddings are fixed continuous entity
representations generated separately, namely in the same manner as presented
in [
For analyzing long context dependencies of mentions, the authors used the
attention model proposed by [
To improve the accuracy of the candidates provided by the EL system, we created a filtering tool based on heuristics and the DBpedia hierarchical structure [13].
Specifically, we used the DBpedia structure to manually specify subsets that represented each named entity type. For instance, the entity type location was associated with categories such as “dbo:Location” and “dbo:Settlement”.
These categories were used to determine whether a candidate provided by the EL system had to be positioned at the bottom of the rankings. In other words, candidates that according to DBpedia did not belong to the named entity type were positioned at the bottom of the ranking.9 For those candidates matching the named entity type10, we extracted their name in the language of analysis. This name was compared with the entity entry using Fuzzy Wuzzy Weighted Ratio11. The most similar candidate to the entity entry was considered to be the most suitable candidate and was positioned at the top.
In the case of person-type entities, we requested to DBpedia their date of birth and extract the year if it was possible.12 Then, we compared the extracted year of birth with the newspaper publication year, which was provided by the organizers, plus ten years more. If the person entity was born ten years after the publication of the newspaper, we removed completely the candidate.
Furthermore, we created a heuristic that consisted in adding NIL as the last possible candidate. This was done for each named entity unless the EL system proposed candidates with a type different from the named entity one. In this last case, a NIL was inserted between the different types of candidates. For example, if the location “Paris, France” had four candidates entries of type LOC, PERS, LOC, the filter would sort them as LOC, LOC, NIL, PERS. When the EL system proposed only candidates that were different from the named entity type, the filter would position on first place a NIL. These heuristics were based on the idea that if the EL system could not provide a candidate of the same type to the named entity, we might be dealing with an entity without an entry in Wikidata.
For RUN3 in the EL-only task, which will be described in Section 5.5, we proposed as well a filter based on DBpedia along with Wikidata. The reason is 9 This included candidates that could not be found in DBpedia as well. 10 In the case the literal and metonymic entities types were discordant, we considered both types as possible. 11 github.com/seatgeek/fuzzywuzzy 12 Certain person-type entities, such as music bands, do not have a date of birth. that the former indexes only a subset of the latter. Thus, to improve the filter, we decided to use Wikidata as a backup knowledge base.
To access DBpedia13 and Wikidata14, we utilized their respective SPARQL
Endpoint query service.
Both entity embeddings and the end-to-end EL method used the pre-trained
multilingual MUSE15 word embeddings of size 300 for all languages in the dataset.
We chose the size of 50 for the character embeddings. The German and French
models were trained on the HIPE split (Table 1. As the HIPE dataset does not
contain training data for English, we trained our English model on the AIDA
dataset [
In order to overcome or reduce OCR problems, we analyzed several mention variations in order to improve the matching with candidates within the probability entity-map. More precisely, we analyze the following variations: concatenation, lowercase, no punctuation, and the Levenshtein distance between a mention and all candidate mentions within the probability table. In the metonymic sense, the approach used was to annotate the corpus consisted in copying the candidates used for the literal sense.
We implemented three configurations of our EL approach for the EL-only
task:
13 wiki.dbpedia.org/public-sparql-endpoint
14 query.wikidata.org
15 https://github.com/facebookresearch/MUSE
– RUN1: for German, French, and English, the output is composed of the
candidate entities proposed by [
We also made three configurations of our end-to-end NERC-EL architecture to recognize and disambiguate entities: – RUN1: for German, French, and English, the output is composed of entities of NERC RUN1 and the disambiguation method of EL RUN1. – RUN2: for German, French, and English, the output is composed of entities of NERC RUN2 and the disambiguation method of EL RUN1. – RUN3: for German and French, the output is composed of entities of NERC RUN1 and the disambiguation method of EL RUN2. For English, the output is composed of entities of NERC RUN3 and the disambiguation method of EL RUN1.
All runs analyze the mention variations and use the filter to select the best five candidate entities among all selected candidate entities by each run. For the EL without prior knowledge of mention types and boundaries, our EL approach depends on the performance of our NERC system to recognize and classify the type of entities in historical documents. The results on all the languages are presented in Table 4. While RUN1 achieved the best results for metonymic, RUN3 outperformed the other configurations on the literal analysis.
For EL with prior knowledge of mention types and boundaries, our system has access to the ground-truth of NERC entities, i.e. correct span and NERC type for all mentions. Table 5 shows the results. As expected, our EL system achieved better results with the ground-truth information (improvement up to 0:09 and 0:31 in the F1 values for literal and metonymic, respectively). All runs achieved similar results for all languages, with the RUN1 being slightly superior to the other runs for literal and metonymic analysis.
The use of the filter based on DBpedia and Wikidata reduced the performance of the EL system in English and French. This might be due to the increment of noise, such as names of disambiguation pages.16 Our filter analyses all candidate entities for each mention to order the list of candidates based on their NERC types and names. Since RUN1 and RUN2 provide up to five candidate entities for each mention, these runs are more likely than RUN3 to provide a NIL entry for a mention. For RUN3, the filtering process has a higher probability to find a candidate of the same named entity type as the mention and disambiguates this mention to a less frequent candidate entity in a KB. 6
For the participation of our team (L3i) to the HIPE lab at CLEF 2020, we proposed two neural-based methods for the tasks of NERC and EL. We conclude, for NERC, that the proposed models generally performed well, and that the stacked transformer-based model with a BERT fine-tuned model and additional transformer layers better learned the characteristics of the HIPE historical dataset.
For EL, our neural model combined with the filtering process analyzed the historical mentions and disambiguated them to the Wikidata KB. Combining information from Wikipedia, Wikidata, and DBpedia allowed a thorough analysis of the characteristics of the entities and helped our method to correctly disambiguate mentions in historical documents. 16 DBpedia does not index disambiguation pages. [13] Lehmann, J., Isele, R., Jakob, M., Jentzsch, A., Kontokostas, D., Mendes, P.N., Hellmann, S., Morsey, M., Kleef, P.v., Auer, S., Bizer, C.: DBpedia - a large-scale, multilingual knowledge base extracted from wikipedia 6(2), 167–195. https://doi.org/10.3233/SW-140134 [14] Makhoul, J., Kubala, F., Schwartz, R., Weischedel, R., et al.: Performance measures for information extraction. In: Proceedings of DARPA broadcast news workshop. pp. 249–252. Herndon, VA (1999) [15] Martin, L., Muller, B., Suárez, P.J.O., Dupont, Y., Romary, L., de la Clergerie, É.V., Seddah, D., Sagot, B.: Camembert: a tasty french language model. arXiv preprint arXiv:1911.03894 (2019) [16] Muller, B., Sagot, B., Seddah, D.: Enhancing bert for lexical normalization.
In: Proceedings of the 5th Workshop on Noisy User-generated Text (WNUT 2019). pp. 297–306 (2019) [17] Neudecker, C.: An open corpus for named entity recognition in historic newspapers. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16). pp. 4348–4352. European Language Resources Association (ELRA), Portorož, Slovenia (May 2016), https://www.aclweb.org/anthology/L16-1689 [18] Padró, L., Stanilovsky, E.: FreeLing 3.0: Towards Wider Multilinguality.
In: Calzolari, N., Choukri, K., Declerck, T., Doğan, M.U., Maegaard, B., Mariani, J., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12). pp. 2473–2479. ELRA, Istanbul, Turkey (May 2012) [19] Pruthi, D., Dhingra, B., Lipton, Z.C.: Combating adversarial misspellings with robust word recognition. In: 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019). pp. 5582–5591. Florence, Italy (2019) [20] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018) [21] Riedl, M., Padó, S.: A named entity recognition shootout for german. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 120–125 (2018) [22] Schweter, S., Baiter, J.: Towards robust named entity recognition for historic german. arXiv preprint arXiv:1906.07592 (2019) [23] Sun, L., Hashimoto, K., Yin, W., Asai, A., Li, J., Yu, P., Xiong, C.: Advbert: Bert is not robust on misspellings! generating nature adversarial samples on bert. arXiv preprint arXiv:2003.04985 (2020) [24] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017) [25] Weber, J.: Strassburg, 1605: The origins of the newspaper in europe. German history 24(3), 387–412 (2006) [26] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Brew, J.: Huggingface’s transformers: State-of-the-art natural language processing. ArXiv abs/1910.03771 (2019) [27] Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)