We report on our participation to the 2020 CLEF HIPE shared task as team Ehrmama, focusing on bundle 3: Named Entity Recognition and Classi cation (NERC) on coarse and ne-grained tags. Motivated by an interest to assess the added value of transfer learning for NERC on historical corpora, we propose an architecture made of two components: (i) a modular embedding layer where we combine newly trained and pre-trained embeddings, and (ii) a task-speci c BiLSTM-CRF layer. We nd that character-level embeddings, BERT, and a document-level data split are the most important factors in improving our results. We also nd that using in-domain FastText embeddings and a single-task as opposed to multi-task approach yields minor gains. Our results con rm that pre-trained language models can be bene cial for NERC on low-resourced historical corpora.
The advent of contextual language models such as Bidirectional Encoder
Representations from Transformers (BERT) [
The increasing abundance of historical text corpora o ers a compelling
opportunity to apply transfer learning. Historical texts pose a set of challenges to the
NLP and Digital Humanities (DH) communities, of which the most general and
pressing are [
Character Recognition (OCR/HTR) errors; b) linguistic change over time; c) language variety, in the absence of mainstream languages such as English. These challenges are not unique to historical texts, but they come to the forefront when dealing with them.
We participated in the CLEF HIPE as team Ehrmama, focusing on bundle 3:
NERC coarse and NERC ne-grained [
Method
Our proposed architecture is composed of two parts: an embedding layer and a
task-speci c layer, in this case for NERC. An illustration is given in Figure 1.
Several embeddings can be combined into a modular embedding layer which
we use to represent input text. We broadly distinguish between (i) pre-trained
(transferred) embeddings and (ii) newly trained embeddings. While pre-trained
embeddings can either be ne-tuned or frozen during learning, newly trained
representations are learned from scratch. Furthermore, embeddings can be
applied at di erent input granularities, including: (i) character-, (ii) sub-word- and
(iii) word-level. These are also modular, and can be used in combination.
The task-speci c layer is a Bidirectional Long Short-Term Memory, using
Conditional Random Field (Bi-LSTM-CRF) as proposed by [
Character embeddings consist of an embedding layer, followed by a bidirectional LSTM. The embedding layer's size and the LSTM hidden size are both hyperparameters with values ranging from 16 to 128 and 16 to 256 respectively. We use a character-level custom vocabularies for each language built from the training and validation data sets. 1 https://github.com/ktodorov/eval-historical-texts. BERT embeddings work on sub-word level. We use bert-base-multilingual-cased for French, bert-base-german-cased for German, and bert-base-cased for English, relying on the HuggingFace Transformers library [16]. This brings the speci c limitation of only working with sequences of 512 tokens in maximum length. As our text sequences are usually longer, we implement a sliding-window splitting of input sequences before passing them through BERT. While splitting, we keep the rst and last 5 tokens of each chunk as overlap among sequential chunks. After embedding each chunk, we then reconstruct the full input sequences by averaging the embeddings of the overlapping characters.
Newly trained embeddings work on sub-word level and their weights are randomly initialised and learned during training. We use the same vocabulary as with BERT. The size of these embeddings is a hyper-parameter and ranges between 64 and 512.
In-domain pre-trained embeddings provided by the task organisers are used for
feature extraction only (frozen). These embeddings have size of 300 and work at
the sub-word level. This model uses the FastText library [
After testing di erent alternatives, we found that the simplest and fastest way to combine these embeddings is by concatenating them, resulting in concatenated sub-word embeddings of a size equal to the sum of the embedding sizes of all enabled modules.
The Bi-LSTM-CRF uses the concatenated sub-word embeddings as its input,
and then merges the output to word level by taking the mean. Finally, the
resulting representation is pushed to a fully connected layer which then outputs
tag probabilities for each token. We tested concatenating embeddings before or
after the Bi-LSTM, or not merging at all, and found that our approach performs
best, also in accordance with previous ndings [13]. A Conditional Random Field
(CRF) [
A multi-task approach is our primary setup. We introduce additional output heads, one for each of the di erent entity types that the task aims to predict. The nal two layers of the model, namely the fully connected layer and CRF, are speci c to each entity tag type, while the rest of the architecture is shared. The individual losses for each task are summed during backpropagation. We compare using single vs multi-task approach in what follows.
Additional resources We use the Annotated Corpus for Named Entity
Recognition built on top of Groningen Meaning Bank (GMB) [
We used an NVidia GeForce 1080Ti GPU with 11GB GDDR5X memory for our experiments. 2.2
In this section, we discuss the remaining pre-processing or hyper-parameter choices which we assessed empirically. 2 https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus [accessed 2020
07-16].
Pre-processing The input data is organised into documents, and each document is split into multiple segments where usually one segment corresponds to one line in the original historical source. The input can thus be split into segments or into documents. Using segments leads to much faster convergence, while document splitting usually yields better results in our experiments. We further analyse the importance of splitting by introducing a multi-segment option which combines more than one consecutive segment. We perform a hard split and pick the maximum length of one multi-segment sequence to be the maximum length allowed by the HuggingFace Transformers library. We do this to avoid any unwanted noise. At document level we overcome this limitation by splitting documents using a sliding window approach where the rst and last 5 tokens for each split are overlapping with the previous and next splits respectively. We perform the cutting before extracting features through BERT after which we concatenate the representations back. We take the average values for the overlapping tokens. Finally, we replace all numbers with zeros, including such that contain more than one digit. Besides, we do not lowercase, nor do we remove any punctuation or other characters.
Fine-tuning vs. freezing There are two possibilities when using pre-trained models: keep them frozen or ne-tune further more. Fine-tuning the model lets us introduce two additional con guration options. The rst one is related to when to start ne-tuning. This is most often performed at the beginning of the training process and until convergence. We try a second approach where rstly the full model with frozen pre-trained weights converges. After, the pre-trained weights are ne-tuned. This is something that we also investigate but nd no di erence between the two approaches. We therefore ne-tune from the start in the reported experiments with ne-tuning enabled.
Manually crafted features Following previous work [
Weighting As it is common with NERC tasks, most of the ground truth is composed of outside or O tags. In our case, these make up for approximately 94.92%, 95.95%, and 96.5% of the total tokens for English, French and German languages respectively. To counteract tag imbalance, we test a weighted loss which we plug into the CRF layer, giving more weight to tags predicted as outside ones but are in fact part of entities, and less on tokens which are predicted as inside an entity but are actually outside. This weighted loss does not prove to be bene cial.
Hyper-parameters We assess Adam and AdamW[
All results are reported in the two scoring approaches used in the challenge | fuzzy and strict. As a reminder, fuzzy scoring works in a relaxed way, allowing fuzzy boundary matching of entities. That is if an entity is only partially recognised, e.g., if 4 out of total of 6 tokens are recognised correctly, this is still considered a successful recognition. Conversely, strict matching requires all tokens to match with exact boundary matching | in previous example this would require 6 out of 6 total tokens to be predicted correctly. For each scoring approach, we provide precision (P), recall (R) and F-score (F), all reported as micro and calculated using the original scorer, used in the competition4. We report the baseline model provided by organisers for reference, reminding the reader that the baseline model always uses a document level split. We also report the baseline model results on our English data.
We order the di erent con gurations for all languages following our ablation studies, which primarily focus on assessing the impact of transfer learning. We start with the simplest Base model which is only using newly-trained sub-word 3 https://github.com/impresso/CLEF-HIPE-2020. 4 https://github.com/impresso/CLEF-HIPE-2020-scorer. embeddings and no pre-trained information of any type. Then we continue by adding Character Embeddings (CE) which use Bi-LSTM (+CE). Due to the signi cant improvements observed by adding character embeddings, we keep them enabled in all of our next reported setups. We further report results that were achieved by adding rstly the (frozen) FastText embeddings provided by organisers (+FT), then (frozen) BERT embeddings (+BERT), and nally both. Whenever BERT is enabled, we also report runs where we disable newly trained embeddings (-newly). Eventually, we report three di erent setups where we unfreeze BERT and ne-tune them on the task at hand. Due to the long sequence lengths when working on document level, we are unable to perform ne-tuning of BERT at the document level. We therefore report the results of ne-tuning BERT only using multi-segment split. All models use the multi-task approach, except for one single-task run, which has all available embeddings enabled (single).
We start reporting results for French, in Table 2. Firstly, adding character-level embeddings and BERT consistently improves results. Better results overall are obtained with a single-task approach and using all available embeddings, including newly trained ones. A document level split, following this con guration, perform best across the board. We also see that most of our con gurations struggle on tasks with sparser annotations such as Metonymic and Nested. Furthermore,
ne-tuning BERT does not seem to improve results. Results for German, shown in Table 3, are consistent with those for French. It is worth noting that our models struggle even more on the German Metonymic and Nested tasks. For nested tags, we are not able to be predictive at all, speci cally on the multi-segment level.
For completeness, we report results for English in Table 4, limited to the Literal coarse task. For a better comparison, we provide results from two baseline models: i) results from the organisers and ii) results training the baseline model on the English dataset we use. Our models are mostly not able to perform beyond the provided baseline. This is likely due to the training data we use.
We clarify that most of these results were obtained after the task submission
deadline. For the deadline, and as reported in [
mance, despite requiring more computation resources. { A single-task approach performs better than multi-task in general, even if the di erences are often minor. It must be noted that, with our setup, six single-task runs require 2.5 times more time on average to converge than one multi-task run using a document split. Instead, six single-task runs using a multi-segment split are as fast as one multi-task run. Comparing one-to-one, a single-task run is on average twice as fast. { A document-level split of the data is in general better than a multi-segment
split, highlighting how larger windows of context are helpful with our model.
When compared to the results of the other task participants [
https://doi.org/10.2200/S00436ED1V01Y201207HLT017 13. Ruder, S.: Neural Transfer Learning for Natural Language Processing p. 329 (Feb
2019) 14. Sang, E.T.K., De Meulder, F.: Introduction to the CoNLL-2003 Shared Task:
2003, Edmonton, Canada. pp. 142{145. Morgan Kaufman Publishers (2003) 15. Sun, C., Qiu, X., Xu, Y., Huang, X.: How to Fine-Tune BERT for Text Classi cation? In: Sun, M., Huang, X., Ji, H., Liu, Z., Liu, Y. (eds.) Chinese Computational
Publishing (2019). https://doi.org/10.1007/978-3-030-32381-3 16 16. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P.,
of-the-art Natural Language Processing. arXiv:1910.03771 [cs] (Feb 2020), http: //arxiv.org/abs/1910.03771, arXiv: 1910.03771