Named Entity Recognition (NER) is a low-level task often used as a foundation for solving higher level NLP problems. In the context of character detection in novels, NER false negatives can be an issue as they possibly imply missing certain characters or relationships completely. In this article, we demonstrate that applying a straightforward data augmentation technique allows training a model achieving higher recall, at the cost of a certain amount of precision regarding ambiguous entities. We show that this decrease in precision can be mitigated by giving the model more local context, which resolves some of the ambiguities.
CEUR ceur-ws.org
The Character Detection task is concerned with detecting which characters appear in a text, and where. We decompose it in two subtasks: • Named Entity Recognition (NER), whose goal is to find character occurrences in a text. • Named Entity Disambiguation (NED), which maps character occurrences to their respective normalized character form.
The result of the character detection task can be used to solve other higher-level problems.
Since its output can afect the performance of other tasks relying on it, such as character network
extraction [
Dekker et al. [
We suspect that these shortcomings come from the datasets used to train the NER systems
assessed by Dekker et al. [
More precisely, there may be a mismatch in style between typical person names from the training dataset and from literary texts. This is particularly the case in the fantasy genre, where character names can have very unusual styles depending on the setting.
In order to try and fix this issue, we showcase the application of a specific data augmentation
technique, mention replacement [
The rest of this article is organized as follows: first, Section 2 provides a few examples of related data augmentation works. Then, in Section 3, we present mention replacement and the experiments we perform to assess its ability to resolve the name mismatch issue we highlighted. We discuss the results of these experiments in Section 4, and perform further analysis to shed light on the precision decrease we observe when using mention replacement. Finally, we conclude with some perspectives regarding NER performance in literary texts.
Generally speaking, data augmentation is the generation of new synthetic training examples.
Data augmentation techniques can be used to address data scarcity problems, or to increase the
diversity of the training dataset to reduce overfitting. While these techniques are ubiquitous in
image processing [
A few techniques have been used to try and fix the domain discrepancy between training
and testing datasets. Ding et al. [
B-PER O O O O O O O
mention replacement
In this section, we first describe mention replacement and how we apply it to texts of the fantasy genre. We then introduce the datasets we use to assess the performance of this data augmentation technique. Finally, we describe the specific setting of the experiment we carry to evaluate its performance, including the model we used and its training parameters.
Dai and Adel [
To perform mention replacement, we compose a list of replacement entities that do not come directly from the evaluation dataset. We scrap the entirety of the names from The Elder Scrolls series of video games that are mentioned on the Unoficial Elder Scrolls Pages , a wiki dedicated to The Elder Scrolls universe3. In total, we retrieve 22,748 first names, 4,879 last names, 647 sufixes and 37 prefixes. We combine these to form new mentions: when performing mention replacement, we randomly compose a new mention by first sampling from a set of valid forms ([first name]+[last name], [prefix]+[first name]+[last name], ...). We weight this sampling by a rough approximation of each form’s frequency. When a form has been selected, we uniformly sample a name part for each of the form’s elements.
Since we are interested in knowing the impact of the number of generated examples, we compare the performance of models trained with diferent augmentation rates. We define the augmentation rate as the ratio of examples generated over the dataset size, so an augmentation rate of 1 would mean generating as many examples as there are examples in the dataset.
Training Dataset We train our model on a modified version of CoNLL-2003 [
Evaluation dataset We use a corrected version of the subset of fantasy novels from the dataset
of Dekker et al. [
Each time one of these heuristics recommends an annotation change, we manually check if it is correct before accepting it. 2. We then consider the diferences between the existing annotations and the predictions of a
BERT model [
The dataset consists of the first chapter of 17 novels. Together, these chapters are composed
of 5,518 sentences and contain 360 unique person names. Inspired by Taillé et al. [
We fine-tune a BERT model [
Results can be seen in Figure 2. We report performance according to the CoNLL-2003
guidelines [
To try to understand the decrease in precision when using data augmentation, we perform some additional experiments. Based on our manual investigation of the false positives, we formulate two hypotheses that we want to test: ℎ1 Adding more examples with PER entities modifies the class distribution in the dataset, progressively leading to a situation of class imbalance where these entities are way more frequent than those of the other classes (LOC, ORG and MISC). As we observe that removing other entity classes from the training dataset is detrimental to PER detection performance (possibly because having more classes means the model is better at distinguishing between them), we suppose that class imbalance can be detrimental as well. precision recall
F1 0:96 0:94 0:92 0:90 0:88 0:86 0:0 0:2
0:4 0:6 Augmentation rate 0:8 1:0
To check if the decrease in precision is due to class imbalance, we test two diferent methods to inject names in the training dataset that should not sufer from the class imbalance problem: • Upsample and balance: After adding newly generated examples to the dataset, we copy some of the original examples and add them to the dataset to restore the original class distribution. Upsample and Balance Replace 0:96 0:94 0:92 0:90 0:88 0:86 0:0 precision 0:2
recall 0:4 0:6 Augmentation rate
F1 0:8 1:0 0:0 precision 0:2
recall 0:4 0:6 Augmentation rate
F1 0:8 1:0
• Replace: Instead of adding newly generated examples to the dataset, we substitute them for the original training samples.
Figure 3 shows the performance of a model trained on a dataset modified with the above augmentation methods. The upsample and balance strategy still increases recall, but does not ifx the precision issue. Meanwhile, the replace strategy still makes precision decrease, and also generally decreases performance for an augmentation rate greater than 20%.
As none of these two methods are able to fix the precision issue, we conclude that class imbalance is not a plausible cause of precision decreasing.
In order to check ℎ2, we analyze the diference in false positives between a model trained with augmentation and a model trained without it. As errors can vary between runs, we perform three training runs for each model and keep the false positives that are consistent between the runs. We then observe the set of false positives of the model trained with augmentation that are not present in the set of false positives of the model trained without augmentation. This allows us to analyze errors that are specific to our augmentation scheme.
We manually find that for 62% (58/93) of these false positives, the model attributes a PER label to some tokens even though the context alone is not suficient to know the correct label of the entity, such as in the following example from Elantris:
Raoden stood, and as he did, his eyes fell on Elantris again.
Here, the model correctly predicts that Elantris is a named entity, but gives it the class PER while Elantris is actually a city (which corresponds to the LOC class). Of course, predicting the correct class of Elantris in this example would only be a matter of luck even for a perfect model, as the given context alone is not suficient.
Overall, this result shows a bias towards the PER class, that was introduced by increasing the variety of possible names at training time. 0:95 0:90 0:85 0:80 0:75 0:94 0:950
Raoden stood, and as he did, his eyes fell on Elantris again.
Resting in the great city's shadow, Kae seemed like an insignificant village by comparison.
We define a context size of as giving the model the sentences that precede the considered sentence, as well as the sentences that follow it.
Figure 4 shows the efect of giving more context to models trained with augmentation ( The Elder Scrolls) or without it (Noaug). As can be seen, context is beneficial to both models, steadily increasing F1-score. Precision increases a lot for the model trained with augmentation, as previously ambiguous entities are now correctly labeled. Using our previously described method for error analysis, we find that only 48% (30/63) of the entities found in false positives are deemed ambiguous for a context size of 1. Meanwhile, the recall augmentation for the Noaug model can be attributed to it predicting PER entities in previously ambiguous cases. While adding more context results in greater recall for the Noaug model, it still never surpasses the recall of the The Elder Scrolls model.
We demonstrated the usage of a simple data augmentation technique to better detect fantasy names when performing named entity recognition. While this technique greatly increases model recall, precision decreases as the model sometimes does not have enough information to predict the correct class of some entities. We showed that this issue can be mitigated by giving the model more local information. However, for the same context size, our technique always results in greater recall but lower precision. In the context of character detection, we argue that this trade is beneficial: while false positives can be filtered after performing NER (automatically or manually), false negatives are not easily recovered. We also note that increasing NER context has a positive impact on recall in general.
Further work might investigate the role of context in NER. While we showed that it can be used to disambiguate some mentions, we were only interested in the local context in this paper. Nevertheless, necessary disambiguation information is not always present near the processed sentence, and might exist in more remote parts of the novel.