Extracting data and knowledge dispersed along Portuguese old medical records is important especially for researchers dealing with historical epidemiology and health sciences. An essential task in Natural Language Processing for processing textual information is Named Entity Recognition (NER). In this paper, our main objective is to test the performance of NER systems for Portuguese for extracting information from XVIII-century medical texts, so that we can provide an annotated version of an important work of this type.
Besides the new advances in Deep Learning, Machine Learning and Neural
Models, text mining techniques ofer an important input for extracting
knowledge from large collections. These techniques can help the work of philology
researchers, historians and physicians that deal with the history of Medicine
and Epidemiology. They provide efective ways to obtain, integrate and
interpret data collected from diferent sources, and they can reduce the required time
to process information from vast textual material in a non-linear approach [
Extracting data and knowledge dispersed among old medical records is
important especially for researchers dealing with the historical epidemiology (HE)
and health sciences [
Within the scope of Information Science and the data systematization of
dispersed texts [
To process and represent this kind of textual information, an essential task
in Natural Language Processing (NLP) is Named Entity Recognition (NER).
It corresponds to the recognition and categorization of entities mentioned in a
text sample or corpus. Examples of named entities are proper names, events,
places, temporal and quantitative data, etc. These extracted entities can then
be further mapped to a knowledge base (KB), in special a Digital Humanities
KB (DHKB) [
In this paper, our main objective is to identify a NER system for Portuguese that would work for extracting information from XVIII-century medical texts4. To this end, we first create a gold standard based on modernized transcriptions of three text samples and then evaluate the systems’ named entity (NE) extraction against this gold standard. As a second step, we contrast the systems’ NE extraction performed on the modernized transcriptions against their own NE extraction from non-modernized transcriptions, which present diferent spelling and syntax. After evaluating the best NER system based on these two experiments, our next objective is to conduct a full NE extraction from an XVIII-century medical corpus.
The remainder of this paper is divided as follows: Section 2 presents previous studies related to NER and old texts; Section 3 describes the corpora, the gold standard, and the three NER systems that we used, while also explaining the procedures for two experiments; Section 4 contains the results of the experiments, presenting a quantitative and qualitative analysis; finally, in Section 5, we recap the main contributions of this paper and discuss future research. 2
Quaresma & Finatto [
Regarding previous work on NER applied to old Portuguese, we have the
work done with the Parish Memories. A digital version of these manually
tran4 An ongoing corpus composed of medical manuals is available on the
Historical Terminology section of the Textecc project http://www.ufrgs.br/textecc/
terminologia/.
scribed texts is freely available through the CIDEHUS Digital Portal5. From
this collection a named entity dataset was automatically built using machine
learning and language models [
There are studies involving NER in other historical languages. Hubkova´ et al.
[
In this section, we describe our corpus. We then go through the process of creating a gold standard from a manual annotation of NEs, we describe the process of using of-the-shelf NLP tools to recognize NEs, and finally we explain our evaluation approach. 3.1
In order to test NER in a broader set of historical texts, we collected three Portuguese text samples from the same period, written by diferent authors, from diferent domains, and using a diferent writing style.
The first sample is from a medical handbook written by Jo˜ao Curvo Semedo
(1635-1719), a Portuguese physician from Monforte, in 1707: Observac¸oens
medicas doutrinaes de cem casos gravissimos [
De hum mercador , a quem repentinamente assaltou huma dor de colica ta˜o intoleravel , que estando na ef´ sacramental para commungar , o na˜o pode fazer ; e sendo eu chamado , conheci dos grandissimos ardores ; e continuos desejos de ourinar , e vomitar , das picadas da bexiga , e do adormecimento da perna direita , que a tal dor era nephritica ; para cujo remedio appliquei hum vomitorio de tres onca¸s de agua benedicta vigorada , e tres ajudas feitas de cozimento de rim de vacca , [...]
The second text sample come from the Gazetas Manuscritas da Biblioteca
Pu´blica de E´vora6 [
Pelas cartas de Vasco Fernandes Ces´ar, se soube a noıct´ia, que aqui todos ignoravam, de que El-rei o tinhafeito Conde de Sabugosa vila junto a Viseu de que n˜ao sabemos se lhe desse senhorio.
Chegou Rodrigo Ces´ar, gordo, mas n˜ao cheio, mostrou grande desinteresse; as minas que descobriu temgrande quantidade de ouro, e se achou um gr˜ao de meia arroba, poerm´e´ mau o clima, [...]
Fr. Antˆonio Vieira (1608-1697) was born in Portugal, and his works are known
to this day for the complexity of the text and the sophistication of the
argumentation. One collection of his sermons is partially available — in a transcribed
version — at the Tycho Brahe Corpus [
These three text samples have approximately the same size (around 2000 words). We used both modernized and non-modernized transcriptions as basis for this study.
We also have 90 transcribed observations available out of the 101 total observations present in Semedo’s medical handbook, and we use this corpus for extracting NEs at the end of this study. 3.2
The first step for testing how NER tools perform in historical texts was to create a gold standard using our existing samples. The gold standard was generated using the modernized versions of our three text samples. Two linguists, authors of this paper, annotated all samples independently and exhaustively.
After the first round of annotation, we compared both lists and agreed upon the entries that would go into the gold standard. This final list has a total of 262 NEs. Table 1 shows the tags in the gold standard, along with a few examples8. In this section, we briefly describe the three NER models that we used. Two pre-trained models were taken from spaCy’s9 NER libraries, and the third one is a BERT-CRF model trained for Portuguese.
The pre-trained NER models for Portuguese ofered by spaCy come with
different language model (LM) sizes. For this study, we selected both the large and
small LMs (spaCy lg and spaCy sm, respectively). These models were trained
on the WikiNER annotation [
Souza et al. [
For the evaluation shown in the next section, we used a glossary of the NEs that considers only the surface form and frequency in each of the text samples. This 8 The complete list contains additional annotations with tags that were not taken into consideration in this study (e.g., MEASUREMENT, SYMPTOM. The complete annotation is available at https://github.com/uebelsetzer/NER_for_Portuguese_XVIII-Century_Texts/tree/main/gold 9 https://spacy.io/api/entityrecognizer 10 SpaCy uses PERSON, LOCATION, ORGANIZATION and MISCELLANEOUS. means that we analyzed whether the systems were able to extract the correct NEs from the samples, without taking the source segments into consideration.
We first compared the systems’ performance on the modernized versions of the samples against the gold standard. Then, as a second verification, we made an intra-system comparison between the extraction from the modernized and the non-modernized samples (see Section 3.1). Finally, we selected the best system to extract NEs from the larger corpus of Semedo’s work. 4
In this section we present the results of a quantitative and qualitative evaluation of the named entity recognition (NER) performed by the tools. At the end we describe the annotation of NEs based on the main corpus. 4.1
Looking at the annotations generated by each system, it becomes clear that BERT-CRF had a better precision for the annotation of NE tags. Some recurrent mistakes of this system were the annotation of locations as persons and also the annotation of any digit as a value. It also missed some values when the number was written in its expanded form (e.g., “150.000 eri´s” was correctly recognized, but “um conto de eri´s” [1.000 eri´s] was not). Regarding both spaCy models, the main errors concentrate in the incorrect annotation of tokens as NE, and also in the annotation of persons as locations (the opposite of BERT-CRF model).
When looking at the annotations across the three text samples, it became clear that the sermon by Fr. Vieira was the most complex for the spaCy systems to properly tag the instances, as many persons were wrongly tagged (16 for spaCy lg and 20 for spaCy sm out of 39). However, it was Semedo’s text the most complex in terms of proper recognition. The whole text had only 14 NEs in the gold standard, and BERT-CRF incorrectly identified 6 extra NEs, while spaCy lg got 14 extra NEs, and spaCy sm got 18 extra NEs, which is more than the number of correct NEs existing in the text. In the Gazetas, which had a total of 171 NEs, all systems worked fairly well, but BERT-CRF again showed the best performance, with 143 fully correct NEs, 14 wrong tags, and only 4 missing NEs. The spaCy lg model had 127 correctly annotated NEs, 17 wrong tags, and 14 missing annotations; while spaCy sm had the worst result, with 105 correct annotations and 42 wrong tags.
Another error that was common for the spaCy models (especially spaCy sm) was the tagging of extra tokens preceding or following a NE. When there was a capitalized word in the proximity, it was often considered as part of the NE, leading to the extraction of NEs such as “Batizou Francisco de Almada” [Baptized Francisco de Almada] and “O Marquˆes de Marialva” [The Marquess of Marialva]11. This was not a problem at all for BERT-CRF.
BERT-CRF had the best result when analyzing the extraction from a modernized version of the texts. However, since the process of modernizing these texts is similar to the process of a translation, it is unrealistic to expect all historical texts to be translated before applying a NER system to them. So we cannot use an NE extraction based on modernized versions as a parameter for old texts. As such, we still had to see how NER systems would work in a non-modernized version. This is what we explore in the next section. 4.2
In this section, we describe the results of our intra-system comparison. This was done to check how the results difered when the sample text varied, and the non-modernized transcription was used for the extraction of NEs.
Full Diferent Partial Extra Missing match tag match instances instances BERT-CRF 221 (83.71%) 24 (9.09%) 13 (4.92%) 6 (2.27%) 14 spaCy lg 189 (53.54%) 32 (9.07%) 24 (6.80%) 108 (30.59%) 15 spaCy sm 195 (63.73%) 45 (14.71%) 18 (5.88%) 48 (15.69%) 17
Table 3 shows how the systems compare among themselves when ran on modernized and on non-modernized versions of the same text. The percentages 11 In the evaluation, these were considered as partial annotations. in brackets are a direct comparison between the annotated NEs in the nonmodernized versions against the ones annotated in the modernized version (e.g. the 221 full-match instances were present in both BERT-CRF annotations with the same tag — albeit with diferent spellings — and they account for 83.71% of the annotations in the non modernized version). The missing annotation are the NEs in the modernized version that where not annotated in the non-modernized version. While here there was no judgment in terms of the extracted NEs themselves being correct or not, it was possible to see some interesting annotations. For instance, in the results from BERT-CRF, where the annotation of the nonmodernized version came up with a few more accurate results (e.g. “senhora Condessa da Atalaja D . Francisca” was a partial, and “senhora Condessa de Arcos” was not present in the annotation of the modernized text). For the spaCy models, however, we see that too many extra instances were added to the annotation, and many of these extra instances were wrong, while there was a larger number of missing instances. 4.3
From a qualitative point of view, the annotation on the non-modernized transcriptions was not as good as in the modernized versions, which was expected. In the non-modernized versions, there are spelling issues that introduce noise in the results. However, even in such adverse context, BERT-CRF annotations were still consistent, and it proved to be the most robust of the three models, as it was able to handle one of the main issues of working with old texts: the non-standard spelling.
Following our objective of retrieving information from historical medical
texts, we automatically annotated NEs in Semedo’s Observac¸oens medicas
doutrinaes de cem casos gravissimos [
We compared the performance of three of-the-shelf NER systems in Portuguese texts written in the XVII and XVIII centuries. We collected three modernized text samples from that period, annotated them with NE tags to create a gold standard, and evaluated the extractions of the three systems against this gold standard. We also compared the extractions from the modernized versions of the samples against their non-modernized versions to see how much the diferences in spelling and formatting would interfere with the systems’ performance. 12 This annotation is readily available at https://github.com/uebelsetzer/NER_for_ Portuguese_XVIII-Century_Texts .
After analyzing the results of both experiments, we concluded that BERTCRF had better performance, even when considering the original spelling of the historical texts. Both spaCy models had issues in recognizing NEs, changing the tag of many entities and adding wrong NEs, especially in the non-modernized versions of the texts. Considering these results, we used BERT-CRF to annotate a large sample of non-modernized texts extracted from Semedo’s work Observac¸oens medicas doutrinaes de cem casos gravissimos .
In future research, we intend to annotate this corpus with other types of entities that are relevant for understanding the medical practices of the XVIIXVIII century. Our plan is to then train a system to identify these entities and extract more information from similar texts from that period.
The authors would like to thank the following institutions for providing funding for this research: Expanding Excellence in England (E3) Fund, promoted by Research England; CNPq and FAPERGS - Brazil (FAPERGS - CAPES - 06/2018 internacionalizac¸˜ao - proc. 19/2551-0000718-3; CNPq 06/2019 - Productivity in research – proc. 308926/2019-6); and the Portuguese Foundation for Science and Technology (FCT), projects CEECIND/01997/2017 and UIDB/00057/2020.