In recent works, several NER models were developed to extract named entities from Portuguese Archival Finding Aids. In this paper, we are complementing the work already done by presenting a diferent NER model with a new architecture, Bidirectional Encoding Representation from Transformers (BERT). In order to do so, we used a BERT model that was pre-trained in Portuguese vocabulary and fine-tuned it to our concrete classification problem, NER. In the end, we compared the results obtained with previous architectures. In addition to this model we also developed an annotation tool that uses ML models to speed up the corpora annotation process.
In recent works, mechanisms were created to extract and identify named entities in Portuguese
archival documents. In [
In this paper, we intend to continue the study already done through the research and testing of new ML architectures, namely BERT models that use the transformers architecture. This approach presents a self-learning technique that can eficiently use the GPUs’ parallel computational power to pre-train models with hundreds of million parameters, making them have a broader and richer understanding of the language used.
During previous research, we identified another problem in creating Portuguese NER models, i.e., the lack of annotated Portuguese data to train ML models. In fact, generating annotated corpora can become challenging and time-consuming. Thus we created a smart annotator tool that uses our ML models to assist the experimenter in the annotation process. an API was developed and documented, allowing the public to use all the NER models as a service, enabling their integration into other systems.
In recent years we witnessed an increase in performance of several NLP tasks due to the new
research in ML applied to text processing. In [
Then, in [
In [
Shortly after that, new models were developed that used the transformers architecture and
achieved SOTA results in several NLP tasks, both on sentence and token level, such as BERT [
This platform was created with the intent of being complemented with new features in the future. Thus, a micro-service architecture was used, promoting looser coupling, more flexibility, and portability.
At the moment, it has two containers that correspond to the Data API and the API Gateway.
The API Gateway is implemented with an Nginx web server containing the client application, developed in Vue.js, a framework that uses reactive interfaces. The use of an API Gateway pattern makes the client less coupled to the micro-services, i. e., it does not need to know the internal structure of the server to communicate with the application. The use of this pattern means that there are no direct references between the client and the microservices, so the refactoring and maintenance of these will have a lower impact to the client. On the other hand, if a gateway server was not used, all microservices would be exposed to the public, which could lead to security issues. Then, Vue.js was used to create the client application. It has a small learning curve, so it is fairly approachable, allowing the creation of maintainable interfaces due to its reusable components mechanism that allows isolating all logic from the views. To complement this framework, Vuetify was used, which consists of a UI library that provides several pre-made reactive components.
The second container is the Data API, which is responsible for receiving, processing and
responding to HTTP requests, in this case, associated with the extraction of entities. For
this, a Node.js server was used, complemented by [
Finally, in order to deploy this platform, each micro-service was wrapped with a docker
container. These containers promote the isolation, scalability, agility and portability of each
micro-service since it is really easy to install a containerized application in any system that has
docker running. At the moment, NER@DI is hosted on the servers of the University of Minho’s
Informatics Department, at [
During the development of the NER models, several support tools were generated. Some of
these tools were selected and implemented in NER@DI in order to make them available to the
public. Most of NER@DI features are described in [
NER@DI can be used by various types of users, for example, historians wishing to extract relevant entities from archival documents or even other developers or researchers with the intent of reusing the annotated datasets in other contexts, or using NER@DI as a service in their own applications.
In order to identify and extract entities from natural text, NER@DI uses several ML architectures, such as Maximum Entropy, Convolution Neural Networks (CNN), Bidirectional Long Short-Term Memory with a Conditional Random Field decoder (Bi-LSTM-CRF) and was recently updated with a new model, Bidirectional Encoder Representations from Transformers (BERT).
In NER@DI previous versions, we used NER approaches that consisted of training models
from scratch. Now we present a BERT model, which consists of using pre-trained models
with hundreds of thousands of parameters and leveraging the knowledge acquired during their
pre-train by fine-tuning them on a specific task, in our case, NER. In this paper we will present
the BERT model in more detail. The other models were described in [
By training models from scratch, we randomly initialize the models’ weights, which means that the model only learns from the training data. However, what if, instead of creating completely new models, we re-utilize pre-trained models and fine-tune them to our concrete classification task? By doing so we could leverage the knowledge that the model obtained during its pretrain, in a huge amounts of textual data, towards our goal. In practice, we would be using the pre-trained model weights, which would provide the model with a statistical understanding of the language vocabulary, and then fine-tune the model with a task-specific classifier. That’s what BERT models do.
Bidirectional Encoder Representations from Transformers (BERT) [
In this paper we intended to process Portuguese text so we used BERT models pretrained in
Portuguese corpora. First we have BERTimbau [
Out of curiosity, at the moment, the largest existing model was created by the cooperation between Microsoft and Nvidia, published in October 2021, Megatron-Turing Natural Language set of 15 datasets consisting of a total of 339 billion tokens.
It is important to note that, by transferring all the pre-trained information to our model,
we are increasing its language knowledge, however, all the errors, noise or even bias are also
transferred. [
In NLP, this technique has been studied and widely used in several areas of token processing, such as NER. In this case, word-based tokenizers are usually used, i.e., defining a fixed size for the vocabulary and then associating an id for the most frequent words of that vocabulary.
This method has shown good results in several contexts, however, it has several limitations. Due to the fact that the number of words is limited, ML models have dificulties dealing with out of vocabulary words or even words that are rarely used. One solution for this problem is to increase the number of vocabulary words ( ), however, this would lead to other problems such as making the computational model heavier and increasing the number of rare words. On the other hand, as each distinct word has a diferent id, similar words have entirely diferent meanings, which causes information about the words’ relationship to be lost during this phase, decreasing the performance of the models.
Thus, in order to solve these limitations and increase the meaning of the numerical representations, models like BERT use a diferent technique, sub-word tokenization. This method aims to decompose rare words into sub-words, keeping the most frequent words intact. In fact, observing a given vocabulary, words like ” t o k e n i z a t i o n ” can be decomposed into two sub-words ” t o k e n ” and ” i z a t i o n ” . Using a word-based tokenizer, the words ” t o k e n ” and ” t o k e n i z a t i o n ” would have representations with entirely diferent meanings, however, with a sub-word tokenizer, by splitting the word ” t o k e n i z a t i o n ” into two sub-words, the model can learn that this word is composed of sub-words that it already knows. In this case, the model associates the word tokenization to the most frequent word ” t o k e n ” thus answering the problem of out-of-vocabulary words and maintaining a vocabulary with a reasonable size [17].
Models that use this method have increased their accuracy, mainly in the classification of unknown words [18].
For the model fine-tuning, we need annotated that so the model learns how to correctly classify
the intended named entities. In this work, we used archival finding aids datasets which we had
already annotated in [
In total, the annotated corpora contain 235875 tokens that make up 7397 phrases. As for entities, we annotated more than 36 thousand of entities in total. All the annotated corpora can be downloaded in the NER@DI platform.
After selecting our models’, we started the data loading and transformation process. First, we loaded the training and validation data into memory and then, we started to tokenization process, mapping all the tokens into their corresponding ids in the pre-trained vocabulary. The training samples must comply with the pre-trained vocabulary, having the same format and structure, which means that the tokenizer used for the pre-training must be the same for the ifne-tuning. Transformer-based models usually use sub-word tokenizers, meaning that the rarest words of the vocabulary are usually split into sub-words.
Listing 1: Text sequence encoding. # O r i g i n a l [ ’ P e l o ’ , ’ q u a l ’ , ’ foram ’ , ’ i d e n t i f i c a d o s ’ , ’ os ’ , ’ b a c h a r e i s ’ , ’ que ’ , ’ s e r i a m ’ , ’ d e s e m b a r g a d o r e s ’ , ’ da ’ , ’ R e l a ç ã o ’ , ’ e ’ , ’ Casa ’ , ’ do ’ , ’ P o r t o ’ , ’ . ’ ] # T o k e n i z e d [ ’ [ CLS ] ’ , ’ P e l o ’ , ’ q u a l ’ , ’ foram ’ , ’ i d e n t i f i c a d o s ’ , ’ os ’ , ’ ba ’ , ’ # # c h a r ’ , ’ # # e i ’ , ’ # # s ’ , ’ que ’ , ’ s e r i a m ’ , ’ desembar ’ , ’ # # g a d o r e s ’ , ’ da ’ , ’ R e l a ’ , ’ # # ção ’ , ’ e ’ , ’ Casa ’ , ’ do ’ , ’ P o r t o ’ , ’ . ’ , ’ [ SEP ] ’ ]
Even if the training data is already tokenized in IOB format, we have to tokenize it again to create the correct correspondence between the training data and the pre-train vocabulary id. In Listing 1 the tokens ”bachareis” and ”desembargadores” were divided into smaller subtokens. Consequently, since we changed the sequence vectors lengths by splitting tokens into sub-tokens, we also have to remap the labels’ vectors to ensure that each token is associated with its correct label. After that, we had to pad and truncate all the sequences according to the pre-trained maximum length parameters.
With the data processed, we started fine-tuning the model for the archival context. In order to do so, we used the Hugging Face library [19] that allows us to fine-tune several state-of-art NLP models with Keras and Pytorch API.
The fine-tuning process removes the pre-trained model head (last layer) focused on the masked language objective and replaces it with a new randomly initialized head. The idea is to create a new classifier specialized in recognizing entities from archival data, with outputs, where is equal to the number of labels.
To fine-tune the model, we only used NVIDIA GEFORCE GTX 1070 Ti GPU. Because of this, we had to adjust the batch size for the bigger models to avoid out-of-memory errors. The ifne-tuning of each model lasted from 30 minutes to 2 hours. During this process, we logged the training and evaluation loss in order to generate the model’s learning curves.
In Figure 2 we can see the loss values per epoch of the three generated models, bert-largeportuguese, bert-base-portuguese and bert-multilingual. In fact, the models start to overfit between 2 and 4 epochs. In order to retrieve the models with the highest capacity of generalisation, the Huggingface library keeps several model checkpoints during the training phase and selects the best one at the end. Only the best checkpoint will be used for the actual Named Entity Recognition.
bert-large-portuguese bert-base-portuguese train loss eval loss
In order to fine-tune and validate our NER models we divided the anottated data into two splits, 70% of the data was used for fine-tunning and 30% for validation. The validation results can be seen in the Table 1. Analyzing this Table we can say that the model which obtained the best results was BERT-large achieving an F1-score of 94.53%, followed by BERT-base with 93.19%, and BERT-multilingual with 92.29% F1-score. Despite this, we can say that all the models achieved similar results even thought they were pre-trained in diferent corpora and have diferent dimensions.
0 5
10 bert-multilingual 15 0 5 10
All Models large train base train mLingual train 15
20 large eval base-eval mLingual eval 0 5 10 15 5 10 15 20
In Figure 3 we have the models’ F1-score results per entity type. Here we can analyze and
compare the disparity between the entity labels recognition results per model (we also added
results from the models trained in [
Then, in Table 2 we validated all the models with a dataset that was not used for training in order to test the models’ generalization.
The BERT models were able to achieve better results in this test. We can analyze the importance of the model’s pre-train. In fact the BERT models were pre-trained with high amount of Portuguese textual data, giving them the advantage when processing out of vocabulary words and unseen documents.
The key to obtaining good results in this subfield is the training data quality. The closer the
training data is to the data in which this technology is intended to be used, the better the results
will be. Good training material is not always available, which creates the need to annotate text.
Normally, this activity is performed manually by an experimenter who is knowledgeable about
the domain of the documents. However, this task can become time-consuming and tedious
despite its low complexity. Thus, the idea of developing an intelligent tool to support text
annotation, ARCANO [
This annotator aims to intelligently assist the entire annotation process using ML models to try to predict named entities of the texts we want to annotate. The idea is to use a generic model to find entities in a small fraction of the target dataset. Then, by correcting the entities found, we teach the model how to classify entities in our specific domain, iteratively.
The sequence diagram in Appendix A illustrates ARCANO’s annotation workflow. Initially, the experimenter imports the entire target dataset for it to be annotated. Then, this dataset is divided into batches that will be used to train the ML model. Firstly, the first batch is sent to the server to identify and classify the entities it can find. Then, the result is sent to the client to be corrected by the experimenter. Now that we have validated training data, we can use it to refine the ML model. Thus, the correctly annotated data is sent to the server to train the model as well as the second batch so that the trained model can extract new entities from it. The more annotated batches, the higher the amount of training data, making the model learn how to classify entities even better. This makes the experimenter annotate fewer and fewer entities manually. The greater the autonomy degree of the model, the lesser the work of the experimenter. Finally, ARCANO joins all the previously annotated data into a file, enabling to export the fully annotated dataset.
This tool was used to annotate a corpus from the Arquivo Nacional da Torre do Tombo, the Arquivo de Oliveira Salazar ’s archival finding aids. In total, 71397 tokens were annotated, making more than 7000 diferent entities. In the end, the annotation process was considerably easier and faster due to the intelligent entity recognition system. 10. Conclusion and Future Work In previous works we obtained high F1-score results in NER using other ML algorithms, however, the BERT model presented a great advance in the generalization. Using the knowledge obtained during its pre-training, BERT models were able to achieve higher F1-score results in a corpus that was not present in their training data (compared to previous architectures). Furthermore, we also concluded that annotating corpora with the developed annotator was much faster than doing it manually.
For future work, it would be interesting to work on the extracted entities, for example creating disambiguation mechanisms, allowing to relate entities between diferent documents. Language models are few-shot learners, volume 2020-December, 2020. [17] HuggingFace, Summary of the tokenizers, 2021. URL: https://huggingface.co/transformers/ tokenizer_summary.html, accessed in 23/08/2021. [18] R. Sennrich, B. Haddow, A. Birch, Neural machine translation of rare words with subword units, volume 3, 2016. doi:1 0 . 1 8 6 5 3 / v 1 / p 1 6 - 1 1 6 2 . [19] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, A. Rush, Transformers: State-of-the-art natural language processing, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 2020, pp. 38–45. URL: https://aclanthology.org/2020.emnlp-demos.6. doi:1 0 . 1 8 6 5 3 / v 1 / 2 0 2 0 . e m n l p - d e m o s . 6 .