Most work on named entity recognition (NER) focuses solely on English. Through the use of training data augmentation via machine translation (MT), multilingual NER can become a powerful tool for information extraction in multilingual contexts. In this paper, we augment NER data from culinary recipe ingredient lists by means of MT and word alignment (WA), following two approaches: (i) translating each entity separately, while taking into account the full context of the list and (ii) translating the whole list of ingredients and then aligning entities using three types of WA models: Giza++, Fast Align, and BERT, fine-tuned using a novel entity-shuffling approach. We depart from English data and produce Italian versions via MT, span-annotated with the entities projected from English. Then, we use the data produced by the two approaches to train mono-and multilingual NER BERT models. We test the performance of the WA and NER models on an annotated dataset of ingredient lists, partially out-of-domain compared to the training data. The results show that shuffling entities leads to better BERT aligner models. The higher quality NER data created by these models enables NER models to achieve better results, with multilingual models reaching performances equal to or greater than their monolingual counterparts.
Named entity recognition (NER) is a sequence labeling task with a long history of works mainly focusing on the recognition of entities such as people, locations, and organizations. Multilingual NER has also attracted research efforts, with recent SemEval campaigns including tasks on multilingual complex NER (MultiCoNER) [1,2]. Despite its popularity and various mono-and multilingual NER datasets being available, specific domains such as the culinary one likely require new annotated data. In addition, NER is often the first step in information extraction for knowledge graph construction and, to the best of our knowledge, all literature in the domain of cuisine on this topic solely focuses on English data [3,4,5,6,7]. Therefore we argue that, given cuisine's multicultural nature, more research in this direction is warranted.
Entity label projection [8] aims to address this scarcity by automating the data generation process for NER. This task consists in taking the labels associated with spans from a source text and automatically applying them to its translation in another language, i.e. the target text. Through this task, we attempt to find an efficient automatic way of developing models for entity projection across languages to produce high-quality multilingual data for recipe Named Entities (r-NE) [4]. Departing from an English-language dataset containing ingredients from culinary recipes, annotated at the span level with entity category labels, we first rely on a MT engine to translate each source entity 𝑠𝑖 individually into Italian, while keeping the full context into account. This results in a first entity-wise (EW) translated EN-IT-ES dataset where entities are linked across languages. 1 Using these synthetic alignments, we train BERT models to align source and target entities, shuffling the latter to prevent the model from learning to simply predict the original entity order. We then test the models on two novel entity alignment datasets, partially out-of-domain compared to the training data, e.g., as regards the used food products, units of measure, and cooking processes. As baselines to evaluate the BERT alignment models, we use Giza++ [9] and Fast Align [10], two statistical word alignment (WA) models. In order to produce higherquality r-NE data, we translate the ingredient lists across their whole length, predicting target entity spans with the best BERT models from the previous step, along with the baseline models. We thus obtain various sentencewise (SW) translated datasets in Italian, trading some alignment accuracy for better translations.
Both types of training data, EW and SW, are then used to fine-tune mono-and multilingual BERT NER models on the task of recognizing entities in food recipes. The models are trained on various combinations of mono-and multilingual data and are tested on the entity annotations from the two aforementioned novel testing datasets.
Our contribution is three-fold: (i) We show the efficacy of fine-tuning alignment models by shuffling entities in contexts where most of the information depends on the presence of lexical items rather than the dependencies linking them. (ii) We showcase the performance delta between mono-and multilingual NER models when finetuning on the synthetic data produced by our alignment. These models can be used to label large datasets in multiple languages at a finer granularity level compared to currently available monolingual resources. (iii) We release code and data to produce data in multiple languages. 2The rest of the paper is structured as follows. Section 2 presents relevant past research on the subjects of cross-lingual entity alignment and recognition. Section 3 introduces the datasets and corpora used in the experiments, along with their annotation process. Section 4 presents architecture, training, and evaluation details for the models comprising our pipeline. Section 5 discusses the conducted experiments and their results. Finally, Section 6 summarizes the paper and draws conclusions. Appendix A shows further results including Spanish. Appendix B presents statistics and gives insight on the additional training data used. Appendix C lists information on the computational requirements.
Word alignment was first approached for statistical MT, with models such as IBM 1-5 [11], used in well-known implementations such as Giza++ and Fast Align. With the advent of Transformers [12] and the BERT model [13], this task has been approached by employing both question answering [14] and token classification [15] models, trained on freely available resources, such as XL-WA [16].
A number of past works have studied label projection following a range of approaches. Jain et al. [8] project PER, ORG, LOC and MISC labels (person, organization, location, and miscellaneous) by translating sentences and then finding potential matches using glossaries. Fei et al. [17] align words using Fast Align and use POS tagging to enhance data for semantic role labeling. García-Ferrero et al. [18] use the AWESoME word alignment model [19] to align machine-translated data from NER datasets in seven languages. Li et al. [15] fine-tune a NER model on English PER, ORG, LOC, MISC data from CoNLL2003 [20] to infer on the source portion of parallel Opus corpora [21] with the aim of creating silver NER data. Subsequently, they train an XLM RoBERTa alignment model by using Wikipedia articles and project the labels on the target portion of the parallel corpus, which they use to train a target-language NER model.
NER can also be approached with large language models (LLM) [22,23,24] by prompting them to extract entities from a given text. For example, PromptNER [25] uses chain of thought [26] along with a list of entity definitions to prompt a variety of LLMs, obtaining results on par with SOTA supervised NER systems. Similarly, [27] use in-context learning [28] to evaluate GPT-3 [22] for NER on the CoNLL2003 [20] and OntoNotes5.0 [29] datasets by using retrieval-augmented generation [30] and comparing the results to BERT and models based on graph neural networks [31].
With regard to data specific to the culinary domain, many English-language resources exist in various forms. RecipeDB [32] is an ontology comprising 118 𝑘 web recipes which can be used to relate foods and cooking processes to taste profiles and health data. FoodOn [33] is a "farm-to-fork" ontology which provides a structure of relationships between food products across the whole industrial supply chain. Bridging the gap between ontologies and NER datasets, FoodKG [34] is a knowledge graph which can be used to find ingredient substitutions based on dietary health requirements. It is built by leveraging FoodOn and Recipe1M+ [35], a dataset originally intended for learning joint text/image embeddings on over 1 𝑀 culinary recipes. Expanding on Recipe1M+, Bień et al. [36] construct RecipeNLG, comprising more than 2 𝑀 recipes. It is the biggest food NER dataset to date, but its granularity stops at the sole food product names. More fine-grained silver labels are obtained by Komariah et al. [37], who propose a new methodology to extract entities from AllRecipes. 3 Doing so, they construct FINER, a dataset comprising 64 𝑘 recipes with labels predicted by what the authors refer to as a "semi-supervised multi-model prediction technique. " The dataset also contains recipe tags such as vegetarian and vegan, which can be useful for training recipe classifiers. Leveraging RecipeDB [32], a large-scale structured corpus of recipes, [38] generate a synthetic dataset of augmented ingredient phrases and compare the NER performance of various rule-based and neural models.
Despite the wide availability of English-language resources in the culinary domain, other languages are largely understudied. To the best of our knowledge, the only study to approach this domain in a multilingual setting was conducted by Radu et al. [39], who obtain NER tags automatically in English, German, and French by using a regex-based tagger. Our work aims to partially address this gap in past research by focusing on Italian.
The entity alignment data used for training is generated through MT starting from TASTEset [40], a dataset comprising ingredient lists from 700 food recipes, annotated at the span level. We use TASTEset because it is human-curated and its annotations are fine-grained. We translate each entity one by one with DeepL, 4 concurrently feeding the whole ingredient list and the single entity as two separate inputs. This provides DeepL with context, improving translation quality and retaining the start and end span indexes in the target text by simply concatenating each translated entity. To the best of our knowledge, DeepL is currently the only MT engine capable of contextually translating a substring taken from a sentence, which is why we are using it in this study. Doing this, we obtain an Entity-wise Machine-translated TASTEset (EMT). Since entities are automatically paired to the source label, the distribution across English and Italian is identical (Table 1).
We also generate shuffled variations of EMT, where the entities within a single ingredient have a probability 𝑝 ∈ {0.1, 0.2, . . . , 1.0} of being shuffled, for a total of ten variations. Figure 1 shows an example where entities have been shuffled in the first and third target ingredients. The rationale behind this approach is that, when training on EMT, if the dataset were to be left as-is, the model would simply learn to associate a source entity to the target entity in the corresponding position, since entities are simply translated and replaced in EMT.
Overall, we have 22 different variations of EMT, i.e. the original and the 10 shuffled versions for each of the two types of tokenization (mBERT's WordPiece [13] vs mDeBERTa's SentencePiece [41]). The datasets have to be tokenized during the generation of the dataset because token indexes depend on the tokenizer being used when converted from character-level span annotations.
We produce a second kind of synthetic dataset by first translating the ingredient lists as a whole, and then aligning source and target entities by using the BERT, Giza++, and Fast Align models presented in Section 4. We refer to this type of dataset as Sentence-wise Machine-translated TASTEset (SMT). As Table 1 shows, the SMT dataset produced by the BERT model trained on both XL-WA and the shuffled version of EMT contains slightly fewer entities than the source material. This is due to the fact that at times the models produce impossible predictions, e.g. predicting the end of an entity to be before its start. 5 This problem does not exist with Giza++ and Fast Align, since their alignments are word-based. As additional training data for the BERT models, we use the EN-IT portion of XL-WA. Table 9 in Appendix B reports the size of each of the partitions we used.
For testing, we annotated an English-Italian dataset of recipes, obtained from GialloZafferano (where the English recipes are translated from the Italian ones). 6 For the annotation process, we recruited a professional translator who is a native speaker of Italian, with an MA in Specialized Translation in both English and Spanish. Figure 2 shows the instructions given for the first multi-class entity annotation task, which consider the same entities as
EMT (en/it) SMT (it) GZ (en) GZ(it) food 4,020 TASTEset, and the second cross-language entity-linking annotation task, carried out by the same annotator at a later time. The annotation was carried out in Label Studio. 7The GialloZafferano (GZ) dataset comprises 597 recipes. The alignments were annotated manually on a subset of 300 recipes, with the possibility of more than one source entity being aligned with one target entity, and vice versa. This is because some recipes contain more than one ingredient option in English but not in Italian (and vice versa), e.g., Cocomero (anguria) 1 fetta vs Watermelon 1 slice. The GZ dataset contains a total of 46,903 NER annotations and 9,842 alignments.
We manually scrutinized GZ and found that the paired recipes do not always coincide completely. Some ingredients may be missing in either language or be an equivalent rather than the same food product. In order to avoid training the alignment models on excessively different recipes, we chose to avoid annotating alignments whenever the number of source ingredients missing from the target recipe surpassed a heuristic threshold of 1/3.
Note that in GZ quantities and units of measure are localized and are thus listed in both imperial and SI units. As shown in Table 1, this is reflected by the lower number of instances annotated as quantity and unit in the Italian portion of GZ, compared to its English portion.
Annotate the ingredients below by assigning to spans of text one of the following categories: quantity, unit, food, color, part, physical quality, process, purpose, taste. Use quantity for numerical values or expressions such as 'to taste', identifying the quantity of an ingredient. Unit stands for "unit of measure", such as grams (g) or ounces (oz). Use color for any color that is not part of a food's own name (e.g. 'red' can be tagged in 'red wine', but not 'black' in 'blackberries'). Part refers to parts of an ingredient, such as 'wings' in 'chicken wings'. Use physical quality for attributes which already characterize an ingredient at the start of the preparation. Process refers to actions that the reader is supposed to carry out. The label purpose answers the question, "What is this entity used for?" Finally, use taste for words referring to a taste, such as 'unsweetened' or 'dry', with relation to a wine. N.B.: The annotations cannot overlap. You can either choose to annotate multiple spans with the same label, or avoid annotating one or more spans of text.
Link each source language entity to its corresponding entity in the target language with an arrow. Entities should only be linked if they share the same use. For example, the "2" in "2 tablespoons chopped onions" does not have the same function as in "2 cebollas largas picadas", since one refers to tablespoons and other to the number of onions. However, "onions" and "chopped" could still be linked, as they are equivalent in the two sequences. Individual source entities can be linked to multiple target entities and vice versa. N.B.: Entities can still be linked if they differ slightly in form or content but still clearly perform the same function in the same ingredient context. For example, "340" and "450" could still be linked if they both refer to the quantity of grams of the same source and target food products.
Entity Alignment As baselines, we use two statistical models: Giza++ [9] and Fast Align [10]. Giza++ combines a HMM [42] alignment model and IBM M1-5 [11]. Fast Align is much more lightweight, only leveraging IBM M2. We use two multilingual BERT models as well: mBERT [13] as the baseline multilingual Transformer model and mDeBERTa [43] because of its larger size (276𝑀 vs 179𝑀 param.) and performance. When using the BERT models, we follow Nagata et al. [14] and treat entity alignment as a question-answering task, enclosing the source word to be aligned within rarely used characters, e.g., '•', feeding the model both the source sequence 𝐴 and the target sequence 𝐵 at once. Figure 1 an entity within a shuffled ingredient's boundaries. We train the models for up to 3 epochs on each dataset with a batch size of 16. The optimizer's learning rate is set at 3 × 10 −4 , while 𝜖 is 10 −8 . Each training run, we select the best model based on the Exact metric 𝐸 [44]:
where 𝑝𝑟𝑒𝑑𝑠 is a list of predictions and 𝑒𝑥𝑎𝑐𝑡(𝑝𝑖, 𝑔𝑖) is the Kronecker delta:
with the predicted and gold strings 𝑝𝑖 and 𝑔𝑖 having been lowercased and stripped of excess punctuation and spaces. We calculate mean Exact and its standard deviation out of five random runs for each model. In order to improve the models' ability to align entities, we optionally train them on an intermediary wordalignment task using the EN-IT training and dev sets of XL-WA. In addition, we train mBERT and mDeBERTa solely using said XL-WA partitions in order to test them directly on GZ. This serves as a baseline which will allow us to gauge the positive effects of fine-tuning on EMT. Entity Recognition For the NER task, treated as token classification, we once again use mBERT. 8To test the efficacy of the multilingual approach, we also use the following monolingual models when training and testing on a single language: bert-base-uncased (henceforth "BERTen") for English [13] and bert-base-italian-uncased ("BERT it ") [45] for Italian. We forgo mDeBERTa for this task, as the focus is showing a comparison between models of equivalent size and performance. Prior to training, the data is preprocessed and labeled using the BIO annotation scheme [46]. We ignore subword tokens when calculating cross-entropy loss, following established methodology. 9We train the models on the EN-IT, EN-ES, EN-IT-ES language subsets of EMT and of the four versions of SMT, produced by mBERT, mDeBERTa, Giza++, and Fast Align. For the BERT models, we use the same hyperparameters used for the alignment task, but with a lower learning rate of 2 × 10 −4 . The models are evaluated using the macro F1-measure. Details on the employed computational resources can be found in Appendix C.
Entity Alignment Table 2 reports the Exact scores for the entity alignment experiment. The entity shuffling approach appears to be very effective for creating data which can make the models better at generalizing. The performance of every single model is greatly enhanced when shuffling ingredients just 10% of the time, with increased shuffling frequency not leading to any significant further improvement. The increase in performance seems to be greater for models which have undergone intermediate training on XL-WA, with mDeBERTax gaining almost 12 points in the Exact metric, when fine-tuned on shuffled data. Unsurprisingly, the larger mDeBERTa performs much better than the smaller mBERT across the board. Although the model obtaining the highest mean performance is obtained at 𝑃 = 0.8, an overlap can be observed between all the confidence intervals for 𝑃 ≥ 0.1. However, this is not true when going from 𝑃 = 0 to 𝑃 = 0.1. Consequently, increased shuffling past 10% does not seem to provide a concrete performance gain, which is why we decided to produce SMT by using the BERT trained on the least-shuffled version of EMT.
In and of itself, the intermediary training step on XL-WA provides a slight performance boost when looking at mBERT vs mBERTx and mDeBERTa vs mDeBERTax. Still, this increase is much smaller compared to the one gained through shuffling. While finetuning the models on a general word-alignment task can be beneficial, the target domain is likely too different from the training data for this to produce a large performance boost. This is especially true as regards the structure of the sentences, since the test data is comprised by short lists of entities separated by semicolons, while the training data is a domain-balanced sample of sentences from Wikipedia. An additional performance boost is provided by multilingual fine-tuning, while cross-lingual settings (e.g., fine-tuning on ES and testing on IT) lead to worse outcomes. Table 6 (Appendix A) shows the results.
Table 3 reports the performance of the best overall models on each class. As the results show, the much lighter Giza++ model surpasses mBERTx, only trailing behind mDeBERTax. The poor scores achieved by the two BERT models are largely attributable to their poor scores on the unit and part classes. We hypothesize that this poor class-specific performance has to do with units of measure often being very short strings. Training mDeBERTa only on the unit instances does not improve its performance, with the model scoring a lower 18.08 Exact metric. Inspecting its individual predictions in this single-class scenario, we noticed that the model does learn to always predict two consecutive tokens, but the enclosed token does not match the original text when converted into characters. This is due to two separate issues: (i) the model selects the wrong span, e.g., selecting an ingredient such as "carote" (carrots) rather than the unit "g" or (ii) the model's prediction is empty when converted to characters. Since mBERT and mDeBERTa both have poor performance on this class while using two different tokenization algorithms (WordPiece vs Senten-cePiece), the problem may lie in the models' tokenizer's token-to-character conversion method. 10 We plan to shed light on this in the future. As regards the part class, the poor performance could be explained by the small More generally, Fast Align, being the simplest and most lightweight model, performs on average well below the other more complex models. 4 reports the results for the NER task. The aligner column indicates which alignment model, out of the best ones listed in Table 3, has produced the SMT training data used to fine-tune the NER model. When no alignment model is specified, the training data being used is EMT. Note that in this case we are not using EMT's shuffled versions, as there is no relation between any two recipes when fine-tuning on the NER task.
When training and testing on Italian data, the best results are obtained for both mBERT and BERT it when fine-tuning on SMT data produced by mDeBERTa. When fine-tuning them on EMT, the performance is noticeably lower, with a 5-point difference for mBERT and an 8-point difference for BERT it . The data produced by mBERT also allows both models to outperform the EMT baseline, although by smaller amounts. Conversely, the data produced by Fast Align and Giza++ worsens the data quality in 75% of the cases. When fine-tuning mBERT on bilingual ES-IT data, the performance on the test set remains essentially unvaried (see Table 8 in Appendix A).
Looking at the baselines at the bottom of Table 4, we can see that fine-tuning mBERT on English data yields worse performance when testing on GZ, compared to fine-tuning on EMT's Italian data. Our data augmentation strategy is thus providing an evident performance boost, with entity alignment producing bigger improvements than machine-translating each entity individually.
In all settings, mBERT performs on par with the monolingual models. This shows that a single multilingual model can suffice when extracting entities from multilingual corpora, saving time and compute.
We explored a simple novel technique to automatically generate high-quality multilingual NER data by combining machine translation and cross-language entity linking. For our experiments, we relied on the Englishlanguage TASTEset dataset, which includes recipes whose lists of ingredients are span-annotated for entity recognition. Moreover, we manually curated a novel English-Italian cross-language dataset, featuring the same kind of annotation, with the addition of crosslanguage alignments.
We machine translated the entities in TASTEset's recipes individually and shuffled them within ingredient boundaries. Leveraging this augmented data, we then fine-tuned BERT entity-alignment models. Using statistical word-alignment models as baselines, we tested these BERT models on our English-Italian parallel corpus. The results showed that models fine-tuned using our novel approach consistently outperform those trained on unshuffled data, along with two statistical baselines.
We then created additional synthetic data by first translating TASTEset's recipes in their entirety, and then aligning the entities in the machine-translated target text using the best models obtained from the first part of the study. These data allowed us to obtain better NER models, compared to the ones we would have obtained by using the original recipes translated entity by entity. We tested monolingual English and Italian BERT models against mBERT, and showed that the latter is capable of obtaining the same performance as its monolingual counterparts when tested on monolingual NER data.
In future work, we plan to extend the annotation of our datasets, both in terms of number of instances and annotators. We will also prioritize solving the token-tocharacter conversion issues encountered in this study. Furthermore, we plan to leverage this data augmentation technique in order to improve multilingual text-to-graph models, since all of the literature in this regard focuses on English-only data [3,4,5,6,7].
In order to test more thoroughly the soundness of our approach, we carry out an equivalent study with Spanish.
We annotated an English-Spanish dataset of recipes obtained from My Colombian Recipes, 11 which we refer to as MCR. MCR is translated from English to Spanish, which is evident from the fact that on the website all Spanish recipes have an English counterpart, but not vice versa. We believe approximately 5-10% of the dataset's instances to be possible MT. A good indication of this is the fact that the English "to taste" is sometimes translated as "para probar", likely an MT mistake, while other times the correct "al gusto" is used. Although using machinetranslated data is not ideal, this was our best choice for a Spanish-language parallel recipe corpus, due to the lack of availability of similar online resources. The use of MT data has implications with respect to the evaluation of the models, as their performance would likely be lower in a real-world scenario involving recipes written directly in Spanish. Nonetheless, given the limited amount of data we hypothesize as being machine-translated, we believe the impact would not be large enough to discredit our results, which focus on the improvement over the cross-lingual EN-ES baseline, rather than the absolute performance of the best model.
MCR contains 276 recipes, 104 of which are bilingual and annotated with alignments. Due to this imbalance between the number of English and Spanish recipes, the number of entities is around 3x for the former, as shown in Table 5. In total, MCR contains annotations for 15,257 entities and 3,565 alignments. Along with the ingredient lists, MCR also contains cooking instructions for all its recipes, along with nutritional facts for 139 of them.
As a monolingual Spanish BERT model baseline to compare against mBERT, we use bert-base-spanish-wwm-cased ("BERTes") [47].
Entity Alignment Table 6 reports the results for the alignment task, complete with the settings including Spanish-language data.
Fine-tuning on the same language as the test set yields better results than cross-lingual scenarios. Furthermore, the best performance on MCR is obtained when finetuning mDeBERTax on both Italian and Spanish. This is not the case for mBERTx and mDeBERTa, whose performance is hindered by the addition of Italian training data. MCR is much narrower in terms of culinary variety, focusing solely on Colombian recipes. On the other hand, GZ contains not just traditional Italian recipes, but an international range of dishes. This is probably the reason why bilingual training is helpful on GZ, but is not beneficial with relation to MCR: adding data from a separate locale helps the models when approaching the more varied GZ, helping them generalize more effectively over its data. Conversely, they are thrown off by the addition of out-of-domain data when tested on MCR's narrower domain.
Comparing the EMT fine-tuning results with the baselines at the bottom of Table 6, we can see that further fine-tuning on EMT does provide a boost, compared to training only on XL-WA. Nonetheless, the difference in performance is much greater when testing on GZ, compared to MCR. When looking at mBERTx, fine-tuned on both Italian and Spanish, the model improves by more than 23 Exact points on GZ, while the gap in performance is just under 16 points on MCR. This effect is even more dramatic for mDeBERTax, with a difference of more than 25 points on GZ, but only 2.48 points on MCR.
Compounded with the fact that, in general, the metrics are much higher when testing on MCR compared to GZ, this points to MCR being a much less challenging test set, compared to GZ. As previously mentioned, part of the dataset is likely machine translated, and since an MT engine is more likely to follow rigidly defined patterns compared to a human translator, this might play a role into the alignment task being easier on these data.
Table 7 reports the performance of the best overall models on each of the individual classes, on both GZ and MCR. Giza++ essentially matches mDeBERTa's performance on MCR, which once again points to entities in MCR being easier to identify compared to GZ. However, the similar performance is largely due to mDeBERTa performing poorly on the unit and part classes, due to the reasons outlined in Section 4. 8 reports the results for the NER task for all language settings. For each language, we use the aligner models which obtained the highest results on the entity alignment task. Note that, since the aligner performance does not significantly improve with increased shuffling (see Section 5), we only train aligner models up to 𝑃 = 0.2 for the Spanish setting due to computational constraints.
In the Spanish monolingual setting, both BERTes and mBERT obtain F1 scores between 0.92 and 0.95 when fine-tuned on SMT, with the models fine-tuned on EMT trailing behind by 11 to 12 points. As all the models perform similarly and the standard deviation is also close to zero, it once again appears that the entities contained in the MCR dataset are not too challenging for both the mono-and multilingual models to identify. Results of the alignment task by class for the best models, using the Exact metric. Best on GZ in bold, best on MCR underlined.
In the bilingual fine-tuning scenario, the training data is a concatenation of the SMT datasets produced by the models obtaining the highest performance on the two test sets. Since this is a bilingual fine-tuning scenario, we only use mBERT, as the monolingual models would not be able to be fine-tuned appropriately on this multilingual data. In this setup, the usefulness of the BERT-based aligners becomes more evident. Indeed, while performance on MCR is largely similar to the other setups, with all models outperforming the baseline by a large amount, the same cannot be said for mBERT's performance on GZ. Finetuning mBERT on the combination of the Italian and Spanish data aligned by Fast Align and Giza++ makes the NER model considerably worse at identifying entities in GZ, with a performance decrease of 20 F1 points with the data created by Fast Align and of 21 F1 points with that created by Giza++. The opposite is true when finetuning the mBERT NER model on the SMT data created by mDeBERTa, with the model achieving an F1 of 0.94, beating the baseline by 5 points. Compared to the model fine-tuned on data created by Giza++, this represents a 26 F1 point increase in performance.
As regards the baseline model fine-tuned on TASTEset's English data and tested on MCR's Spanish entities, we can see that, unexpectedly, the model obtains a 0.88 F1 score, outperforming the mBERT (0.83 F1) and BERTes (0.84 F1) models fine-tuned on the monolingual Spanish EMT data. Despite this, fine-tuning on SMT data produced through our alignment approach allows the NER models to beat this 0.88 F1 baseline, reaching scores as high as 0.95 F1, as previously mentioned.
In all three scenarios, mBERT achieves performances comparable to those of the monolingual models. This shows that, when inferring on multilingual corpora to extract entities, a single multilingual model can be used, saving time and computational resources both during training and inference.
As additional data for intermediate word-alignment training, we use XL-WA [16]