This paper documents two sequence-labeling models for NER in Spanish: a conditional random field model with handcrafted features and a BiLSTM-CRF model with word and character embeddings. Both models were trained and tested using CAPITEL (an annotated corpus of newswire written in European Spanish) and were submitted to the shared task on Spanish NER at IberLEF 2020. The best result was obtained by the CRF model, which produced an F1 score of 84.39 on the test set and was ranked #6 on the shared task.
In this section we describe the two models that were explored for Spanish NER on the CAPITEL corpus: a CRF model with handcrafted features and a BiLSTM-CRF model with word and character embeddings.
A CRF model was built usingp y c r f s u i t e 1 [9], a Python wrapper focrr f s u i t e 2 [10] that implements CRF for labeling sequential data. The model also usedTtohk een and S p a n classes from s p a C y 3 library1[1].
The following handcrafted binary features were used for the model: • Bias feature: a feature that is active on every single token to support setting per-class bias weights. • Token feature: the string of the token f(e a t u r e s [ t o k e n = “ a g u a ” ] = 1 . 0 ). • Uppercase feature (y/n): active if all characters in the token are uppercOaNseG(, IBM ) (f e a t u r e s [ i s _ u p p e r c a s e ] = 1 . 0 ). • Titlecase feature (y/n): active if only the first character of the token is capitalized (Gobierno, Alcalá) (f e a t u r e s [ i s _ t i t l e c a s e ] = 1 . 0 ). • Character trigram feature: an active feature for every trigram contained in the token (f e a t u r e s [ t r i g r a m = “ a g u ” ] = 1 . 0 ). • Punctuation feature (y/n): active if the token contains any type of punctuation mark (,-/ ) (f e a t u r e s [ i s _ p u n c t u a t i o n ] = 1 . 0 ). • Word sufix feature : last three characters of the token
(f e a t u r e s [ s u f f i x = “ d a d ” ] = 1 . 0 ). • POS tag feature: part-of-speech tag of the token provided bsyp a C y library (e s _ c o r e _ n e w s _ m d model) (f e a t u r e s [ p o s t a g = “ V E R B ” ] = 1 . 0 ). • Digit feature (y/n): active if the token contains a number20(14, salud2)
(f e a t u r e s [ i s _ d i g i t ] = 1 . 0 ). • Word shape: shape representation of the token provided sbpya C y library (e s _ c o r e _ n e w s _ m d model) (f e a t u r e s [ s h a p e = “ X x x x ” ] = 1 . 0 ). • Lemma feature: lemma of the token provided by thee s _ c o r e _ n e w s _ m d model froms p a C y (f e a t u r e s [ l e m m a = “ r e í r ” ] = 1 . 0 ). • Word embedding (see Table1). 1https://github.com/scrapinghub/python-crfsuite 2https://github.com/chokkan/crfsuite 3https://spacy.io/
The model was trained on the training set of the CAPITEL corpus and tuned on the validation set using grid search. The following hyperparameters were explored during tuning: c1 (L1 regularization coeficient:0.01, 0.05, 0.1, 0.5, 1.0), c2 (L2 regularization coeficient:0.01, 0.05, 0.1, 0.5, 1.0), embedding scaling (scaling factor applied to each dimension of the embedd0i.n5g,: 1.0, 2.0, 4.0), and embedding type (from a set of diferent Spanish word embeddings; see Tabl1e) [12, 13, 17, 14, 11, 15, 16]. A window of two tokens in each direction was set for the feature extractor. Optimization was performed using L-BFGS. The threshold for the stopping criterion delta was set to delta 1=e − 3.
The best results were obtained with c10=.01, c2 = 0.5, scaling =0.5 and FastText Spanish embeddings [13] that were trained on the Spanish Unannotated Corp4or[1a8]. These hyperparameters produced an F1 score of 83.60 on the validation set (precision = 84.23, recall = 83.03). The lowest F1 result obtained on the validation set during grid search experiments was 78.90, which illustrates the substantial impact that hyperparameter tuning can have on the model’s performance.
A feature ablation study was then performed on the tuned model by removing one feature at a time and testing on the validation set. The results obtained during the feature ablation study were consistently worse than those obtained with the full set of features, which demonstrates that all features contribute positively to the model’s performance. The embedding feature seemed to be the one that contributed the most to the result (see Tab2l).e
A BiLSTM-CRF model was also trained for the same task using the CAPITEL corpus. This neural model was created using the librNaCrRyF++5 [19]. NCRF++ is a PyTorch-based framework that implements a neural sequence-labeling model in three layers (character sequence layer, word sequence layer and inference layer). The architecture chosen for our neural NER model 4https://github.com/josecannete/spanish-corpora 5https://github.com/jiesutd/NCRFpp was a character CNN + word LSTM + CRF model. This neural architecture has previously demonstrated to be succesful on other NER task7s].[
For this model, we used FastText Spanish embeddings with 300 dimensions from14[]. The dimensions of the character embeddings were set to 50. The learning rate was 0.015, the batch size was 20. L2 regularization coeficient was 1e-08, the number of hidden dimensions was 200, dropout was 0.5, the optimization algorithm used was SGD. Iterations were set to 100, the best results on the validation set were obtained at iteration 81, which produced an F1 score of 85.04 (precision = 85.66, recall = 84.42), almost two points higher than the best result obtained by the non-neural model on the validation set.
The best performing version of both the CRF and the BiLSTM-CRF model were then run on the test set of the CAPITEL corpus. The CRF model produced an F1 score of 84.39 on the test set (precision = 84.75, recall = 84.12, ranked #6 on the shared task), which is almost 1 point higher than the best result obtained on the validation set. The BiLSTM-CRF model, however, obtained an F1 score of 83.01 (precision = 84.33, recall = 81.82, ranked #8 on the shared task), two points lower than the best result obtained on the validation set and 1 point less than the CRF model. This gap between the neural and non-neural model (the non-neural model outperforming the neural model) can perhaps be explained by the diferences between the test set and the training set, but also by the lack of tuning on the BiLSTM-CRF model (as the neural model could have benefited from a more exhaustive tuning). Either way, both results are modest compared to the best performing model of the shared task (which produced an F1 score of 90.2300]). [
In this subsection we will document some of the errors that were produced by the CRF and the BiLSTM-CRF model:
The tag OTHER was behind many of the disagreements in tag selection: for instance, the name of newspapers and TV channels (such aLsa Vanguardia or TVE) were labeled on the gold standard asORG when they referred to the institution, but labeledOaTHs when referring to the actual publication or channel (see guideline 5.4.1.7 from the task annotation guide2l1i]n)e.s [ These subtleties in meaning were not captured by the presented systems, as both models tended to tag these entities asORG, regardless of the context.
Similarly, the name of countries, regions and other geopolitical units (that have sometimes been considered undeGrPE in other annotation schemes2[2, 23]) were a frequent source of error. According to the annotation guidelines, countries were labeLlOeCdoarsORG depending on whether the name was referring to the political institution or the geographical location (see guideline 5.2.1. on the annotation guideline2s1[]). These nuances were not well-captured by our models and also produced frequent tag selection errors.
Nested entities also produced some issues, particularly to the CRF moSdemela: na de la Moda de China was labeled as one singlOeTH entity in the gold standard, but was labeled as two separate entities by the CRF modelSe(mana de la Moda as OTH, China as LOC).
The scope of the entities was also a common source of error, as when taggSaintéglite GOES-16 (instead of justGOES-16).
Additionally, film and book titles (such aLsa llegada o La peste) should have been labeled as OTH but were consistently ignored by our models (this entities could perhaps have been captured by the CRF model had a quotation feature been included).
Finally, person names with unusual shapes were sometimes mislabeled, aEsliBnigotes. Likewise, the actress and singeIrmperio Argentina (that appeared once in the background set) was labeled aOsTH by the CRF and asORG by the neural model, and is a good example of the dificulty that Spanish artistic nicknames can pose to NER systems.
In this paper we have presented two diferent sequence-labeling models for Spanish NER: a CRF
model with handcrafted features and a BiLSTM-CRF model with word and character embeddings.
These models were applied to the CAPITEL corpus, an annotated corpus of journalistic texts
written in European Spanish. Both models were submitted to the CAPITEL shared task on NER
at IberLEF 2020. The CRF model produced an F1 score of 84.39 on the test set and was ranked #6
on the shared task, while the BiLSTM-CRF model obtained an F1 score of 83.01 and was ranked
#8.
[1] E. F. Tjong Kim Sang, F. De Meulder, Introduction to the CoNLL-2003 shared task:
Language-independent Named Entity Recognition, in: Proceedings of the Seventh
Con[19] J. Yang, S. Liang, Y. Zhang, Design challenges and misconceptions in neural sequence
labeling, in: Proceedings of the 27th International Conference on Computational Linguistics
(COLING), 2018. URL: http://aclweb.org/anthology/C18-13.27
[20] R. Agerri, G. Rigau, Projecting Heterogeneous Annotations for Named Entity Recognition,
in: Proceedings of the Iberian Languages Evaluation Forum