Danish natural language processing (NLP) has in recent years obtained considerable improvements with the addition of multiple new datasets and models. However, at present, there is no coherent framework for applying state-of-the-art models for Danish. We present DaCy: a unified framework for Danish NLP built on and integrated with SpaCy. DaCy uses efficient multitask models which obtain state-of-the-art performance on named entity recognition, part-of-speech tagging, and dependency parsing. DaCy contains tools for easy integration of existing models such as for polarity, emotion, or subjectivity detection. In addition, we conduct a series of tests for biases and robustness of Danish NLP pipelines through data augmentation. DaCy large compares favorably and is especially robust to long input lengths and spelling variations and errors. All models except DaCy large display significant biases related to ethnicity while only Polyglot shows a significant gender bias. We argue that for languages with limited benchmark sets, data augmentation can be particularly useful for obtaining more realistic and fine-grained performance estimates. We provide a series of augmenters as a first step towards a more thorough evaluation of language models for low and medium resource languages and encourage further development.
1.1. DaCy
With this motivation we present DaCy: an efficient end-to-end framework for Danish NLP
with state-of-the-art performance on POS, NER and dependency parsing. DaCy fills the
gap in Danish NLP by providing a consistent interface that is easily extendable and able to
integrate other models. DaCy is built on SpaCy v.3 which comes with a range of advantages:
the framework is optimized, user-friendly, and well-documented. DaCy includes three
finetuned language models: DaCy small, based on a Danish Electra (14M parameters) [
Besides models fine-tuned for DaCy, the package includes convenient wrappers to add other
models to the pipeline. For instance, Danish models for detecting polarity, emotion, and
subjectivity classification can be added in a single line of code, and any HuggingFace Transformers
[
Fine-tuned language models are commonly evaluated by testing performance on a gold-standard
benchmark dataset. The most commonly used benchmark for Danish is the DaNE dataset
[
To provide an additional layer of validation, we propose evaluating models on augmented
gold-standard data. Data augmentation entails generating new data by slightly modifying
existing data points [
Although data augmentation is most commonly used for increasing the amount of training
data, it can just as well be used for evaluation purposes [
The contributions of this paper are three-fold. 1) We introduce new state-of-art models for Danish dependency parsing, NER and POS. 2) We introduce the DaCy Python library as a unified framework for state-of-the-art NLP in Danish. 3) We evaluate Danish NLP pipelines
using data augmentation and provide directions for future model development.
To train the candidate models for DaCy, all publicly available Transformer-based language
models for Danish were fine-tuned on the DaNE corpus [
Table 1 shows the performance of all fine-tuned models evaluated on DaNE’s test set. The
three best performing models in each size category, XLM-Roberta, DaBERT, and Ælaectra
Cased are included in DaCy as the large, medium and small, respectively. In line with previous
ifndings [
To evaluate the robustness of DaCy and other Danish NLP pipelines, we assessed their
performance on multiple augmented version of the DaNE test set. All Danish models are trained on
the DaNE corpus which consists of a mix of textual data of both spoken and written origin from
the years 1983–1992 [
The augmenters presented here are not meant to be exhaustive, but rather a first step towards more thorough validation of new language models. We argue that the bar for inclusion of a new model should be set higher than a slight increase in benchmark performance. Language models are used in a variety of contexts which current benchmarks tasks, especially for low resource languages, do not capture. Our aim with these experiments is to provide an extra layer of insight into the performance of language models that more closely mimics naturalistic use cases, and encourage the development of further augmenters. Augmentation not only provides insights into when model performance breaks down, whether certain models are more suited for specific use-cases than others, but can also be used for identifying specific areas to improve upon.
The augmenters developed for this paper are designed in accordance with the SpaCy framework, and are thus not necessarily tied to DaCy or Danish in particular and can be used both during model validation and training. Comprehensive tutorials are provided on the DaCy Github repository.
We tested small, medium, and large SpaCy [
e) Abbreviate all first names to the first character including a full stop.
The stochastic augmentations, i.e. name and keystroke augmentations, were repeated 20 times.
Previous evaluations of Danish NLP tools have used the gold-standard tokens instead of using a tokenization module. While this allows for easier comparison of the specific modules it inflates the performance metrics of the models and is unlikely to reflect the metric of interest, namely, the performance during application.4 All models were tested using both their own tokenizer (if they have one) and the SpaCy tokenizer for Danish. The performance reported in
section 3 uses the best peforming tokenization module for each pipeline. For all models except Stanza and Polyglot this was found to be the SpaCy tokenizer.
This paper has introduced the DaCy models and presented a thorough evaluation of Danish
NLP models on a battery of augmentations. DaCy models achieve state-of-the-art
performance on Danish NER, POS, and dependency parsing, and are robust to augmentations such
as keystroke errors, name changes, and lowercasing. The results from training DaCy
underline three well-known trends in deep learning and NLP, 1) larger models tend to perform
better, 2) higher quality pre-training data leads to better models, as illustrated by the
superior performance of Ælaectra compared to DaELECTRA, and 3) multilingual models perform
competitively with monolingual models [
Our experiments with multiple augmenters revealed diferent patterns of strengths and weaknesses across Danish NLP models. In general, larger models tend to be more robust to data augmentations. Several models are highly sensitive to casing, which limits their usefulness on certain domains. Evaluating models on augmented data provides a more holistic and realistic estimate of the expected performance, and can reveal in which use cases one model might be more useful than another. For example, it might be better to use DaCy medium on social media as opposed to DaCy large as its performance is not afected by casing.
The purpose of the data augmentation experiments was to evaluate the robustness of Danish models and to open a discussion on how to present new models going forward. As more models are developed for low and medium resource languages, properly evaluating them becomes vital for securing robustness, transparency, and efectiveness despite limited benchmark sets. We do not posit data augmentation as the only solution, but demonstrate that it can efectively reveal performance diferences on important factors such as casing, spelling errors, and biases related to protected groups. As researchers, we bear the responsibility for releasing adequately tested and robust models into the world. With the increasing ease of deployment, users must be made aware of the level of performance they can realistically expect to achieve on their problem, and when to choose one model over another. Social media researchers should know that certain models are sensitive to casing, historians should know that some models handle old text variations such as ae, oe, aa poorly, and lawyers should be aware that models might not be able to identify abbreviated names as efectively. In this regard, transparency and openness as to when and how models fail are crucial measures to report. Such evaluation requires the development of infrastructure and tools, but is fast and easy to conduct once in place. For instance, it only takes 8 minutes to test DaCy large on all augmented datasets including bootstrapping. As part of the DaCy library, we provide several augmenters and utility functions for evaluation that integrate with SpaCy and encourage new NLP models to use and expand upon them. For the continued development of low and medium resource NLP in a direction that is beneficial for practitioners, it is vital to conduct more thorough evaluation of new models. We suggest these augmenters not as an evaluation standard, but as preliminary guiding principles for future development of NLP models for low and medium resource languages in particular.
We would like to especially thank Martin C. Jespersen for early conversions on biases and covert weaknesses in Danish language models. [21] E. V. Meldgaard. Muslimske fornavne i Danmark. 2005. url: https : / / nors . ku . dk / publikationer/webpublikationer/muslimske%5C%5Ffornavne/.