This paper describes our contribution to the SEPP-NLG Shared Task in multilingual sentence segmentation and punctuation prediction. The goal of this task consists in training NLP models that can predict the end of sentence (EOS) and punctuation marks on automatically generated or transcribed texts. We show that these tasks benefit from crosslingual transfer by successfully employing multilingual deep language models. Our multilingual model achieves an average F1-score of 0.94 for EOS prediction on English, German, French, and Italian texts and an average F1-score of 0.78 for punctuation mark prediction.
The prediction of EOS and punctuation marks in automatically generated or transcribed texts is a relatively novel task. While sentence segmentation is a core, and low-level, natural language processing (NLP) task, punctuation has, in the past, primarily been studied in the context of error correction and the normalisation of automatic speech recognition (ASR) output. However, with the recent rise of conversational agents and other NLP systems that are able to generate new texts, the injection of punctuation and EOS marks has gained wider interest. This is hardly surprising because punctuation affects the readability of the text produced by the NLP system and, thus, its perceived overall performance. The SEPP-NLG Shared Task offers two subtasks, namely: • Subtask1 – Sentence segmentation: Fullstop prediction on fully unpunctuated, lowercased documents. • Subtask 2 – Punctuation prediction: Prediction of all punctuation marks on fully unpunctuated, lowercased documents, where the possible punctuation marks are members of the set p = {: −, ?.0}, with 0 indicating no punctuation.
The task is carried out on the German, English,
French, and Italian sections of the Europarl corpus
Earlier studies on EOS and punctuation prediction reflect the various fields of application of this technology. The task is mostly modeled as token-wise prediction. Over the last few years, consistent performance improvements have – unsurprisingly – been achieved with the help of neural network approaches and large-scale neural language models.
The work by Attia et al. (2014) constitutes a rather traditional approach to spelling and punctuation correction, in this case for Arabic. The authors report that in their data set, punctuation errors constitute 40 % of all errors. The task is modeled as token-wise classification with context windows varying between 4-8 words. Classification is carried out with Support Vector Machines and Conditional Random Field (CRF) classifiers, using part-of-speech (POS) and morphological information. The authors obtain the best result, an F1-score of 0.56, with the CRF classifier and a window size of five tokens.
Che et al. (2016) experiment with three
different neural network architectures, using pretrained
GloVe
Treviso et al. (2017) study sentence segmentation – not punctuation – in narrative transcripts that were generated in the context of examining patients for symptoms of language-impairing dementia. They work on three different Portuguese data sets. Input data is modeled by means of POS features, word embeddings, and prosodic information. They then combine convolutional and recurrent neural network layers, achieving F1-scores between 0.7 and 0.8 on two evaluation data sets.
Schweter and Ahmed (2019) also experiment with the Europarl corpus, however, their task is different from the task presented here, i.e. they model only sentence segmentation by predicting, at each full stop in the input text, whether it is an EOS marker or forms a part of another linguistic unit (for instance, it could mark an abbreviation). Predictions are produced by character-level models that are fed not only the token to disambiguate, but also local contexts in the form of context windows. Working on a wide variety of languages – including often overlooked languages such as Bosnian, Greek, or Romanian, – they achieve F1-scores between 0.98 and 0.99, with their BiLSTM model performing best on average.
Sunkara et al. (2020) also work in the clinical
domain, more precisely, on the output of
medical ASR systems. They jointly model punctuation
and truecasing by first predicting a punctuation
sequence and then the case of each input word. The
authors use a pretrained transformer model
The task consists in predicting EOS and punctuation marks on unpunctuated lowercased text. The organizers of the SeppNLG shared task provided 470 MB of English, German, French, and Italian text. This data set consists of a training and a development set. For system ranking, a test set with in-domain and a surprise set with out-of-domain texts were used.
Figure 1 shows the distribution of the punctuation labels for subtask 2, for all languages. As can be seen from the Figure, the distribution of the labels is quite skewed, even if we disregard that the majority of tokens in each data set has the label ”0” (omitted in Figure 1 for better readability). All languages follow the same distribution pattern, however, they exhibit subtle differences. For instance, the difference in frequency between commas and fullstops is particularly pronounced for German and German, in general, has a higher proportion of commas, indicating complex sentence structures. For other language pairs, we observe slight differences in the distribution of hyphens and colons.
Earlier versions of subtask 2 also required predictions for the punctuation marks ”!” and ”;”. During the training phase, the task organizers mapped these symbols to the fullstop to account for strongly skewed distributions and potential HTML artefacts. Sentences containing other punctuation symbols than those already mentioned – parentheses, for instance – were removed by the task organizers because not all instances of parentheses were wellformed (i. e. not for every opening parenthesis there also was a closing parenthesis). These issues leave avenues for future research. 4 4.1
The transformer architecture
However, to assess the performance gain enabled by a transformer-based language model, we also trained (for German sentence segmentation) a first, non-neural baseline: a CRF model on the basis of bag-of-words, POS and local context (+/- 2 tokens) features. This model seemed to perform much better than the spaCy2 baseline provided for sub2https://spacy.io/. 1,290,282
2,208,970 1,223,802 1,657,880 , ? : . , . , ? : . , 0 0.5 1 1.5 2 2.5 0 0.5 1 1.5 2 2.5 (c) Subtask 2 French (d) Subtask 2 Italian task 13, however, since it was outperformed by all transformer-based models by a large margin, we decided to not explore this direction any further.
As a second baseline, we trained a vanilla multilingual Bert model and explored techniques to improve this baseline. In particular, we focused on three different options, namely data augmentation, hyperparameter optimization, and the selection of different architectures and pre-trained models. We have also tested various preprocessing steps to remove special characters and HTML artefacts, but this had no significant effect on our results.
As a first step towards model selection, we
trained a set of mono- and multilingual models
on 10% of the training data for each task. We then
selected the best models per language and the best
multilingual model and trained them on the full
training data set. This approach helped us to iterate
quickly by avoiding long training times (up to 20
hours on a single GPU) just for model selection.
We then selected the following architectures for our
3https://sites.google.com/view/
sentence-segmentation/.
tests:
• Bert
0.849048 0.867502 0.872033 0.885560 0.901298 0.903943 0.921170 0.932057 0.935672
We trained a 10 % and 100 % model for all architecture types to ensure that the architectures scale well with the increased data. Comparing the results from Table 1 and 2, we found that the models for task 1 gain between 0.1 % to 1 % by scaling from 10% to 100% and the model for task 2 gain between 3 % to 5 %. 4.2
All selected architectures are limited with respect to the number of tokens they can process, typically 512. Since most documents are longer than this limit (see Figure 1), we needed a strategy to handle longer sequences.
The simplest method to achieve that is by splitting the text into chunks of 200 words before processing. The number of 200 words was chosen empirically to account for the fact that words get tokenized into more than one token. The disadvantage of this approach is that it is inefficient since most sequences will not utilize the full 512-token capacity of the model.
0 10 100 200
We therefore chose to first tokenize each document and then split it into sequences of 512 tokens. However, this approach, just like the first one, can produce sequences that start with the last word of a sentence or end with the first word of a sentence, giving the model no context for the prediction. To address this issue, we used a sliding window approach and ran experiments with different step sizes similar to the stride parameter in convolutional neural networks. This method ensures that the model has additional context for making predictions. For training, we ran a grid search to find the optimal length of the overlapping window, using an English Bert base model on 10% of the data. Based on the results shown in Table 2, we choose an overlapping window size of 100 for training our models. The loss was calculated for the whole sequence, including the overlapping part. Since this method also generates new training sequences, it also acts as a data-augmentation. 5
Table 1 shows the results of the 10% model comparison training. All the models that performed best on task 1 also performed best on task 2. For English, we selected two models, XLM RoBERTa Large and RoBERTa Large since their scores were about even. An Electra-based model achieved the best results for the German language, whereas, surprisingly, English and Italian Electra models scored below baseline Bert models. For French, we selected Camembert large, a 335 million parameters RoBERTa-based model which scores notably better than Camembert base using 110 million parameters. The digital library team at the Bavarian State Library (dbmdz) published two different Italian Bert-based models, the XXL version of the model was trained on the larger corpus and achieved the best result. The multilingual XLM RoBERTa base model achieved better scores than the older multilingual Bert model using the same number of parameters. The larger 335 million parameter version of this model achieved the best multilingual model score, on par with the language-specific models. Note that the scores of the multilingual models are evaluated on a multilingual development set.
We trained the selected models on the full training set for each task and evaluated them on the development sets. The results of this evaluation can be found in Table 3 for both subtasks 1 and 2. For both tasks, the large multilingual XLM RoBERTa outperformed all language-specific models. Therefore we submitted our XLM RoBERTa based models for task 1 and 2. For the Italian language, the XLM-RoBERTa-based model scored notably better than the best language-specific model. However, for the other languages, the performance gains are not that significant. The scores of the German Electra-based model are comparable to those of XLM RoBERTa, despite using 110 million parameters in contrast to the 550 million parameters of XLM RoBERTa large. This indicates that there is room for possible performance improvements. 5.1
Since the multilingual models outperformed almost all monolingual models, we selected these for subtasks 1 and 2. Furthermore, we submitted one smaller monolingual model to evaluate its performance on the test set and out-of-domain test set (surprise test).
is based on the 550-million-parameter XLM RoBERTa large model and was trained on the labeled data of task 1. Across all four languages this model archived an average F1 score of 0.94 on the test set and an average F1 score of 0.78 on the surprise test set.
based on the 110-million-parameter German Electra base model. It was trained on the labeled data for task 1 and an additional data set consisting of data from speeches of the German parliament (Bundestag, 134 MB4) and a text crawl from the Leipzig corpora collection (245 MB5), containing a mixture of news texts and Wikipedia articles. For the German language, this model archived an F1 score of 0.95 on the test set and an F1 score of 0.80 on the surprise test set.
FullStop Multilingual Task 2: This model is also based on XLM RoBERTa large and was trained on the labeled data for task 2. As shown in Figure 2 and Table 4, the model performs well on EOS marks across all languages. In contrast, the performance for colons and hyphens is lower. We suspect that this is due to the properties of the data set as described in section 3. We have seen that hyphens and colons are not only infrequent in the training data for all languages, they also exhibit unstable 4https://github.com/Datenschule/offenesparlament-data 5https://wortschatz.uni-leipzig.de/de/download/German Model roberta-large xlm-roberta-large electra-base-german-uncased electra-base-german-uncased with data augmentation camembert-large bert-base-italian-xxl-uncased xlm-roberta-large distribution patterns across languages. Intuitively, this is not surprising as hyphens and colons, in many cases, are optional in the sense that they can be substituted by either a comma or a full stop, i. e. the rules for their usage are not only grammatical and syntactic, but also stylistic. Performance increases might be achieved through targeted training with adversarial examples. The model achieves an average F1 of 0.78 on the test set. Similar to the other models, the performance degrades to an average F1 of 0.61 for the out-of-domain surprise set.
Inference on the complete test and surprise set (470 MB) takes about 1 hour for each multilingual FullStop model using an Nvidia 3090 GPU. 6
The type and amount of data used for pretraining has a significant impact on the final model’s performance. Table 1 shows that, for Italian, there is a 5% difference for task 2 between the two monolingual Bert-based models. Both models use the same 110 million parameters of the Bert architecture, but were trained on different corpus sizes. The ”bert-base-italian-uncased” model was trained on a 13GB corpus and the ”bert-base-italian-xxluncased” model was trained on a 81 GB corpus. The positive effect of larger corpus sizes on model performance has also been verified for other transformer architectures, for instance by Conneau et al. (2020) and Clark et al. (2020).
for different languages. Electra is the bestperforming monolingual German model, but for English and Italian, results obtained with Electra are well behind those obtained from mono- and multilingual Bert models. We conducted a series of tests with different hyperparameters for the English Electra models, but could not further improve the results.
Both Tasks benefit from multilingual models
and training data. To our surprise, the
multilingual XLM-Roberta-based model outperformed all
monolingual models, even though earlier
multilingual Bert models were, in most cases,
outperformed by their language-specific counterparts. We
suspected that this could be explained by the much
larger number of parameters used by XLM-Roberta
large. To test this hypothesis, we trained a
monolingual English model based on XLM-RoBERTa and
another English model based on the monolingual
RoBERTa. As shown in Table 3, both models are
outperformed by the XLM-RoBERTa model,
showing that the model benefits from multilinguality.
Although we have no direct explanation for the
superior performance of the multilingual model, we
would like to accentuate that it is in line with earlier
work
What are the optimal hyperparameters for each model? We ran a hyperparameter search for the Adam optimizer using the Akiba et al. (2019) framework with a budget of 200 trails on the German Electra base model. For the hyperparametersearch, we configured the following search space: learning rates between 1 · 10−2 and 1 · 10−5, 1 to 5 training epochs, batch sizes from 22 to 27, a weight decay from 1 · 10−1 to 1 · 10−12 and Adam epsilon from 1 · 10−6 to 1 · 10−10.
We have compared these settings with Adafactor
The labels of task 2 are a super-set of the labels for task 1, therefore one can use a model trained for task 2 on task 1. We changed the classification result of task 2 by mapping the sentence end labels ”.” and ”?” to label 1 and all other labels to label 0. The results in Table 6 show that this method decreases the final scores only marginally. For many applications, it is sufficient to train one model that processes all four languages for both tasks. For this shared task, we trained and submitted two different models, since a dedicated model for task 1 slightly improves the results.
en de fr it Task 1 Model 0.945746 0.958591 0.941974 0.934144
For the purpose of the shared task, we did not aim at optimizing inference and training efficiency. However, we tested if it is necessary to use all the 12 Bert base layers. To this end, we trained a set of models on 10% of the English data using 3,6, 9 and 10 layers on task 1. To keep the results comparable, we used the same hyperparameters as with all other models, described in section 4. The results in Table 7 show that with this simple layer pruning approach it is possible to retain 99% of the model’s performance while removing 1/4 of the last layers. We suggest to explor more advanced optimization techniques in further studies.
In this paper, we have shown that transformerbased architectures can be successfully applied to the tasks of punctuation mark and sentence end prediction. To our surprise, monolingual models are outperformed by multilingual models, showing that these models can transfer knowledge across languages. For the future, we plan to improve on two main aspects. Firstly, we want to reduce the size of our models. Both ”FullStop Multilingual” models use 550 million parameters which leads to computationally expensive inferencing. In our ablations, we have demonstrated a first approach to reducing the number of parameters. Secondly, we would like to improve the out-of-domain performance of our models. The shared task surpriseset showed that there is a performance degradation on texts from unseen domains. We will address this issue in future research.
This research has been funded by the European Social Fund (ESF), SAB grant number 100339497 and the European Re-gional Development Funds (ERDF) (ERDF-100346119). AnneKathrin Schumann has received funding through the SAB’s technology startup scholarship (Technologiegru¨nderstipendium).