This paper describes our approach (URmSBD) to address the shared task on Sentence End and Punctuation Prediction in NLG Text (SEPP-NLG) organised as part of SwissText 2021. We participated in Subtask 1 (fully unpunctuated sentences - full stop detection) and submitted a run for every featured language (English, German, French, Italian). Our submissions are based on pre-trained BERT models that have been fine-tuned to the task at hand. We had recently demonstrated, that such an approach achieves state-of-the-art performance when identifying end-of-sentence markers on automatically transcribed texts. The difference to that work is that here we use languagespecific BERT models for each featured language. By framing the problem as a binary tagging task using the outlined architecture we are able to achieve competitive results on the official test set across all languages, with Recall, Precision, F1 ranging between 0.91 and 0.96 which makes us joint winners for Recall in two of the languages. The official baselines are beaten by large margins.
Text normalization has always been a core
building block of natural language processing aimed at
converting some raw text into a more convenient,
standard form
The SwissText 2021 Shared Task 2: Sentence End and Punctuation Prediction in NLG Text is concerned exactly with this problem area. The goal is to develop approaches for sentence boundary detection (SBD) in unpunctuated text. Providing suitable solutions means fostering readability and restoring the text’s original meaning.
We took part in Subtask 1 (fully unpunctuated sentences – full stop detection) of this challenge and did so for all featured languages. This report starts by contextualising the task as part of a short discussion of related work. We will then introduce our methodology, briefly describe the data and report results. Finally we present some discussion and conclusions. 2
Sentences are considered as a fundamental
information unit of written text
Sentence Boundary Detection (SBD) is an
important and actually well-studied text processing
step but it typically relies on the presence of
punctuation within the input text
The system architecture we use is adopted from our
previous work that achieved state-of-the-art
performance on a very similar task
We fine-tuned a pre-trained BERT model on the problem and obtained high F1 scores for the desired positive class (sentence end) outperforming 1https://github.com/doGregor/SBD-SCD-pipeline alternative approaches on different datasets. We use a softmax classification head predicting the label (EOS or O) by the highest probability at each token. 3.2
We apply two changes to the model fine-tuning process for this shared task as follows: • First of all, we are faced with four different languages and not just English texts. The two obvious options would be to use a multilingual language model or to choose a different language-specific pre-trained model for each of the languages, i.e. German, French, English and Italian. We decided to adopt language-specific BERT-base models as Nozza et al. (2020) reports that this yields better results than using mBERT, pre-trained on a multilingual corpus. • Secondly, we change the process of sample construction. We handle the unpunctuated input text as one long chain of words. We originally split this chain in samples of 64 words and fine-tuned the model with a maximum sequence length of 128 BERT-specific tokens. Further experiments have shown that utilizing token sequences as long as possible (512 BERT tokens) yields the best results. Therefore, we pre-process the raw text data by sending it through the model’s tokenizer first. Each time a batch of iterated words fits 512 BERT tokens we create a sample from it. If a word at the transition between two samples would be ripped apart (as adding it entirely to the current sample would exceed 512 tokens), we put it at the beginning of a new sample and pad the rest of the previous one with special PAD tokens.
All other hyperparameters are kept in line with Donabauer et al. (2021), namely using an epoch number of 3 and a batch size of 8 per device. Since we run it on 3 GPUs simultaneously the batch size per iteration increases to 24. We also evaluated our approach on the datasets with tuned hyperparameters. However, it turned out that increasing the number of epochs to 5 leads to a deterioration of results.
We participated in Subtask 1 (fully unpunctuated sentences – full stop detection) of SwissText’s SEPP-NLG Shared Task 2.
Before addressing the experimental setup we briefly describe the provided data sets. The challenge’s domain are NLG texts. Since there are no corpora that feature such data nor manually corrected versions the organizers selected Europarl2 as source. This corpus includes transcribed text data originating from spoken text in many different languages. The data come along in lowercase format and are already split up into tokens. Sentence boundary punctuation is removed. Instead labels are assigned that mark upcoming sentence ends. The last token of each sentence is labeled with ’1’, all remaining with ’0’.
The data are provided as multiple tab-separated
value files grouped by each language and set. The
number of tokens per language and dataset is
reported in Table 1. We explain our pre-processing
with respect to a single set for a single language,
e.g. the English evaluation set. Firstly, we read
each tsv file one after the other and concatenate all
tokens and labels as two long lists. During reading
we save the order and length of the input files. By
that we are able to reconstruct the original
structure of the files later on. The list of tokens is fed
into the model-specific tokenizer. If tokens are not
recognized properly we replace them with ’nan’.
Each time a batch of 512 BERT tokens is filled, we
create a sample from it. Data are saved in
CoNLL2003 format
As mentioned earlier, we make use of language specific models rather than mBERT. We briefly describe the respective models and the corpora they are trained on.
• English: Classic BERT base uncased model,
trained on English lowercase text
We make use of the PyTorch4 version of the Python huggingface5 transformers library to access models and run fine-tuning. We execute the scripts on 3 Nvidia GeForce RTX 2080 Ti GPUs with an overall memory size of 33GB. 5 5.1
The official baseline is produced using the spaCy NLP package. The organisers report scores for different pipeline versions and we are describing the best performing one for every language in Table 2. The official evaluation metrics are Precision, Recall and F1-score of the positive class label (i.e., sentence end).
As Table 2 illustrates, F1-scores for English, German and French are ranging from 0.32 to 0.47. For Italian the F1-metric is collapsing to 0.01, caused by a very low Recall of 0.00. 5.2
We summarise the results obtained when running our system, UR-mSBD, on the test data. For each language we also include scores obtained on the dev set as well as the surprise test set that was introduced to check the generalizability of the different approaches.
Table 3 presents the results for the English data, Table 4 for German, Table 5 for French, and Table 6 presents the results for the Italian test data.
We see overall consistently high scores for all three metrics and across all languages when looking at the official test sets. An average of F1=0.93 aggregated over all languages places us just one percentage point behind the top performance. Looking at Recall, we actually end up being joint winners for the German and French test data.
3https://github.com/dbmdz/berts 4https://pytorch.org/ 5https://huggingface.co/ sentative of the data the system was trained on.
Across the board all the baselines were beaten by large margins. 6
For all featured languages our fine-tuned
BERTbased predictions are performing very well with
results for all three metrics (P/R/F1) in the 90s and
being very competitive when looking at the other
submissions for this shared task. This first of all
demonstrates the power of transformer-based
models and confirms findings we reported previously
The fact that the baselines were outperformed by such large margins is perhaps a sign that nonneural approaches are not competitive for the task and data at hand.
We note that our approach performed best for German texts which might be caused by high similarity between the data the model was pre-trained on and the data sampled to be training, dev and test sets for this task. It will be worth exploring whether for different data samples we observe a similar pattern or whether the differences are in fact not significant.
Taking a slightly broader perspective, we
observe that the scores obtained here are similar
to what we obtained when running our sentence
boundary detection algorithm on a dataset
comprising transcribed lectures given at Stanford
University first proposed by Song et al. (2019), and
the DailyDialog dataset
Another general pattern we read into the results is that there are only small differences when comparing results on the dev sets with the results on the test sets. We conclude that our approach can generalize to unseen data as long as the training data is representative for the data used for testing.
The highest scores are reported for German (with Precision at 0.94, Recall at 0.96 and F1 at 0.95). All the scores for the test sets are above 0.90. For the surprise test set the results drop quite a bit but are still reasonably high given the data is not repreThe approach does however generalise less well to out-of-domain (’surprise’) data with F1-scores dropping between 0.15 and 0.18, compared to the Europarl sets. We still consider the results to be reasonably well though given they are on average over all languages only 0.03 behind the top-performing system. 7
We framed the task of full-stop prediction (Subtask 1 of Shared Task 2 at SwissText 2021) as a binary classification task over all input tokens identifying whether each of these tokens should indicate the position of a full stop or not. Fine-tuning language specific pre-trained BERT models for each of the four tasks resulted in competitive results. Given the small difference in F1 of 0.01 compared to the top results reported for this competition for three of the languages (as well as aggregated over all languages) we will await statistical significance tests as our results may well turn out to be on par with the top results in this task.
This work was supported by the project COURAGE: A Social Media Companion Safeguarding and Educating Students funded by the Volkswagen Foundation, grant number 95564.