This paper describes the first Sentence End and Punctuation Prediction in Natural Language Generation (SEPP-NLG) shared task1 held at the SwissText conference 2021. The goal of the shared task was to develop solutions for the identification of sentence boundaries and the insertion of punctuation marks into texts produced by NLG systems. The data and submissions2, and the codebase3 for the shared tasks are publicly available.
Sentence End Detection, also known as Sentence
boundary disambiguation (SBD) or boundary
detection, is the Natural Language Processing (NLP)
task of recognizing where a sentence begins and
ends. A period is the most common end of
sentence indicator in written English as well as many
other Indo-European languages. However, a period
may be used in a decimal point, an abbreviation,
an email address, or other possible cases as well
which makes sentence boundary detection a
challenge. Other forms of punctuation such as question
and exclamation marks, semicolons, comma, etc.
add to this challenge. Although sentence
boundary detection is considered an almost solved issue
for formal written language
The goal of the SEPP-NLG shared task is to build models for identifying the end of a sentence by detecting an appropriate position for putting an appropriate punctuation mark. 2
Similar to the system proposed by Grefenstette and
Tapanainen (1997), the earliest attempts for
sentence boundary detection utilize a set of rules or
regular expressions. In a different direction,
Reynar and Ratnaparkhi (1997), and Kiss and Strunk
(2006) proposed an information-centric approach
based on the Maximum Entropy model, and an
unsupervised method based on collocation
statistics respectively. Decision tree classifier
Our task is closely related to Tilk and Aluma¨e (2016) and follow-up work that uses the Europarl and TED talk corpora for punctuation prediction. Similar to our goal, Z˙elasko et al. (2018); Donabauer et al. (2021) investigate sentence boundary detection in unpunctuated ASR outputs of spoken dialogues based on textual features. Cho et al. (2017) propose a method to predict sentence boundaries and punctuation insertion in a real-time spoken language translation tool. In a similar setting, Klejch et al. (2017) include acoustic features to improve punctuation prediction in a speech translation system, and Yi and Tao (2019) combine lexical and speech features for punctuation prediction in a traditional ASR setting. Finally, Rehbein et al. (2020) investigate the annotation and detection of sentence like units in spoken language transcripts. 3
Ultimately, the goal of SEPP-NLG is to predict
sentence ends and punctuation in NLG texts. However,
there are no corpora that feature NLG texts and
their manually transcribed and corrected versions.
Therefore, we approximate the setting by using a)
transcripts of spoken texts, and b) lower-casing the
texts and removing all punctuation marks. While
there are multiple corpora of transcribed spoken
language, we choose the Europarl corpus4
We offer the following subtasks: • Subtask 1: (fully unpunctuated sentencesfull stop detection): Given the textual content of an utterance where all punctuation marks are removed, correctly detect the end of sentences by placing a full stop in appropriate positions. • Subtask 2: (fully unpunctuated sentencesfull punctuation marks): Given the textual content of an utterance where all punctuation marks are removed, correctly predict all punctuation marks.
Participants were free to choose for which languages and subtasks they contributed a submission, but were encouraged to participate in all languages. 3.1
We leverage the open parallel corpus (OPUS)
version of the Europarl corpus5 (Tiedema
5https://opus.nlpl.eu/Europarl.php boundaries in the corpus are automatically generated, they are quite reliable as the data and the models trained to detect the boundaries contain all the original punctuation symbols of the transcripts.
In the spirit of the “Swissness” of the SwissText conference where SEPP-NLG 2021 is co-located, we select 3 of the 4 official languages6 of Switzerland, i.e. German, French, and Italian and complement the selection by incorporating English.7
The Europarl corpus contains multiple punctuation symbols. For subtask 2, we gauged which subset of them represents a realistic and feasible goal for their automatic prediction in a stream of unpunctuated, lower-cased tokens. Also, we considered which punctuation marks improve the readability of a text the most. Hence, we consolidated the selection of punctuation symbols for subtask 2 to : ; ?:0 (0 indicating no punctuation) and mapped the symbols !; to :, the period. We removed all sentences from the data that contain other punctuation symbols such as parentheses, as there is no straightforward way to remove punctuation without interfering with the naturalness of a sentence. This removal affected the data for both subtasks and resulted in removing less than 10% of the data per language. We also removed HTML artifacts, and special (non-visible) characters (zero width space, soft hyphen) from the data. Finally, we omitted sentences with fewer than 3 tokens and documents with fewer than 2 sentences.
The data format is as follows: Lower-cased tokens per file are listed vertically, and the labels for subtask 1 (binary classification) and 2 (multiclass classification) are appended horizontally, separated by tab. The labels encode whether a token emits a sentence end (subtask 1) and a punctuation symbol (subtask 2). Table 1 shows an example.
Per language, we randomly selected 80% of the documents for the training and 20% for the test set. From the the training set, we then randomly sampled 20% of the documents as the development set.
Table 2 shows several statistics of our data. We see similar properties for all languages: Most sentences are unique, and there are few sentences that occur both in the train and test sets.8 German fea6The forth, Romansh, is not represented in Europarl. 7Incorporating further languages from the OPUS corpus using our scripts is seamless as the data format is consistent across languages.
8Duplicate sentences are often formulaic, administrative ones, like ”The session is adjourned.” etc. tures the largest vocabulary, as is expected due to its morphological richness, and the vocabulary overlap between train and test sets is roughly 50% for all languages.
Concerning the labels, the data is highly skewed towards the 0 label for both tasks, as most tokens do not emit a sentence end or punctuation symbol after them. For example, there are 9’618’776 tokens with the label 0 and 420’446 with label 1 subtask one in the English test set, which yields an average sentence length of almost 24 tokens. Table 3 shows a breakdown of the label counts in the English test set for subtask 2. It shows that the period and comma symbols have similar counts and are the most frequent labels among the non-0 labels. The remaining labels occur less than an order of magnitude less frequently. These label distribution properties are similar across all languages. 3.2
The Europarl corpus covers domain-specific
language, i.e. political statements in the European
parliament. To measure how well the participating
systems trained on our data generalize to
out-ofdomain data, we incorporated a surprise test set
comprised of TED talk transcripts9
For each language, we sampled 500 TED talks, favoring those that have the lowest vocabulary overlap with our Europarl test sets to maximize the vocabulary shift. The document-based average percentage of the vocabulary overlap ranges from 85
to 90, meaning there are on average 10-15% of tokens per document in the surprise test set that are not in the Europarl test set.
While being one order of magnitude smaller than the Europarl test set, the surprise test set is also highly and similarly imbalanced regarding the label distribution. In the English surprise test set, there are 67’446 tokens with label 1 and 1’014’464 tokens with label 0. This yields an average sentence length of 16 tokens, which is significantly lower than the 24 tokens in the English Europarl test set. The label counts for subtask 2 follow an almost identical distribution in both test sets. 4
ZHAW-mbert: We provided a baseline based on
the multilingual BERT model
ZHAW-adapter-mbert: To contrast the
resource-intensive fine-tuning of mBERT with a
computationally cheaper approach of task adaption,
we apply the adapter-transformers library11
OnPoint: In their study of sentence segmentation, Michail et al. (2021) proposed a majorityvoting ensemble model consisting of several Transformer models trained in different ways. The models’ predictions are leveraged at test time using a sliding window to obtain the final predictions. They offered their system as language-dependent models for all four languages of the shared task and both sub-tasks.
10https://github.com/ThilinaRajapakse/ simpletransformers
11https://github.com/Adapter-Hub/ adapter-transformers/ train\test #tokens
Unbabel-INESC-ID: Rei et al. (2021) extend the architecture proposed by Rei et al. (2020) to develop a multilingual model for sentence end and punctuation prediction. Their system is designed based on pre-trained contextual embeddings and built on top of a pre-trained Transformer-based encoder model. They propose their method as a single multilingual model for all languages and subtasks of the shared task.
UR-mSBD: Donabauer and Kruschwitz (2021) propose a system based on a pre-trained BERT model and fine-tuned for the first sub-task. They use language-specific models for each of the four languages of the shard task. They consider sub-task 1 as a binary classification problem by identifying tokens that indicate the position of a full stop.
oneNLP: Applying multi-task Albert for English and multi-lingual Bert for other languages Mujadia et al. (2021) explored the impact of using contextual language models for sentence end and punctuation prediction. They modeled the problem in both subtasks as a sequence labeling task. They presented the results of employing a baseline CRF, as well as the results of applying a fine-tuning method over contextual embedding.
HULAT UC3M: Based on the Punctuator
framework
HTW: Guhr et al. (2021) modeled the task as a token-wise prediction and examined several language models based on the transformer architecture. They trained two separate models for the two tasks and submitted their results for all four languages of the shared task. They advocated transfer learning for solving the task and showed that the multilingual transformer models yielded better results than monolingual models. By pruning the Bert layers, they also showed that their model retains 99% of its performance without 1/4 of the last layers. 5
In section 3.1 we showed that our data is highly imbalanced regarding the label distribution. Accuracy or Macro F1 scores are not suitable metrics in this setting, as majority class prediction would yield an accuracy of 96% for subtask 1 on the English test set, e.g. Therefore, we applied the following metrics to evaluate the participants’ submissions: • Subtask 1: F1 score of the label 1 (the positive class, i.e. sentence end) • Subtask 2: Macro F1 of the selected punctuation symbols
We observe that a) most systems achieve a very high score for subtask 1 for all languages on the Europarl data, and b) the F1 scores are almost identical (with seemingly minor differences in precision and recall) for the top-ranking systems for both tasks. Further, the top-ranking systems are the same ones for both tasks. This is to be expected to some degree, as it can be argued that subtask 2 subsumes subtask 1.
While the F1 scores for subtask 2 seem low compared to subtask 1, a more detailed results analysis reveals that the lower (Macro) F1 scores mainly stem from the labels with the lowest counts in the data. Table 6 gives the detailed classification report for the top three ranking system for the English test to the Europarl dataset, we train the ZHAW-mbert set. It shows that the systems are able to predict approach on the remaining TED talks that were not periods, commas, and question marks reliably, but selected for the surprise test set and then test the that they struggle with hyphens and colons, which system on the surprise test set. Table 7 shows that lowers the Macro F1 scores.
Label htw+t2k
OnPoint
Unbabel 0 , . : ? 0.99 0.82 0.95 0.42 0.57 0.88 0.99 0.82 0.95 0.41 0.57 0.91 tems on the English test set for subtask 2.
All systems perform significantly worse on the surprise test sets for both tasks. To gauge the difficulty of the task on the TED dataset compared the average F1 score does improve by 11 percentage points when training the ZHAW-mbert system on domain data. Still, the 0.66 F1 score is 9 percentage points behind the average F1 score on the Europarl data. Hence, the drop in performance of Europarl-trained ZHAW-mbert on the surprise test set can both be accounted for by the domain shift and by the increased difficulty of the target domain (TED talks).
We expect that this applies for the performance drop of all systems.
ZHAW-mbert
Prec. for subtask 2 (averaged over all languages).
We expected some submissions to use linguistic features such as part-of-speech tags or partial syntax parse trees and hypothesized that such systems would fare better on out-of-domain data. However, all participating systems applied neural encodings of the surface tokens and did not encode linguistic features explicitly. Still, the ranking of the systems remains intact on the surprise test sets.
The top three systems in both tasks all use transformers-based approaches and tackle the tasks in a similar manner. We hypothesize that this is the main reason for near identical performance of the systems in terms of F1 scores. Based on the task results, these three systems seem to produce near-identical output. To better gauge their similarities and differences, we evaluate their outputs for subtask 2 in a pair-wise manner on the English test set. We apply the evaluation metric such that one system output takes the role of the ground truth and the other the one of the system prediction, which yields the F1 scores per class that we leverage as an indicator of the similarity or agreement of the per-token predictions. Table 8 shows the results. While the macro F1 scores and even the per-class F1 scores in Table 6 are highly similar, there are significant differences in this analysis. For example, for the hyphen class, the systems have different predictions in over 30% of the cases, and for colon in roughly 20%. For the majority classes of the non-0 classes, the systems disagree in about 10% of the cases for comma, but their predictions are highly similar for period (96% agreement). htw+t2k vs Unbabel
OnPoint vs Unbabel
OnPoint vs htw+t2k
Following Tuggener (2017), we can take the comparison a step further and analyse the type of differences per label. For example, the OnPoint submission’s F1 score for hyphen is 4 percentage points higher than the one of Unbabel, and their prediction agreement for hyphen is 68%. This does not indicate, however, whether OnPoint’s predictions are always better. The aforementioned comparison takes a ground truth label G, the predicted label A of one system, and the predicted label B of another system and defines three types of differences for the cases where A 6= B: • correction: G = B • new error: G = A • changed error: G 6= A 6= B
Table 9 shows the results. We see that the predictions of commas makes up a large portion of the differences. When OnPoint’s prediction differs from Unbabel’s for comma, OnPoint is correct and Unbabel incorrect in nearly 70% of the cases, which explains the 2 percentage point higher performance of OnPoint in Table 6. Still, Unbabel is correct in almost 30% of the cases where the two predictions differ. While we showed that there are differences in the outputs of the top three systems that are not reflected in the averaged F1 scores, the declared criteria for winning the task are the averaged F1 scores in Tables 4 and 5. Since the top three systems in these tables are practically indistinguishable based on these F+ scores, we declare OnPoint, htw+t2k, and Unbabel as the joint winners of the SEPP-NLG 2021 shared task. Congratulations!
We presented the setting and results of the first Sentence End and Punctuation Prediction in NLG text (SEPP-NLG 2021) shared task. We found that all participants explored neural networks-based models (particularly transformers) to tackle the task. The results for the in-domain Europarl data were high for the most common punctuation symbols, but the performance decreased significantly when the models were faced with out-of-domain data.
The discussion of the task results during the session at the SwissText conference yielded the following desiderata for future iterations of the shared task: • More heterogeneous data (more domains) • Add truecasing as an additional task • Add other language families • Take inference time / computational costs as an additional evaluation criteria, or create a separate track that puts emphasis on a lowresource/low-latency setting
We thank the participants for their submissions and their valuable feedback on early versions of the data and task details. This work was funded by Innosuisse under grant project nr. 43446.1 IP-ICT.