This paper presents our solutions for the SwissText 2021 shared task “Sentence End and Punctuation Prediction in NLG Text”. We engaged with both subtasks (i.e., sentence end detection and full-punctuation prediction) and built systems for English, German, French and Italian. To tackle the punctuation prediction problem, we ensemble multiple differently trained Transformer models (BERT, CamemBERT, Electra, Longformer, MPNet, XLM-RoBERTa, XLNet) and leverage their results using a sliding window method during inference time. As a result, we achieve an F1 score of the positive class of 0.94 for English, 0.96 for German, 0.93 for French, and 0.93 for Italian for the subtask 1 “sentence end detection” on the respective test sets. Furthermore, Macro F1 results on test sets for subtask 2 “full-punctuation prediction” for English, German, French and Italian are 0.78, 0.81, 0.78, 0.76 respectively.
Transcribed or translated texts often contain erroneous punctuation. Correct punctuation, however, is crucial for human understanding of a text, as shown by T u¨ndik et al. (2018). Rightly placed punctuation not only makes the text more readable and intelligible but can change the meaning of sentences, as well. Translated texts pose another challenge: Different languages expose different sentence structuring conventions and hence use punctuation very differently.
However, systems for automatic transcription of
speech nowadays focus on minimizing the Word
Error Rate (WER), which omits punctuation
Therefore, attempts in improving the quality of such texts must also focus on a more precise prediction of punctuation. Consequently, this is an ongoing research effort in the NLP community. Recent developments in NLP (such as Transformers) offer new possibilities to tackle punctuation prediction effectively. Some of these attempts are discussed in Section 2.
Following recent attempts, we propose an ensemble system based on the Transformer architecture, where multiple models predict the punctuation symbols of a given text. The results are then combined and the final predictions are made. Our languagespecific systems are able to predict punctuation for English, German, French, and Italian texts and are on par – if not better – with current state-of-the-art models that participated in the shared task.
Our main contributions include 1. the exploration of different Transformer-based models and identification of the most important features which affect the performance for this task, and 2. a showcase that the ensembling of differently trained models enhances the performance for the punctuation prediction task. 2
Punctuation prediction tasks pose many challenges. One of them is the restricted input length thus restricted context for the Transformers. To solve the above mentioned limitation, Nguyen et al. (2019) used an overlapped chunk method (i.e., an overlapping sliding window) combined with a capitalization and a punctuation model to tackle the punctuation problem in long documents. First, the text is divided into chunks with overlapping segments. Second, a punctuation model (seq2seq LSTM, Transformer) predicts punctuation and capitalization for every segment. Lastly, overlapped chunk merging combines chunks by discarding a defined number of tokens per overlapped chunk.
Courtland et al. (2020) changed the usual framework of punctuation prediction to predicting punctuation for the whole sequence rather than for single tokens. The authors used a feedforward neural network. Similar to Nguyen et al. (2019), they find that using a sliding window approach improves prediction performance. However, instead of producing multiple predictions for the same token, they sum activations before prediction and make inference afterwards.
Sunkara et al. (2020) used a joint learning objective for capitalization and punctuation prediction. The model input are sub-word embeddings. The authors used the pre-trained BERT model (BERT base truncated to the first six layers). They finetuned the model on medical domain data because the medical domain was in the main scope of this paper. They also fine-tuned the model for the punctuation prediction task. The authors used masked language learning objective while forcing half of the masked tokens to be punctuation marks.
Similarly, Nagy et al. (2021) also leveraged pretrained BERT models (BERT base cased and uncased and a smaller version for English; multilingual and Hungarian-specific BERT versions for Hungarian). They added a two-layer multi-layer perceptron network with a soft-max output layer. The model also used a sliding window approach to enhance the results further. This model is trained to predict four labels: empty (no punctuation), comma, period and question mark.
Our approach differs from the above mentioned in using ensembling of multiple pre-trained Transformer-based models fine-tuned for the given task. Very importantly, our systems predict six different punctuation symbols for the punctuation prediction task.
Additionally, a multilingual Transformer was used as a part of our ensemble. We hypothesize that it would be able to capture more accurately the multilingual content of the EuroParl data. Furthermore, low-resource Latin languages might benefit from pre-training on more data, e.g., including other Latin languages.
In Section 3 we will discuss the datasets we used and the challenges they provide. The problem is described in Section 4 together with detailed description of our approach. Section 5 contains explanation of used hyperparameters, technical details and experimental setup. Section 6 presents our results and discusses the impact of used methods. Finally, a conclusion is drawn in Section 7. 3
The Europarl Parallel Corpus
Subsequently, we outline challenges that we believe are especially relevant in solving this shared task and thus directly influenced our proposed system architecture. 3.1
As shown in Table 1, the mean token length is many orders of magnitudes longer than what typical Transformer architectures can process at once (typically up to 512 subtokens). It should be noted that some of the documents are especially long and can contain up to 100,000 tokens. The most obvious solution would be just to split documents into smaller sequences and subsequently merge predictions. However, this approach lowers the context with which a model is confronted and might lead to lower prediction quality (presumably at the beginning and end of a sequence). 3.2
To some extent, documents in the EuroParl Corpus contain multilingual content. As shown in the examples in Table 2, many documents contain names of people and areas that reflect the multilingualism of the participants of the European Parliament, i.e., members come from all over Europe. Therefore, using pre-trained models trained on monolingual
i agree completely with mr po¨ ttering and with you too mr swoboda the president of the european commission jose´ manuel dura˜o barroso however data only may result in an inaccurate representation of this content. The class distribution of the training and evaluation set, as shown by Table 3, presents a rather typical situation in machine learning: Some of the classes have very few examples compared to the biggest classes. Neglecting this circumstance will likely lead to low performance for minority classes. Using typical techniques such as class-specific loss weights or data augmentation might improve performance to some extent. We have tried to reduce this problem by adding a model with altered loss weights to the ensemble.
Punctuation : ? . , 0
We modelled this problem as a token classification task. More precisely, each token is assigned a label representing the following punctuation symbol (if any). We concentrated our main efforts and focus on the full punctuation prediction. As such, we built all of the models to be able to predict all punctuation symbols. For the end of sentence prediction task, we mapped predictions of ‘.’ ‘?’ to 1 and the rest to 0.
The corner-stones of our systems are pre-trained Transformer models. We trained four different finetuned models for each language and combined the predictions using majority vote ensembling (see Section 4.5). Table 4 provides an overview.
Electra
Both MPNet
Longformer
XLM-RoBERTa
The authors of CamemBERT
BERT
As discussed earlier, documents in the corpus can be rather long, and typical Transformers cannot process such documents at once. Therefore, instead of simply splitting the documents into smaller segments, sequences are overlapped for inference. In other terms, a sliding window is applied, as suggested by Nguyen et al. (2019). Subsequently, the overlapped sequences are merged back together by discarding half of the overlapped tokens at the begging and end of each sequence. Our experiments have shown that an overlap of 40 tokens performs best. Consequently, we chose this overlap length for the final models. 4.4
For the German, French and Italian ensembles, we retrained the best performing model with weighted loss. We set the weights to three for the two least performing classes (‘-’, ‘:’) and left them unchanged for the other classes (i.e., a weight of one). The idea is to increase recall for these classes by sacrificing overall performance, which, in return, helps an ensemble to create more accurate predictions.
Initially, we used inverted class frequencies as loss weights. However, this approach turned out to be too aggressive (worse minority class and overall performance). Further, we experimented with increasing minority class (‘-’, ‘:’) weights. Initial experiments showed showed that weights set to three for minority classes and one for majority classes performed best on the development set. Our approach is rather heuristic, and further experimentation may lead to better results. 4.5
We did preliminary experiments in separate stacking models as mentioned in Wolpert (1992) as well as ensembling using the arithmetic average of class probabilities of single models as described in Goodfellow et al. (2014). However, one technique was shown to be more effective: majority vote ensembling. More concretely, all the models predict (i.e., vote) and the most voted label is then used as the final prediction. In case of a tie, the least common
0.943 0.955 0.933 0.926
0.777 0.812 0.778 0.755
Electra 0.940 Electra 0.954 Electra 0.923 Electra 0.922
0.769 Electra 0.803 Electra 0.758 Electra 0.746 Longformer 0.934 XLM-RoBERTa 0.952 XLM-RoBERTa 0.926 XLM-RoBERTa 0.918
P R F1 0.82 0.75 0.78 0.82 0.80 0.81 0.80 0.76 0.78 0.77 0.74 0.76
P 0.81 0.82 0.78 0.77
0.75 0.80 0.77 0.74 F1 0.77 0.81 0.77 0.75
P R F1 0.65 0.59 0.62 0.66 0.65 0.65 0.63 0.60 0.61 0.57 0.55 0.56 label is chosen. Additionally, predictions for a hyphen are counted twice – mainly to increase the performance for the worst-performing label (which was the case for all languages). Our experiments on the development set have shown that this leads to an increase of 1-2% Macro F1 score for all languages compared to the single best performing model. 5 5.1
At the beginning of development, we empirically determined what characteristics of the model and fine-tuning correlate with better performance. For fine-tuning, five epochs performed consistently well for all transformer architectures. Due to the large document size, the larger the maximum sequence length, the better the performance. To our surprise, there were no significant differences between the performance of cased vs. uncased Transformers on our lower-cased data. 5.2
For the training of our models, we used the Simple Transformers 1 library, a wrapper for the Hugging Face 2 library, that allows for fast experimenting. As the Simple Transformers library does not support weighted loss training, we have adapted the relevant code for this purpose.
1https://simpletransformers.ai 2https://huggingface.co 5.3
We trained all of the models on a single T4 GPU instance. Our final models shared some of the hyperparameters, namely a learning rate of 4e−5, a batch size of 16 (four for Longformer) and the maximum sequence length (512, 4096 for Longformer). We trained each model for five epochs. 6
Our results for sentence end prediction and full punctuation prediction can be seen in Table 7 and Table 8, respectively. They demonstrate the high capability of using Transformers in predicting punctuation marks. Especially for sentence end prediction, the F1 scores are well above 90% for all languages. We hypothesize that it is because usage of sentence end punctuation is less ambiguous – it is consistently and grammatically correctly used in the data. For full punctuation prediction, the overall performance is significantly lower for all languages. The full punctuation prediction task is more difficult not only because of the existence of more labels, but also because some of the labels might not follow strict grammatical rules. For example ‘-’ or a ‘:’ can be used differently due to different styles of linguistic expressions, while a label such as a comma might be misplaced due to human error.
With respect to our system, sliding windows are a simple way to improve performance when an input sequence is much longer than what a model can actually process. However, this performance gain is limited, and as of now, it is not clear how this compares to a model that can process much longer sequences. Observing results we have obtained from single models at Table 5 and 6 for both subtasks we can see that the model architecture has an effect on performance. Within our experiments, majority vote ensembling further enhances performance. 7
In this paper, we showed that the ensembling of diversely trained Transformers can yield significant improvement and allows for good generalisation for punctuation prediction in out-of-domain examples. From this work, it can be seen that combining different Transformers can be really beneficial. However, further work is needed to determine if more advanced ensembling techniques could further increase the quality of the predictions.
We want to thank Simon Clematide and Phillip Str o¨bel for their valuable inputs and the Departement of Computational Linguistics for providing us with the necessary technical infrastructure.