Generally, the output of automatic speech recognition (ASR) systems ignore the prediction of punctuation marks. Similarly, output of OCR systems (Nguyen et al., 2019) need automatic validation for punctuation. Apart from the omission of punctuation markers, some automatic tools generated texts e.g. PDF to text extraction may erroneously displace sentences for several reasons. Here, detecting the end of a sentence and placing an appropriate punctuation mark significantly improves the quality of such outputs by preserving the original meaning. Thus, missing punctuation or inappropriate punctuation degrade the readability of the presented text and leads to poor user experiences in real-world scenarios (Che et al., 2016;Ueffing et al., 2013) as well as erroneous input to the subsequent automatic systems such as Machine Translation, Summarization, Question Answering, NLU etc. Therefore it is necessary to restore or correct punctuation marks for these automatic outputs.
Traditionally, automatic punctuation marking approaches (Lu and Ng, 2010) can be divided into three broad categories (Vandeghinste et al., 2018) based on the used features. They can be prosody based features (Kim and Woodland, 2001;Christensen et al., 2001), lexical features (Augustyniak et al., 2020;Peitz et al., 2014) or combined or hybrid features of the previous two features based methods. Recent lexical based punctuation prediction methods build upon deep neural networks where it is modeled as a sequence to tag prediction task (Li and Lin, 2020) or a sequence to sequence prediction task (Vandeghinste et al., 2018).
The simplest and basic form of punctuation prediction is the discovery of sentence boundaries, here the problem is the binary classification (where classes are period or empty as label). An incremental and a bit harder problem is the prediction of each individual punctuation, here the class labels for subtask2 are ": -, ? . 0" (0 indicating no punctuation). SEPP-NLG 2021 presents both these tasks as a challenge for the English, German, French and Italian languages.
In a recent advance of deep learning, pretrained language models such as ELMo (Peters et al., 2018), ULMFiT (Howard andRuder, 2018), OpenAI Transformer (Lee andHsiang, 2020) and BERT (Devlin et al., 2018) have resulted in a massive jump in the state-of-the-art performance for many NLP tasks, i.e text classification (Büyüköz et al., 2020), natural language inference and question-answering, dialogue system (Budzianowski and Vulić, 2019) etc. All these approaches pre-train an unsupervised language model on a large corpus of data such as all wikipedia articles, news articles and then fine-tune these pretrained models on different downstream tasks.
Here, for our experiments on the two punctuation prediction tasks, we try to use multi-lingual Bert and ALBert (for English) as a fine-tuning task along with the baseline experiments with CRF. casing and removing all punctuations in the transcripts available in multiple languages. Table 1 describes the corpora details for Training and Development corpus for all languages in terms of sentences and Tokens. Table 2, Table 3, Table 4 and Table 5 describe the training corpora details in terms of punctuation classes and their respective distribution. Here data numbers are given after ''! ;'' are mapped to ''.''.
We primarily used two broad categories of approaches. We model the problem as a sequence labeling task. In Machine Learning approaches, we trained a CRF model to identify the different kinds of labels correctly. Transformer based BERT fine tuning is also used as the other technique.
We split the training data in English into sequences of 25 tokens each. This decision of setting the maximum sequence length to 25 was based on the average sentence length of the training data in English. We only used words as features and utilized a continuous window of 5 words over the full corpus as the required features for the CRF.
Multi-task learning (MTL) is a technique which aims to improve generalization, strengthen representations and enable adaptation in machine learning (Worsham and Kalita, 2020) for related tasks. For our case, we enabled multi-task learning for our AlBert and mBert based contextual experiments as presented in Figure 1 where contextual embeddings are shared across the sub-tasks. We applied a sequence to tag classifier on the output contextualized token embeddings of Albert/mBert for the tag prediction. Here, we have used Albert2 for English
As the results of CRF with word level features for English were poor (shown in table 7), we did not conduct CRF experiments on other languages.
We could observe that the results of Bert with Multi task learning is superior to the results of CRF. This is due to the better sentence or sequence representations learnt from the transformers. Simple surface level word features fail to capture the end sentence or punctuation markers in CRF.
We have successfully applied contextual embedding for the task of punctuation prediction and achieved comparable results on both of the subtasks. We believe that fine-tuning Bert on more data would benefit the overall punctuation task. Also,
Lang Pr Re F1 the language specific contextual embedding would improve performance in other languages. We will be incorporating both of these points in our future work.