This paper describes our team oneNLP's (LTRC, IIIT-Hyderabad) participation for the SEPP-NLG 2021 tasks1, Sentence End and Punctuation Prediction in NLG Text-2021. We applied sequence to tag prediction over contextual embedding as fine-tuning for both of these tasks. We also explored the use of multilingual Bert and multitask learning for these tasks on English, German, French and Italian.
Generally, the output of automatic speech
recognition (ASR) systems ignore the prediction of
punctuation marks. Similarly, output of OCR systems
Traditionally, automatic punctuation marking
approaches
The simplest and basic form of punctuation prediction is the discovery of sentence boundaries, here the problem is the binary classification (where classes are period or empty as label). An incremental and a bit harder problem is the prediction of each individual punctuation, here the class labels for subtask2 are “: - , ? . 0” (0 indicating no punctuation). SEPP-NLG 2021 presents both these tasks as a challenge for the English, German, French and Italian languages.
In a recent advance of deep learning,
pretrained language models such as ELMo
Here, for our experiments on the two punctuation prediction tasks, we try to use multi-lingual Bert and ALBert (for English) as a fine-tuning task along with the baseline experiments with CRF. 2
As a part of SEPP-NLG 2021, the organizers released an Europarl corpus of spoken texts by lowerClass : , ? .
0
Class : , ? .
0 Total casing and removing all punctuations in the transcripts available in multiple languages. Table 1 describes the corpora details for Training and Development corpus for all languages in terms of sentences and Tokens. Table 2, Table 3, Table 4 and Table 5 describe the training corpora details in terms of punctuation classes and their respective distribution. Here data numbers are given after ‘’! ;‘’ are mapped to ‘’.‘’. 3
We primarily used two broad categories of approaches. We model the problem as a sequence labeling task. In Machine Learning approaches, we trained a CRF model to identify the different kinds of labels correctly. Transformer based BERT fine tuning is also used as the other technique. 3.1
CRF
We split the training data in English into sequences
of 25 tokens each. This decision of setting the
maximum sequence length to 25 was based on the
average sentence length of the training data in
English. We only used words as features and utilized a
continuous window of 5 words over the full corpus
as the required features for the CRF.
Multi-task learning (MTL) is a technique which
aims to improve generalization, strengthen
representations and enable adaptation in machine
learning
EN DE FR
IT AVG EN DE FR
IT AVG EN DE FR
IT AVG Pr 0.73 0.71 and mBert3 for other languages.
The following configuration is used for Bert finetuning.
• Input : Subword tokens (same as Bert/Albert vocab) • Embedding size : 512 • Transformer Config : layers (6), hidden size (2048) attention heads (8) • Dropout : 0.30, Optimizer : Adam • max word sequence length : 300 • Fine Tuning Steps : 90K with 40 Batch size Due to the algorithmic limitations, we were not able to apply MTL for CRF as it does not facilitate the joint learning of multiple classification tasks at one go. 4
As the results of CRF with word level features for English were poor (shown in table 7), we did not conduct CRF experiments on other languages.
We could observe that the results of Bert with Multi task learning is superior to the results of CRF. This is due to the better sentence or sequence representations learnt from the transformers. Simple surface level word features fail to capture the end sentence or punctuation markers in CRF. 5
We have successfully applied contextual embedding for the task of punctuation prediction and achieved comparable results on both of the subtasks. We believe that fine-tuning Bert on more data would benefit the overall punctuation task. Also, 3https://tfhub.dev/tensorflow/bert multi cased L-12 H768 A-12/4 Pr 0.92 0.93 0.9 0.88 0.91 0.81 0.85 0.77 0.78 0.8 0.92 0.94 0.9 0.88 0.91 the language specific contextual embedding would improve performance in other languages. We will be incorporating both of these points in our future work.