In this paper, we propose an evaluation of a Transformerbased punctuation restoration model for the Italian language. Experimenting with a BERT-base model, we perform several ne-tuning with di erent training data and sizes and tested them in an in- and crossdomain scenario. Moreover, we o er a comparison in a multilingual setting with the same model ne-tuned on English transcriptions. Finally, we conclude with an error analysis of the main weaknesses of the model related to speci c punctuation marks.
Nowadays, Automatic Speech Recognition (ASR) and Speech-to-Text technologies and services have reached an incredible level of accuracy in transcribing recorded (or live) speech audio streams. A simple but e ective test can be run, with any modern smartphone, by using the dictation feature to write a text message.1 However, we can immediately notice that the audio stream is transcribed as a word stream, lacking any punctuation or sentence segmentation, and sometimes pieces of text are di cult to understand without some attempts to mentally insert punctuation marks in the ow of words.2
Lack of punctuation may be a minor problem in everyday short-text
messaging, but correctly inserted punctuation is crucial in long speech transcription,
live subtitling or any NLP processing of speech data, especially for downstream
processes such as parsing, information extraction, dialog modeling. Many major
Copyright © 2021 for this paper by its authors. Use permitted under Creative
Commons License Attribution 4.0 International (CC BY 4.0).
1 Obviously, the speech must have a close-to-standard accent without using dialectal
or slang words.
2 Other than di cult, unpunctuated text can be also ambiguous. Here is an
amusing example of two completely di erent letters, with the same words but di erent
punctuation: https://www.nationalpunctuationday.com/dearjohn.html.
commercial services such as Google Cloud3 or Microsoft Azure4 o er the option
of including automatically generated punctuation. As well, it is possible to train
a public ASR model, such as wav2vec [
The rest of the paper is organized as follows: in Sec. 2 we present related works, in Sec. 3 we introduce the setting, models and data used for the experiments, in Sec. 4 discuss the obtained results and in Sec. 5 we conclude the paper.
Contributions In this paper we: i) investigate the impact of di erent training sizes on the performance of a punctuation restoration model based on the Transformer architecture; ii) we test the performance of the model in di erent scenarios (in- and cross-domain); iii) we compare the results obtained in Italian with those obtained with an English model; iv) we inspect the most common errors emerged during the experiments. 2
Punctuation restoration is a well known task, especially in Speech Processing
and Machine Translation, where many approaches have been tested to tackle
the problem in the past decades. In early attempts, acoustic features has been
exploited to train nite-states or Hidden Markov Models [
To solve this problem, multimodal models have been proposed, making use
of parallel audio and transcripts as training [
Obviously, multimodal models need a discrete amount of parallel audios and
texts, and outside the English World it is not trivial at all to nd such data.
Given that, many works have focused on textual-only approaches [
More recently, the potential of Transformer-based Neural Language Models
(NLMs) have been exploited in several studies [
While the vast majority of this research is focused on the English language,
relatively little work has been done to inspect the potential of these models on
other languages. [
We explored the potential of transformer based language models for the
punctuation restoration task on the Italian language. Speci cally, we de ned two sets
of experiments. The rst consists in evaluating the impact of the ne-tuning set
size on the task performances. For that purpose, we tested the performance of
a state-of-the-art transformer based architecture for punctuation restoration [
In the second set of experiments, we compared the performances of two differently ne-tuned models on 4 test datasets, as explained in 3.2. Moreover, we proposed an error analysis in order to investigate strength and weakness of the proposed methodology.
Model and datasets used for the experiments are described below. 3.1
We relied on the architecture previously de ned in [
The pre-trained Transformer used in our experiments is the XXL uncased
version of the BERT model for the Italian language developed by the MDZ
Digital Library Team and available trough the Huggingface's Transformers library
[
The model has been ne-tuned on two corpora, in order to evaluate divergences in
the results with respect to the domain variation deriving from di erent data. The
rst corpus is a large collection of authentic contemporary texts in Italian derived
from the web, and it is the de-facto reference corpus for Italian in many NLP
applications: the Italian Web as Corpus (ItWaC) [
The second corpus used for the ne-tuning is the Italian sub-corpus of the
Opensubtitles Multilingual Corpus [
By creating two ne-tuned models, we want to investigate if the language diversity observable in the two corpora (i.e. average written language and multiregistry from ItWaC, close-to-spoken but highly regular from Opensubtitles) is re ected in the way the models handle punctuation.
Dataset Sentences Tokens Commas Periods Questions
ItWaC 765,491 19,226,715 (25.12) 1,403,527 (1.83) 729,806 (0.95) 35,685 (0.05) Opensubtitles it 1505279 14,468,346 (9.61) 754,951 (0.5) 1,265,306 (0.84) 239,973 (0.16) ParlaMint it 134,887 3,203,374 (23.75) 238,960 (1.77) 130,386 (0.97) 4,501 (0.03) TEDx it 1,139 21,667 (19.02) 1,823 (1.6) 1,070 (0.94) 69 (0.06) TEDx en 1,210 21,383 (17.67) 1,636 (1.35) 1,142 (0.94) 68 (0.06) Table 1. Statistics on the datasets used for ne-tuning and test. In parenthesis, the average distribution per sentence.
Moreover, we considered other two resources for the purpose of evaluating
the two models performances in a cross-domain scenario. The rst resource is
the Italian part of the ParlaMint Comparable Corpora [
With the second test-only resource we also introduce a multilingual setting,
useful to compare models performances with reference systems available for
English language. We used the Italian-English alignment of the Multilingual TEDx
Dataset [
Table 1 reports some numbers about the datasets herein described. These statistics refer to the whole set of texts processed, and for all the experiments conducted with di erent size of ne-tuning a random selection of sentences has been collected.
Data pre-processing The model implemented in our experiments is trained on a classi cation of tokens on the basis of the presence or absence of a punctuation mark immediately after the target token. It is important to remember that punctuation is a feature of the written language modality, and it is used to mimic oral pauses in the transcription of speech: commas are used for short pauses, periods for long pauses at the end of an utterance and question marks for questions. For this reason, we collapsed all the possible punctuation marks to these 3 classes, reducing the complexity of the ne-tuning data. 10 The complete collection of comparable corpora in 17 languages is available at: https://www.clarin.si/repository/xmlui/handle/11356/1432 11 The full dataset is available at: http://www.openslr.org/100/ We rst investigate the impact of di erent training sizes on the performance of BERT-BiLSTM. In order to do so, we ne-tuned our punctuation restoration models in parallel, with increasingly large portions of the two corpora, from 100k to 10 million tokens, and then tested them on a previously unseen portion of the two datasets consisting of 200k words. Results (in terms of micro F-score) are reported in Figure 1. As a general remark, we found that, for both models, the curve tends to atten out when the ne-tuning process is performed with portions larger than 2 million tokens. As regards the di erences between the two datasets, we can notice that the model ne-tuned on Opensubtitles performed slightly better than the one trained on ItWac. For instance, focusing on the results obtained in the last run (10 million tokens) we can observe that the di erence between the two models in terms of F-score is about 0.05 points (0.75 vs. 0.70). Moreover, it is interesting to note that while the Opensubtitles model obtained quite good results even with very small portions of the dataset (e.g. 100K), the itWac model requires at least one million words to achieve comparable results. This behaviour is quite predictable due to the fact that Opensubtitles texts are extremely regular and minimum variation is appreciable through the whole set of data. On the contrary, using ItWaC that is more heterogeneous, the system need more data to start correctly modelling the distribution of punctuation marks.
To better investigate their performances, we report in Table 4 the results obtained by the two models ne-tuned with 10 millions words from Opensubtitles/ItWac and tested in two di erent scenarios: i) in-domain, i.e. testing on the same dataset; ii) cross domain, i.e. testing on the other domain. Moreover, in order to provide a direct comparison between the two models, we tested both their performance on the Parlamint datasets.
As it can bee seen by looking at the average scores (column Avg ), the indomain con guration always achieves the best results (ItWaC: 0.65; Opensubtitles: 0.73). By focusing on the cross-domain con gurations, it is interesting to notice that the high variability of ItWaC texts strengthens the model and enables it to handle punctuation with better performances with respect to the model ne-tuned on Opensubtitles. Speci cally, observing the performances of ItWaC model on Opensubtitles testset and viceversa, we notice a di erence of 0.05 points. While, looking at both models (ItWaC and Opensubtitles) tested on ParlaMint, the gap increases to 0.07 points in favour of the ItWaC model. We can explain this behaviour on the basis of the nature of the ParlaMint dataset, where regularity and formality leads to longer sentences with punctuation usage closer to average written texts. Thus, ItWaC model, which is based on an heterogeneous collection of texts larger than Opensubtitles, is capable of predicting punctuation in a more robust way.
Looking at per-class scores, it is possible to notice that all systems perform better in predicting the PERIOD class: with exclusion of the Opensubtitles model tested on ItWaC, all scores are above 0.70. This result is encouraging because periods, exclamation marks and other full stops are used to signal the end of a sentence, thus a similar model can be e ectively exploited to tackle the task of segmenting the continuous ow of speech transcription, enabling better subsequent sentence-based methods of analysis (e.g. part-of-speech tagging, dependency parsing and so on).
We register lower gures on the QUESTION class. It is probably due to the unpredictability of these in Italian only on the basis of transcribed text, without considering intonation. We further investigate this problem in 4.2. 4.1
As already mentioned in Sec. 3.2, we also decided to compare the performance of
our ne-tuned models with a reference system available for the English language.
Speci cally, we compared the results obtained with the ItWaC/Opensubtitles
models when tested on the Italian transcriptions of TEDx speeches with the
ones obtained by the system devised in [
As noticed in the previous experiments, the model ne-tuned on ItWaC data achieve better results when tested in a cross-domain scenario. In fact, we can observe a di erence of about 0.03 points in terms of average F-scores. In this respect, it is interesting to note that the main di erence between the performance of the two models is due to the classi cation of question marks. Focusing instead on the comparison between Italian and English models, we can clearly observe that the latter outperforms the Italian ones. Also in this case, the classi cation of question marks is the one that contributed most to the score di erence between the models. This result is quite expected, since a question in English, beside the presence of a question mark at the end of the sentence, is usually characterised by an inversion of the subject and the verb in the principal clause. On the contrary, in Italian the punctuation mark is the only discriminating feature of questions in the written modality, while intonation plays the main role in spoken Italian. Therefore, the identi cation of question marks tends to be much easier for the English language. 4.2
In order to further deepen our analysis, in this section we investigate in more detail the main errors made by the two models when predicting the di erent punctuation marks. In Figure 2 we report the confusion matrices (in terms of accuracy) of the results obtained by the model ne-tuned on ItWaC and tested on Opensubtitles and vice-versa. As a general remark, we can highlight that the COMMA class is the most confused in both models. Due to the unbalanced distribution of the O (Other) class with respect to the punctuation classes, the high confusion of every class with this one is easily predictable. Thus, if we exclude the O class from the gure, we can notice that the class with which the COMMA is often confused is PERIOD, for both models tested on the opposite dataset (e.g. model ne-tuned on ItWaC and tested on Opensubtitles and vice versa). We can ascribe this problem to the average length of sentences, that diverges between the two: in ItWaC, the average sentence counts 25.12 tokens with about 1.83 commas per sentence; in Opensubtitles, the average sentence is 9.61 tokens long, with a distribution of commas of 0.5. For this reason, we can assume that the Opensubtitles model tends to create shorter sentences, thus using the full stop mark more frequently than the ItWaC one.
Figure 3 reports instead the confusion matrices of the results obtained by the two ne-tuned models (ItWaC and Opensubtitles) on ParlaMint test data. As we have seen previously, the model ne-tuned on ItWaC is the one that achieved better results regardless of the class taken into account. In fact, with the exception of Other (O ), in all the other classes we observe a performance gap that goes from 0.2 (PERIOD ) to 0.11 (COMMA) accuracy points. Focusing on the mismatched classes, we can see once again that commas are often mistaken as other tokens (O ), while question marks are classi ed as periods, as in the following example:
Original: Vorrei ricordarvi i fallimenti ai quali siete andati incontro e state continuamente andando incontro con i bonus. Devo ricordarvi, forse il bonus vacanze? [en. I would like to remind you of the failures you have experienced and you are continually experiencing with the bonuses. Should I remind you the holiday bonus? ] ItWaC/Opensubtitles: Vorrei ricordarvi i fallimenti ai quali siete andati incontro e state continuamente andando incontro con i bonus. devo ricordarvi, forse il bonus vacanze.
If we look at the di erences between the two models, we can clearly notice that the one ne-tuned on Opensubtitles tends to wrongly classify in 0.15 of the cases a comma also as a full stop, as in the following example: Original: Scusate la digressione: pure io sono un mancato operaio, due braccia rubate all'agricoltura - allora lo si diceva in senso denigratorio, mentre oggi tale de nizione si e quali cata un po' di piu - e ho potuto permettermi di studiare e di laurearmi. [en. Sorry for the digression: I am too a nonworking class person, two arms stolen from agriculture - at the time this was said in a derogatory sense, whereas today this de nition has been requali ed - and I was able to a ord my studies and my degree.] ItWaC: Scusate la digressione, pure io sono un mancato operaio, due braccia rubate all'agricoltura, allora lo si diceva in senso denigratorio, mentre oggi tale de nizione si e quali cata un po' di piu e ho potuto permettermi di studiare e di laurearmi Opensubtitles: Scusate la digressione. pure io sono un mancato operaio. due braccia rubate all'agricoltura. allora lo si diceva in senso denigratorio, mentre oggi tale de nizione si e quali cata un po' di piu e ho potuto permettermi di studiare e di laurearmi
From the previous example, we can also highlight that the hyphens were correctly classi ed as COMMA by the ItWaC model (punctuation marks to class mapping in Table 2), while they were identi ed as full stops by the Opensubtitles one. This could be due to the fact that since the Opensubtites dataset is composed of shorter sentences (derived from transcribed dialogic turns), the model tends to extend this behaviour on its inferences. Conversely, the colon were correctly identi ed as PERIOD by the Opensubtitles models. 5
In this paper we veri ed if it could be possible to obtain good results in restoring punctuation in raw transcription texts by means of a ne-tuned Transformers model. We chose to exploit 2 corpora as ne-tuning, namely ItWaC and Opensubtitles, in order to observe the di erences emerging from domain variety and their projection on performances.
First, we evaluated the impact of di erent sizes of ne-tuning datasets, and we observed that the model ne-tuned on highly regular data (i.e. Opensubtitles) need less information to start modelling punctuation with regards to the model ne-tuned on more heterogeneous data (i.e. ItWaC); for both models, the curve tends to atten out with ne-tuning portions larger than 2 million tokens. Moreover, the model ne-tuned on ItWaC obtains the best results when tested cross-domain on ParlaMint dataset, which is used as neutral testing eld for both models.
Lately, we o ered a comparison between the Italian models herein ne-tuned
and the English model originally presented in [