PESInet is an Automatic Prosody Recognition system aiming at classifying Information Units as Statement, Question or Exclamation. PESInet adopts a modular architecture, with a master NN evaluating the results of two independent BLSTM NNs that work on audio and its transcription. PESInet has been trained with our own three-class, balanced corpus composed of about 1.5 million text phrases and 60 000 utterances of recited and spontaneous speech. PESInet reached an accuracy of 80% on three classes, and 91% on two classes (Question vs Non-question). Finally PESInet, compared against human listeners on a two-class test based on a different corpus, reached a better Accuracy (89% for PESInet, against 80% for human listeners).
The goal of PESInet was to investigate whether
clues derived from text could improve the
recognition of simple prosodic forms in Information Units
1http://www.polisocial.polimi.it
2https://www.esagramma.net
(IUs). In particular, we focused on Statement,
Question, and Exclamation which are
proposition’s structures and are independent of the
pragmatic function of the corresponding IU: each one
can assume a large set of illocutionary acts, as
explained into the Language into Act theory (L-AcT)
described in Cresti (2014). An IU is composed
of a textual realisation (i.e., a written phrase) and
an acoustic realisation (an audio recording of a
speaker uttering such a phrase), and conveys a
specific informative intention
The majority of studies on prosody regards the
automatic recognition or detection of single prosodic
clues
Copyright 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). eral RNN and Bidirectional RNN (BRNN) models, trained on a simulated call-centre recordings consisting of just 2850 Question and 3142 Nonquestion IUs. The best result is an F1 score of 85.5%.
The work described in Yuan and Jurafsky (2005) focuses on Question and Statement detection, from text and audio, for Chinese; authors investigate the influence of text in prosody comprehension, on a telephone corpus (with transcriptions). Their classifier achieves an error rate of 14.9% with respect to a 50% chance-level rate. Quang et al. (2007) use decision trees to automatically detect Questions in a small elicited French and Vietnamese corpora, leveraging both acoustic and lexical features (unigrams, bigrams, and presence of so-called “interrogative terms”). The best result is an F1 of 80% for the Vietnamese language.
Finally, the work described in
None of the works mentioned above is perfectly comparable with ours and, on the other hand, all of them are based on ah-hoc corpora (as we did). This makes impossible to compare the results we obtained against other approaches. We, however, validated our results comparing our model against human listeners. 4
Our own corpus is composed of eBooks, EPUB3
audio-books (an EPUB3 audio-book contains both
text and audio recording, time-aligned at the level
of sentence), and the LIT/DIA-LIT corpus
From eBooks, the textual part of EPUB3 audiobooks, and transcriptions of LIT/DIA-LIT we extracted about 1.5 million sentences, balanced on the three target classes: Statement, Question, and Exclamation.
From LIT/DIA-LIT audio recordings and the audio part of EPUB3 audio-books, we collected about 60 000 utterances (again, balanced on the three target classes). Both sentences and utterances were tagged with the correct class, leveraging the punctuation marks we found in text/transcriptions. Of course, we removed such punctuation marks from the textual part of the corpus. Moreover, we discarded all the sentences containing a sub-phrase or other complex syntactic structures. In doing so we aimed at retaining plain simple examples of statements, questions, and exclamations.
We are aware that leveraging punctuation marks for tagging sentences can lead to confounds, as exclamation marks is also used for Vocatives and Orders, while the full stop is also used for Orders. Anyway, it was simply not possible to manually review the text collection and manually solve the problem. Thus, we assume our corpus is affected by a small amount of noise (in other words, we assume Exclamations and Statements are way more frequent than Vocatives and Orders).
Notice that the question marks might be used for different question typologies (rhetorical, information-seeking, confirmation-seeking, biassed), and that question could be further partitioned into open questions, polar questions, etc. Thus, the question mark is used to tag sentences with wildly divergent phonetic forms. This is not, however, a blocking issue: it only makes harder for the classifier to learn the input/output correlation. In particular, this is one of the reasons that lead us to the idea of leveraging text to improve the classification of IUs.
Summing up, we built three corpora:
ACorpus: audio corpus composed of about 60 000 .wav labelled samples.
TCorpus: textual corpus composed of about 1.5 million .txt labelled samples.
MCorpus: mixed corpus composed of all the ACorpus files, with their transcriptions (from the TCorpus); about 60 000 labelled samples. 5
From acoustic and textual samples we derived a set of features that our NNs leveraged for training and recognition. 5.1
With a sample rate of 44.1 kHz, we adopted a window of 2048 samples with a hop-size of 1024 samples (i.e., every 23 ms a new vector of acoustic features is produced). Notice that our window is larger than the one usually adopted by ASRs; in fact, we are not interested in phone recognition and, on the other hand, prosody phenomena appear in larger temporal scale than the one involving individual phones.
We tried several window sizes, and several
acoustic features; in particular we experimented
with different combinations of Cepstrum
coefficients. At the end, we come up with the following
129 acoustic features, normalised (to minimise
dependency on speakers and recording settings) and
calculated by means of Praat
To feed the model with textual samples we used
the usual word embedding technique, which
represents the vocabulary in a continuous vector space
of 300 dimensions
Summing up, each sentence was transformed into an array that contains a column of 300 real numbers for each token. Notice that punctuation marks were discarded and no lemmalisation was applied.
Available at: wordembeddings/ http://hlt.isti.cnr.it/
PESInet is composed of three different NNs: Acoustic and textual features defined in Section 5 generated low-level pieces of information, looking at very local phenomena. For considering higher-level phenomena, both the Audio-based and the Text-based NNs relied on the same architecture, leveraging an initial multi-layer convolutional block.
A convolutional layer is composed of several kernels with a predefined width, which “scan” the input array. Each kernel, after the training phase, specialises in finding certain patterns in the input sequence. The network learns “high level” features (i.e., common prosody contours, for the Audio-based NN, or particular word sequences for the Text-based NN) from our low-level feature set.
Features related to prosody unfold along
different time extents
In particular, our convolutional block is composed of three layers, which “scan” at three different temporal granularity levels. In general, if s is the stride adopted for kernels at any temporal granularity level and di is the kernel height at the i-th temporal granularity level, the kernel height at the (i + 1)-th temporal granularity level is di+1 = di + s; see Figure 1, for a simplified example with two levels. Stride is chosen so that, after each shift of the filter, the kernel will include a small subset of the previously analysed input.
Finally, padding is applied to the input sequence, so that the shorter kernels (and, by construction, all the other, longer kernels) fit the sequence length.
Being the kernels of different heights, they will cause the outputs to have different dimension as well, relatively to the layers they come from. These dimensions are adjusted in the following layer of the network. Figure 2 shows a simplified schema with two differently-sized kernel groups. 6.2
Both the Audio-based and the Text-based NNs relied on a multi-layer network. The general architecture is composed of three BLSTM layers on top of the convolutional block. We connected the first convolutional layer to the first BLSTM layer; then, the second convolutional layer is connected, together with the output from the first BLSTM, to the second BLSTM layer; finally, the third convolutional layer is connected, together with the output of the second BLSTM layer, to the third BLSTM layer. Figure 3 shows the way in which the convolutional block is used.
The Softmax layer shown in the Figure 3 is used during the training phase and then removed, as the Text-based and Audio-based NNs are combined together with the Master NN. 6.3
The Master NN is composed of a fully-connected layer, and a Softmax layer. PESInet, the resulting network, is shown in Figure 4. Notice that PESInet is supposed to works on utterances, while the text is generated by means of an ASR; in fact, this is the setting we expect to be adopted during actual usage of PESInet. Our corpus, conversely, was based on human-generated text; we are aware that in doing so we did not consider the errors due to the ASR and, as a consequence, overestimated the figures obtained during the training/validation procedure. The rationale was highlighting the contribution of text-related features to the recognition of prosodic forms, and thus we decided to avoid the “noise” introduced by ASR-related errors.
As a final remark on the ASR, notice that it is supposed to not add any punctuation mark to the transcription it generates. 7
The architecture was implemented, trained, and tested using the TensorFlow library along with Python 3.6. The code itself was run on a machine equipped with 32GB of RAM, a Xeon Intel processor and a Nvidia Titan X (Pascal) GPU. During training, we adopted the early stopping (using Accuracy as reference index); moreover, to improve the learning effectiveness, we used the variational drop-out on recurrent layers. We started training, independently, the Audio-based and the Textbased NNs, on 80% of their respective corpora: ACorpus and TCorpus. Then, once removed the final Softmax layer from them, these NNs where attached to the Master NN, and a further training –involving 80% of the MCorpus– was performed on PESInet. In particular, we investigated three approaches: 1. Allowing PESInet to train only the Master
NN weights (all the others remain fixed). 2. Allowing PESInet to change all its internal weights (also those already trained). 3. Training PESInet from scratch, skipping training of Audio-based and Text-based NNs. 8
Validation was performed using 20% of the corpus. We experimented with several feature combinations, hyperparameter values, and network structures, before reaching the final models.
The Audio-based and Text-based NNs gave the following Accuracies: 0.68 and 0.79. It’s interesting that the Text-based NN gave a better Accuracy than the Audio-based NN. This was surprising, as, after all, prosody is an acoustic phenomenon. Nevertheless, data seem to show that the words composing the utterance are indeed a good predictor of prosody. Moreover, considering that ACorpus was much smaller than TCorpus, the surprisingly low results of Audio-based NN can be explained.
Table 1 and Table 2 show the confusion matrices for the two NNs. It’s interesting to notice that Audio-based NN predicted Statements much better than the other two classes, while Text-based NN was also very good in recognising Questions.
About PESInet, Table 3 shows that the approach 2 obtained, as expected, the best results. As the confusion matrix of Table 4 shows, audio and text cooperated to improve recognition of all the three classes.
As a further experiment, we trained and tested PESInet on two classes: Question vs Nonquestion, adapting the same PESInet architecture to handle 2 classes. The corpus tags Predicted
Excl. 205 1242 215 racy for our NN (or surprisingly bad Accuracy for human listeners) could be caused by decontextualisation: in the experiment each IU was given in isolation, without any dialogue context; probably, listeners were more affected by that lacking of context than our NN. Anyway, this is just a hypothesis that should be investigated and deepened with further experiments, as the comparison could be tainted by a large number of other confounds, such as the non ecological nature of the task and the stratification of the repertoire of Italian speakers. fExclamation, Statementg were rewritten as Nonquestion, and we randomly extracted a number of Non-Question samples equals to the Question samples. Then, we used 90% of such dataset for training and 10% for testing. Accuracy reached 91% (Table 5). 8.1
Finally, to validate the results we obtained, we
conducted a perceptive experiments with 302
Italian speakers
The experiment was divided into several tests; each test was about a specific prosodic form: users were asked to listen a set of IUs and select which of them carried the expected prosodic form. In that experiment we used an ad-hoc audio/textual corpus called SI-CALLIOPE, where 14 professional actors spoke a set of 139 sentences, for a total of 1946 IUs. Notice that SI-CALLIOPE did not share anything, in terms of sentences and speakers, with corpora we used to train PESInet.
In particular, for the Question/Non-question test, each user listened to a set of audios randomly extracted from 714 question IUs and 1232 nonquestion IUs. The average accuracy was 80% (std. dev.: 7.24%).
Running the two-class version of PESInet on the same test, we got an Accuracy of 89%.
We argue that this surprisingly good AccuPESInet got an Accuracy of 80% on three classes and and 91% on two classes. Moreover, PESInet reached very good results when compared to human listeners on a totally different corpus. Although this human/NN comparison should be taken with a grain of salt, we believe that it is a hint that the network works well and the results are truly promising. As a future work, more recordings should be added to ACorpus and MCorpus to improve the performance of the Audio-based NN and, as a consequence, of the whole PESInet.
Currently, we are working for cleaning the code and streamlining the training procedure, as we plan to release the code.