The result of automatic speech-to-text conversion is a sequence of words contained in a working dictionary. Hence each number must be added to the dictionary, which is not feasible. Therefore we need to introduce a post-processor block extracting numeric sequences by speech recognition response. We describe a sequence-to-sequence converter that is a finite state transducer ini tially designed to generate phoneme sequences by words for Ukrainian using the expert-specified rules. Further, we apply this model to extract numeric sequence by speech recognition response considering word sequences as well as time and speaker identity estimations for each word. Finally, we discuss experimental results and spot detected problems for further research.
Human speech contains, depending on a domain, a significant amount of numeric sequences, which express cardinal numbers, time and date, addresses currency expressions and so on.
A speech-to-text system produces a sequence of items that are, typically, words contained in the system’s dictionary.
The system’s productivity depends on the dictionary amount. Taking more space and computational resources, a larger vocabulary induces additional hypotheses, which is a source for error increase.
If we consider each number as a valid word, this means that vocabulary expands as much as numbers might by expressed. Therefore, covering numbers between 1 and 1000000 would hypothetically mean that at least a million of words must be intro duced to the vocabulary. For highly inflective languages, like Ukrainian, this amount is multiplied by the mean number of word forms. Moreover, most of these number are unseen for the component of an ASR model constraining hypothetical word orders. So the data sparsity grows drastically.
From the other hand, quite a limited sub-vocabulary of lemmas (stems) is used to compose a spoken numeric. For Ukrainian, 20 lemmas are sufficient to compose spoken numbers from 0 to 19, nine stems are used for tens, nine stems are used to express hundreds and, finally, several lemmas name greater digit groups like thousand, million and billion. Therefore, in speech-to-text output all numbers are spelled as word sequences and finding their numeric form looks as a productive way.
The recent works aims to minimize the supervision, which varies much in dependence of the specific language [1,2,3]. The models using an end-to-end recurrent neu ral network are effective for English language, as an example, however, as reported, it does make errors with respect to the numeric value of the expression for highly inflective languages. Even such extremely rare cases would mislead about the message being conveyed that is completely unacceptable. The second type of models uses finitestate transducers constructed with a minimal amount of training data per inflectional form, on average, that is crucial for highly inflective languages like Ukrainian.
The referred approaches intensively exploit the number verbalization provided by a text-to-speech system and huge amount of synthesized speech as for the end-to-end model. That is what is paid to minimize the supervision, which requires huge computational resources and is not applicable to the matured and generally more productive HMM/DNN approach [4]. Instead, we retain a reasonable amount of supervision for tuning the finite-state automata based on [5] and use widely available language knowledge. This work reports the current state of the research applied to Ukrainian. 2
Selection of Hypothetical Numeric Subsequences In general, we consider recognition response that includes, beside a word sequence, estimations for beginning and duration of each recognized word as well as speaker diarization labels. Therefore, we may avoid including into hypothetical numeric word sequences speech and speaker disruptions, since a long pause between speech segments as well as a speaker change likely cut a numeric sequence. Particularly, our assumption is that a speaker never continues pronouncing the number started by the previous speaker.
Each word is assigned with either numeric or generic or both labels. In Table 1 we can see a sequence of 10 words, (w1, w2, … , w10), recognized in the beginning of the real news episode. The first word meaning “eighteen” starts at 15.08 s and its duration is 0.65 s as estimated by a speech-to-text converter. The second word is ambiguous and means either a number or an inflectional form of “magpie” word. As one can see, in Ukrainian, several numbers are homographs. Also among them are certain forms of words meaning two, three and five. In this work we label such words as a numeric word and include them to hypothetically numeric subsequences.
Hence, we selected two numeric word subsequences (w1, w2, w3) and (w7, w8, w9). From the first subsequence we intend to extract numbers, 18 and 45, whereas the second subsequence contains just one number, 2019. A key issue in modeling the conversion between sequences is the question of how we define the correspondence between elements of source and target sequences. We consider a finite sequence of source elements a1N=(a1 , a2, .. . , an, . .. , aN ) where each element is taken from the set of input elements, A. Let us construct the conversion of this sequence to a set of sequences for output elements taken from B.
Consider an elementary correspondence f that maps a subsequence of a1N, starting from its n-th element, to an element from B set or an empty element:
f ( anN )=b , anN ∈ Def ( f )⊂ A , b ∈ B ∪ ∅ , 1 ≤ n ≤ N.
2
3
4
5
6
7
8
9
10
3
(
F ( anN )={(f k1 ( anN ) , f k2 (anN ) , .. . , f kLk (anN ))∈ BLk∪ ∅ ,1 ⩽ k ⩽ K F}.
Here Lk is length of k-th target subsequence and the number of the target
subsequences is KF . Introduced correspondences (
Now we define an operation that concatenates over the sets produced by F and G taken from F as all possible combinations of target sequences generated by F followed by G:
F ∘ G={(f 1u , f 2u, . .. , f uLu , g1v , gv2, . .. , g vLv ), 1 ⩽u ⩽ KF , 1 ⩽v ⩽ KG}.
Additionally, we assume that the connection result is empty if at least one of F and G
is empty. Further, we specify ordered correspondences (
F=( Fi,di ,δi) , F ∈ F ,1 ⩽ i ⩽|F|, 0< di , δi={0 ,1 }, (
GJ , M=
∘ u∈ J ,v ∈ M
Fu,du,δu ( avN ).
Let j and m be the last elements of J and M respectively. Then connecting the next
N
correspondence, Fi,di,δi ( an ), we proceed in accordance to (
∘ u∈ J ,v ∈ M m+ di=n ; δr , 1 ⩽r ⩽ i ;
Fu,du,δ u( anN ) ∘ Fr ,dr ,δr ≠ ∅ ,1 ⩽ r ⩽i , if δi=1 .
Otherwise the connection is not applicable.
By means of expression (
We illustrate this process on the graph in Fig. 1. The sequence of Ukrainian words “сто”, “двадцять”, “три”, “тисячі” (“hundred”, “twenty”, “three”, “of thousand”) is accomplished with word sequence boundary elements “_”.
Thus we have a sequence of six elements a1N = (“_”, “сто”, “двадцять”, “три”, “тисячі”, “_”), N = 6. All valid correspondences for this example, Fi,di,δi ( anN ) , 1 ⩽n ⩽ N, as it follows from the graph, are:
F1,1,0( a16)={( _ )},
F2,1,0( a26)={(100)},
F6,2,0( a26)={(120)},
F7,5,0( a26)={(123000, _ )}, F3,4,0( a36)={(23000, _ )}, F4,1,0( a36)={(20 )}, F5,3,0( a46)={( 3000, _)}.
Moving alongside the arrows we generate expressions of the form (
On practice, we do not need to consider the entire subsequence anN. Normally, we
narrow the context to ann−1+TF, where TF ⩾1 depends on the specific correspondence
(
120
_
0
0
0
3
2
20
_
0
0
0
3
_
0
0
0
3
2
1
2
3
4
5
6
7
i
The expert can specify parameters of correspondences (
Numeric Sequence Extraction Structure To extract numeric sequences from recognition response, firstly, we analyze word sequences and select all sub-sequences hypothetically containing numbers. Then we apply the conversion procedure described in Section 3 to the selected word subsequences and extract possible numeric sub-sequences. Finally, we connect all subsequences to get hypotheses of text with extracted numbers.
To select all word sub-sequences that may contain numbers we generated a list of valid numerical word forms proceeding from [6]. Then all words in recognition response [7,8] matching one of the generated word forms are marked as numeric. Thus, we obtain the input numeric word sequences each of which is split in accordance to the boundaries induced by pauses longer than a threshold and speaker alterations derived from the speaker diarization procedure [9].
The selected word sequences are processed further by the multilevel rule-based se quence-to-sequence converter as shown in Fig. 2.
At first stage, the language-dependent rules are applied to obtain language-independent symbol sequences. These symbols replaces Ukrainian words for ones (e), tens (d) and hundreds (c) and correspondences for digit groups of thousand (t), million (M), billion (B) and trillion (T). For instance, “дві тисячі двадцятий” (“two thousand twentieth”) is directly mapped to “2et2d”.
At the next stage the cases when two or more numbers are pronounced in row are handled. Language-independent symbol sequences are segmented assuming that same or greater digit group starts a new number. This approach, however, might work improperly for descending numeric sequences. Fig. 3 illustrates number boundary detection for word sequence that means the time of 18:45. Three recognized words “вісімнадцята” (eighteen), “сорок” (forty) and “п’ять” (five) are mapped to language-independent symbol sequence “1d8e”, “4d” and “5e” and each word is connected to the first respective symbol with an arrow before and after segmentation on two numbers “1d8e” (18) and “4d5e” (45).
At the last stage the extracted segments belonging to different numbers are accom plished with missing zeros and ones and, finally, cleaned from symbols that names digit positions and digit groups. So the example given in Fig. 3 does not require any mentioned accomplishments and can be mapped directly to numbers from the language-independent numeric presentation, | 1 d 8 e | 4 d 5 e | → | 18 | 45 |. In turn, if we consider the language-independent presentation of verbalized current, 2020, year, it requires inserting zeros, skipped by verbalization, at proper positions: | 2 t 2 d | → | 2 t 0 c 2 d 0 e | → | 2020 |. 5
The rules are specified in text form as fields separated by tabulation, <tab>: <source_subsequence_pattern><tab>[<condition><tab>] <analysis_step1><target_subsequence_pattern1> [<tab><analysis_step2><target_subsequence_pattern2> ...] Here optional components of the template are shown in square brackets. The terminated with ellipsis block might be repeated with different values that will induce multiple decisions. We will refer to examples in Table 3 for rule specification illustrations.
<source_subsequence_pattern> consists of explicit characters as well as wildcards replacing any one symbol, ?, and one or more symbols, *, like in examples 1 through 3 in Table 2. Furthermore, the expert may define a subset of characters tak ing them in square brackets (samples 4 and 5). Sequence elements are separated by whitespace.
The only <condition> is exclusivity introduced in (
The next pairs of parameters may repeat as many alternative conversions are valid for the source subsequence.
<analysis_stepX> value stays for the analysis step introduced in (
No 1 2 3 4 5 6
Source subsequence тисяч* п’ятис* дванадцят* The module that provides word-to-number extraction in accordance to section 4 is written in Perl and derived from the implementation of bidirectional text-to-pronunciation conversion [5]. The rules are specified as described in Section 5, one level per file. The file that corresponds to the next level is indicated in header.
The basic implementation is deployed online [10] and may be tested alongside with other rule-based sequence-to-sequence conversions.
For experiments, the data is read and written in time-marked conversations (ctmfile) format. In Table 3 the aligned input and output lines are presented for a real broadcast transcript leveraged by means of automatic speech recognition for Ukrainian broadcast media transcribing system [7]. The numbers, indicated with bold, are extracted as expected. A speech-to-text system produces a sequence of items that are, typically, words contained in the system’s dictionary. 7
The described multilevel rule-based system allows for generating hypotheses of wordto-number conversion. Best hypothesis selection is the subject of analysis of lexical, syntactic and prosodic contexts by large corpora.
To introduce a new language an expert just need to fill the language-dependent rules mapping to a language-independent number spelling presentation as illustrated in Table 2, rows 1 trough 3. This way a multilingual content might be introduced as well.
Further modeling will include appending a suffix for ordinal numbers and extraction of fractions, compound words (like “20-year-old”), time, sport scores and other numerical types. 6. Shirokov, V., Manako V.: Organization of resources for the national dictionary base.
Movoznavstvo 5, 3–13 (2001). 7. Sazhok, M., Selyukh, R., Fedoryn, D., Yukhymenko, O., Robeiko V.: Automatic speech recognition for Ukrainian broadcast media transcribing. Control Systems and Computers 6 (264),p. 46-57 (2019). 8. Povey, D., Ghoshal, A., Boulianne, G. et al.: The Kaldi Speech Recognition Toolkit. In:
IEEE 2011 Workshop on Automatic Speech Recognition and Understanding (2011). 9. Zewoudie, A., Luque, J., Hernando, J.: The use of long-term features for GMM- and i-vector-based speaker diarization systems. EURASIP Journal on Audio, Speech, and Music Processing, 14 (2018). 10. Bidirectional text-to-pronunciation conversion tool, www.cybermova.com/labs, last access 2020/02/20. 11. Allauzen, C., Riley, M., Schalkwyk, J., Skut, W., Mohri, M.: OpenFst: A General and Effi cient Weighted Finite-State Transducer Library. In: Holub, J., Žďárek, J. (eds) Implementation and Application of Automata. CIAA 2007. Lecture Notes in Computer Science, vol 4783. Springer, Berlin, Heidelberg (2007).