Classification of language units (sounds, phonemes, lexemes) is an urgent task of computer linguistics. Its effective solution will allow, for example, to automate of the painstaking and time-consuming work of transcribing speech signals, which is necessary when creating speech corpora. Existing approaches to solving this problem mostly come from the field of automated speech recognition and are characterized by either extremely high requirements for the hardware component of the corresponding information system (with local implementation), or a low level of information security and saturated traffic (with network implementation). We also note that the a priori tendency of such systems to take into account the results of predicting the appearance of language units in speech signals in the process of classifying the first ones becomes a drawback when transcribing speech, for which a sufficiently developed universal background model is not available. In the thesis, a method of classification of language units is proposed, based on the Markov interpretation of parametrized cepstral patterns of the short-term representation of speech signals. The described method formalizes both the computationally efficient process of classifying language units based on the stationary distribution of the hidden Markov model of speech, and the training process of such a model, formulated with an orientation to the rational use of memory. Testing of the proposed method of classifying language units in the balanced metric of qualitative indicators showed its significant advantage over the classical approach in conditions where the number of speakers is relatively small and the size of the training sample is limited compared to the size of the test sample. Also, testing showed that the proposed method outperforms the classical method in terms of time spent on training and classification by at least two orders of magnitude.
The majority of users and specialists inextricably link the task of converting audio recordings of speech into text with the field of speech recognition [1-5]. At first glance, they are right, but the devil is in the details. For example, the result of the "phonogram-text" transformation of the form "eats shoots and leaves" is semantically different from "eats, shoots and leaves". To notice the difference, the information system must not only use an adequate language model but also distinguish stress and intonation. Another "English case" – "ship" and "sheep" are pronounced the same, but, agree, they mean different things. Therefore, the conversion of "phonogram-text" is a multifaceted scientific problem, the solution of which is still ongoing even for one language. These theses are focused on transcribing [3, 6-8]. Automation of this procedure is necessary to solve the ever-present problem of creating representative language corpora.
Many popular speech recognition frameworks can potentially be adapted for automated transcribing. For instance, an excellent framework from the Nvidia corporation is called NeMo [8, 9]. This framework has many useful features in the context of our investigation. For example, the "Punctuation and Capitalization Model" unit determines for each spoken lexeme whether its text equivalent should be written in upper or lower case, as well as which punctuation mark should be used after it. At first glance, NeMo is the answer to all our questions, but it has one big drawback – the entire "smart" component of the framework is implemented on BERT [9-11]. This transformer is not just resource-hungry – even its profile training takes place only on the side of the corporation. Household gaming "supercomputers" are weak for this, not to mention mobile gadgets. The second problem is the closedness of the code and the implementation of most recognition operations "in the cloud", which does not seem rational from the information security point of view.
However, there are analogues with open source [3, 12, 13]: CMU Sphinx, Julius, RWTH ASR, Hidden markov model ToolKit (HTK), and Kaldi. The article [3] gives the results of the comparison of these systems. They were trained on 160 hours of English-language audio recordings and verified on a 10-hour test sample. It was Kaldi who won in accuracy, yielding speed. Kaldi uses MelFrequency Cepstral Coefficients [14] and Perceptual Linear Prediction [15] to parameterize shortterm phonogram fragments. The Kaldi language model is built based on hidden Markov models [15], a Gaussian mixture model [16] and a deep neural network of the Time-Delay Neural Networks type [17]. The well-known Finite-State Transducer (FST) [3] is responsible for speech-language modelling, and the forward-inverse algorithm [3] is responsible for decoding. As you can see, all these technologies have been known for over forty years. The source of Kaldi's effectiveness is not manufacturability but closed training data. And, again, all these technologies are focused on speech recognition based on transcribed language corpora, and not on automating the process of creating such corpora.
Taking into account the strengths and weaknesses of the mentioned methods, we will formulate the necessary attributes of scientific research.
The object of investigation is a process of classifying a parameterized pattern of a short-term representation of language units.
The subject of investigation is the theory of probability and mathematical statistics, the theory of relativity, the theory of pattern recognition, methods of digital signal processing and mathematical programming.
The aim of the investigation is to formalize the computationally efficient process of classifying language units based on the stationary distribution of the hidden Markov language model.
The research tasks are: - to formalize the computationally efficient process of classifying language units based on the stationary distribution of the hidden Markov language model;
- to formalize the process of training the language model, formulated with an orientation towards the rational use of memory, necessary for the classification of language units;
- justify the adequacy of the proposed mathematical apparatus and demonstrate its functionality with an example.
We formalize the process of classifying language units at the junction of the paradigms of digital signal processing [18, 19] and pattern recognition [20, 21]. Let the spoken unitary speech unit X t [n] (sound, phoneme, lexeme) be stored in the phonogram X [n] for a time interval of duration t [ms]. It is necessary to find the index w of this language unit in the corresponding dictionary of capacity W .
The parameterization of the phonogram X [n] is performed by representing the last one with a sequence of intervals of stationarity with a duration of 10 [ms] each (without overlapping): i = 1, mt , mt = t 10 .
The result of parameterization of the contents of the і -th stationarity interval of the original phonogram will be the characteristic vector st(i) ∈ . The procedure for forming a characteristic vector st(i) includes a sequence of such operations as: Xˆ t [n] =X t [n] − 0.95 X t [n − 1] ; - spectral alignment of the harmonic component of the analyzed stationarity interval: - application of the short-time Fourier transform [20] to represent the content of the stationarity interval in the frequency space;
- determination of cepstral characteristics [3, 14, 15, 20] (mean values and standard deviations of р Mel-cepstral coefficients, mean value, standard deviation and slope of the spectral centroid, mean value and standard deviation of the spectral decay) for the analyzed stationarity interval. For visual assessment of the content of the analyzed stationarity interval, we will use the normalized chrominance energy diagram [22].
Parametrization of the phonogram will be preceded by its division into harmonic (tonal) and
percussive (transitional) components. The result of the parametrization of the harmonic component of
the phonogram X [n] with duration t will be the matrix of characteristic vectors St = ( st(
V : st(i) → ot(i) , ∀i ∈[1, mt ] , st(i) ∈ , ot(i) ∈ .
The focus of our investigation is the definition of a language model λ , which generalizes to a set of models of language units λ ( j) from a dictionary with capacity W : λ = {λ ( j) } , j = 1,W . During training, W discrete hidden Markov models are created:
λ (w) = {π (w) , A(w) , B(w) } , w = 1,W , where π (w) = {π i(w) } , i = 1, N (w) is a stochastic vector of initial states, the number of which is limited by the value of N (w) ; A(w) = {ai(jw) } , i, j = 1, N (w) is a matrix of probabilities of transitions between states; B(w) = {bi(w) ( k )} , i = 1, N (w) , k = 1, K is the matrix of output probabilities.
Based on the defined model λ , the classifier returns the index of the language unit wt∗ ∈W for the input parameterized pattern Ot of the empirical phonogram X [n] of duration t . At the same time, the generation probabilities of each of the models of the complex pattern Ot are analyzed. The final decision is made under the rule wt∗ = arg max ( P (Ot λ (w) )) .
w=1,,W 2.2.
Mathematical formalization of the investigated process
Let us investigate analytically the operation of processing the unknown parameterized pattern
O = (o(
Pj = limn→rij∞( n) . (
Extending condition (
By definition: if the process described by the hidden Markov model is in the state j , then the probability of the value k appearing at the output of the model can be characterized by the stochastic parameter E ( k ) :
E ( k ) = ∑ Pjbj ( k ) , k ∈{1, K} , (
j
where bj ( k ) are the elements of the matrix of output probabilities В , described when defining the
Markov model (
We present the probability of generation by the hidden Markov model λ
parameterized pattern O = (o(
i=1
To increase the computational efficiency of the process of calculating expression (
m ln P (O λ ) = ∑ ln E (o(i) ) . (8)
i=1
The characteristic ln P (O λ (w) ) presented by expression (8), calculated for ∀w ∈{1,W } , is
generalized by rule (
w=1,,W
The classification rule (9) relies on the computational procedure (
Considering that the Markov model λ π (w) =const∀λ (w)
A(w) is imposed in terms of the Bakis model [23] so {Qn =i,Qn+1 =i} , {Qn = i,Qn+1 = i + 1} , {Qm =i,Qm+1 where m is the final state. The most difficult thing is to choose the values of the matrix of output =1} , probabilities B(w) . To formalize this process, we will apply the Baum-Welsh algorithm [14, 15]. This iterative algorithm allows finding the local maximum probability P (O (w) λ (w) ) for the model λ (w) based on the training sample size T .
d
We choose the initiating values of the nonzero elements of the matrix A(w) in such a way as to make the transition {Qn = i,Qn+1 = i + 1} more likely than {Qn =i,Qn+1 =i} : ai,i+1 = 1 − ai,i∀i ∈{1, N − 1}, aN ,1 = 1 − aN ,N , ai,i ≤ 0.5∀i ∈{1, N}. (10) (11)
We choose the initial values of the elements of the matrix B(w) based on the calculated ratio of the value of the parameter U (w) ( k ) , which represents the number of occurrences of the index k in the training sample of the word w , to the total number of indices in this training sample, represented by the value of the parameter U (w) :
bi(w) ( k ) = U (w) ( k ) U (w) , i = 1, N .
Note that all elements of the matrix B(w) must be greater than zero, which is dictated by the use of a logarithm in expression (9). At the same time, in addition to condition (11), the equality of ∑ bi(w) ( k ) = 1∀i ∈{1, N (w)} . (12) K k =1 must hold for the resulting nonzero initiating elements B(w) .
After the initial values of elements of the tuple π (w) , A(w) , B(w)
are determined, the sought
parameters P(w) and Е(w) are calculated by expression (
Section 2 summarizes the theoretical results in the form of profile information technology for the
classification of language units. The input parameters of the technology are the trained language
model λ ={λ (w) =E(w) } w = 1,W
O = (o(
О and each λ (w) -th model-element of the language model λ m in the parameterized pattern O . Classification results w∗ will be obtained in O ( KW ) , where K is the capacity of the codebook V . Implementation of the proposed technology on the platform of parallel computing has great potential. During the parallel calculation of the characteristic ln P (O λ (w) ) expressed by expression (8) ∀w ∈W , the classification result w∗ will be obtained in O ( m) iterations, not in O ( mN (w) ) iterations.
Experimental studies of the proposed method of classification of language units were conducted using the materials of two language corpora – the Ukrainian-language corpus GRAC and the Englishlanguage corpus TIMIT. The General Regionally Annotated Corpus (GRAC) is a corpus of the Ukrainian language & speech with a volume of more than 800 million lexemes, intended for linguistic research on grammar and vocabulary. GRAC works based on the morphological analysis system developed by the r2u group. This system analyzes the spoken text and determines lemma (lexeme) and tags (grammatical features) for each word form. The distribution of speech material by region of origin (territorial units of Ukraine, Ukrainian-speaking diasporas, etc.) is specific to GRAC. Such an organization allowed us to focus our broadcast material on the Vinnytsia region. This narrowing was chosen deliberately to investigate the functioning of the proposed method in conditions where the number of speakers is relatively small and the size of the training sample is limited compared to the size of the test sample. TIMIT is a speech corpus that contains verified results of phonetic and lexical transcription of spoken American English. Phonetic materials in the TIMIT corpus are grouped by sex, region, and date of recording. A large number of available materials in the TIMIT corpora made it possible to investigate the functioning of the proposed method in conditions where the number of speakers is large, and the volume of the training sample significantly exceeds the volume of the test sample. In Fig. 1, the speech base formed based on the GRAC and TIMIT language corpora are presented in a visual form, which the authors used for experimental investigation of the proposed method of classifying language units.
1000 800 rbe600 m u N 400 200 0
Number of language unit classes
Number of phonograms per class
Number of speakers the t -th language unit; y (Ot ) is the true class to which the parameterized pattern Ot of the t -th language unit belongs; w∗ (Ot ) is the class to which the author's classification method, generalized by expression (9), assigned the parameterized pattern Ot of the t -th language unit; I (e) is an indicator function: if the argument-condition е is true, then I (e) = 1 otherwise I (e) = 0 .
We will obtain the empirical functional dependence ρ = f (l, K , N (w) ) , where l = {GRAC,TIMIT} is the data source, K is the capacity (number of language unit classes) of the code book V , N (w) is the number of states of models λ (w) , w = 1,W . The result of the experiment K = {2i } , i = 5,12 ; N (W ) = {3, 6} is shown in Fig. 2.
1,00 0,95 0,90 ρ 0,85 0,80 0,75 l=GRAC,Nw=3 l=TIMIT,Nw=3 l=GRAC,Nw=4 l=TIMIT,Nw=4 l=GRAC,Nw=5 l=TIMIT,Nw=5 l=GRAC,Nw=6 l=TIMIT,Nw=6 ( K (l =TIMIT , K =212 , N (W ) =6).
4 5 6 7 10 11 12
13 =3)and max (ρ TIMIT ) is reached at ( K =212 , N (W ) =6). Further investigation was conducted with these sets of controlled parameters: (l =GRAC, K =210 , N (W ) =3),
In addition to the already defined qualitative characteristic (13), such metrics as precision P , recall R , and F1 -metric are informative for evaluating the quality of the result of solving the classification problem [18, 19]. This "qualitative triple" is calculated based on the content of the confusion matrix C = {ci, j } , an arbitrary element ci, j which represents the number of cases of classification of the object j as і . Operating with the content of the matrix C , we formulate expressions for calculating
F1(i) =2PiRj ( Pi + Rj ) , i, j = 1,W . (16)
Metrics (14)-(16) are defined so that when evaluating an ideal classifier in the qualitative space P, R, F1 , we obtain Pi =Ri =F1(i) =1 for ∀i ∈{1,W } . Fig. 3 presents an averaged evaluation results of the classical Forward Algorithm method [24, 25] and the author's method in metrics (14)-(16). This experiment took into account the optimal controlled parameters determined by the results of the previous experiment (see Fig. 2).
Rj =P ( w∗ (Ot ) = j y (Ot ) = j ) =c j. j
Pi = P ( y (Ot ) = i w∗ (Ot ) = i ) = ci,i
P, R.F1 in the context of the terminology used in Section 2:
W ∑ ci, j , j=1
W ∑ ci, j , i=1 (14) (15) s e lu0,6 a V G V A0,4 0,2 0,0
Note that the total time T spent by the test computer system on the classification of all patterns from GRAC and TIMIT data sets by the Forward Algorithm method FA and the author's method AM amounted to {TGRAC,FA ,TGRAC,AM ,TTIMIT ,FA ,TTIMIT ,AM } = {0.0845,0.0057,13.9352,0.4521} [s]. (17)
Let's start the discussion with the analysis of those shown in Fig. 3 results. These results prove that for the same input data and values of controlled parameters, the author's method and the classical method show close results in the balanced metric of quality parameters (14)-(16). Shown in Fig. 1 information is a confirmation that the amount of input data is statistically representative, and the availability of the source of their origin allows to ensure the reproducibility of the empirical results presented in Section 3. Therefore, the adequacy of the method of classification of language units proposed in Section 2 is empirically proven.
Note that GRAC -data reproduce a situation in which the number of speakers is relatively small,
and the size of the training sample is limited compared to the size of the test sample. Instead, TIMIT
data reproduce a situation in which the number of speakers is large, and the volume of the training
sample significantly exceeds the volume of the test sample. Fig. 3 shows that in the classification of
GRAC -data, the author's method 1.0 − 1.5% is superior to the classical analogue in all (14)-(16)
metrics. At the same time, when classifying TIMIT -data, the classical method prevails. These
empirical results can be explained by the fact that when determining the stationary distribution of the
hidden Markov model, we deliberately used the most computationally efficient method presented by
expression (
The results presented in Fig. 2 are also interesting. It can be seen that the function ρ =f (l =GRAC, K , N (w) ) has an obvious extremum w = 10 . This circumstance can be interpreted in two ways. Either the ratio "number of classes"-"number of instances of the training sample per class" is information lossy, or the k-means method used to implement transition st(i) → ot(i) ∈ Ot (see Section 2.1) is suboptimal for generalization. The search for a scientifically reliable answer to this collision is a promising direction for further research.
Classification of language units (sounds, phonemes, lexemes) is an urgent task of computer linguistics. Its effective solution will allow, for example, to automate of the painstaking and timeconsuming work of transcribing speech signals, which is necessary when creating speech corpora. Existing approaches to solving this problem mostly come from the field of automated speech recognition and are characterized by either extremely high requirements for the hardware component of the corresponding information system (with local implementation), or a low level of information security and saturated traffic (with network implementation). We also note that the a priori tendency of such systems to take into account the results of predicting the appearance of language units in speech signals in the process of classifying the first ones becomes a drawback when transcribing speech, for which a sufficiently developed universal background model is not available.
In the thesis, a method of classification of language units is proposed, based on the Markov interpretation of parametrized cepstral patterns of the short-term representation of speech signals. The described method formalizes both the computationally efficient process of classifying language units based on the stationary distribution of the hidden Markov model of speech, and the training process of such a model, formulated with an orientation to the rational use of memory. Testing of the proposed method of classifying language units in the balanced metric of qualitative indicators showed its significant advantage over the classical approach in conditions where the number of speakers is relatively small and the size of the training sample is limited compared to the size of the test sample. Also, testing showed that the proposed method outperforms the classical method in terms of time spent on training and classification by at least two orders of magnitude.
Further research is planned to be directed at increasing the informativeness of the parameterized pattern of the short-term representation of the phonogram of the speech signal in the context of the task of classifying language units.
The authors would like to thank the Ukrainian Army for providing security to perform this work. This work has become possible only because of the resilience and courage of the Ukrainian Army.