Methods of voice command (VC) recognition in heavy noise environments are required for precise work of speech information systems on the factory floor and in transport. The paper considers a speaker-dependent way of VC recognition for VCs belonging to a limited vocabulary and being recognized in heavy noise environments. For this purpose, VCs are transformed into cross-correlation portraits (CCPs), i.e. special images. The VC under recognition is referred to a class with a minimal distance (metric) between CCP of this command and model CCPs of the class. The authors elaborated algorithms for VC transformation into CCPs, a method for defining VC boundaries, ways of model command optimization and metric choice. As a result, a rather precise VC recognition in heavy noise environment was obtained.
The growth of production and transport intensity leads to increase in operator burden. To reduce such workload,
speech information systems are used. However, these systems often have to recognize VC precisely, especially for
noisy environments. At present, a large number of speech recognition systems functioning in nearly noiseless
environment have been developed. They include, for example, IBM Via Voice, its recognition accuracy is reported to
be 97% and its recognition vocabulary includes up to 2,000 VCs; Dragon NaturallySpeaking or Dragon for PC, this
software package accurately recognizes 70% of the vocabulary, which includes nearly 60,000 words; L&H Voice
XPress, its accuracy is in the range of 90%-98% and its vocabulary size is nearly 1,000 words, etc. There are also
user-friendly systems of continuous speech understanding and processing, such as VocalIQ, Siri, Google Now and
Cortana. To compare VocalIQ with Siri, Google Now and Cortana the systems were given multiaspect requests in
a natural language [
VC recognition system developed by SRT International. The developers report the recognition accuracy to be 98%. A multipurpose 4-th generation Eurofighter is equipped with a voice control system developed by Logica. The vocabulary consists of 250 words, and the average VC accuracy is not less than 95%. The developers declare, that for the export version of the Rafale Block 05t, Thales Avionics has developed a speech control system with recognition accuracy not less than 95% for a 300 VC vocabulary, but no information on its implementation is available. Patent US 6529866 B1, 4 March 2003, The United States of America as Represented by the Secretary of the Navy, describes a method and system for transformation of an audio signal into speech. Audio signals are said to contain both VC units and noise, but test and implementation information is not available. Patent WO 1999040571 A1, 3 February 1999, Qualcomm Incorporated, describing a system and method for improving speech recognition accuracy in noisy environment also provides no test or implementation data. Among home-grown technologies the following ones should be noted. First of all, it is a VC recognition system tested on the Mikoyan MiG-29 (Fulcrum). Recognition accuracy is reported to be 56-81%, no information on the vocabulary is available. Patent RF 2267820 1, 25 April 2006, Ulyanovsk State Technical University. Recognition accuracy is reported to be 92%, vocabulary size is 23 VCs, and noise level is 3dB. No information on implementation is available. Patent RF 2271578 2, 10 March 2006, Speech Technology Center. The invention relates to speech analysis under adverse environmental conditions, e.g. in moving transport or high level noisy workplaces. No test information is available. Despite the available developments, there is no information on the actual application of VC recognition systems in avionics, since in real flights the systems developed showed substantially less efficiency than anticipated. Thus, developing VC recognition systems for noisy environments remains a challenging task. This paper examines a speaker dependent technique of VC recognition for a limited vocabulary. A method of VC transformation into portraits, i.e. images, is used.
The problems of speech recognition, in particular VC recognition, are widely discussed in modern literature.
The first methods of automatic sound recognition were obtained in the first half of the 20-th century [
This paper deals with VC recognition based on their transformation into portraits, i.e. flat images, and further
implementation of image processing techniques [
Let S = s0, s1, s2, s3, ..., sN−1 be digital VC readouts. Then, a two-dimensional image X(i, k) = {xik : i = 1, 2, 3, ...; k = 1..K} will be its autocorrelation portrait (ACP). This image is obtained in the following way. Let us divide VC S into M segments and perform the following transformations
X(i, k) =
Cov(S n, S n+k) , σnσn+k (1) where Cov(S n, S n+k) is a sample covariation of signal S intervals S n, S n+k, which are spaced kΔt apart, σ2, σ2n+k are n sample dispersions of segments S n, S n+k respectively. Thus, the k−th element of the i−th ACP line is equal to the correlation coefficient between the i−th segment S i and the segment shifted left with respect to S i on k readouts. Fig. 1 shows ACP examples.
Let us note some ACP characteristics, which make them favorable for VC recognition. VC portraits are unique,
i.e. ACPs of different VCs are unlike, whereas ACPs of the same VCs pronounced at different time intervals are
the same. Autocorrelation transformation normalizes a signal, as a result ACPs are nearly insensitive to noisiness
and slowly varying additives. If we consider additive white noise with dispersion σθ2, then its ACPs and VC ACP
readouts distorted by noise will differ by a constant factor. However, ACPs also have some negative characteristics,
e.g. the dependence of element brightness on the differences in the tone of VC pronunciation, as well as geometric
ACP distortions due to variations in speech rate. These distortions can be steadied by modifying ACP development,
e.g. taking into account loudness extremum. VC recognition by their ACPs is conducted in the following way. ACPs
of model VCs are stored in the memory. VC under recognition is transformed into ACP and it is referred to the class
with a minimal distance between its model portrait and ACP of a recognized VC. This distance (metric) between
two ACPs (i.e. images) is calculated as follows. At first, two images are aligned, i.e. for each line of one image a
corresponding line of another image is found. The average distance (e.g. Euclidean) between the corresponding lines
is considered to be the distance between the portraits. Such a correspondence for ACP of one and the same command
means the proximity of VC fragments, so the distance is relatively small, since it only occurs from the difference in
pronunciation and surrounding noise. If ACPs of different VCs are compared, then this distance is usually much more
visible due to the larger difference in sounds. In the process of command alignment dynamic programming based on
minimum distance criterion was applied. While testing the accuracy of VC recognition, commands were pronounced
by the speaker in real time. The vocabulary used consisted of ten VC groups, and there were 4-23 aviation commands
in each group. In total, the vocabulary included more than 100 VCs. Aircraft engine noise recorded in a flight mode
was used as a background and reference noise, the signal-to-noise ratio was 5-0 dB. Four male speakers took part in
the tests. Before the experiment each speaker recorded model VCs, each VC belonging to the given vocabulary was
pronounced twice. During the experiment on VC recognition each speaker pronounced all the commands from the
given vocabulary three times, all in all, more than 1,200 VCs were recorded during the experiment. Average command
accuracy was more than 95%. However, further processing has shown that the probability of accurate VC recognition
can be significantly reduced in the course of time. This problem is connected with model aging, i.e. speaker’s voice
pattern can change with time, and previously pronounced command models will not reflect the peculiarities of the
speaker’s voice at the very time of VC recognition. Therefore, it is required to update the commands from time
to time (e.g. before the flight), which, of course, has certain inconveniences. One VC model does not reflect all
the possible variants of its pronunciation, so the number of VCs was increased, i.e. the speaker pronounced each
VC more than once at different periods of time. The totality of all these patterns somehow reflected pronunciation
diversity. However, the increase in model number complicates and slows down the recognition algorithm, but it is
permissible only to a certain extent. Therefore, the model number should be limited. Besides, these models should
reflect the pronunciation diversity as much as possible. It turns out, that recognition accuracy depends greatly on the
correctness of model choice, and recognition deviations can be more than 10%. Thus, among several pronunciations
it is necessary to choose a certain number of VCs as model ones, so that the obtained model library contributed to the
best VC recognition accuracy. This problem of model library optimization was examined in [
Another way to decrease the impact of VC pronunciation variability is to use a different kind of portraits instead
of ACPs. In the process of ACP development correlation coefficients between the segments of the same VC
(autocorrelation) are found. When ACPs are used for recognition, the distances between the ACP of a recognized command
and the ACP of a model command are found. If the distance between the ACP of the command under recognition and
the ACP of its model is found, the ACPs of two different pronunciations of this command will be compared. These
ACPs can significantly differ from each other (the distance will be large). Therefore, when comparing portraits it is
desirable to minimize the difference in pronunciation. For this purpose, it is necessary for pronunciation variability
to be somehow reflected in portraits. Let us consider a cross-correlation portrait (CCP), which consists of correlation
coefficients between segments of two VCs (cross-correlation) [
σ1iσ2i,k μ1i = N1 P Nj=−01 S 1i·N1+ j, μ2i,k = N1 P Nj=−01 S 2i·N2+ j+k, σ1i2 = N1 P Nj=−01 S 1i2·N1+ j − μ1i2, σ2i2,k = N1 P Nj=−01 S 2i2·N2+ j+k − μ2i2,k. (2) (3) (4) (5) (6) While choosing parameter K, it is necessary to take into account the following fact: if its value increases, than value xik decreases. It is connected with correlation reduction of VC readouts along the line. This property proves the inadvisability of using large values K while developing CCPs (large K means that K > N).
Obviously, if CCPs of the same pronunciation (S 1 = S 2 = S ) are developed, we get the ACP of a VC S . It is desirable to examine the CCP of two pronunciations of one and the same command. It depends on two pronunciations, so the pronunciation variability affects the portrait form. Fig. 2 shows CCPs of several VCs. For example, in the picture Manevr3 + Manevr4 ’plus’ means that this very CCP was obtained from the third and fourth pronunciations of the VC ”Manevr”.
Note, that CCP characteristics are similar to those of ACP. But CCPs are less pronunciation dependent, as they combine two different pronunciations. VC recognition by means of CCPs is carried out in the same way as recognition by means of ACPs. For each VC, a model CCP made of two pronunciations of this VC is developed. These model CCPs are stored in the memory. For the VC under recognition CCPs are developed with one of pronunciations of each ”Navigatsiya1+Navigatsiya2”
”Navigatsiya3+Navigatsiya4” ”Noised Navigatsiya1+Navigatsiya2”
”Noised Navigatsiya3+Navigatsiya4” command group, then the distance between this CCP and the model CCP is found. The recognized VC is related to the group with the smallest distance.
The CCPs used have a number of characteristics, which come from both the properties of the speech signals themselves and the structure of their CCP development. Let us consider some techniques increasing the recognition accuracy by means of CCPs.
If a VC under recognition is too noisy, it increases the distance from its CCP to its model portrait formed by means of noiseless pronunciations. Therefore, ’noisy models’ were used in the experiment, i.e. artificial noise was added to the model commands. It came from the microphone placed far from the operator. As a result, the distorted models and the command under recognition contained approximately the same noise, which significantly increased the recognition accuracy.
While developing portraits, it is desirable for the VC time boundaries to be defined as precisely as possible. Then a more accurate portrait alignment can be attained. Among several known techniques of useful signal detection, the one, which shows the most accurate recognition results on the background of noise, was chosen. Besides, after definition of VC boundaries by means of this technique some boundary adjustments were made, which resulted in recognition accuracy.
In some VCs, e.g. those consisting of two words, there are micro-pauses between speech units. These pauses can differ in duration, but they do not contain any information. So, a special method for their removal was developed.
Portrait width, i.e. the line length, can be chosen arbitrary. So, it is desirable to choose the optimal length, which contributes to the best recognition accuracy. It turned out, the line length in the portrait of a VC under consideration should be equal to K = D/(5M), where D is the length of the recognized VC, M is the number of lines in a portrait. The line lengths of model CCPs are a bit longer, but they are no less than K.
VC recognition by means of their CCP is based on detection of the portraits, which are as similar as possible. Hence, there appears a problem to define the distance between two CCPs, i.e. metric defined on CCP. This distance is considered to be equal to the average distance between the corresponding CCP lines. Moreover, any metric defined on the lines, i.e. on finite sequences or vectors, can be used. Twelve known metrics (namely, Euclidean, Hilbert, Zhuravlev, etc.) and their variants were tested. For the purpose of the problem under consideration, five metrics showed the best results: Zhuravlev method (for ε = 10, ε = 20 and ε = 30), the Ruzicka distance and the Bray-Curtis distance. Besides, analyzing the recognition results obtained while using these metrics it was found out that certain recognition errors corresponded to certain metrics. Therefore, it is possible to improve recognition accuracy by using, for example, two metrics. If the recognition results coincide, then the command is considered to be recognized; if the recognition results differ, the command should be considered unrecognized. In such a case, the speaker should pronounce the command once again.
As in the case of VC recognition by means of CCPs, the words included in the model library significantly affect the recognition accuracy. Therefore, it is required to optimize the model portrait library while recognizing VCs by means of CCPs.
Each CCP line is a sequence of correlation function sample values. Because of speech signal quasi-periodicity, the correlation function turns out to be similar to periodic. This quality was used to improve the portrait quality by removing insignificant harmonics from each CCP line spectrum. This operation was performed by means of FFT. The isolation of the most fundamental harmonics for each CCP line reduced the influence of speech signal pronunciation variability.
The experiments showed that using CCPs with the described above modifications significantly reduced the effect of VC pronunciation variability and model aging. The recognition accuracy was nearly the same as in the ACP recognition on newly-pronounced models. Thus, to evaluate the efficiency of the suggested method, an experiment was conducted. The recognition was tested on two groups of VCs consisting of 10 commands each. The first group included single-word commands, the second group of VCs included both single-word and two-word commands. Each VC was pronounced 100 times by a woman-speaker. The experimental results are represented in Table 1. The maximum VC recognition accuracy was 95.6%.
The present work suggests and examines a speaker-dependent method for recognizing voice commands from a limited vocabulary in conditions of intense acoustic noise, e.g. on the background of an aircraft engine. This method implies transformation of digitized commands into certain images and further application of image processing methods. The method underwent various modifications in order to increase the recognition accuracy. Tests on a large number of voice commands showed rather high efficiency of the suggested method.