Hypervideo usage scenarios like physiotherapy trainings or instructions for manual tasks make it hard for users to use an input device like a mouse or touch screen on a hand-held device while they are performing an exercise or use both hands to perform a manual task. In this work, we are trying to overcome this issue by providing an alternative input method for hypervideo navigation using speech commands. In a user test, we evaluated two different speech recognition libraries, annyang (in combination with the Web Speech API) and PocketSphinx.js (in combination with the Web Audio API), for their usability to control hypervideo players. Test users spoke 18 words, either in German or English, which were recorded and then processed by both libraries. We found out that annyang shows better recognition results. However, depending on other factors of influence, like the occurrence of background noise (reliability), the availability of an internet connection, or the used browser, PocketSphinx.js may be a better fit.
INTRODUCTION
Using speech input, users are nowadays able to control
smartphones, navigation systems, and Smart-TVs without touching
them. Depending on the system, either certain commands are
recognized (for example in TomTom navigation systems [
However, up to now, only few homepages and Web
applications have built-in support for speech input. Especially
hypervideo players could benefit from speech control. Hypervideos
consist of interlinked video scenes which are enriched with
additional information. Playing such videos requires special
players that provide additional means of navigation in
additional information, in scenes, and between scenes [
SPEECH RECOGNITION FRAMEWORKS
Several speech recognition APIs exist having varying features
and limitations. Available APIs are, for example, Google
Speech API [
Web Audio API and PocketSphinx.js
The Web Audio API is a “high-level JavaScript API for
processing and synthesizing audio in web applications” [
Web Speech API and annyang
The Web Speech API enables the incorporation of voice
data into web apps [
IMPLEMENTATION In order to test the annyang and the PocketSphinx.js projects, we installed a reference platform. It consists of a Web server (Apache 2.4.10) and a database (MySQL 5.5.38). We used Perl (Version 5.14.2) for the implementation of the dynamic test Web page, which shows contents in the selected language (German or English) and provides a log-in system to avoid abuse and falsification of the test results. We only used Google Chrome for our tests, because annyang is built on the Web Speech API which was only available for Google Chrome at the time of the tests. annyang For an implementation of speech detection and recognition with annyang, it was only necessary to include the JavaScript library into the Web application. In order to allow the usage of the microphone it was mandatory to install an SSL-certificate. We furthermore rewrote the onResult function of the annyang project to make the implementation conform with the PocketSphinx.js implementation described hereafter. To test German words, the language only had to be set to German using the setLanguage function. Further modifications and additions were not necessary.
PocketSphinx.js
The implementation of speech detection and recognition with
PocketSphinx.js required more effort compared to the
implementation with annyang, because the source code of
PocketSphinx.js only comes with an English acoustical model. To
avoid having to generate our own acoustical model for
German words, we used the one provided by VoxForge [
Test System The web page used for the tests consisted of a index.pl file in Perl (which generated the HTML code), some JavaScript files, and a MySQL database. The database stored user names and passwords, test words and their pronunciation, test results, as well as data that might be displayed on the dynamic web page. JavaScript was used to start and stop voice recording and recognition in the different technologies. In our tests, we used a fixed set of words which was shown in our test application and had to be spoken out loud by the participants. We furthermore used a timer which limited the recognition time per word and showed each word for 10 seconds. If a word was recognized correctly during the first attempt, the next word was loaded. If it was not recognized, a second attempt with a new timer was started. The user then had to repeat the word. The Web page informed the user whether the word was recognized correctly. The system used versions of annyang and Pocketsphinx.js available in November 2014. STUDY/METHOD To find out if annyang or PocketSphinx.js performs better, we conducted a study with 58 participants.
Procedure/Data Collection We used the 18 words shown in Table 1 which represent the key functions of our HTML5 hypervideo player. The words were presented to the participants in random order to avoid exercise effects towards higher word IDs. Each recorded word was tested with both technologies, annyang and PocketSphinx.js. Before starting the tests, the users had to select which language they wanted to do the test in. As a result, 33 participants used the German version of the test and 25 users participated in the English version. Participants The participants in our study were mainly between 18 and 60 years old. 34 of the participants were male, 24 were female. The test was mainly distributed in Germany, so most of the participants were native German speakers. The tests were conducted on desktop computers or laptops, whereby 33 participants used internal and 25 participants used external microphones. See Table 2 for more precise demographic data.
ANALYSIS AND RESULTS We analyzed the frameworks in two different ways. On the one hand, we analyzed the number of recognized words per language and per framework. On the other hand, we compared the two frameworks in different categories relevant for practical usage in our hypervideo player.
Recognition of Words We analyze the recognition of the words for the two languages first separately and then together. Taking a look at the recognition of the German words, it can be said that annyang has a better recognition rate than PocketSphinx.js (see Table 3 and Figure 1, blue and gray bars). Out of 594 words (18 words spoken by 33 test users), annyang recognized 527 words in the first and 27 in the second attempt which results in 554 recognized words. PocketSphinx.js in contrast recognized 399 words in the first and 62 in the second attempt which results in 461 recognized words. The annyang library failed to recognize 37 words, while the number of not recognized words Summarizing the results over all languages, it can be noted that annyang showed better overall results than PocketSphinx.js (see Table 5). Annyang had a recognition rate of 90.61 % while PocketSphinx.js recognized only about three quarters (73.94 %) of the words. The rate of not recognized words was around 10 % for both libraries. One reason for the worse results for PocketSphinx.js may be background noise which has a greater influence on PocketSphinx.js than on annyang. Taking a look at the recognition performance of individual words (see Figure 2), it can be stated that the recognition for 1Partially recognized words are words that either are only a part of the given word or contain the given word (but also other letters), meaning the recognized word contains more or less letters than the given word. for PocketSphinx.js was 56. The biggest difference was in the number of partially recognized words1, where the number for annyang was quite low, but PocketSphinx.js recognized 77 words partially. Taking a look at the results for the English words (see Table 4 and Figure 1, orange and yellow bars), the results are similar to those of the German words. Out of 450 words (18 words spoken by 25 test users), annyang recognized 367 in the first and 25 in the second attempt resulting in 392 correctly recognized words. In contrast, PocketSphinx.js recognized 269 words in the first and 42 words in the second attempt resulting in 311 correctly recognized words. Only 1 word was recognized partially using annyang, whereas PocketSphinx.js recognized 88 words partially. For the English words, annyang showed slightly worse results (57 not recognized words) than PocketSphinx.js (51 not recognized words). One reason for the higher number of not recognized words might be the fact that the words were not spoken by native speakers. The level of correct pronunciation is unfortunately not known in this case. the German words for annyang did not show huge differences between the words. The English words in contrast showed larger differences. The words “fade in” and “journal” were recognized correctly less than 15 out of 25 times compared to annyang. The results for German words with PocketSphinx.js are worse for all words, especially for the word “abspielen”. Taking a look at the results for the English words, it can be said that especially the words “fade in”, “search”, “full screen”, “picture”, and “picture gallery” showed worse results with 10 or less out of 25 recognized words.
Practical Comparison While the results in the user tests regarding word recognition performance clearly were in favor of annyang, the decision for using one of the libraries in real world HTML5 hypervideo players requires further thoughts. We examined 5 factors further, namely: dependencies and integration, reliability, availability, browser support, and supported languages. Dependencies and integration as well as supported languages may be of less interest. Assuming that no large changes are made in the Web application that uses the speech recognition, the integration only has to be implemented once. Regarding language support, both libraries show a large number of supported languages or provide the possibility to extend or create language models in case they do not exist aready. Reliability, availability, and browser support play a more important role. Depending on the hypervideo application area, background noise may occur, an internet connection may not be available at all times, or company restrictions may not allow to use certain browsers. Please refer to Table 6 for a comparison what to use best in a given scenario.
CONCLUSION In this work, we describe the implementation of a test framework for the speech recognition libraries annyang and PocketSphinx.js. We wanted to test the quality of the recognition of certain words that could be used to verbally control hypervideo players. As a result, it can be noted that annyang provides better recognition results both for English and German words.
noise is reliably dis- tion is reliable as long tinguished from lan- as the surroundings guage; recognition of have no background spoken words is reli- noises able in most cases
support
except Internet Explorer However, recognition may not be the only factor to consider when integrating one of the libraries into a hypervideo player. Depending on the application area, the occurrence of background noise (reliability), the availability of an internet connection, and the used browsers may influence the selection of the library.
In the tests described in this work, we only used Google Chrome, due to a missing support of the libraries in other browsers. In future work a test with other browsers and the test of other libraries may bring further results that may influence the selection of one of the libraries.
The voice control should be integrated into the hypervideo
player and tested in a real world scenario measuring user
frustration due to speech recognition performance in a real
world setting. Depending on the scenario the hypervideo
player is used in, another hypervideo control approach may
also be helpful. In case of a physiotherapy or fitness training,
for example, it is helpful to show main video contents on
a bigger screen. A solution to enable an easier control of
the hypervideo in this specific case may be a second screen
application that splits contents from control elements [