In this paper we address three limitations of our previous implementations in human-machine spoken interaction in Slovak: low prosodic variability, limited naturalness, and deployment in real acoustically challenging situations. We designed a word-guessing game in which subjects provide verbal cues for a target animal and social robotic head Furhat guesses the animal. We then deployed it in an open-air science festival with over 60 subjects playing the game. We describe the implementation and initial observations from the design, deployment, and user evaluation.
Spoken interactions between humans and machines (HMI) are becoming ever more common in everyday lives. Predictions for near future include common deployment of not only personal assistants in smart phones but also companions for social well-being for senior citizens, applications participating in diagnosing health issues, or teaming between humans and robots for various tasks. While most of the research in this area is done in English or other major languages, understanding both cognitive as well as engineering aspects of deployment in less studies languages is also warranted. In this paper we address limitations of our previous implementations in humanmachine spoken interactions in Slovak and describe the implementation and initial observations from a novel research tool for spoken interaction between humans and robots. Our long-term goal is to understand better the intricacies and challenges of HMI in Slovak and inform thus designers developing real-world applications in this area.
In previous experiments with spoken HMI in our lab we
used a simple card game GoFish, with a closed set of
questions the user could ask the machine, or one-person
adventure game motivated by Harry Potter, with a less
constrained options for user to respond to the prompts from
the machine. Our experience pointed to three areas of
limitations relevant to the current study. First, when
humans interacted with spoken dialogues systems in these
task-oriented game-like scenarios ([
Second, the limited naturalness of the dialogues in our previous scenarios could also be attributed to the lack of backchanneling and conversational fillers that are so ubiquitous in human-human dialogues but were missing in our previous implementations.
Third, the experiments were conducted in laboratory environment without testing the deployment capability in Copyright ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). real environments and situations and in adverse acoustic conditions. Transferring HMI applications from the laboratory to real environment and testing its usability is important if spoken HMI technology should be usable in a wide scale of users and situations.
All of these potential limitations are addressed in our new experimental setup in which we designed a simple games ‘Guess my animal’ and tested how people interact with the socially expressive robotic head Furhat with implemented backchanneling behavior in real acoustically challenging condition of open-air science festival with high babble noise and loud-speaker noise. Additionally, we tested the relevance of small talk prior to the target interaction, which was hypothesized to facilitate the establishment positive social contact, engagement of the users and perceived naturalness of the interaction. 2.
We designed a simple game ‘Guess an animal’ in which a user selects a card with a name of the animal, provides cues and information about the animal without reveling its identity, and a machine guesses the animal. We employed the robotic head Furhat equipped with Slovak TTS and ASR and deployed in an open-air science festival with over 60 subjects playing the game. In this section we describe in turn the hardware, the robotic head and its setup, the software in terms of ASR, TTS and the implementation of the game, and the procedure for data collection. 2.1. Hardware
The robotic head FURHAT is a 3D humanoid agent that
employs the optical projection of an animated facial model
([
In terms of hardware, FURHAT is a computer with a mounted model of human head, the face is back-projected on the front mask; see Fig 2. In our experiment we used the 1st generation of FURHAT [https://docs.furhat.io/gen1]. Camera or Microsoft Kinect sensor using face recognition allows tracking the face of the user(s) who enter or remain in the Furhat’s visual field and subsequent controlling of the head movement and gaze for eye contact functionality. We used the default artificial face ‘Bertil’ and the control of the visual modality was kept at a minimum in the current experiment.
FURHAT comes pre-equipped with various software features. Primarily, it uses commercially available speech recognition (ASR), e.g. by Google, and speech synthesis (TTS), e.g. by Amazon. Our long-term goal is to use Slovak ASR and TTS in experiments investigating how modifications of TTS and improvements in ASR affect user experience. For this reason, although Slovak ASR is available with Google, we equip FURHAT with our own Slovak TTS and ASR so that we have greater control over the characteristics and deployment of Slovak ASR and TTS.
The robot can be programmed using FURHAT Legacy
SDK [http://www.furhat.io] statechart XML-based
framework for developing multi-modal interactive systems
developed from the original IrisTK system [
In the current experiment, the architecture of the hardware is illustrated in Fig. 1. The human user is sitting in front of the robot at a distance of about one meter and his/her speech is recorded with a head mounted microphone that is connected to a laptop. The recorded sound is sent to the ASR server that outputs the text transcript of the recognized utterances. The game itself and the dialogue manager (see section 2.2.3) processes the ASR output and selects the robot’s behavior consisting of speech, non-verbal audio expression like hesitation or a backchannel, visual expression of the face, or a combination of the above.
Slovak ASR was implemented based on Kaldi toolkit for
speech recognition [
To make speech synthesis fast we used a statistical
textto-speech system wherein speech is synthesized directly
from hidden Markov models [
The game is construed as a research tool for investigating various aspects of human-machine spoken interactions that can be deployed and adjusted to various situations. For this reason, in this first implementation we aimed at a simple algorithm to test the functionality of the basic building blocks rather than a sophisticated and complex dialogue manager.
The basis of the game is the table that lists the animals and describes their keywords. There were 12 animals in total in this version of the game. We have grouped the keywords into seven attribute categories: {legs, food, area, species, characteristics, look, alternative}. We have pretested with several naïve subjects and added the keywords used by the pilot subjects. A subset of the table relevant for ‘tiger’ is illustrated in Table I. attributes are assumed to be unique to a single animal, e.g. ‘meow’ should be used only with the cat. After pre-testing we have also included special keywords such as reptile, mammal, amphibian, bird, insect, carnivore, four-legged, double-legged, bark, meow, wing, fly, ride, etc. These form a subset of the keywords such that the animals including these keywords will add an extra point but the animals without these keywords will have one point deducted from their score.
The progress from the start to the FURHAT’s guess is controlled with several internal parameters. First, the player must use a minimal number of words. If this limit is not reached and FURHAT detects silence, it prompts the player with general cues like ‘Tell me something more’ or ‘What else?’. Second, a minimal number of questions from FURHAT has to be asked. In these questions FURHAT determines the animal with the highest score of the keywords as the most probable guess, checks the attributes for which it detected no keywords, and asks randomly about one of these attributes. For example, ‘Tell me how many legs this animal has’ or ‘Where does this animal live’. Third, FURHAT tries to achieve a minimal difference between the top animal with the highest score and the next best one. The interaction of these three parameters keeps the game to a manageable length, and guarantees dialoguelike turn-exchanges and sufficient input from the player.
FURHAT then proceeds to the guess, which is either unique, if a single animal reaches the highest score or includes alternatives if multiple animals reached the top score FURHAT checks with the player if its guess was correct and based on ASR of the response appends the count of the correct responses.
In addition to the internal parameters, two external parameters were also implemented. First, to test the effect of positive social and emotional engagement between the layer and the robot on the subsequent spoken interaction, we designed introductory small-talk. Prior to playing the game, the user can interact with FURHAT in a short minidialogue. It consists of (1) FURHAT introduction and the prompt for the user’s name, commenting on the beauty of the name, and if the name is correctly recognized, addressing the user with this name throughout the game, (2) FURHAT’s asking about the user’s birthplace, using it in its response if correctly recognized, and commenting positively about the place, and (3) asking about recent icehockey world championship, the preferred team, and again expressing positive comment regarding that team. We hypothesized that this small talk establishes positive emotion and social rapport of the user toward FURHAT since it expressed positive comments about user’s name, birthplace, and favorite team. The interaction could thus be initiated with this small-talk or without it directly proceeding to the game.
Second, to enable testing the effect of backchannel use on the naturalness of the interaction, we selected four instances of backchannel ‘mhm’ from the corpus used to train the female TTS voice (section 2.2.2). If a short silence was detected in the running ASR, FURHAT produced one of these backchannels randomly and simultaneously nodded its head. We hypothesized that this behavior will naturally prompt for more input by the user and would be unobtrusive even if resulting in simultaneous speech of the user and the FURHAT. In the current setup, this parameter was always set to be active.
If a passer-by was interested in interacting with FURHAT and FURHAT was not engaged with another player, the experimenter seated the subject in front of the FURHAT and fitted him/her with a head-mounted microphone. After briefly explaining the rationale of the game, the experimented started the external recorder and prompted the subject to test the game with the experimenter. The goal was not only for the subject to try out and get comfortable with giving cues about animals, but also to obtain the baseline of the user’s speaking behavior. For later experiments, we plan to use this baseline in testing how the speaking behavior changes when interacting with an experimenter and with FURHAT.
After this trial run, the experimenter explained that the subject will play the game three times with FURHAT, that the experimenter will not participate in the game, and prompted the subject to use full sentences and rich input for the FURHAT. Importantly, motivation of a small gift was indicated if FURHAT is successful at guessing the animals. $
Before each game the subject selected a card with the target animal from a list of cards offered by the experimenter.
After completing the three rounds of the game, the experimenter asked the subject to fill out a brief post-test questionnaire primarily assessing two domains on a Likert scale from 1 (positive) to 5 (negative): FURHAT’s speaking behavior (natural – not natural) and FURHAT’s abilities (smart – dumb). 3.
We start with the only quantifiable result available at the moment regarding the effect of small talk on the interactions and continue with informal observations regarding the functionality of the proposed set-up.
Our logs show that there were 73 subjects that completed the full three rounds of the game and we have 61 filled out post-test questionnaires. We calculated the total word and syllable count in all the recognized speech of the subjects from the logs. We take this as a proxy measure for user’s engagement in the interaction with FURHAT. Additionally, we have the subjective evaluations present in the post-test questionnaire.
Of the 73 complete logs, 45 subjects started with the small-talk and 28 proceeded directly to the game. Welch two-sample t-test showed that interactions with small-talk included significantly more speech from the player than those without it; t[69.95] = -2.93; p = 0.0046 for syllables and t[69.37] = -2.65; p = 0.01 for words. This is shown in Fig. 3 and suggests that small-talk has positive effect on players’ engagement with FURHAT in this implicit measure.
Of the 61 post-test questionnaires, 43 subjects played with small-talk. We observed a tendency that their experience with FURHAT’s speech was slightly less natural than for the subjects without small-talk (t[30.4] = -1.77, p = 0.087). It might be that small-talk affects differently the explicit perceived naturalness of FURHAT’s speech and the implicit measure of engagement. When people establish greater social rapport with FURHAT in small-talk, it increases their expectation of the naturalness of its speaking behavior, which is not always met in the current implementation. This speculation is supported by the results from the effect of small-talk on the perceived abilities of FURHAT. Subjects with small-talk preceding the game rated FURHAT as significantly less capable and smart than those without small-talk (t[43.9] = -2.63, p = 0.012). $
The experience with a 2-day deployment of FURHAT with Guess my animal game in an open-air science festival yielded the following observations. First, we discuss the limitations that we wanted to address. The implementation of the game with FURHAT was perceived very positively and the interaction as quite natural. This was the case both in the questionnaires (mean perceived speech naturalness 1.87 and abilities 1.71 on a scale between 1 and 5), and informal reactions of the subjects. The implementation of small-talk proved to affect engagement of people and their evaluation of the interaction. The speaking behavior with backchaneling and conversational fillers was also perceived as very natural. Finally, the fact that the game was functional in such adverse noise conditions (speech from passer-bys, loud music, loud presentations for big audience) is encouraging for future implementations of FURHAT with general public.
Additional observations relevant for future work include:
The temporal management of the turn-taking should be improved as this was the only feature commented on by the subjects sometimes in negative terms. The main reason for this deficiency was that sometimes FURHAT’s speech served as the input to the ASR, which resulted in asynchrony between the subject and FURHAT. This stems from the absence of information regarding the end of the TTS utterance sent to FURHAT. We will implement Voice-activity-detection and experiment with FURHAT’s broker functionality to address this issue.
The audio output from the speakers was more natural than using headphones but in high noise people sometimes did not hear FURHAT very well. The use of more powerful speaker system for open-air events is advisable.
The recorded baseline when subjects interacted with the experimenter is not usable in the current setup due to great noise of the environment and the difference between the game external and internal speech recording. Adjustments should be made to use either directional microphone or include the pre-test baseline interaction into the game to unify the recording of entire session.
T h e i m p l e m e n t a t i o n o f b a c k c h a n n e l s a n d conversational fillers was well received but some deployments were not natural. Particularly, backchannels after questions from the subjects or subjects utterance such as ‘I don’t know’.
In this initial setup, the three renditions of the game played in succession did not vary basic utterances of FURHAT; especially those initiating and concluding the game. For greater naturalness we will vary these utterances to limit the perception of ‘mechanical’ speech by FURHAT.
Several people also commented on the confusing biological sex of FURHAT. TTS was a female voice, the facial features were gender-neutral, but in Slovak morphology the name ‘FURHAT’ is associated with the male gender and in some of FURHAT’s utterances male gender in self-address was also used. We will implement interactions with either male or female consistent persona for FURHAT in voice quality, visual representation of the face, and speaking behavior.
This work was funded by the Slovak Scientific Grant Agency VEGA “Automatic assessment of acute stress from speech”, grant No. 2/0161/18.