Behinds the capability of a person to be an interlocutor of a conversation there lies many human capabilities, many of which are carried out unconsciously and very naturally in childhood. Human-like turn-taking in the human-machine interactions (HMI) can be seen as a critical issue to achieve natural conversational interaction. The production of the listener response starts before the speaker turn is finished, which means, that listener is often able to anticipate the remaining content of unfolded speaker turn. The ability to anticipate can be identified as a very important human capability, which supports rapid turn-taking. This anticipation process, which occurs during listening relates to sentence comprehension and turn-taking. The proposed paper study this phenomenon on the small corpus of Slovak interviews, where the attention is focused on overlapping segments and discussed possible applications, where anticipatory behavior on the machine side can bring benefits.
Nowadays spoken communication between human and machine has become obvious, what relates to the new types of devices, which start to be a part of our everyday life. Good examples are Amazon Echo, Google Home, TV with voice control, Voice search applications, social robots.
We can classify that form of communication mostly as simple dialogues, what means question-answer scenarios and task-oriented domain-specific dialogue.
Unlike mentioned scenarios, in case of human-human spoken interaction, the situation is significantly more complex. Here, information exchange is often performed very quickly, a lot of information is omitted, but supplemented by the listener based on the common background and history (short-term, long-term, topicrelated, speaker-related, situation-related). Behind a capability of a person to be an interlocutor of a conversation lies a lot of important human capabilities, many of which are carried out unconsciously and very naturally. People are easily able to track the content, to detect a speech act (dialogue act) behind a speaker’s turn, to perform effective and rapid turn-taking, to provide feedback in a role of listener, or incrementally construct their turns.
In the proposed work, we focus on the three related aspects of the spoken communication - turn-taking, sentence comprehension and anticipation.
“A turn is the time when a speaker is talking, and
turntaking is the skill of knowing when to start and finish a turn
in a conversation. It is an important organizational tool in
spoken discourse.” [
Stivers et al. in [
Figure 1 Turn-taking timing
Anticipation relates to the sentence comprehension. It can be inferred, that instead of the word sequence, the meaning is anticipated. We can identify, that in the moment, when a person is able to anticipate remaining part of the speaker turn, there is partial or complete comprehension, which can be signalized using one of the following behavior: providing a feedback (backchannel signals) or attempt to take the floor.
The attempt to take the floor can result into the
overlapping speech. Overlapping speech is a segment of the
conversation, where both interlocutors speak
simultaneously. The speaker tries to finish his turn, but the
listener starts his own turn. Several studies quantified a
significant occurrence of overlaps in dyadic human-human
spoken interactions (e.g 18.9% in [
We supposed that overlaps are the best place to observe anticipatory behavior of the listener.
The reason, why we decided to study the relation between rapid turn-taking, comprehension and anticipation is the lack of fluentness in case of human-machine spoken interaction. We can observe that machines are still not enough skilled in turn-taking.
Rapid turn-taking in HMI is especially important, when we consider the fact, that in case of humanoid robots, social and family robots people tend to expect more natural dialogue interaction, often called “conversation” due to their human-like embodiment.
Human-like turn-taking in the human-machine
interactions can be seen as a critical issue to achieve natural
conversational interaction in HMI [
Together with observation of anticipatory behavior in human-human interactions, several questions regarding human-machine spoken interaction arise. E.g.:
Would machines let their “predictable” turns unfinished, when they observe a comprehension on the side of human listener? Can unfinished turns increase a natural character of human-machine interactions?
Or:
Would machines be able to interrupt human speaker turn and if yes, in which situations?
How can be anticipation integrated on the side of machine?
How can machines catch the moment when the listener has enough information to comprehend?
And finally:
Could be helpful if the machine would be able to interrupt the human interlocutor?
To answer proposed questions, we decided to collect human-human interactions to analyze turn-taking, overlaps and anticipatory behavior.
The paper is organized as follows: The second section deals with anticipation in human-human spoken interaction, description of the prepared corpus and results of its analysis. Section 3 provides discussion of applications, where anticipation can bring benefits. 2
interactions in human-human spoken
Anticipation play a very important role in the
humanhuman as well as human-machine spoken interaction,
because it influences the speed of response in conversation
(see [
Anticipation is routinely used by human interlocutors in dyadic interactions. Expect the exact results that indicate anticipatory behavior, according our opinion, there can be identified also other indicators of interlocuters anticipation: Unfinished turns: We believe that in case of unfinished turns speaker considers that the listener can anticipate remaining part of his turn.
Overlaps and interruptions: We believe that the listener can try to take floor, when he has come to the comprehension before the end of the speaker turn.
Interruptions differ from overlaps in the timing. Listener
usually use a small pause in the speaker turn to interrupt the
speaker and to take a floor without causing an overlap.
Such pauses are usually marked as “Transition relevant
places” (TRP) as defined by Sacks et al. in [
Unfinished turns and occurrence of overlapping segments can have several other reasons expect comprehension based on anticipation. Therefore, the careful analysis needs to be done.
The mutual understanding or comprehension on the
listener side can be indicated by backchannel signals,
which are usually produced by human listeners.
Backchannel signals was defined by Yngve [
Corpus of the investigative interviews of the TV program “Na rovinu” and episodes of TV discussions “Pod lampou” were selected for the analysis of turn-taking mechanisms.
The overall length of the analyzed corpus was approx. 8 hours. There were 9 speakers (8 males, 1 female), two of them play a role of the moderator (male).
Recordings processing have two main levels. In the first
level, recordings were transcribed. The first transcriptions
were generated by our automatic transcription system [
In the first step of the analysis we focused mainly on overlapping segments. To classify different types of overlaps, we designed 13 categories according intent behind the overlaps (see Fig. 2.)
Two basic categories – concurrent and cooperative overlaps are further divided according their function.
Concurrent/competitive overlaps represent overlaps, where we can identify an attempt to grab the floor. On the other side cooperative/non-competitive category mark overplaps, where listener’s goal is to assist the speaker to continue his turn. before the end of the speaker turn, what causes overlapping speech.
More than 84% of overlaps were caused by the moderator, who jumped into respondent speech. Respondent jumped into moderator turns approx. in 16% of all cases. The explanation may be that the moderator has a responsibility for interaction, and he need to maintain topic, topic shifts and timing (duration) of the interaction.
Table in Fig.3 shows the distribution of intentions behind the overlaps, which we detected manually.
According obtained results, we can conclude significant differences between moderator and respondent. On the side of the moderator, the most numerous categories were: supporting the speaker (approx. 32%), request for completing information or providing additional information (around 20%) and asking the additional question (18%).
On the side of the respondent, the most numerous categories imagine overlaps, which occur when a listener expressed involvement and agreement (more than 57%).
Obtained results show that roles, which play interlocutors affects the number of overlaps and also intentions behind attempts to take the floor.
Next issue is that designed categories contain several items, which can be identified as backchannel signals (express involvement/agreement/disagreement, express noninterest). As was concluded above, backchannel signals indicate rather partial understanding, than anticipatory behavior. However, information about this dialogue parts is important too, to analyze backchannel mechanisms in different scenarios and roles, which interlocutor plays.
Designed categories of overlaps correspond wih
speech/dialogue acts, which are conveyed through
utterances that interrupt the speaker turn. Confirmation
request, Clarification request, Agreement, Disagreement
are typical dialogue acts (see e.g. [
2.2
Total number of both speakers performed turns was 1646. In 34.8% of cases, turns were taken by the listener The main reason, why anticipation needs to be considered is to support rapid turn-taking, which enables smooth and fluent human-machine spoken interaction. But anticipation can enable also other human-like capabilities in HMI.
In case of application anticipation in human-machine
dialogue interactions, there exists only few works that deals
with this phenomenon. We can mention the work of
Dominey et al., described in [
The anticipatory behavior on the machine side means the ability to find an enough reliable hypothesis of unfold utterance just uttered by human interlocutor. Machines that can communicate with user through spoken language implements a human machine communication chain. Modules of this chain have anticipatory potential, because they use resources with related data, as are recognition network, language models or dialogue model. 3.1
While, the anticipatory behavior can be considered as non-important for simple task-oriented dialogue systems, it can bring more human-like character of the interaction and advantages in case of systems for multi-party conversations, for human-machine collaboration scenarios or for crisis scenarios, where can be useful or necessary for the machine to be able to rudely interrupt a human speaker(s), to be able to take a floor and to propose ideas or solutions. 3.1.1
Machine in the role of discussion moderator can be the next example of application, where anticipatory behavior can play important role. Emotionally colored multi-party spoken interactions as are e.g. political discussions often require a lot of effort on the side of moderator to lead the interaction and to manage turn-taking for all speakers. Moreover, he often needs to enforce good behavior or compliance with specified time. 3.1.2
Anticipating in simultaneous interpreting simply means
that interpreters say a word or a group of words before the
speaker actually says them [
Nowadays, machine translation is a common application. There also exist applications, which perform interpretation, but they interpret whole sentences after their pronunciation. But, if we imagine machine as the simultaneous interpreter, it must be able to anticipate, to produce fluent simultaneous interpretations without meaningless gaps. In that case, anticipation can enhance perceived quality and clarity of the translation.
Anticipatory function can help also human interpreters in the way, that it can suggest them hypothesis about next words in the speaker utterance. Architecture of such machine-supported simultaneous interpretation system is sketched on the Fig. 4.
One of the scenarios of machine-supported simultaneous interpretation is that the system will listen to the speaker and try to provide early predictions of the next words according just recognized partial utterance. Speaker speech will be continually recognized by Automatic Speech Recognition (ASR) module, where recognition hypothesis will be provided as often as possible. Each recognition hypothesis will serve as the input into Hypothesis construction module, in which, next words will be predicted according general, speaker-dependent and topic models. Statistical n-gram language models/DNN networks that are used in ASR module can also serve for final hypothesis generation.
The same scenario can be used also on the side of the interpreter, where the system can suggest next words of his interpretation.
Especially interesting can be an automatic transfer of
emotions, which can help interpreter to choose the most
appropriate words, which consider emotional coloring of
interpreted speech. Work proposed in [
The aim of the paper is to stimulate discussion on the use of anticipation and anticipatory behavior in the HMI and in the practical applications. Certainly, anticipation lies behind the smooth human-human interactions and rapid turn-taking and we believe that it can significantly accelerate human-machine spoken interaction and to support human-like character of the HMI.
We believe that machine-supported anticipation can decrease cognitive load in several applications, e.g. in simultaneous interpretation, where such a system can suggest hypothesis for the interpreter in advance or to support preparation of the interpretation result.
We realize that obtained results are relevant only to the interview scenario and the distribution of analyzed overlaps category will change according roles, relationship, emotions of interlocutors and according discussed topics. The first challenge of our future work will be collecting and analysing of dialogue interactions in other scenarios.
The research presented in this paper was supported by the Slovak Research and Development Agency projects APVV SK-TW-2017-0005, APVV-15-0731, Ministry of Education, Science, Research and Sport of the Slovak Republic under the research project VEGA 1/0511/17 and by Cultural and Educational Grant Agency of the Slovak Republic, grant No. KEGA 009TUKE-4/2019.