-
Since the advent of computers, the scope of the conversation partner in the human–
artificial agent dialogue system has been developed in various ways. The external
appearance of a digital human has reached a level high enough to be recognized as a real
human, and we often encounter them on the Internet, kiosks, and TV screens. In
addition to message information, human dialogue interactions include tone, pitch, and
nonverbal dialogue cues that constitute the context of speech intent and contain emotional
expressions.[
1
] Therefore, human–digital human interaction should be as natural and
intuitive as actual human interaction to avoid miscommunication. If a virtual agent’s
appearance is unnatural, it may offend the user’s feelings (e.g., the Uncanny Valley
Effect). In this sense, a digital human should resemble a real human being and produce
natural sounding/nonverbal responses to the human user. Nonverbal communication
makes up a large proportion of human-to-human communication, comprising about
two-thirds of human-to-human contact Nonverbal expressions are made up of the gaze,
facial expression, and gestures.[
2
] In addition to verbal expressions, nonverbal
expressions provide important communication functions such as providing information ahead
of spoken language in face-to-face interactions, controlling interactions, and expressing
intimacy. Therefore, the perception and social effects of digital human behavior should
be treated as important.[
3, 4
]
Nonverbal expressions can sometimes be misleading in meaning transfer because
they are represented by symbolic symbols reflecting the culture of each country. People
also express their feelings unconsciously and instinctively. Because nonverbal
expressions appear as symbolic symbols reflecting the culture of each country, it can
sometimes cause misunderstandings in conveying meaning. Sometimes, the latent content of
communication can play a more decisive role through these unrecognized nonverbal
expressions. Information that is not conveyed through language gives the impression
of more than the cognitive activity required for language generation. It can help
improve the reliability of digital humans by giving an impression of the mind.[
5
] In
addition, related studies have found that nonverbal communication can improve the
likeability of, interest in, and satisfaction with virtual agents.[
6, 7
] Natural nonverbal
communication serves as a key function of relationship formation and is a means by which
communication can embody information beyond messages. In particular, since digital
humans have a higher degree of freedom of expression of emotions than other types of
conversational agents and have the characteristics of manipulation in digital space, it is
expected that the nonverbal expressions of digital humans will have a significant impact
on design.
The purpose of this paper is to find out why digital humans as virtual agents have
greater potential as future conversation partners than robots through a case study
comparing virtual agents and robots. This paper consists of the following: Section 2. A
comparative study case analysis of virtual agents and robots. Section 3. Deriving the
strengths as a good listener of the digital human based on the results of research case
analysis. Section 4. Proposed the response four functions of a digital human as a listener
(good listener). Section 5. Discussion, conclusions, and future work.
2
2.1
Related Work
Concept and Application of Digital Human and Robot
A digital human (or, virtual human) is an artificial agent with both a human-like body
(expression or natural body movement) and intelligent, cognitively-driven behavior.[
8
]
A set of joints is added to the 3D face for expression and movement. The 3D face has
features such as eyes, teeth, tongue, and skin. Current research on virtual agents is
largely conducted in four areas: environmental design, training, culture and education,
and medical care.[
9
] Recently, digital humans have helped humans by taking various
roles such as advertising models (e.g., Lil Miquela, Oh Rozy, Imma), virtual idols,
teachers, counselors, coaches, and bankers. Digital human production companies aim
to replace most corporate chatbot services with a digital human. Digital human
production companies aim to replace most corporate chatbot services with digital humans.
Digital humans cannot perform tasks beyond the environment outside the interface. The
representative virtual agent Greta can display hand gestures, but her lower body is
motionless, i.e., the activity space is limited to the virtual world. If digital humans can be
used to perform tasks or collaborate in real environments for humans, it is expected that
the scope of their contribution will be expanded further than now. Robots can have a
variety of capabilities, mimicking human emotional states, intention and behavior
recognition, interpretation of contextual information, communication, and contextual
behavior.[
10
] Current applications of robots include a variety of areas such as
counseling,[
11
] education and training,[
12
] security and rescue operations,[
13
] social services
and business,[
14
] entertainment,[
15
] and industrial assistance,[
16
] Robots are
becoming more and more advanced in human interaction with humans and compared to the
virtual agent, the biggest advantage is that they can exhibit a physical presence. They
are now designed to be supported in a personal environment, such as at home. However,
the application of robots is still limited in cooperating with humans or in carrying out
social tasks for human welfare. This is because robots lack both physical dexterity and
expressive ability to imitate simple expressions. In general, robots must meet space and
cost requirements and have less hand degree of freedom,[
17
] The expressions displayed
through most postures and facial expressions are also limited, and only a few systems
can respond to visual response demands. Physical limitations include the robot’s angle,
joint speed and torque limitations, awkward arm composition or trajectory, and
excessively fast movement. Performing tasks using these action systems can make humans
uncertain and anxious. Even state-of-the-art humanoid robots are still unnatural,
unhuman, and expensive.[
18
]
2.2
The concept and function of the listener’s nonverbal response
Listener feedback may be defined as a response by the listener to the content of the
speaker’s utterance. The listener can switch to the speaker’s position at any time and
can express what they think or feel. During the speaker’s turn, they can provide
feedback without interfering with the utterance.[
19
] The expression of the listener’s
reaction is collectively referred to by various terms such as listener feedback, listener
response, backchannel, and nonverbal communication strategy (NVCS). It is used as
feedback on receiving the communicative behavior of the interlocutor, and through
language and gestures, the listener can indicate the level of participation in the
conversation. For example, the speaker may stop the conversation or restructure the sentence if
the listener is not interested.[
20
] Among many feedback types, listener feedback is an
important feature in face-to-face interactions because it represents the willingness to
continue to hear or invites speakers to continue with the conversation.[
21
] It can also
be used to express evaluations such as surprise, interest, and sympathy. If there is no
feedback, the speaker may feel anxious about whether the communication is going well
and the listener may feel as if they are talking to a hard “machine,” so it should be
handled with interest in the communication process. The listener can switch to the
speaker’s position at any time, express what he or she thinks or feels, and can provide
feedback without interfering with the utterance during the presenter’s turn.[
19
] Fig. 1
reconstructed a Shannon-Weaver-based model to supplement our current
understanding of some basic concepts. The listener creates the meaning as code and then transmits
it through the message. The speaker receives the code and understands the meaning.
The speaker sends a code back to the listener’s area in response.[
22
] In this way, the
listener and the speaker are mutually cyclical, suggesting that if you become a good
listener, you can become a good speaker at the same time. This paper focuses on the
nonverbal listener feedback as the ultimate goal of digital human development as a
dialogue listener and the ability to generate appropriate nonverbal expressions.
Method
Cases of Comparative Studies of the Virtual Agents and Robots
In human–robot interaction (HRI) studies, there have often been comparative studies
of physical and virtual implementations; therefore, we assume that the more
humanrelated social characteristics digital humans present, the more likely they will lead to
natural communication, so we would like to look at comparative studies of existing
virtual agents and robots. In the following, we list and explain the results of various
previous studies focusing on communication between robots and virtual agents.
[
23
] that when giving recommendations to users in a color selection experiment,
robots were less convincing than virtual characters on the screen. Participants had to
choose one of four colored square names displayed on a computer monitor. Before
making a decision, a robot or virtual character recommended one option to the user,
noting that it was the option chosen by other users. As a result, participants followed
virtual character recommendations more than robots. In the post-questionnaire
response, the subjective “familiarity” factor was found to be different from the subject’s
behavior. The familiarity factor of the robot group was much stronger than that of the
agent group, and the subject accepted more recommendations from the agent.
[
24
] compared people’s responses to robots, projected robots, and agents in health
interviews to help them understand differences in people’s social interactions with
agents and robots. The researchers hypothesized that robots would have more social
impact than agents simply because of their physical proximity. The results showed that
the robot had more social impact, but the participants who interacted with the agent
remembered more key information in the recall test than the participants who interacted
with the robot. studied whether humanoid robots in real life could elicit stronger
anthropomorphic interactions than software agents and whether physical presence
modulates this effect. The researchers predicted that subjects would anthropomorphize with
more anthropomorphic humanoid robots than less anthropomorphic agents. As a result,
the participant interacted more with the robot as a person than with the agent, and the
more anthropomorphic, the more subjects treated artificial agents as a person.
[
25
] demonstrated the importance of non-functional aspects that can enhance the
level of enjoyment and social presence of older people. It was hypothesized that the
more natural and human the conversation with the conversational agent, the higher the
perceived pleasure and acceptance. The virtual agent used in this study is “Steffie,” and
Steffie is a virtual 3D agent that can use various facial expressions, hand/arm gestures,
lip-syncing, and voice repetition in the form of a woman. The robot used Philips
Electronics’ iCat. iCat can make a variety of facial expressions using lips, eyes, eyelids, and
eyebrows, has a female voice and is in the shape of a cat. Statistics show a stronger
relationship between intention and use of virtual agents than robots. M Heerink
revealed that the two agents could not explain why the virtual agent had a stronger
influence on intention and use than the robot because of the fundamental difference in the
appearance and action system of the two agents.
[
26
] studied how the physical presence of a robot affects human judgments about a
robot as a social partner. Subjects participated in a simple book-moving task with either
a physically present robot or a humanoid robot displayed via live video. The Nico robot,
which was used in the experiment, was a humanoid robot in the upper body, wearing
children’s sportswear and a baseball cap. Nico’s head has a total of seven degrees of
freedom and six degrees of freedom (two on the shoulder, elbow and wrist) on each
arm. In the experiment, subjects easily approached Nico in video and augmented
conditions, while avoiding face-to-face encounters with the physically present Nico. The
researcher identified two causes for these results. First, physical robots can be perceived
as more expensive than monitors used in video display conditions, so robots may be
reluctant to come closer. Second, the granting of personal space between the robot and
the subject can be interpreted as a sign of respect. However, space was also created
between Nico in video and augmented conditions, indicating that the first case would
be more appropriate.
[
13
] invited Brazilian subjects to interact with two types of receptionists with
different appearances (agent vs mechanical robot) and voices (human vs mechanical) to
investigate factors related to designing a receptionist robot for deployment in Brazil. In
the interaction experiment, participants interacted with two receptionists with different
characteristics (a conversational virtual agent and a humanoid robot) and voice
(humanlike vs robot). Two receptionists directed the participants to a specific room where the
assessment was conducted via questionnaire. The researchers found that when
comparing Ana and Kobiana through all categories of questions, they preferred Ana in both
groups of participants and that the main reason was its human appearance.
3.2
Cases of Comparative Study of Virtual Agents and Robots
Through the review of related prior studies, we derived the following.
1. Although it was shown that people had stronger behavioral and attitudinal responses
to physically existing agents, when both physically implemented agents and virtually
implemented agents were presented, each had different results depending on the
purpose of the study (Table 1). Depending on the appearance of the virtual agent and its
degree of freedom to express, it is assumed that each resulted from a different result.
2. The nonverbal expression of the virtual agent usually has a positive effect on users
in the experiment, but unnatural expression or repetition may give a feeling of dis-
traction or discomfort. Therefore, it is necessary to provide natural and appropriate
feedback.
3. Users expect a natural conversation response from these anthropomorphic agents.
Therefore, the more similar to a person the agent is, the more likely the user will be to
treat the agent as a person.
4. Finally, since the difference in the social reality given by the implementation
environment is greater than the difference in appearance and function, it is necessary to
consider how this sense of presence can be realized in digital humans.
Based on the above four points, we felt that for a digital human to become a good
conversational partner, a new design unique to a digital human is needed that is
different from the existing robot design. The current nonverbal representations of virtual
Interactions with agents show a
stronger relationship between
intent and use than with robots.
Overall, participants preferred the
robot but easily approached the
virtual Nico while avoiding
faceto-face interaction with the robot
Nico.
Preferring virtual agents that
resemble humans to robots that do
not resemble humans.
Philips iCat
Steffie
[
27
]
Humanoid Nico
3D Modeling Nico
[
13
]
Humanoid Kobiana Ana
agents are not as natural as the real appearance of the digital human. The above study
results are experiments that exclude realistic human-like virtual agents (digital
humans), and since they did not focus on subjective evaluation criteria or use
representative evaluation scales, there is a possibility that participants’ responses may be
different. What virtual agents and robots have in common is that they have a body. Whether
virtual or physical, due to differences in the physical specifications of the agent, the
nonverbal expression and implementation method of the two are different, and various
nonverbal expressions can be generated due to this transformation.
3.3 Cases of Comparative Study of Virtual Agents and Robots
Efforts to communicate with humans naturally are continuing as in previous studies and
several improvement methods have also been proposed.[
28, 29
] Allwood proposed four
feedback functions: contact, perception, understanding, and attitudinal reactions. In this
paper, we reconstruct the overlapping functions of the four feedback functional
elements of the preceding studies as “attention,” “understanding,” and “opinion.”[
30, 31
]
Here, we would like to examine the case of a listener’s reaction studies by additionally
using the element of “timing” (Fig. 2).
Attention is an expression of the listener’s willingness and ability to recognize a
message.[
30
] Attention can help users interact with agents to provide even more listener
feedback.[
32
] A representative expression of attention is staring at the speaker. Gaze
can signal usually that the speaker’s continuing encouragement of utterance and that
communication channels is open. Yoichi found in healthcare studies that patients pay
more attention to agents when returning listener feedback while the patient is
speaking.[
33
] Oh studied the degree of attention conveyed by nodding, audio and
audio-visual feedback.[
34
] The robot was evaluated more positively when it displayed hand and
arm gestures with words and asked participants to pay attention to the robot during the
interaction.[
35
]. Allwood and Cerrato found that nodding the head conveys that the
listener is paying attention and further triggers a sympathetic reaction.[
36
]
The listener understands the speaker’s intentions through the language information, and
the speaker monitors the listener to see if what the speaker wants to convey has been
achieved.[
32
] The listener can express understanding by nodding or staring.[
31, 37
]
Nakano et al. found that nonverbal cues perceived as positive evidence of
comprehension were context-dependent. They also found that staring at the speaker was
interpreted as evidence of incomprehension that provoked further explanation from the
speaker.[
31
]
The speaker checks how the partner receives the message. Listeners can express their
opinions (acceptance, consent, preference, etc.) to make communication livelier. The
expression of opinion may take an expression form similar to the above understanding
element. Understanding, however, is simply focused on the listener’s understanding of
the information, and expression of opinion is an implicit confirmation of the
understanding. Nodding proved to be very important because all participants responded
“agree” when displayed alone. Smiling, nodding and raising eyebrows also received
high marks as signs of consent.[
31
] When virtual agent Billie requests confirmation,
nodding is considered an acceptance and shaking the head is considered an expression
of rejection. On the other hand, if the user nodded while agent “Billie” presented the
information, nodding was interpreted as evidence of understanding.[
38
]
Timing needs to be considered a digital human listener feedback element because it can
provide an unnatural feeling and a sense that we are indeed talking. For human-like
communication, proper timing of the response to feedback is important.[
39, 40
] Even
a good expression of consent can cause misunderstanding in the process of conveying
meaning if it appears when it is not appropriate. Also, timing can be a signal of
turntaking and can contribute to creating a natural and realistic digital human.[
21
]
[
41
] scrutinized when and how the listener inserted responses in line with the speaker’s
context. They suggested that the speaker’s gaze mediates this cooperation. [
42
] The
“Rapport Agent” creates rapport by providing feedback to the person speaking about
the comics they have seen before. The camera analyzes the speaker and determines the
appropriate moment to provide feedback with head nods, head shakes, head rolls, and
gaze.[
39
] Previous investigation of nonverbal feedback from avatars or robots revealed
that cues or reactions, such as head turns, are effective when they occur at meaningful
times rather than at random times
4
Discussion
Existing studies have not explored the potential mediated effects of digital human
appearance as no work has been done using digital human beings to any degree.
Therefore, we felt the need for a new guide to digital humans in line with the rapid
commercialization trend of digital humans. For natural communication, we showed that people
had stronger behavioral and attitudinal responses to physically present agents as
opposed to a virtual presence, but when both the physically implemented agent and the
virtually implemented agent were presented, the potential for development as a good
listener compared to the robot was found respectively. It is presumed that different
results were derived depending on the appearance of the virtual agent and the degree of
freedom to express it. Also, depending on the appearance and degree of freedom of the
virtual agent, the response can be linked to the agent’s overall satisfaction. Unnatural
expressions or excessive repetition may cause discomfort, so it is necessary to consider
providing natural and appropriate feedback. Based on this, we defined four feedback
A New Good Listener, the Digital Human 45
functional factors for interactive agents to become good listeners and proposed a wide
range of concepts that can be applied. The four functional feedback elements presented
were summarized into three elements (attention, understanding, and expression) that
overlap or have the same meaning in previous studies, then redefined a total of four
elements by adding the “timing” elements that stand out in other nonverbal
communication studies.
First, an expression that pays attention to the speaker is required. Second, whether the
understanding of the content of the ignition is successful or not. Third, it should be
possible to express the listener’s opinion about the content. Finally, the expression of
the listener’s attention, understanding, and opinion should be expressed in a timely
manner. Digital humans, which are currently commercialized, may be suitable as
lowcost personal assistants because they can be less expensive than robots and less
constrained by their environment of use. The digital human can be generally better than a
robot in that it can represent behavior, emotions, gestures, and expressions like humans.
All told, it suggests that the digital human has sufficient potential to be utilized as a
virtual listener. From the robots and agents used in this study alone, it is not clear to
what extent the results will be the same for each evaluation in different implementations
with different agent types. In future research, it will be necessary to check whether there
is an empirical effect as a good listener through the feedback presented.
5