In recent years, with the outbreak of Covid-19, there has been an increasing opportunity to move communication with others to online venues. In addition, job interviews, which are important for many students with regard to communication, have been moved online. In order to avoid tension and anxiety during the interviews, it is important to train using the VR interview simulator. However, the effects of the interviewer's facial color and expression on the interviewee's sense of tension and anxiety have not been clarified. With regard to facial color and expression, it has been found that a red face enhances the perception of anger. In this study, we conducted tense VR interview training focusing on the relationship between red face and anger facial expression perception. Then, we investigated the effects of the interviewee's perception of interviewer's anger by the interviewer's red face and angry expression on the interviewee's tension and anxiety. We then developed a more effective interview simulator. In the experiment, the interviewer avatar's complexion was made to turn red or change to an angry expression in response to what the subject said during the interview training. The subjects' tension and anxiety changes before and after the interview were then surveyed using a questionnaire. The results showed that in the VR interview training, there is a synergistic effect of the interviewer avatar's facial color and expression that significantly affects the interviewee's tension and anxiety due to the interviewee's perception of interviewer's anger, which may be an important factor in developing effective, tension-filled interview training.
In recent years, with the global outbreak of Covid-19 and the rapid development of virtual reality (VR) technology, there have been increasing opportunities to move communication with others, such as meetings and interviews, which used to be conducted face-to-face, to online venues such as telework and video conferencing. Furthermore, social VR contents represented by VRChat [1] are currently attracting attention and are one of the useful tools for communicating with others remotely. There have been many studies on avatar-mediated communication using VR here, and in a study [2] that compared multi-person interaction in the video conference and avatar-mediated communication in a VR space using intellectual, subjective judgment, and negotiation tasks between subjects, it was found that suggests that VR can be used to achieve communication similar to face-to-face communication. Furthermore, a number of studies have shown that communication in a virtual space using VR can also affect individual personality traits. A previous study [3] investigated the impact of virtual space experiences on individual communication compared to face-to-face communication. In addition, the impact of personal personality traits such as shyness on the communication experience was also investigated, and it was found that shy participants were less anxious when communicating online in virtual space than they were when communicating in person.
Furthermore, many studies have been conducted on the effects of visual characteristics such as the appearance and facial expressions of the partner avatar on subject's impressions of the partner avatar. In a previous study [4] that investigated the involvement of nonverbal visual features in impressions given by manipulating the size of the avatar's pupils, blink frequency, and viewing angle, the avatar with larger pupils and less frequent blinking, as well as the avatar viewed from below, were rated as the most sociable, confident, and attractive. In addition, a previous study [5] that investigated the effects of the facial expressions 0000-0002-2190-4629 (M. Sato) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). of the partner avatar on subjects in communication using a trust game in a VR space suggested that the facial expressions of the partner avatar, regardless of whether they were positive or negative, influenced subject's trust and decision-making behavior.
In addition, in terms of communication with others,
interviews in job hunting and examinations are
important for many students. With the recent outbreak
of Covid-19, the global economic situation has worsened,
and problems such as the employment crisis and
unemployment have made many job-seeking university
students anxious [6]. Therefore, interview training is
considered to be effective in order to avoid being driven
by tension and anxiety in the interview situation. An
interview simulator using VR is an effective and efficient
tool for improving the interviewee's skills, providing the
interviewee with an opportunity to practice in a virtual
space and feedback from a virtual interviewer to relieve
tension and reduce interview anxiety. In a previous
study [7], a virtual mock job interview simulation was
developed as a career support for college students
preparing for job hunting, and VR exposure therapy was
used to reduce anxiety about job interviews. They then
investigated the impact of the level of reality of the
virtual interviewer's graphics during the process. The
results suggested that the higher the level of graphics,
the greater the sense of presence, but did not have the
significant effect on the sense of anxiety. In a previous
study [8], it focused on job interviews, which cause
anxiety in people of all occupations, and investigated
what level of reality is necessary for subjects to cope
with their anxiety by conducting interviews with virtual
avatars of various levels of reality, such as realistic,
cartoonish, and actual photographs. As a result, subjects
showed more anxiety depending on the avatar's attitude
than on the avatar's reality. In a previous study [9],
factors that potentially influence interviewee behavior
in VR interview training were examined separately.
Then, a virtual reality job interview simulator was
developed to investigate what factors influence
interviewee's anxiety. The results showed that the type
of interview questions had the most significant effect on
anxiety. Questions related to the interviewee's college
expertise, such as computers, algorithms, and
programming, may have caused a cognitive load on the
interviewees, leading to increased anxiety and decreased
communication skills in their interview performance.
The Trier Social Stress Test (TSST), which measures
subject's acute stress response, has also been VR-ized,
suggesting that the VR-TSST is effective in inducing
psychosocial stress, although the effect is smaller than
the traditional TSST [
Many studies have investigated the factors that
influence interviewee`s anxiety in VR interview
simulators. However, the relationship between the facial
color of the virtual interviewer and expression has not
been clarified in terms of its effect on interviewee`s
anxiety. Several studies have been conducted on the
recognition of facial color and facial expression. In a
previous study [
The relationship between communication with
avatars using VR and facial color has also been studied;
a previous study [
In this study, we focused on the relationship between anger perception by facial color and expression revealed in previous studies, and clarified the effects on tension and anxiety when subjects were made to strongly perceive the interviewer avatar's angry emotion. Then, we intentionally conducted virtual interview training with a sense of tension to verify what effect it has on subjects' interview performance and developed an effective VR interview simulator. To this end, we created a VR interview simulator that changes the facial color and expression of the negative interviewer avatar, which affected subjects' anxiety in previous studies, to blush or angry expression depending on subjects' answers.
The experimental program used in this study was created using Unity (2019.4.12f1). Two PCs were used in this experiment: one for the subject and one for the experimental assistant who played the interviewer avatar. The subject's PC was equipped with Windows 10 OS, Intel® Core i7-11700KF 3.60GHz CPU, NVIDIA GeForce RTX 3060 Ti GPU, and 32GB memory. The PC for the experimental assistant was equipped with Windows 10 OS, Intel® Core i7-4790F 4.00GHz CPU, NVIDIA GeForce GTX 2070 GPU, and 16GB memory. The subjects wore a head-mounted display (HMD: VIVE Pro, HTC, resolution 2880*1600, refresh rate 90 Hz, viewing angle 110°) on their heads and trackers (VIVE Tracker 3.0, HTC) on their right and left wrists, and the position and rotation information of the subject's head and wrists were obtained from the base station (SteamVR Base Station 2.0, HTC), which is an infrared sensor. The movements of the avatar were synchronized with those of the user by tracking the movements using Final IK, a Unity asset. In this experiment, the subject and the avatar played by the assistant communicated in the same virtual space, so it was necessary to synchronize the movements of the subject's avatar from the perspective of the interviewer avatar. To this end, we created a communication environment in Unity and imported PUN2 (Photon Unity Networking 2), an asset that enables multiplayer play, to construct an experimental system in which the subject and the other party can each assume the role of an avatar and face each other and communicate. Figure 1 shows a subject participating in the experiment, and Figure 2 shows each avatar facing each other in the created virtual space.
MakeHuman (The MakeHuman team.), an open-source 3D character creation software, was used to create the avatars used in this experiment. This software allows the user to adjust the height, gender, age, skin color, and skin aging of the avatar by setting parameters. Using this system, we created an avatar that looks like a Japanese male, based on a questionnaire.
In this study, we also examined the possibility of using face tracking to express the interviewer avatar's angry facial expression. However, we found that facetracking could not express the anger that was conveyed to the subject, and that the assistant who played the role of the interviewer avatar required considerable training to perform the expression of anger. Therefore, animated facial expression changes were substituted. The created avatars were imported into Blender (Blender Foundation), and animations were created for mouth movements, blinking, and transitions to angry expressions. Figure 3 shows the interviewer avatar with normal and angry faces. We considered having the experimenter's assistant actually speak the voice and dialogue of the interviewer's avatar. However, the assistant's voice as it was would have revealed the identity of the assistant to the subject, so it was necessary to use a voice changer. However, the voice changer was not compatible with an angry speech style, which could make subjects feel uncomfortable, so we used Japanese text-to-speech software. In this study, we referred to previous studies [8, 9] and carefully selected
In this experiment, it was necessary for the interviewer avatar to blush and show angry expressions in response to the subject's statements during the interview training with the interviewer avatar. For this reason, the skin color of the avatar created by MakeHuman was dyed red using Adobe Photoshop 2022 (Adobe Systems Inc.). It was necessary to investigate what level of skin redness was appropriate to express anger in a more realistic manner. To this end, we manipulated the CIE Lab a* (red-green) values of MakeHuman's skin images using Adobe Photoshop 2022, and presented avatars with skin of different redness for a questionnaire-based survey. The CIELAB color space is modeled after the human visual system and is designed to be perceptually uniform. As a result of the survey, an avatar with skin set to 0 points for a* input and 15 points for output in the CIELAB color space of Adobe Photoshop 2022 was judged to have an appropriate skin tone for an angry expression. Regarding the animation of the interviewer avatar blushing, a questionnaire was used to create one that more realistically conveyed the change in skin tone. Figure 4 shows the interviewer avatar blushing with normal and angry faces. 2.4. Task In this experiment, the speech task was performed in which subjects answered questions from an interviewer a v a t a r . T h e s u b j e c t ' s v i e w p o i n t d u r i n g the speech task is shown in Figure 5. After confirming the operation of the HMD and tracker, the subject followed the instructions of the experimenter and faced the interviewer avatar, which was played by an assistant. After the interviewer avatar said, "Please introduce yourself," the subject introduced his or her name and affiliation, and then answered the questions posed by the interviewer avatar. The subject was instructed to look at the interviewer avatar's face at all times while answering. The time for answering the questions was two minutes, which was not told to the subjects during the experiment. The questions were randomly selected from the following four types of questions: "What are your strengths?" "What are your weaknesses?" "Please tell me about your dream," and "What kind of person are you said you are?" The interviewer avatar was randomly selected from the following conditions: "Normal face normal skin," "Normal face - red skin," "Angry face normal skin," and "Angry face - red skin. The interviewer avatar said "Why can't you answer?" when the subject could not speak for 5 to 10 seconds after being asked a question, and if there was extra time to answer, he said "Is that all you have to say? Talk more!" After that, the interviewer avatar shifted into an angry mode.
In the angry mode, the interviewer avatar in the angry face condition frowned and the interviewer avatar in the red skin condition blushed. The interviewer avatar continued to make intimidating comments in response to the subject's answers, such as "Please be more specific," and "Not yet? Answer quickly!" After two minutes had elapsed, the interviewer avatar said, "Unfortunately, time is running out," and terminated the speech task. Table 1 shows the correspondence table between the inputted keys and the interviewer avatar's lines.
The procedure of this experiment is described next. First, informed consent was obtained from the subjects, and an overview of this experiment and its procedures was provided. Next, a preliminary questionnaire was administered. The pre-questionnaire asked about the subject's name, age, VR experience, and job interview experience, and then asked for answers about the subject's pre-experiment state anxiety and mood state.
Next, the flow of the speech task was explained so that after meeting the interviewer avatar, the subject would introduce himself/herself and then answer questions posed by the interviewer avatar. The subject was then asked to put on the HMD and tracker, and after immersing himself/herself in the virtual space, he moved his own avatar to check the arm movements. After that, the interviewer avatar, played by an assistant, entered the virtual space and faced the subject's avatar. In this trial, we did not conduct a practice run because we thought that if we conducted a practice run, subjects would become accustomed to the task and this would affect the results. In this trial, subjects faced one of four types of avatars, introduced themselves, and answered questions from the avatars. To take into account the effect of habituation, the avatars and questions were presented randomly, and the same avatars and questions were never presented.
Subjects answered the in-experiment questionnaire after the interview with the interviewer avatar was over. The questionnaire asked for the subject's name, age, impressions of the appearance of the interviewer avatars, impressions of the interaction itself, state anxiety after the interview with the interviewer avatars, and mood state. This trial was conducted four times, one interview question per interviewer avatar. There was a break of at least 2 hours between each trial, and subjects had interview with all interviewer avatars over the course of two days. At the end of the experiment, subjects were asked in a post-test questionnaire to freely express their opinions and impressions about whether facial color or facial expression was more important in expressing the interviewer avatar's anger, based on this experiment.
In this experiment, the State-Trait Anxiety Inventory
(STAI) [
Twenty subjects participated in this experiment. Eighteen subjects were male and two were female, with a mean age of 22.45 years and a standard deviation of 0.86. Of the 20 subjects, 17 had experience with VR and seven had experienced with job interviews.
With respect to the experimental data, the data obtained were considered as within-subjects factors and were analyzed. The analysis was based on the difference in scores of the total scores of each subscale of the STAI state anxiety and TMS before and after the interview with each avatar; after confirming normality using the Shapiro-Wilk test, the paired t-test was used to test for significant differences. For the convenience of the graphs that appear in the following sections, we refer to - no
on the condition of the interviewer avatar.
A box-and-whisker diagram representing the amount of change in STAI state anxiety total scores before and after the interview with each avatar is shown in Figure 6.
Based on the amount of change in STAI state anxiety total scores before and after the interview with each avatar, a Shapiro-Wilk test was conducted and found to be normal in all conditions (p > 0.05), so a paired t-test was conducted, resulting in a strong significant difference in the Avatar 4 (t = -3.5621, p < 0.01; df = 19) condition, very strong significant differences in the Avatar 2 (t = -4.44, p < 0.001; df = 19) and Avatar 3 (t = 5.1278, p < 0.001; df = 19) conditions.
A one-way analysis of variance was performed to compare the change in STAI state anxiety total scores across each avatar condition, but multiple comparisons were not performed because no significant differences were obtained (F (3.76) = 1.392, p > 0.05). 3.2. TMS
Restless On edge Hopeless
Sad Gloomy Exhausted Fatigued
Tired Energetic
Lively Active Grouchy Anger
Peeved Uncertain About Things Unable to Concentrate
Bewildered Q1 Q2 Q3 Q4 Q5 Q6 Q7 obtained for Avatar 2 (t = 2.9066, p < 0.01; df = 19) and Avatar 3 (t = 3.2259, p < 0.01; df = 19), and very strong significant differences were obtained for Avatar 4 (t = 3.9428, p < 0.001; df = 19); strong significant difference were obtained for Avatar 3 (t = 2.6569, p < 0.01, df = 19), and significant differences were obtained for Avatar 1 (t = -2.7407, p < 0.05; df = 19) and Avatar 2 (t = -2.3424, p < 0.05; df = 19). No significant differences were found in the scores of the other scales.
One-way ANOVAs were conducted to compare the change in total scores for each TMS subscale across each avatar condition but multiple comparisons were not performed because no significant differences were F (3.76) = 0.083, p > 0.05),
F (3.76) = 0.456, p > 0 0.322, p > 0 Vigor F (3.76) = 1.293, p > 0 (F (3.76) = 0.641, p > 0.05), and p > 0.05).
F (3.76) =
We compared the Avatar 1 and Avatar 2 conditions. Since significant differences were obtained in the Avatar 2 condition, it was possible that the redder skin color made subjects more anxious when the interviewer avatar had a normal face. With regard to their impression of the interviewer avatar's appearance in the questionnaire administered during the experiment, some subjects responded to the Avatar 2 condition that they thought the interviewer avatar was angry because his face turned red, suggesting that the redder skin even in the normal face condition affected the subjects' perception of interviewer's anger.
Next, since strong significant differences were obtained for both Avatar 3 and Avatar 4, it was possible that subjects felt strongly anxious feelings when the interviewer avatar's facial expression was angry. In addition, the Avatar 3 condition might have made the subjects felt more strongly anxious, since the p-value in the paired t-test was the lowest and a very strong significant difference was obtained for Avatar 3. Comparing these two avatars, here again, with regard to impressions of the interviewer avatar's appearance, some subjects had the impression that the interviewer avatar with an angry face and red face was angry and scary, while others could see that he was angry with furrowed brow, but his skin suddenly turned red, giving the impression that he was more surprised than scared.
Based on the above, the STAI analysis suggested that subjects might feel more strongly anxious when the interviewer avatar blush for normal faces and do not blush for angry faces. However, in the post-experiment questionnaire asking whether the color of the interviewer avatar's face or facial expression was more important for the perception of interviewer's anger, 8 of the 20 subjects answered that facial expression was more important, 5 answered that facial color was more important, and 7 answered that both were important. It was difficult to give a clear answer as to whether facial expression or complexion had more influence on anger perception based on the STAI results alone. 4.2. TMS Next, we proceed with a discussion of each TMS scales. The experimental results showed that significant Vigor conditions except for Avatar 1. For Depression, Avatar 1 had the least effect on anxiety based on the STAI results, suggesting that the subjects felt depressed because they could not answer the questions well rather than because they were anxious about the interviewer avatar. st subjects had negative impressions of their interactions with the interviewer avatar, and said that the interviewer avatar had a coercive attitude and that the interview felt like a pressure interview, which made them anxious. Since this study aimed to train subjects to interview with a negative interviewer which had affected their anxiety in p r e v i o u s s t u d i e s [ 8 , 9 ] , we believed that we were able to replicate this to some extent in this regard. In other words, in this experiment, the subjects' vitality after the experiment was reduced due to the effects of the interview with the coercive and negative interviewer avatar. Next, with respect to the T A but Avatar 4, and tension only in Avatar 4. Therefore, in Avatar 4, subjects might have been more tense than angry with the interviewer. Considering the STAI results, it was possible that the interviewer avatar with an angry face and red skin had the most significant effect on the subjects' tension and anxiety during this interview training.
In recent years, people have shifted their communication to online, and job interviews, which are important and anxiety-provoking for many students, are no exception. The VR interview training is a tool to assist such anxious students and has been the subject of much research. However, there was no clarification regarding the effects on interviewees' perception of interviewer's anger when they were interviewed with the interviewer avatar who blushed or had angry facial expressions and intentionally made them feel tense or anxious. Therefore, in this study, we investigated and measured anxiety and mood changes before and after communication with each avatar using four types of avatars whose facial colors and expressions changed according to the subject's responses in interview training, and investigated which of the changes in the interviewer avatar's facial color and expression was more effective for the perception of anger.
In the task, the interviewer avatar, which was to perform negative attitudes and statements to the subjects' responses, intentionally reproduced an intimidating interview by changing its facial color and expression in response to the situation.
The experimental results indicated that the interaction between facial color and expression might be an important factor in the interviewee's perception of interviewer's anger.
Future prospects include the development of more effective interview training by incorporating real-time facial expression changes through facial tracking and physiological indicators such as heart rate and pupils.