In virtual reality (VR), the interactions of users with embodied agents when the users are anxious or when they do not accept an agent are not yet completely understood. Gaze can be indicative of the user's anxiety and acceptability of an embodied agent. An agent's expressions or actions can, in turn, be used to accommodate the user's anxiety. Previous work on social anxiety disorder (SAD) found evidence of avoidance or hyper-vigilant gaze patterns in relation to agents or people the participants were gazing at. Thus, we investigated if there are specific gaze patterns for normal individuals experiencing anxiety in the moment when gazing at an embodied agent. We focused mostly on avoidant gaze patterns. Based on evidence of gaze patterns in SAD and autism, we designed an experiment where normative individuals interact with an agent showing a neutral, happy and angry expressions. We aim to examine if normal anxious participants have similar gaze patterns or avoidance patterns to those with SAD. We also investigated if the user's acceptability or preference of the virtual agent's display of emotions had an efect on the avoidance via eye gaze. In particular, we investigated the user's eye patterns in relations to the agent's eyes, face or body to see if there were similarities to people with SAD. Using correlation analysis, we found a significant positive correlation between the acceptability of the participant to the virtual agent's expression and their ifxation on the agent's eyes. We also found a significant correlation between fixations on the agent's body and how anxious the participant was at the experiment's start. Later, these results can be used to find a link between acceptability, anxiety and SAD.
and were usually conducted using still photographs [
Individuals with Social Anxiety Disorder (SAD) react
difIn the field of virtual reality (VR), embodied agents are ferently to facial displays of emotion. This happens in
commonplace as non-player characters (NPCs) or as VR too. It happens independently of the avatar fidelity
other users (avatars) in-game. Thus, determining how [
We hypothesize that we can use gaze location on cial expression of the embodied agent. We would like to the agent to measure the degree of comfort towards an also observe if this afects normative individuals and if it agent’s facial expression or degree of user anxiety. To mimics those with SAD. this aim, eye gaze was measured using the VIVE Pro Eye tracker while participants looked at a VR agent with vary- 2.2. Hypotheses ing expressions. The main contributions of this paper are: Our hypotheses are as follows.
• Analyzing the correlation between general anxiety of a normative user and their gaze patterns on the embodied agent. • Analyzing the correlation between the user acceptance to the embodied agent’s emotional display and their gaze patterns on the embodied agent. • Comparing the findings to those found in SAD
using similar studies.
wards virtual agents. The main inspiration for this study
stems from gaze analysis of individuals with HSA (High
Social Anxiety) towards the facial expressions of other
individuals in social situations. Previous gaze studies
showed that individuals with HSA averted their gaze
when shown photos of individuals expressing positive or
negative emotions [
Wieckowski et al. explored variability in bias
toward social stimuli in the form of vigilant attention and
avoidant attention using eye gaze techniques instead of
a traditional probe task technique often used to study
attention bias in anxious youth with clinical SAD [
For H1, the Self Assessment Manniken (SAM) was used to assess if the facial expressions had an efect on the participant’s afect, to see if the avatar’s expressions afected the user. Regarding H2, while participants were answering questionnaires in 3.6 about the diferent emotional display of the agent, not all of them accepted the agent’s emotional display in the same manner. E.g. while some people highly disliked the happy face, other people were comfortable with it. We assessed if there is a pattern between acceptability of the emotional display and the number of fixations on the agent. For H3, we assessed if there is a relation between the anxiety of the participant and their fixation behaviors on the agent’s diferent body parts. We expected the more anxious participants to be avoidant of the agent’s face and eyes. The anxiety state in this case is the user’s default state before and during the experiment.
participated in the study; 10 Japanese, 1 Kenyan, 1
German, 2 Nepali, 1 Colombian, 4 Chinese, 1 Thai, and 1
Malaysian. No participant tried our system before. The
participants were asked to wear glasses if their vision was
not good. Sources of error were accounted for by
removing three participants in which there was missing data
(e.g., the VIVE Pro Eye tracking was disabled accidentally
for one of the faces). Three participants that were
extremely fatigued were excluded using a fatigue score in
the pre-questionnare. After the exclusions, the number
of participants was 15. The experiment was approved by
the ethics committee of our institution.
3.2. Experiment Design hyper realistic [
The possibility that there is an uncanny valley in the or number of fixations thus we did not account for the virtual agent’s appearance is higher with agents that are baseline when observing gaze patterns.
the uncanny valley, the avatar’s blinking was animated.
The frequency of the blinks was randomized, between
0.5 and 4 seconds. The agent’s gaze was fixed during
the run of this experiment. The agent was also given
a breathing animation using Maximo to give it a more
realistic feel [
Eye Metrics We implemented a gaze ray to determine the intersection of the participant’s gaze with the avatar’s face. To determine the number of collisions between the gaze ray and parts of the face, we added colliders to the face, eyes, and body (Fig. 5). When the ray did not collide with the avatar, it was recorded as ‘other.’ We defined that a constant gaze on the same body part for Questionnaires The participants answered questionnaires at diferent points during the experiment. Sometimes a questionnaire like SAM was repeated more than once so were questionnaires about the acceptability of the avatar. To identify each of which point of the experiment the participant answered the questionnaires we assigned codes as follows:
The questionnaire about the avatar was used to measure the participant’s acceptance aka contextual comfort of the virtual agent’s expression after each trial. The questionnaires were answered thrice per participant after each face. From the answers of the acceptability questionnaire, we created a variable known as acceptability by taking as follows: : (1) − (2)
accepted, while a negative value indicated that the agent Pre-questionnaire Before the experiments, partici- was rejected by the user. We calculated the acceptability pants reported their demographics and their current fa- per face per participant. This score was then used to find tigue and anxiety. They also reported their general anxi- a correlation between the acceptability of the agent. Each ety level on a 9-point Likert scale. facial expression was compared to the participant’s eye on fixations diferent locations of the embodied agent Self Assessment Manikin (SAM) A SAM question- (head, body, eye, etc). naire with 9 point likert scale is used to measure va- The acceptability questionnaire was conducted three lence,arousal and dominance. (Fig. 6) shows a sample of times per participant, after the participant viewed every the valence and arousal questionnaires used. SAM was expression of the agent for one minute. For instance, given to the participant before the experiment and after after the participant views the happy expression for one every face. SAM was used to measure how the avatar minute, the experiment stops and the participant answers afected the participant. The SAM questionnaire was the SAM and the acceptability questionnaires. This propresented prior to and after looking at the avatar. cedure is then repeated with the other two expressions. The questionnaires are not validated which might be a limitation of the study.
Acceptability Questionnaire This was a more detailed questionnaire about the avatar. The questionnaires were given a code AH, AA, or AN, after each diferent 4. Analysis and Results avatar with diferent facial expression. It asked how people felt about the avatar. It consisted of several questions Despite being given no instructions on where to dianswered on a 9-point Likert scale. The scores are on a rect their gaze, the majority of participants gazed at scale from 1 to 9 to match the same format of the SAM. the agent’s face or body. However, the gaze patterns From this questionnaire, we only used two questions for changed according to the facial expression shown, the further analyses: participant’s acceptability score, and the anxiety score, as detailed below. Deviations away from the face changed Q1 : I felt comforted by the avatar. according to the acceptability and anxiety scores. Q2 : I felt disturbed by the avatar respectively. SPSS was used to analyze the data. To test H1, we measured if the agent’s facial expression had an efect on the [a] Valence: A higher value means that a person is content while a lower value means that a person is upset participant’s afect as follows. The SAM questionnaire was taken four times and symbolized with experiment codes described in the Questionnaire portion of Sec. 3.6 as follows: Before the experiment (B), after the neutral agent(AN), after the happy agent (AH), after the angry agent (AA) for each participant.
A Shapiro-Wilk test showed a significant departure from normality W(98) = 0.94 , < 0.01; W(98) = 0.94, < 0.01; W(98) = 0.95, < 0.01 for valence, arousal and dominance respectively. We thus ran the KruskalWallis as a non-parametric test to compare if there were significant diferences between conditions B, AH, AA, [b] Arousal: Measures the state of alertness. A higher value and AN. A Kruskal-Wallis rank-sum test carried out on indicates higher alerntness. arousal valence and dominance showed that there was a statistical significance in valence ( 2(3) = 21.48, < 0.001) and arousal ( 2(3) = 8.867, < 0.05) between the diferent embodied agent expressions when compared to the baseline. The mean rank valence scores of 41.07, 45.16 , 28.67, and 62.5 for B, AN, AH, and AA, respectively.
For arousal, the mean rank scores were 39.98, 41.61, 57.6, 37.84 for B, AN, AH, and AA, respectively. The results for dominance showed no statistical significance.
Pairwise Wilcoxon rank sum tests with Benjamini- Figure 7: Boxplot showing [a] Valence and [b] Arousal values Hochberg (BH) -value adjustment were carried out for entered by participants after viewing each facial expression the SAM valence and arousal scores since they showed (AA, AH, AN) compared to the Baseline, B. The starred brackets significance. Experiment codes (B, AN, AH, AA) were show statistically significant pairings using the Wilcoxon rank used to represent the experiment stage as detailed in 3.6. sum tests.
The following score pairs were compared to each other: (B, AN), (B, AH), (B, AA) representing the comparison between the baseline state and the state of the participant (AA, AH) with ( < 0.01, Cohen’s = 0.25) and (AN, after viewing agents of diferent emotions. This is to test AH) with ( < 0.01, Cohen’s = 0.25); = 15 for all if the diferent emotional agents had a significant efect the test concerning valence and arousal on the afect of the participant. The diference between We tested H2 and H3 and see if anxiety and acceptabilthe scores of the following face pairs were compared to ity afect where the participant looks most at the avatar each other: (AN, AH), (AN, AA), (AH, AA). and whether they are avoiding the agent’s face or eyes as
The significant results for valence and arousal are sum- a result. We compared those results to previous findings marized using starred brackets in Fig. 7. Significant re- for socially anxious individuals. Kendall’s b correlation sults were as follows: The valence was larger after intro- was used to run the correlation tests since the sample size ducing the happy agent (AH), compared to the baseline is small, and the normality tests showed that the sample (B): ( < 0.05, Cohen’s = 0.25). The valence was did not follow a normal distribution as before. conversely lower than the baseline after introducing the To test H2, when running the tests on all the particiangry agent (AA): ( < 0.01, Cohen’s = 0.25). There pants, we found no correlation between the acceptability was no significant diferences in valence when comparing score and number of fixations on all portions of the agent. the score of the baseline condition to introducing the neu- We ran another Kendall’s b correlation using only anxtral agent (AH). The valence score was also lower after viewing the angry face as compared to after viewing the neutral face ( < 0.05, Cohen’s = 0.25). Conversely, Table 1 the valence score was higher after the happy face as com- Correlation values between acceptability score and fixations pared to the neutral face ( < 0.05, Cohen’s = 0.25). on diferent portions of the agent for N = 13 (anxious particiThe diference in valency values between the AH and AA pants) (*: < 0.05, **: < 0.01), pair was significant ( < 0.01, Cohen’s = 0.25). headf bodyf eyesf other
The results for arousal were as follows: the diference bodyf .04 between arousal values for the following score pairs with eyesf .51* − .22 the following experiment codes showing statistical sig- other .59** .21 .16 nificance: (B, AH) with ( < 0.05, Cohen’s = 0.25); Acceptability .31 .03 .51* .22 ious participants. The results are summarized in Table 1.
A positive correlation between the participant’s acceptability score and the number of fixations on the eye were found which was statistically significant ( = 0.510, < 0.05). The acceptability score was standardized between − 1 and 1.
To investigate H3, a Kendall’s b correlation was run [a] Angry Expression to determine the relationship between the participant’s anxiety score and amount of fixations on a certain portion of the avatar, as shown in Table 2, with = 45, regardless of the face used. There was a strong, positive correlation between the participant’s anxiety score of and the number of fixations on the agent body per minute, which was statistically significant ( = 0.30, [b] Neutral Expression < 0.001). Additionally, there was also a strong nega- Figure 8: Scatter plot illustrating the correlation between the tive correlation between the participant’s ( = − 0.22, anxiety score of the participant and the number of fixations < 0.01). The results are summarized in Table 2. detected on outside of the agent’s face, when observing the
A Kendall’s -b correlation was run to determine the avatar with the (a) angry expression as opposed to number relationship between the anxiety score of the participant of fixations on the agent’s body, when observing the agent and the number of fixations in a certain section of the with the (b) neutral expression over one minute. agent, among 15 participants, per facial expressions. We observed if there are any patterns for specific facial expressions. We found no significant correlations between tion between the anxiety score and the upper face with the anxiety score and the body fixations when present- statistical significance ( = − 0.263, < 0.05). For ing the participants with the happy face. However we the result per facial expression, only the happy expresfound a strong correlation between the number of body sion showed a strong negative correlation between the ifxations (bodyf) and the anxiety score when participants anxiety and the upper face, with statistical significance were presented with the neutral face, which was statis- ( = − 0.408, < 0.05). tically significant ( = 0.411, < 0.05). There was also a strong correlation between the number of fixations detected outside the agent and the anxiety score 5. Discussion when participants viewed the angry face, there was also a significant correlation between the number of fixations outside the agent when participants were viewing the angry face, which was statistically significant ( = 0.420, < 0.05). The values were Benjamini-Hochberg corrected. The results are summarized in the scatter plot detailed in Fig. 8. We ran two additional Kendall’s -b correlation tests to see if there was a correlation between anxiety score and if the participant gazed at the upper or lower part of the avatar’s face, first regardless of the face with = 45, then per facial expression (angry, neutral, happy) with = 15. For the results, regardless of the face presented there was strong negative correlaIn H1 we suggested that the agent afects the user valence and arousal. The results in Fig. 7 show that the agent has a significant efect on the valence and arousal of the user according to the emotion displayed. This shows that the model of the avatar actually works to afect the participant. The diferences between valence and arousal of the participant when viewing the neutral agent compared to the baseline were not statistically significant. This is expected because there was no emotion conveyed with the neutral facial expression. The angry and happy facial expressions afected the valence and arousal of the participants significantly compared to the baseline. This showed that there the facial expressions of the agents afected the participants’ valence and arousal.
The angry facial expression also induced a lower valence score while the happy facial expression induced a higher valence. This confirmed that the facial expressions of the avatar were perceived correctly and supports H1.
In H2, we predicted that user’s acceptability of the
agent emotions afects gaze patterns on the agent. SAD
individuals do not want to confront the emotions of
others, thus exhibiting behaviors such as avoiding the face
or being hyper-vigilant to cope with emotional display.
In addition to the anxiety score, we measured if the par- the agent, when anxious, matching the pattern of those
ticipant’s acceptance or rejection of the agent’s emotions with SAD as per [
The acceptability score can be used to change how The study observes the correlation between anxiety of agent’s faces are studied. It can be used in the future to normative users, their acceptability of the agent’s emostudy the root cause of SAD or even cultural diferences tions and the respective gaze patterns on an agent with in social norms when both accepting and reacting to varying emotions. Our results suggest that individuals varying facial expressions. The acceptability measure with anxiety in the moment have similar gaze patterns to can be used as an extra factor in future SAD studies. those with SAD. The similarities between normative indi
For H3, we hypothesized that user’s overall anxiety viduals facing anxiety in the moment, their gaze patterns
afects gaze patterns on the agent. In this study, we took and their acceptability of the agent, can unlock a better
the number of fixations on each part of the agent as understanding of the way SAD individuals operate. The
an indication of acceptance or avoidance of the facial techniques used for SAD individuals can also be used
expressions of the avatar. On one hand, an increased to accommodate normative anxious individuals facing
number of fixations on the eyes or head suggest that social situations in VR.
the participant accepts the avatar. This is consistent In this model the participant is allowed to look away
to previous studies in [
We also explored the relationship between agent avoid- parts of the agent’s face, were also analyzed. The more ance and the participants’ anxiety level. The initial anxi- negative the expression, the further the anxious particety score represents an approximation of the participants’ ipant strayed away from agent’s eye, then the face and overall state before facing the agent. We found signifi- entire body respectively. These findings are an important cant correlations between the user’s initial anxiety and indication in designing future systems. E.g. Nonverbal body fixations, invariant of facial expression as shown in agents with positive afect might be a better choice for Table 2. When analyzing the correlations, on per facial an anxious normative individual as they were still more expression basis, there was a strong correlation between likely to gaze at the face regardless of their anxiety. The the anxiety score and the number of fixations on the body studies also show that users are more likely to gaze at the of the agent or outside of the body when presented with agent’s eyes if they accept the agent’s emotional display. the neutral and angry expressions respectively. When A Metaverse VR avatar and VIVE tracker were used viewing the angry expression, the fixations were com- in the experiment. The technique can be easily applied pletely outside the body showing that the participant to a more ecologically valid setting to find avoidance is more likely to avoid the agent completely the more patterns of anxious users in real-time. This is useful to negative the expression is. adjust a non-verbal agent’s expression to accommodate
We also found a strong negative correlations between for the user’s anxiety and facial preference. Other senthe participants’ anxiety score and the number of fixa- sors can also be added to find stronger patterns e.g heart tions on the agent’s eyes, invariant of facial expression. sensor or facial tracker. The studies sufer some limitaThis indicates that the participant avoids the agent’s eyes tions due to limited participant count and non-validated and is more likely to look at the agent’s body or outside of questionnaires.