With the rise of large language models, virtual humans (VHs) are becoming increasingly prevalent as conversational service agents across various domains. Consequently, kiosks equipped with large 2D displays have become the primary platform for deploying VH systems. However, VHs presented on 2D displays introduce the Mona Lisa efect, which hinders precise eye contact interaction with users. In this paper, we propose a straightforward method to alleviate the ambiguity in VHs' eye gaze direction. To evaluate the efectiveness of our method, we conducted an experiment involving 30 participants. The results revealed a statistically significant improvement in gaze perception accuracy within 2D VH systems when employing our approach. This improvement has the potential to enhance user engagement and the overall quality of human-virtual human interactions.
while non-verbal communication cues like gestures, eye
contact, and facial expressions are also crucial [
Recently, virtual humans (VHs) have been utilized in Among these non-verbal cues, eye contact plays a vital
various fields, including education, counseling, and en- role in expressing interest, concentration, facilitating
diatertainment. This is partially attributed to VH modeling logue in multiparty conversations, and enabling smooth
tools1 that have simplified the creation of VHs resembling turn-taking [
Ongoing studies are utilizing these technologies to in- VH as making eye contact [
Previous research has demonstrated that people have the VH systems. a tendency to expect human-like interactions from VHs. In this paper, we propose a technique to mitigate the In such interactions, verbal expressions hold significance, Mona Lisa efect in 2D display environments without the need for specialized equipment such as a parallax APMAR’23: The 15th Asia-Pacific Workshop on Mixed and Augmented barrier or stereoscopic glasses. Our method involves Reality, Aug. 18-19, 2023, Taipei, Taiwan subtly rotating the virtual camera, which significantly * Corresponding author. improves the perception of the gaze direction of a VH ($J. Ksuunmg)h; umny@upnughsaon.l.eaec@.krpn(Su..eJudnug()M;j.uLneeeg)old12@pusan.ac.kr displayed on a 2D screen. The rest of the paper discusses 0009-0007-0987-7190 (S. Jung); 0000-0003-3943-3463 (J. Kum); related work, details our proposed method, and presents 0000-0002-9421-8566 (M. Lee) an experiment conducted to validate the efectiveness of © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License our approach. 1 hCPWrEooUrctkReshtdoinpgpssIhStpN:/c1e:6u1r3-w/-0s.o7r3g/wACwttErwibUu.trRioenaW4l.0louInrstekironsanhti.oocnpoal m(PCCr/ocBhYce4a.0er).adcintegrs-c(CreEaUtoRr-/WS.org) 2https://www.nvidia.com/en-us/on-demand/session/gtcfall21d31017/
dressing the mitigation of the Mona Lisa efect, some relevant areas of work are summarized in this section.
The Mona Lisa efect is a phenomenon in which
individuals perceive that the gaze of a static image, depicting
a person or animal, is following them. This efect is
particularly noticeable within an angle of 5 degrees in both the
left and right directions [
In response to the Mona Lisa efect, researchers have
proposed various methods to enhance eye contact inter- Figure 1: The top view in the Unity 3D virtual environment
action with VHs in 2D display environments. Some of in the 5-degree condition with our proposed method, when
these approaches involve modifying devices and applying VH looking at the left participant
content-based techniques to achieve a more natural eye
contact experience. One such method includes the
addition of an assistive device that rotates the display to align real-world position. However, the position of their faces
with the user’s position, enabling a more natural and im- can vary depending on their sitting posture or height,
mersive eye contact interaction [
Our method is based on Fish Tank VR, proposed by Ware
et al. [
We employed YOLO8 [
We used a within-subjects design to examine the efectiveness of our proposed method. The experiment was conducted at 5 and 15-degree angles, with all participants ifrst experiencing the 15-degree condition. In both condi
(a) (b) (c) (b) Figure 4: (a) The experimental setup and the schematics of the experimental space and (b) actual experimental space tions, the virtual human (VH) appropriately rotated her spine, head, and eyes to match the position of the target it was looking at (Figure 2 and Figure 3).
The 15-degree condition was conducted to verify that the participants understood the experiment by assessing the VH’s gaze outside the range where the Mona Lisa efect occurs. There were no other visual cues except for the VH’s gesture. Then, the 5-degree condition was conducted to examine our proposed method in the narrow range where the Mona Lisa efect could occur. In both conditions, the VH looked at the left participant, the middle between both, or the right participant. A set of 30 gaze judgment tasks consisting of ten of each gaze direction was used for the 15-degree condition, while in the 5-degree condition, we used a total of 60 gaze judgment tasks. In the latter condition, our proposed method was applied for half the time (see Figure 3(a)), and the other half was not (Figure 3(b)) The order of VH’s gaze directions was counterbalanced and randomized. Additionally, five VHs were used in the experiment, and the order was also randomized. For the eye gaze of the VH, we exploited the Wizard of Oz paradigm, where an experimenter in another space controlled the position of the target for the VH to look at using a GUI.
We organized our experimental environment with a 65-inch TV, two tables, and two chairs (Fig 4(b)). The distance between the 2D display and the center of the two participants was 2.4 meters. Additionally, the participants sat at the same distance from side to side based on the VH; 0.64 meters in the 15-degree condition and 0.21 meters in the 5-degree condition (Fig 4(a)). A desk was placed in front of the participants so that they could fill out a questionnaire with their smartphones.
We used the Unity game engine 3 to render the VH and an ofice-like environment on the 2D display. For the VHs, we used rigged 3D human models created by Character Creator 3 4. To eliminate the gender efect of the VH, we made all five VHs female. We modified the appearance of the VHs to have diferent facial features, hairstyles, and clothes. The VH was placed behind a table with a monitor and other ofice supplies. In the real experimental space, we placed an actual desk in front of the 2D display. We used Final IK 5 for natural gaze behavior.
4https://www.reallusion.com/character-creator/ 5https://root-motion.com/ 4.3. Participants We found statistically significant results by using the one-way ANOVA. First, we compared the 15-degree conDue to the experimental configuration, we recruited two dition and the 5-degree condition (Figure 5(a)). The reparticipants at a time. In cases where only one participant sults showed that participants were more accurate at was available, one of the experimenters pretended to guessing where the VH was looking in the 15-degree be the participant and joined the experiment. The data condition, outside the Mona Lisa efect ( < 0.000). submitted by the experimenter were excluded from the Additionally, to assess the efectiveness of our proanalysis. We recruited 30 participants (13 males and 17 posed method, we used a dataset from the 5-degree confemales) from a local university. The average age of the dition for analysis (Figure 5(b)). In this result, we only participants was 22.4 (SD=2.10). Before the experiment, utilized the dataset when the VH looked at the left or right an experimenter tested the participants’ dominant eye participant since there was no camera rotation when the through a simple test (right eye 18, left eye 12). VH looked at the center. The participants were more accurate at guessing where the VH was looking in the 4.4. Questionnaire narrow range when our proposed method was applied The pre-questionnaire included questions about demo- ( = 0.033). graphics and the dominant eye. During the experiment, the participants selected whether they thought the VH 6. Discussion&Conclusion was looking at them, the center, or the other participant.
We counted the correct responses.
tary visual cues efectively diminished the Mona Lisa 4.5. Procedure efect. Not only did participants recognize eye contact as a distinguishing feature of the VH, but the subtle camera After the participants completed the pre-questionnaire, rotation also introduced subtle disparities between the the experimenter briefly explained the experiment. The desk and chair-like background. Building upon these participants were then seated in the appropriate seats findings, we anticipate that incorporating multiple visual based on the conditions, and the experiment began. cues will enhance the likelihood of accurately perceiving
After a 3-second black screen, the VH was rendered on eye contact. the display, looking either at the left participant, the cen- For instance, displaying the complete body of the VH ter, or the right participant, depending on the condition. can provide insight into the character’s body orientation, After both participants answered where they thought the while incorporating lines or patterns into the background VH was looking, the experimenter pressed the next but- can aid in establishing eye contact. These additional cues ton on the GUI to activate the black screen. The direction contribute to a more comprehensive visual context that the VH looked at was indicated on the GUI in a prede- facilitates the perception of eye contact. termined order. While the black screen was displayed, However, a notable limitation of this experiment is the experimenter clicked on the location of the target for its focus on two user cases centered around a specific the VH to look at in order. The VH and the main camera basis, casting doubt on whether the significant efect rotation were controlled in Unity automatically. For each observed would be consistent with a larger number of condition, we repeated this process 30 times. users or under non-central conditions. Consequently, further investigation is needed to determine whether these 5. Result limitations can be overcome and whether this method remains efective for more than three users.
Additionally, our experiments solely considered gaze.
Further tests are required to ascertain whether these minute diferences in eye contact can be perceived during a dialogue with a Virtual Human (VH). To enable these experiments, we need to equip the VH with the ability to recognize user speech and maintain eye contact with the correct user during conversation. Our objective is to test whether this enhancement enables users to detect these subtle diferences.
(a) (b) Figure 5: Each graph shows the score in (a) 5-degree and 15-degree conditions and in (b) camera rotation on / of in 5-degree condition.