Directing attention to critical areas of the road plays a key role in facilitating drivers' comprehension of their surrounding environment. Prior research has shown that explicit gaze guidance using visual cues can effectively enhance visual focus. This study explores whether incorporating gamification into gaze guidance can further improve attentional direction when observing real-world driving scenarios. We also examine the impact of such guidance on situational awareness. To this end, we developed gamified gaze guidance content tailored for passengers in autonomous vehicles. An experimental study was conducted with thirty participants who viewed recorded driving videos through a head-mounted display while their eye movements were tracked. The results demonstrate that gamified gaze guidance more effectively directs visual attention than conventional methods, and participants who experienced it also demonstrated an improved understanding of the driving environment.
Renner et al. have shown that displaying guiding objects can result in a gaze guidance effect.
Renner et al. found that using Augmented Reality (AR) to show the location of a real object can
reduce the time needed to find it [
However, there are challenges associated with using explicit gaze guidance. Studies have shown
that while it can enhance guidance, it may also increase cognitive load by adding extra objects to the
eyesight. Also, only displaying gaze guidance objects can lead to boredom and difficulty in gaining
a continuous gaze guidance rate. Furthermore, if target objects are obscured or blocked by the
guiding objects, it can cause distractions and make it difficult to understand the guiding information [
Gamification involves adding game-like elements, such as earning points or leveling up for players, to tasks to encourage participation and achieve goals more quickly. By integrating the narrative of gamification into tasks, we believe a significantly higher gaze guidance rate can be achieved compared to merely displaying gaze guidance objects, as this helps to keep participants’ attention focused longer on the road. Additionally, the anticipation of increased participation may enhance participants’ understanding of their surrounding environments as well.
Furthermore, Muguro et al. measured the reaction time of the driver to traffic hazards in a driving
simulator [
As illustrated in Figure 1, the purpose of this research is to improve the comprehension of specific areas in the surrounding environment for drivers by providing gaze guidance to an appropriate area. To achieve this objective, two experiments were conducted: one to assess the effect of gaze guidance through gamification in real-world driving videos and another to evaluate the comprehension of gaze-guided information.
There are mainly two ways to realize gaze guidance: implicit and explicit. In implicit gaze guidance,
gaze guidance is conducted by altering the color and resolution of the video being viewed. A key
advantage of this method is that it facilitates natural gaze guidance, as users may not realize the
intention behind it. Miyajima et al. utilized this approach in their research aimed at reducing motion
sickness by controlling gaze direction. As a result, they found that gaze guidance was possible while
not letting participants know the intention of the guidance [
In contrast to implicit gaze guidance, explicit gaze guidance employs visual cues such as arrows
or pointers. The advantage of this method is that it not only has a strong effect on gaze guidance but
also allows for intentional gaze guidance, where attention is directly guided. Sasamoto et al. applied
this approach in caregiving training. By showcasing the gaze of expert physiotherapists in training
videos, they observed that learners’ intentional gaze increased. Moreover, the learners gained a
deeper understanding of the essential points in the training video [
Furthermore, McCay-Peet et al. have studied how visual catchiness (saliency) affects user
engagement. The experiment tested how the visual prominence of important details affects user
engagement, focused attention, and emotional response by comparing two conditions: a
highsalience condition, where text was presented with a large font size, bold, or italicized styling, and a
low-salience condition, where text was displayed in a standard format without bold or italicized
styling. As a result, they have found that a visually noticeable location helps the users to process
information faster and more efficiently [
Some research on gaze guidance has been conducted in the context of autonomous vehicles. Han et
al. investigated the effect of gaze guidance on detecting hazardous situations faster during takeovers
by comparing the difference between high- and low- saliency levels. In this research, a driving
simulator was used for experiments. They found that gaze guidance with a high salience level, where
a flashing red bounding box appeared around the side mirror to help with blind spot detection, was
more effective in reducing crashes during takeovers. In contrast, gaze guidance with a low saliency
level, where a static red bounding box was used around the side mirror to assist with blind spot
awareness, was less effective [
Similarly, Laura et al. used a driving simulator to examine whether gaze guidance can help drivers
avoid pedestrian collisions by measuring reaction times and accident rates. As a result, they found
that gaze guidance not only decreases crashes involving pedestrians but also promotes safer driving
behavior overall [
Several studies have integrated gaze guidance with gamification. Bishop et al. developed a virtual
reality (VR)-based gamification system designed to train young cyclists aged 11 to 14. Participants
rode a virtual bicycle and earned points for looking at specific areas, such as intersections or
approaching cars. The results indicated that participants developed better safety habits [
Muguro et al. investigated how VR/AR-based gamification in autonomous vehicles can maintain
user engagement while ensuring road awareness. In the experiment, participants controlled a paddle
using a joystick to intercept objects while also responding to pop-up traffic hazards, allowing
researchers to measure reaction time, gaze behavior, and cognitive load. The findings show that
gamification helps direct drivers’ visual attention to relevant areas and maintains their engagement and
road awareness [
Steinberger et al. sought to reduce driver boredom and enhance focus through gamification. If the
driver’s gaze is focused on road hazards, pedestrians, and important objects, points will be given. If
the driver looks away for too long, the points will be deducted. This approach proved effective in
decreasing driver boredom and increasing engagement during extended periods of driving [
However, it is worth noting that most of these studies have been conducted in simulated
environments rather than real-world settings [
Therefore, to address the limitations of simulated environments and excessive complexity in gamification, our research utilizes real-world recorded driving videos and introduces a straightforward gamification system. This system is designed to help drivers easily understand their environment while still providing strong gaze guidance effects.
As shown in Figure 2, gaze guidance processing with gamification is applied to 360-degree driving videos on a frame-by-frame basis to guide the passenger’s eye to the intended location. The overall method is divided into two units: the image processing unit and the human interface unit. The image processing unit is responsible for analyzing the recorded driving scenes, extracting frames, and detecting important objects through segmentation. Once a target object, such as a pedestrian, vehicle, or signboard, is identified, the system places a sphere at the object’s coordinates so that the intended area of focus is clearly highlighted within the immersive environment.
The human interface unit builds on this foundation by introducing elements of gamification. Instead of passively displaying the sphere, the system actively responds to the user’s gaze: when the gaze aligns with the sphere, it enlarges in size, creating an immediate sense of interaction and feedback. If the gaze is held long enough, the object disappears, and a new one is generated shortly after, maintaining continuous engagement. This gamified cycle not only directs visual attention to critical areas of the road but also prevents fatigue or boredom that may occur with static cues. In the following subsections, each unit will be explained in detail, highlighting how they work together to achieve effective and engaging gaze guidance. As shown in Figure 3, the image processing unit serves as the initial stage of the gaze guidance method, managing the preprocessing of recorded 360-degree driving videos. It takes these videos as input and extracts individual frames. Each frame represents a snapshot of the driving environment, providing a structured basis for subsequent analysis and object tracking.
Once the frames are extracted, the next step involves selecting an object for the user to focus on in each frame. Object selection, such as for pedestrians and road signs, is performed using image segmentation, which di3vides the visual data into distinct regions based on shared characteristics such as color, texture, or shape at the pixel level. This process will be discussed in more detail in the implementation section. The segmentation process isolates the target object from its surrounding environment, ensuring that the gaze guidance system can accurately track the object’s location across consecutive frames.
After the target object is identified and segmented, the system proceeds to extract the object’s coordinates from each frame. These coordinates serve as spatial reference points necessary for placing the gaze guidance object within the video. Initially, the coordinates are extracted in two-dimensional Cartesian form. Once all the object coordinates are obtained, the system converts them into polar coordinates to match the 360-degree display environment used in the HMD.
To guide the user’s gaze, the system places a visually distinct sphere at these polar coordinates, ensuring that the object is easily noticeable within the immersive 360-degree environment. The sphere’s placement is carefully calibrated to match the object’s position across frames, maintaining consistency as the video plays. By the end of the image processing unit, 360-degree video with a gaze guidance object is generated as the output. As shown in Figure 4, the human interface unit is responsible for interacting with the user’s gaze behavior. The user’s gaze information is obtained from the HMD in polar coordinates. When the user’s gaze aligns with the coordinates of the gaze guidance object, the object undergoes a visual transformation by enlarging. The scale of this enlargement increases linearly according to the following equation:
Scale() = 1 + - / × ( − 1), (1) where represents the duration of time the user maintains their gaze on the sphere, is the required gaze duration for the sphere to disappear, and is the maximum scale of the sphere. In this experiment, the values are set to = 0.3 second and = 2. When the user continuously gazes at the sphere for one second, the sphere disappears. If the user’s gaze moves away from the sphere’s coordinates, the unit resets and starts counting from the beginning again. A few seconds after the sphere disappears, a new sphere is generated at a different location. This approach effectively directs the user’s attention by strategically placing gaze guidance objects within the driving videos.
4.1.
This section provides a detailed explanation of how gaze guidance videos are generated. The Segment
Anything Model 2 (SAM2) [
For this research experiment, three different types of videos are used, which are outlined below: • • •
Original 360-degree driving videos
360-degree driving videos + gaze guidance object
360-degree driving videos + gaze guidance object + gamification feature
The first video is the original recorded driving scene, specifically filmed in New Orleans [
5.1.
The objective of the first experiment was to evaluate the effect of gaze guidance through gamification in real-world driving videos. Three different types of videos are used, as mentioned in the previous chapter: original videos, videos with gaze guidance objects (no gamification features), and videos with gaze guidance objects plus gamification features. Also, three different driving scenes were prepared: turning right, stopping at a red light, and driving straight. Therefore, there are nine videos in total (3 original videos, 3 without gamification videos, and 3 with gamification videos). Table 1 provides a quick layout of the videos. Each participant views three different types of videos, each depicting different driving scenes. Each video lasts approximately 20 to 30 seconds, with a resolution of 3840 pixels × 2160 pixels at 30 frames per second (fps). The videos are presented in the following order: original, without gamification, and then with gamification. Each participant’s eye gaze information is recorded while watching the video.
Two experiments are conducted in this research. Both experiments are conducted with 30 people. All the participants had normal eyesight and normal color vision. Also, Meta Quest Pro was used in these experiments, which allows for tracking of the participants’ gaze information. Figure 5 shows the experiment scene with an HMD. Table 2 and Figure 6 show the results for the average gaze guidance rate for each video type and scene. The gaze guidance rate measures how many frames the participants focused on the gazeguided object across all video frames. The results indicate that original videos have the lowest gaze guidance rate, while videos featuring red spheres demonstrate a significantly higher gaze guidance rate. Furthermore, the combination of red spheres and gamification results in an even higher gaze guidance rate, suggesting that gamification has a substantial impact on guiding participants’ gaze. A Kruskal-Wallis test was conducted to determine if there were significant differences among the groups. The analysis revealed a significant difference (p < 0.001) among the different types of videos within the same scene.
The second experiment aimed to investigate whether users can enhance their understanding of their surrounding environment through gaze guidance. In this experiment, only two videos were asked to watch: original and gamification videos. Both videos are driving straight scenes. Each video lasts approximately 10 to 15 seconds, with a resolution of 3840 pixels × 2160 pixels at 30 fps. The order in which the videos were presented was randomized (either the original video first or the gamified video first). Figure 7 shows scenes from the gamification videos. After watching each video, participants answer three questions about the video content. Examples of these questions are as follows: • •
On the left road, there is a signboard with one alphabet written on it. What is the alphabet? Multiple choices: F, G, H, J On the left, there are two people: one person is dancing. What is the other person doing? Multiple choices: sleeping, playing guitar, painting a picture, watching a phone.
Table 3 shows the results of average accuracy rate and standard deviation for each video. The results indicate that the gamified videos achieve a higher average accuracy rate compared to the original videos. Table 4 and Table 5 illustrate the varying order in which participants viewed the videos. Table 4 shows results when the original videos are viewed first, while Table 5 shows results when the gamified videos are viewed first. From these two tables, it can be observed that videos watched later tend to have a higher accuracy rate. This phenomenon is likely due to a learning effect; after the first video, participants understood the question format and knew what details to attend to in the second video. Nevertheless, even after accounting for this learning effect, the gamified videos consistently show a higher accuracy rate. This finding suggests that gamification effectively aids participants in deepening their understanding of their surrounding environment.
This research aims to improve the understanding of the surrounding environment for drivers by providing gaze guidance to an appropriate area and specific location. Two experiments were conducted to achieve this goal using a video with gamification features for intentional gaze guidance. The first experiment explored the impact of gaze guidance through gamification in real-world driving scenarios. The second experiment examined whether this gaze guidance helps users gain a deeper comprehension of their surroundings. As a result, gamification significantly improves gaze guidance compared to traditional methods of explicit gaze direction, even in real-world videos. Additionally, gamification has been shown to enhance drivers’ awareness of their environment by guiding their gaze to a specific location. In conclusion, the findings indicate that gamification can intentionally enable gaze guidance to a specific location and effectively improve drivers’ understanding of their surroundings.
One limitation of this study is that in Experiment 1, the condition order was fixed (original, without gamification, with gamification), which may have introduced a learning effect as participants became increasingly familiar with the driving scenes. Although Experiment 2 randomized the order and confirmed the superiority of the gamified condition regardless of sequence, future work should adopt a fully counterbalanced design to eliminate potential order effects and further strengthen the validity of the results.
Special thanks are extended to Hitesh Pandya, Tiago Rodrigues, and Bruno Coelho (Capgemini Japan K.K., Engineering and RD Services) and to Chokiu Leung (University of Tsukuba, Institute of Systems and Information Engineering) for their valuable advice and insightful feedback throughout the research. This research received partial funding from the JSPS Grant-in-Aid for Scientific Research (Grant Number 24K02978, 25K03146).
During the preparation of this work, the author(s) used Grammarly in order to: Grammar and spelling check. After using these tools/services, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. [3] P. Lindemann, T.-Y. Lee and G. Rigoll, "Supporting Driver Situation Awareness for Autonomous Urban Driving with an Augmented-Reality Windshield Display," in IEEE International Conference on Intelligent Transportation Systems (ITSC), Hawaii, 2018. [4] R. Lin, L. Ma and W. Zhang, "An interview study exploring Tesla drivers' behavioral adaptation,"
Applied Ergonomics (Elsevier), vol. 72, pp. 37-47, 2018. [6] H. Somerville and D. Shepardson, "Uber car's 'safety' driver streamed TV show before fatal crash -police," Reuters, 23 June 2018. [Online]. Available: https://www.reuters.com/article/world/ubercar-s-safety-driver-streamed-tv-show-before-fatal-crash-police-idUSKBN1JI0LB/. [Accessed 20 August 2024].