In this paper, we propose a remote collaboration system to assist the person with visually impaired in object-finding. The system consists of a 360-degree image centering on a person with visually impaired and presented as a panoramic image on the PC screen of a supporter in a remote location. By clicking on the PC screen, the supporter can present the AR audio (auditory indicator) superimposed on the real space that the person with visually impaired perceives, using spatial sounds. Auditory indicator enables the person with visually impaired to understand the location of the object intuitively. We conducted experiments to clarify the efect of the proposed system on the performance time of the object-finding task and the phrases of the supporters. The results of the experiment showed that the auditory indicator enabled the supporter to guide the simulated person with visually impaired by using demonstrative pronouns such as “this” and “here”.
in language, sighted people describe the route based on
their location. In contrast, PVI describes the route based
According to 2012 World Health Organization (WHO) on the locations of landmarks. Furthermore, for route
report, there are approximately 285 million person with descriptions between specific points, audio navigation
visually impaired (PVI) worldwide1. PVI sufer many in- created by PVI is subjectively more satisfactory for PVI
conveniences in their daily lives due to their inability to than that created by sighted people [8]. The reason was
recognize visual information. Various assistive technolo- that PVI felt more secure, oriented, and clear when the
gies have been studied, such as navigation aids [
However, these remote collaboration systems do not According to Kraut et al. [11], in a remote
collaboraconsider the diferences in spatial perception between tion among sighted people, sharing the local user’s visual
PVI and sighted people. According to Tsai et al. [
sighted people using AR audio. 2. Related work remote location. A 360-degree camera is used to present a panoramic image of the environment around PVI on the screen of the supporter’s PC. When the supporter clicks any point in the panoramic image, auditory indicators are presented to PVI from the direction corresponding to the panoramic image. By presenting auditory indicators with spatial sounds, PVI can intuitively perform object-finding.
This chapter describes studies on object-finding assis- 3.2. Interface
tance for PVI. Kaul et al. [
In order to improve the reliability and usefulness of source is placed on the computer’s three-dimensional
assistive technology, Bigham et al. developed VizWiz [
Be My Eyes2 is a mobile application that allows PVI to tial sounds to PVI.
ask for assistance from a remote location using a video Spatial sound means that meta-information such as
chat on their smartphones. PVI does not need to be distance, direction, and spatial extent is represented in
aware of the object’s location in advance to receive as- the sound reproduction. The perceived information, such
sistance. However, the field of view that can be shared as distance and direction, is called sound image
localizawith the user is limited due to smartphone camera use. tion. Sound image localization can be applied to a sound
Jones et al. [13] show that in remote collaboration us- source by giving volume, time, and frequency response
ing smartphones, users who receive video sharing from diferences to the left and right voices. In this paper, to
their smartphones intentionally use information from present spatial sound, sound image localization is
conthe camera images when asking questions. However, the volved with a sound source using Unity and Steam Audio.
narrow field of view and the inability to control the
direction of the camera was found to be stress factors. In
Be My Eyes, since the person being assisted is PVI, the 4. Experiment
supporter cannot easily convey the information obtained
from the visual images, which may cause redundant com- In order to clarify the efect of using the proposed system
munication. Wang et al. [10] suggest that redundant on the performance time of the object-finding task and
communication is a stress factor in remote collaboration. the phrases of the supporter, we conducted an
objectifnding task experiment with a simulated person with
visually impaired (SPVI) and a participant playing the
3. System design role of a supporter in a remote location, based on the
experiment by Kual et al. [
used the proposed system, which enables the sharing of 360-degree images, auditory indicators, and voice instructions (Condition proposed.) The remaining two groups of four shared the 360-degree images and performed the task by voice instructions without auditory indicators (Condition control.).
In the laboratory, desks are arranged in four directions around the SPVI, and seven objects (targets) are placed on the desks. Figure 1 shows an overview of the laboratory.
In the experiment, SPVI is instructed by the experiment supervisor on what to find (targets). Although there are seven possible targets, SPVI is only informed of them once the experiment supervisor instructs SPVI to look for them. SPVI cooperates with the supporter through the system to find the target and carry it to the designated position. Three trials were conducted per pair. Since the eye mask blocked SPVI’s vision, the experiment was conducted with SPVI sitting on a swivel chair with wheels to avoid the risk of falling.
Participants in Pairs #1 and #2 were presented with auditory indicators, while those in Pairs #3 and #4 were not presented with auditory indicators.
rectly in the third trial of pair #1 due to the problem with the output of the panoramic image. Therefore, the task performance time was excluded as an outlier. In addition, the task of the third trial of pair #3 was also excluded from matching the number of tasks.
As a result of the experiment, the task performance time tends to be shortened when auditory indicators are presented.
When auditory indicators were presented, phrases such as "right/left/straight" were also confirmed. On the other hand, we could confirm the use of demonstrative pronouns such as "this" and "here". In addition, it was confirmed that SPVI responded correctly to the direction of the target when demonstrative pronouns were used. Table 2 shows the number of times demonstrative pronouns, the phrases "left," "right," and "straight" were used.
Pro.
Left Right Straight Table 1 were presented. In addition, it was also confirmed that the Performance time. #1, #2 are using auditory indicator, #3, #4 auditory indicator made the supporter use more demonare not using auditory indicator. strative pronouns such as "this" and "here". We concluded
Performance time[s] that these results are because the presentation of the tarPair 1st 2nd 3rd Ave get direction by auditory indicators functioned in the #1 56.3 41.0 - 48.7 same way as gaze sharing in the remote collaboration #2 72.6 57.5 40.7 56.9 between sighted people. The results of this paper sug#3 52.0 74.3 - 63.2 gest that demonstrative pronouns can be used in remote #4 99.6 55.3 52.8 69.2 collaboration with PVI, indicating a new interaction between PVI and sighted people using AR audio. However, Table 2 one limitation of this paper is that the participants of Number of times demonstrative pronouns, the phrases “left”, the experiment were SPVI. Since SPVI is a blindfolded “right” and “straight” is used. Condition proposed: Using audi- sighted person, it is not representative of PVI, and future tory indicator, condition control: Not using auditory indicator. studies should be conducted with PVI as a participant.
In the future, it is necessary to investigate whether the presentation of auditory indicators has the same role as gaze sharing and to clarify the mechanism by which the demonstrative pronouns were used.
We conclude that the reason for the shorter task performance time is that the auditory indicator enables an intuitive presentation of the target direction. In the remote collaboration between sighted people, task eficiency was improved by sharing the gaze of the supporter by pointing [12]. We consider that a similar mechanism is responsible for shorting task performance time. We also note that demonstrative pronouns were used. Previous work [12, 11] confirmed that sharing the visual space enables supporters to use demonstrative pronouns, improving task eficiency. The results of this paper are consistent with these results.
The use of demonstrative pronouns is considered to be because the auditory indicators functioned in the same way as pointing for the sighted people, and a pseudovisual space was shared between SPVI and the supporters.
This is concluded from the fact that in the study of remote collaboration among sighted people by Kraut et al. [11], the supporter started to use demonstrative pronouns after the visual space was shared.
In this paper, we propose a system that uses auditory indicators to provide directions to assist PVI in objectifnding through remote collaboration. In order to clarify the efectiveness of the proposed system, we conducted an object-finding task experiment. We investigated the efect of the auditory indicator on the task performance time and the phrases of supporters. As a result of the experiment, it was confirmed that the task performance time tended to be shortened when auditory indicators spatial voice navigation for people with visual impairment and without visual impairment, in: Proceedings of the 2019 7th International Conference on Information and Education Technology, 2019, pp. 295–300. [8] Y.-H. Hung, K.-Y. Tsai, E. Chang, R. Chen, Voice navigation created by vip improves spatial performance in people with impaired vision, International journal of environmental research and public health 19 (2022) 4138. [9] Y. Hato, S. Satake, T. Kanda, M. Imai, N. Hagita, Pointing to space: modeling of deictic interaction referring to regions, in: 2010 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI), IEEE, 2010, pp. 301–308. [10] B. Wang, Y. Liu, J. Qian, S. K. Parker, Achieving efective remote working during the covid-19 pandemic: A work design perspective, Applied psychology 70 (2021) 16–59. [11] R. E. Kraut, D. Gergle, S. R. Fussell, The use of visual information in shared visual spaces: Informing the development of virtual co-presence, in: Proceedings of the 2002 ACM conference on Computer supported cooperative work, 2002, pp. 31–40. [12] K. Gupta, G. A. Lee, M. Billinghurst, Do you see what i see? the efect of gaze tracking on task space remote collaboration, IEEE transactions on visualization and computer graphics 22 (2016) 2413–2422. [13] B. Jones, A. Witcraft, S. Bateman, C. Neustaedter, A. Tang, Mechanics of camera work in mobile video collaboration, in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015, pp. 957–966.