Multisense Blind Shooter: Auditory mobile application to assist blind and visually impaired people in target shooting

Abstract

The Multisense project aims at making sport, and in particular modern pentathlon, include blind and visually impaired individuals (BVI) in ordinary club. With that perspective in mind, a first prototype of a mobile application, called Multisense Blind Shooter, was developed to make laser-run, especially target shooting, accessible to BVI. The smartphone has to be attached on an ordinary gun thanks to a 3D printed fixation. The application is based on auditory feedbacks to inform the shooters about the gesture correction they have to apply to hit the target. Four designs of auditory feedback paradigm are proposed.

Keywords

• eol>Human-Machine Interface
• Visual Impairment
• Assistive Technology
• Accessibility
• Auditory Display
• Auditory Guidance
• Sports and Video Games

1. Introduction

The Multisense project, carried out in partnership with an association for awareness of visual impairment, MixHandi-Cap sur la Vie (http://www.mcv-slb.fr/) aims to help the inclusion of visually impaired people in sport, particularly in modern Pentathlon. The demonstration we propose here concerns more specifically the laser-run discipline and more precisely the target shooting task. In this task, the constraints are both precision and time as the shooter must hit the target, using a laser weapon (without any projectile), 5 times in less than 50 seconds. Although adapted weapons already exist in other sports shooting practices [ 1 ], their cost do not allow ordinary clubs to buy them and the target shooting practice therefore remains reserved for adapted sports clubs poorly distributed over the territory. Furthermore this forces blind and visually impaired people (BVI) to keep among themselves instead of practicing with able-bodied individuals in ordinary clubs. In addition, the sound substitution used in these weapons is very dificult for beginners to access because it relies on memorizing a reference musical note and sound cues that only give access to a single dimension (distance to target) without indicating in which direction to redirect the weapon. As a solution, the Multisense project seeks to provide digital assistance based on auditory feedbacks to make target shooting more accessible to people with visual disabilities. This will enable the practice of Laser Run in ordinary sports clubs more inclusive, mixing able-bodied people and people with disabilities and thus will participate in the well-being and social integration of people "diferently competent" while promoting the evolution of representations, mentalities and learning to solidarity with respect for diversity for able-bodied people. Our proof of concept, named Multisense Blind Shooter, is based on sonification, i.e. representing non audio information through non verbal sounds (equivalent of visualisation for graphical representation of data). Our main question remains in the selection of the optimal auditory parameters to make shooting more accessible to beginners and still allow them to progress from recreational practice to competition. To answer this question, Multisense Blind Shooter currently proposed four diferent sonification approaches. This will allow us to later compare the actual sonified version of the target shooting assistance currently used in international sports shooting competition (i.e. a single dimension, only based on pitch and the reaching of a reference note pitch) with three other new methods. The first one is an adaptative method which aims to improve the accuracy of target acquisition in the zone close to the center of the target. The second one relies on 2D or even 3D sound guidance methods which have been developed very recently for other use contexts such as surgery [ 2 ]. Finally, we propose to include a sound spatialization system which has already shown its efectiveness in auditory guidance here again in another context of use, namely racing video games accessible to the blind [ 3 ].

2. System Overview

The Multisense Blind Shooter mobile application was developped with ease of access in mind. Four sound renderings are currently available on the application until they can be compared and eventually mixed and optimized. The smartphone fits into a 3D printed holder that is attached to a conventional shooting weapon (1). This simplifies the equipment needed for clubs by allowing visually impaired people to use their own phones.

2.1. Apparatus

Our mobile application is developed on Unity 3D, allowing us to access the phone’s sensors to retrieve yaw, pitch and roll. To calibrate the center of the target (align the virtual target that appear on the phone and the real physical target), a sighted person will need to aim at the real target once to set the center in the application. Once done, the user will be able to shoot with the laser gun on the real target. The sensors data are used to calculate the rotation of the aim in the application. By projecting a ray from the Unity’s scene camera on a plane, the coordinates of the sight are obtained in relation to the virtual target center. The distance information in abscissa and ordinate are then transformed in sound through one of the sound feedback algorithms before sent to the user in real time through headphones (required for spatialization). For our future experiments purposes, we are also able to shoot on the virtual target directly by touching the screen to retrieve the coordinates of the sight.

2.2. Sonification methods

Sonification is possible thanks to the many properties of sound that can be modified to obtain diferent returns. Based on former research on auditory guidance [ 4, 5, 2 ], the sonification methods integrated in Multisense rely on spatialization, pitch and the acoustic phenomenon of beats and roughness. Spatialization refers to the localisation of the sound in space that can be simulated by various audio techniques like stereo for 2D renderings and binaural for 3D renderings. Pitch defines the psycho-acoustical parameter that allows the diference between low and high sounds. It is directly related to the perception of the fundamental frequency of the sound and therefore to the perception of musical note played (example note A4 correspond to 440Hz, the lower the frequency the lower the pitch). This property is for example used in the Shepard tones which is made by adding sinusoidal frequencies separated by an octave creating an auditory illusion of a sound that constantly rises or falls. Beat occurs when two signals of close frequency are superimposed, a periodic modulation of the volume is then perceived. This modulation becomes more and more rapid when the frequencies are distant until a limit of approximately 15 Hz where the beat is so rapid that the sound becomes buzzing, it is the roughness [ 6 ].

1D guidance through Pitch [ 1 ]. The first method of sonification is inspired by the systems currently in place in world competitions of adapted shooting sports. It uses only the variation in pitch. The closer the user gets to the center, the higher the sound becomes. A reference note informs the shooter that they reached the center of the target. This method does not allow to know exactly where you are on the target but only the distance from the center. It requires a lot of training to learn the reference note and is therefore not dedicated to novices. This is why we want to compare its performance with other existing sonification strategies.

2D guidance through Pitch, Roughness and Beatings [ 5, 2 ]. This sonification, inspired by the work of Ziemer et al., uses multiple properties of sound to guide the user to a given point in surgery. This method is interesting since it sonifies each part of the target (top, bottom, right and left) in a diferent way, giving a more precise idea of the direction of the weapon. The x-axis is sonified in such a way that the user hears a range of Shepard’s tones rising more slowly as they approach the target from the left and falling more rapidly as they move away from the center to the right. The y-axis uses the beat and roughness of the sound so that when approaching the target from above, the beat becomes slower and slower. While when one approaches from below, the sound becomes less and less rough. When the user is in the center, the volume as well as the height of the sound remain constant.

Adaptative 2D guidance through pitch and Beatings. This is a 2-dimensional sonification that uses beats as well as pitch to transmit the position of the user according to the center of the target. On the x-axis, the closer you get to the center, the faster the beat, on the y-axis, the closer you get to the center, the higher the pitch is. In the center of the target, a distinct sound called white noise is created to distinguish it from the rest of the target. This method is therefore inspired by the two previous ones. However, this system uses a logarithmic dichotomy algorithm to obtain a more and more precise sonification as the user approaches the center of the target. Indeed, the range of sound modulation adapts according to the distance to the center. When the user travels half the distance to the target, the range is remapped so that this new distance becomes the new maximum distance. This method is the only one that have been pre-tested. Unfortunately pre-tests shown that the remapping is dificult to handle for the non-initiated. This is why we decided to add another sound parameter to our rendering : spatialization, as described as follow.

Spatialized Sonifications. Our application takes the 3 previous sonification methods and allows to add a binaural spatialization so the target appears localized in space. The users have then to try to place the sound in front of them to hit the center of the target. This methods is inspired from race gaming accessibility method [ 3 ] where the sound of the car engine had to stay in the center to avoid barriers and other obstacles.

3. Conclusion and Future work

Our next step will be to test our diferent audio feedbacks to ensure their usability and to determine their limits in terms of accuracy and rapidity of shooting. Another interesting question will be to determine the learning curve, i.e. the evolution of shooting performance after diferent types and duration of training, for each sonification strategy. Actually, we may even voluntarily choose to mix the diferent sonification strategies and personalize the audio rendering to better address preferences and needs of each user. These sonification methods can also be tested in the context of video games accessibility as the task of target acquisition is similar in both contexts of First Person Shooter games and target shooting sports except for their constraints. Indeed, video games do not require a harsh accuracy compared to sport performances, while on the contrary the target can be moving and shooting back in video games but will stay fix in sport.

We would like to thank Tim Ziemer for having shared his own code from [ 2 ] so we can integrate it directly in our Unity app. We are also thankfull to Matthieu Aussal and Morgane Besnier for their technical help on other sonification methods and in the project in general.

References

[1] B. B. Sport , A guide to visually impaired friendly sport , 2018 .

[2] T. Ziemer , N. Nuchprayoon , H. Schultheis , Psychoacoustic sonification as user interface for human-machine interaction , CoRR abs/ 1912 .08609 ( 2019 ). URL: http://arxiv.org/abs/ 1912 . 08609. arXiv: 1912 .08609.

[3] B. A. Smith , S. K. Nayar , The rad: Making racing games equivalently accessible to people who are blind , in: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI '18 , Association for Computing Machinery, New York, NY, USA, 2018 , p. 1 - 12 . URL: https://doi.org/10.1145/3173574.3174090. doi: 10 .1145/3173574.3174090.

[4] G. Parseihian , M. Aramaki , S. Ystad , R. Kronland-Martinet , Exploration of Sonification Strategies for Guidance in a Blind Driving Game , in: Technology with Swing, Revised papers of the CMMR2017 , 2018 . URL: https://hal.archives-ouvertes.fr/hal-01933559.

[5] T. Ziemer , H. Schultheis , Psychoacoustic auditory display for navigation: an auditory assistance system for spatial orientation tasks , Journal on Multimodal User Interfaces ( 2019 ) 205 - 218 . URL: https://doi.org/10.1007/s12193-018-0282-2. doi: 10 .1007/ s12193-018-0282-2.

[6] E. Zwicker , H. Fastl , Psychoacoustics. Facts and Models , Springer Series in Information Sciences, 2nd . ed., Springer Berlin, Heidelberg, 1999 .