108 Developing a Video Game Accessibility Toolkit for the Deaf and Hearing Impaired Omar Naranjo, Gonzalo Pérez, Manuel López Ibáñez, and Federico Peinado Departamento de Ingenierı́a del Software e Inteligencia Artificial Universidad Complutense de Madrid c/ Profesor José Garcı́a Santesmases 9, 28040 Madrid (Spain) onaranjo@ucm.es - gonzap03@ucm.es - manuel.lopez.ibanez@ucm.es - email@federicopeinado.com www.narratech.com Abstract. Accessibility is an aspect of video game development that has not been worked on to a great extent yet. It can have a great impact on gameplay, for those with some type of disability, and another large number of players who use those features to enhance their gaming expe- rience. This potential is something that industry is beginning to consider seriously, as attested by the Accessibility in Video Games award at The Game Awards ceremony. Some of these features, such as subtitles, have been implemented in video games for a long time, but in a simplistic way and without the support of other tools that allow accessibility issues to be addressed from several fronts. The goal of this project is the develop- ment of an Accessible Audio Toolkit that extend the Unity environment allowing creators to easily introduce configurable hearing accessibility techniques into their games, showing sound direction using a compass or a 3D radar, and displaying more configurable subtitles. The results have been experimentally verified, adding these features to a simple game where sound is very important, putting players to play it without sound, in order to confirm the effectiveness of our tool. The software has been published on the Unity Asset Store, allowing the developers community to integrate it into their own projects. Keywords: Game Development · Player Accessibility · Hearing · Sub- titles · Unity · Development Tools · Game Industry 1 Introduction Accessibility in video games can be defined as the ability to play a game under restrictive conditions, such as disabilities or functional impairments [3]. About 15% of the world’s population suffers from any kind of impairment, and this percentage is increased to a 20% when it comes to video game players, according to some experts [7] and market surveys [14]. Some specific conditions, however, can limit a person’s ability to play a game. The following list contains the most relevant ones, classified in four categories: Copyright © 2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 109 – Visual: Conditions affecting vision, such as blindness, partial visual impair- ments and colour blindness. – Auditory: Conditions affecting sound perception, such as deafness and hear- ing loss. – Motor: Conditions affecting a person’s ability to move, such as Parkinson’s disease and partial paralysis due to nerve failure. – Cognitive: Conditions affecting a person’s brain, such as dyslexia and at- tention deficit hyperactivity disorder. In this paper, we will focus only on auditory disabilities, which are particu- larly common among video game players. An auditory disability can be defined as any quantitative variation in sound perception. It can often be classified as hearing impairment or deafness. People with hearing loss often have trouble when trying to follow a regular conversation, particularly in noisy environments. As a result, dialogue in a video game may become practically inaudible for them due to an accumulation of other sound sources (e.g.: music [2], sound effects, etc.). The purpose of this work is to encourage the adoption of accessibility mea- sures by video game developers. We aim to do this in two ways: firstly, by exposing problems faced by people with disabilities, as well as potential benefits for studios that address them [15]; and secondly, by offering practical, real-life solutions that any developer could implement in their game. We specifically focus on small-to-average video game studios, often lacking resources and knowledge regarding this subject. This is why we offer a simple solution, easy to implement, that allows for including accessibility measures for people with a wide range of hearing disabilities. The rest of the paper is structured as follows: the next section introduces the concepts used in our research and summarizes the related work in the field. Section 3 presents the explanation of the development process of our tool, the Accessible Audio Toolkit (AAT), and how it works. Next, Section 4 explains the validation with real users and discuss their impressions and thoughts. Finally, we close the paper with some conclusions and future lines of research. 2 Accessibility in Games For many years, video games did not contain options or features to help people with hearing impairment to follow the plot of the game, the most that could be found were subtitles which could not be configured by the player, if not only they could be activate and deactivate [11]. Nowadays, many large video game companies are beginning to worry about this issue and are already adding a wide variety of accessibility options in their video games. It is indeed an interdisci- plinary field which requires a lot of research still. It was 2020 the first year that the The Game Awards recognized the work of the developers most committed to accessibility with the Innovation in accessibil- ity award, the nominated games were the following: HyperDot [12], Grounded [4], 110 Assassin’s Creed Valhalla [9],Watch Dogs: Legion [8] and The Last of Us Part 2 [6]. Where we highlight The Last of Us Part 2 [6] who was the winner of this award, due to the large number of different configuration options it presents. The developers of the video game, to meet the needs of all players, designed three preset options that include the recommended settings for visual, motor and hearing accessibility. In the part of hearing accessibility we find options such as an arrow that activates and indicates the direction from where you have received damage. An element of the HUD that alerts you when enemies are about to detect you. If the player has gotten lost in any area, he will have the possibility to press a button and a visual clue will come out where he has to go. Picking up ammo, crafting ingredients, or other resources displays a notification in the HUD. Fig. 1. Subtitles from The Last of Us Part 2 There is also the ability to customize subtitles. You can adjust the size, show a dark background to make it easier to read, change the color of the font, the name of who is speaking appears and you can add the possibility of indicating the address where that one is located. Apart from the games nominated for The Game Awards, we found other games that have implemented some accessibility options, as in the case of Fort- nite Battle Royale [16], the players they have the possibility to make a configu- ration to better detect enemies or objects in Fortnite. Since if you activate this option, the player will have a circular indicator of some sounds coming from the game, which will help players to be able to know more exactly where an enemy that has made noise or a chest is located. During the game the player will be able to capture the footsteps of the enemies, the shots, when a player builds, the sounds of unopened chests, of vehicles and other sounds. These sounds are indicated with an icon depending on the sound and the distance from which they 111 are produced is also shown in an approximate way, changing the colors, their intensity and opacity [5]. Fig. 2. Sound indicator in Fortnite Battle Royale Today large companies are starting to create games with a large number of accessibility options. But many indie games lack these features because it is resource-intensive and there are hardly any tools in stores to help them stream- line their work. That is why AAT is necessary. 3 Accessible Audio Toolkit The main difficulties encountered by people with hearing problems were identi- fied, contacting groups and associations of people with disabilities and conduct- ing an interview with a colleague of ours who suffers from bilateral hearing loss. We find it necessary to create a set of tools to translate all the sound information visually, facilitating its configuration and implementation for developers. 3.1 Subtitles It is used to inform, through text, what is playing in the video game. We wanted to help developers avoid common errors such as matching the font with the background, very small font size or a large amount of text per line. All these errors make it difficult for players to read the text [10]. We found two different types of subtitles: – Dialogue subtitles: which deal with a text that provides information about what a person is narrating or discussing. We want to give freedom to be able to configure them as you wish, being able to change the position, the size, 112 the number of characters per line, the color, the active time, and all this following the recommendations of the experts, such as Netflix [13], in order to help developers create subtitles that can be read clearly. – Event subtitles: These subtitles are used to inform players of non-word sounds, such as the sound of a gunshot or a mobile phone. These subtitles show the name of what is playing and together with an arrow it will indicate to the user the location where it is playing. Fig. 3. Subtitles by events detecting two sounds 3.2 Compass The design of the compass clearly originates from the tools of the same name widely used in games mainly of the RPG genre. These compasses represent the relative direction of a marker in game space with respect to the player’s position, and its distance through the size of the icon and its transparency. In the tool we use these characteristics to represent a good part of the sound information that would otherwise be difficult to display. Each perceived sound is represented on the compass by a marker whose characteristics, listed below, are dynamically modified. – Position: The position of the marker on the compass represents the player’s relative direction in relation to the source of the sound. – Size: The size of the marker represents the intensity with which the sound in question is perceived from the player’s position. – Color: The marker color indicates the type of object that is emitting the sound, as configured by the developer. – Altitude icon: An up or down arrow is displayed if that source is out of the camera’s field of view above or below the player respectively. 3.3 Radar 3D The purpose of 3D radar is to offer a solution to the graphical representation of sound in three-dimensional space. This is represented in the kit as a sphere whose surface is dynamically deformed according to the direction and intensity of the sounds it perceives. 113 Fig. 4. Compass rendering angle The points on the surface of the sphere that are closest to each of the audio sources will be identified, using the position of the marker to indicate the di- rection to the origin of the sound. As long as the sound intensity is zero, these points will remain static at a fixed distance from the center of the sphere (the radius of the sphere). When the sound in question increases in intensity, the dis- tance from the point to the center of the sphere will be increased, deforming the sphere and creating a sort of “spike” that points in the direction of the origin. Each “spike” will be represented in a different color depending on the type of object that is emitting the sound, as configured by the developer. In Fig 5, we show a screenshot where we can see our tools added in a video game, called textit Death Impact [1] created by first-year students of textit Video Game Development at the Complutense University of Madrid. In the central and upper part of the screen is the textitCompass, in the lower part is the Dialog subtitles and in the upper right the Radar 3D. 4 Experimental Validation To evaluate the usefulness of the project, we launched a campaign on social networks with which we got 25 users with whom to carry out tests and receive some feedback on their experience with the Compass and Radar3D tools. At this point in our research project we still don’t have a large enough group of subjects with hearing disabilities, so we perform a simulated deafness by disabling all the audio of the game. We carried out a type A/B test with the users, where one party would test the muted Death Impact game with our tools and another without them. The objective of the test was not simply to see which group played better, since that 114 Fig. 5. Screenshot of the game Death Impact [1] using our tools is trivial, but to evaluate if our decisions throughout the development of the tools had been correct, or how to improve if that was not the case. After completing the game, both groups answered a form on the question of accessibility in video games and their experience during the test, and group B also on their opinion of the tools. The majority of respondents voted “Shooting” as the most crucial genre on which to implement this type of measure. And almost all of the respondents in group A, who at no time saw the tools in action, asked for some kind of sound radar and/or subtitles. As for testers in group B, the acceptance of AAT is very positive, especially the compass. The Radar 3D still has almost a 75% approval rating, a slightly lower value that reflects that its design still has some potential for improvement. In addition, they were asked what sounds they were able to easily locate, with each tool, from each of the directions. The Radar3D is a somewhat more general tool, which allows us to perceive sounds from all directions with some precision, while the Compass is an ideal tool to represent frontal or lateral while it is unable to represent those coming from behind. 5 Conclusions In the research that we carried out at the beginning of this work, we came to the conclusion that small companies hardly used hearing accessibility tools, due to the cost or the scarcity of resources of this type that are available in the market. We are listening to people with real hearing disabilities to learn about their varied experience in video games and their opinion regarding accessibility in video games, and in the future we carry out studies on them. We found it necessary to develop a toolkit for developers that allows them to be easily implemented in any game developed in Unity. In addition, we wanted 115 to ensure a level of configuration of the tools that allows their use by as many people with disabilities as possible, without sacrificing their simplicity of use for both them and the developers. After developing the tools, we created a test environment that allow us to show the tool’s capabilities and integrate it into a real video game created by our Video Game Design course classmates. We demonstrated the usefulness of the tools through user testing, where the vast majority of testers respond that the tools apparently fulfill their function, and that these types of measures are necessary in modern games. Finally, we published AAT on the Unity AssetStore1 as a free asset, that we can gradually improve and add new tools, both for hearing accessibility and for other types of disabilities. References 1. Alonso Sánchez, J., Casiano Flores, D., del Castillo Rubio, I.: Death Impact (2021) 2. Bierre, K., Chetwynd, J., Ellis, B., Hinn, D.M., Westin, T., Ludi, S.: Game Not Over: Accessibility Issues in Video Games (2005) 3. Bierre, K., Hinn, M., Martin, T., McIntosh, M., Snider, T., Stone, K., Westin, T.: White paper on Accessibility in games: Motivations and approaches. igda (2004) 4. Brennecke, A.: Grounded (2020) 5. David, H.: Fortnite: cómo detectar mejor a los enemigos con la opción visualizar efectos de sonido. hobbyconsolas (2019) 6. Druckmann, N., Newman, A., Margenau, K.: The Last of Us Part 2 (2020), https: //www.playstation.com/es-es/games/the-last-of-us-part-ii/ 7. Hamilton, I.: Ian Hamilton Web (2013), http://ian-hamilton.com/ 8. Hocking, C., Hudson, K.: Watch Dogs: Legion (2020) 9. Ismail, A., Baptizat, E.: Assassin’s Creed Valhalla (2020) 10. Mangiron, C.: Reception of game subtitles: An empirical study. The Translator 22(1), 72–93 (2016), https://doi.org/10.1080/13556509.2015.1110000 11. Mangiron, C., Orero, P., O’Hagan, M.: Fun for All: Translation and Accessibility Practices in Video Games. Bern: Peter Lang (2014) 12. McGregor, C.: Hyperdot (2019) 13. Netflix: Timed Text Style Guide: General Requirements (2021), https://partnerhelp.netflixstudios.com/hc/en-us/articles/ 215758617-Timed-Text-Style-Guide-General-Requirements 14. PopCap: PopCap Games Mobile Gaming Research (2012), https://www.gamesindustry.biz/articles/ popcap-games-research-publisher-s-latest-survey-says-that-casual_ games-are-big-with-disabled-people 15. Reverter Oliver, B., Martı́nez-Sierra, J.J., González Pastor, D., Carrero Martı́n, J.F. (eds.): Panorama Actual de la Accesibilidad a los Videojuegos, pp. 101–110. Granada: Comares, Colección Interlingua (2021) 16. Sugg, Darren: Fortnite (2017) 1 Accessible Audio Toolkit, https://assetstore.unity.com/packages/add-ons/accesible- audio-toolkit-194392