Deaf and hard-of-hearing individuals face challenges in identifying the direction and type of sound sources. Enabling sound localization enhances safety, and understanding the type of sound sources improves awareness of the surrounding environment. However, few studies have explored methods to present both sound direction and type information to hearingimpaired individuals. In this study, we designed a sound localization aid device using vibrators suitable for daily use and investigated tactile perception through changes in vibration frequency to represent diferent sound types. We prototyped two sound localization aid devices: a headphone type and a hat type. For the presentation of sound types, we examined a method of converting sound frequencies to vibrations. We integrated these two systems into a new aid device and evaluated its efectiveness through experiments. The results showed no significant diference in sound localization, but a trend toward significance was observed in sound type identification. Diferent sound types could be perceived through variations in vibration.
80 phon 60 phon 40 phon 20 phon threshold
Frequency[Hz] with both sound localization and sound source identification using vibration stimuli, designed to accommodate various types of hearing impairments.
In the case of hearing impairment in both ears, the degree of physical disability is classified into Levels 2, 3, 4, and 6 according to the “Degree of Physical Disability Classification Table” in the Enforcement Regulations of the Act on Welfare of Physically Disabled Persons [9]. At Level 2, both ears have a hearing level of 100 decibels or more (complete deafness in both ears). At Levels 3, 4, and 6, the hearing level decreases by 10 dB for each grade. 2.2. A wearable device that transmits sound vibrations to the body ] L P S B d l[ e v e L e r u s rs e P d n u o S ] B d [ d l o h ttrs e h n e m e c a l p s i d e d u itl p m A
A device developed for hearing-impaired individuals is Fujitsu’s Ontenna (in Japanese) [7]. Ontenna converts sounds from 60 to 90 dB into 256 vibration and light intensity levels, allowing users to perceive sound characteristics such as rhythm, pattern, and volume. However, since it consists of only one device, it is unsuitable for FAcⅡor︓puPsaccleini sound localization. Additionally, it is not ideal for sound source identification because it only presents vibration intensity.
The support device proposed by Yumiba et al., which focuses on the sound localization issues of individuals Frequency[Hz] with unilateral hearing loss (in Japanese) [8], provides two distinct vibrations for dangerous sounds and com- Figure 2: Vibration detection threshold curve (taken from munication sounds. The device has four vibrators placed [11]) on the left and right, front and back of the shoulders, allowing for a five-directional sound presentation when the rear vibrators on both sides activate simultaneously. the higher frequency range, tends to decline with age. While this study demonstrated usefulness in the sound Figure 1 shows the equal-loudness contour [10]. The localization task, the device is designed for unilateral equal-loudness contour represents the sound pressure hearing loss. Therefore, it does not provide a system for levels at various frequencies that are perceived to have presenting sound characteristics like volume or rhythm, the same loudness. This is regarded as one of the most which is a limitation. fundamental characteristics of human hearing. According to Figure 1, human hearing is most sensitive to frequencies between approximately 2,000 Hz and 4,000 Hz.
The range of frequencies that can be perceived as sound is called the audible frequency range. Humans perceive variations in air pressure as sound through the sensory organ known as the ear. Even with the same energy fluctuations in pressure, the perceived loudness of sound is not consistent across diferent frequencies. Generally, for healthy young people, the audible frequency range is from 20 Hz to 20,000 Hz, and hearing, particularly in
Skin sensory receptors are classified into mechanoreceptors, thermoreceptors, and nociceptors. Mechanoreceptors are responsible for sensing touch and pressure and include Merkel cells, Pacinian corpuscles, and Meissner’s corpuscles [12]. Figure 2 shows the vibration detection threshold curve, illustrating the sensitivity of diferent skin receptors to various frequencies [11]. In this study, we focus on the curve related to Pacinian corpuscles, the receptors that detect vibrations in human skin. According to Figure 2, human skin can perceive vibrations in the range of approximately 5 Hz to 500 Hz, with the highest sensitivity between 200 Hz and 400 Hz.
We will present not only the direction of the sound source In this section, we define the fundamental requirements but also "what kind of sound is being made" to convey that the proposed system must meet. The key require- various information contained in sound. Vibration stimments include sound localization, sound source identifi- uli will convey sound characteristics such as pitch and cation, and form factor. We explain below how achieving loudness, and we will explore methods to convert sound these requirements contributes to realizing the system’s information into vibration stimuli. goals. In summary, for the three main elements of sound: loudness will be represented by the strength of the vibra3.1.1. Requirements for sound localization tion, pitch by the speed of the vibration, and timbre by a combination of frequencies.
Deaf and hard-of-hearing individuals may have dificulty or be unable to perceive the direction of sound sources.
This can lead to dangerous situations, especially if they cannot notice sounds coming from outside their field of vision. Therefore, at a minimum, it is necessary to be able to identify the sound source positions from the left and right rear. Additionally, being able to identify sound sources from the left and right front would further enhance safety.
For deaf and hard-of-hearing individuals, it is extremely dificult to distinguish between diferent types of sound sources, such as human voices, warning sounds (e.g., sirens), and general environmental sounds. For example, they may not be able to recognize the diference in urgency between an ambulance and a police car siren or notice a car horn from behind. Being able to perceive all sounds as they are, similar to people with normal hearing, would likely improve safety and enhance the quality of daily life.
The appearance, size, and shape of the system must match the environment in which it is used, its social acceptability, and the needs of the user. Considering long-term use in daily life, the device should be designed to be lightweight, easy to wear, and not cause embarrassment. As possible designs that meet these criteria, we considered headphone-type and Beret-type devices. To achieve sound localization, a small microphone array, a microcontroller, and small vibrators are used. The sound source direction is communicated by activating the vibrator corresponding to the detected sound source direction from the microphone array.
As shown in Section 2.3, the frequency characteristics of human perception difer significantly between auditory perception and tactile perception (skin sensation). Therefore, it is necessary to convert frequency ranges that are easily heard by the human ear into frequency ranges that are more easily perceived as vibrations on the skin. When examining the equal-loudness curve from 0 Hz to 4000 Hz and the Pacinian corpuscle curve from 0 Hz to 500 Hz, we found that the curves exhibit a very similar trend. Therefore, for simplicity, this study adopts a linear conversion. The equation is shown below: max − min = max − min Where: : is the vibration frequency, : is the sound frequency, max: is the maximum value of the vibration frequency range, min: is the minimum value of the vibration frequency range, max: is the maximum value of the sound frequency range, and min: is the minimum value of the sound frequency range. In this study, max is set to 500 Hz, maxto 4000 Hz, and both minand minare set to 0 Hz. For example, when the sound frequency is 1000 Hz, the vibration frequency is converted to 125 Hz.
v振ib動ra⼦tor Figure 4 shows the appearance of the prototype berettype device. This device weighs 353 grams in total, with Figure 4: Beret-type device (left), appearance (top right), the built-in battery alone weighing 130 grams. The com- microcontroller (bottom right), and microphone array ponents used include a beret, six disk-shaped vibration motors (DVM150), an Arduino UNO, and a sound direction sensor board (CRESCENT-031). The control program for the device was also developed using Arduino IDE. 3. Convert the frequency using the conversion func
The beret-type device can convey the direction of tion sound sources in six directions. The vibrators are ar- 4. Apply Inverse Fourier Transform (IFFT) to the conranged, when viewed from above, at 0 degrees (forehead), verted data 60 degrees (right front), 120 degrees (right rear), 180 de- 5. Output the transformed data to the vibrator grees (back), 240 degrees (left rear), and 300 degrees (left This method is similar to that used in real-time voice front). Depending on the detected sound source direction, changers. the corresponding vibrator is activated. For example, if In these steps, we prepared audio data in advance and the sound source is detected from the front at an angle confirmed the operation of the Haptuator by converting of 0 ≤ < 30 or 330 ≤ < 360, the forehead vibra- this audio source. For the implementation, we used a tor is activated. Similarly, for angles 30 ≤ < 90, the Haptuator (TactileLabs / TL002-14R), a computer (Macright-front vibrator operates; for 90 ≤ < 150, the Book), an audio amplifier, and a stabilized DC power right-rear vibrator; for 150 ≤ < 210, the back vibra- supply. tor; for 210 ≤ < 270, the left-rear vibrator; and for First, we downloaded "Ambulance Siren 1" from Sound 270 ≤ < 330, the left-front vibrator. Efect Lab [ 13]. The audio file was divided into 0.1-second segments, and Fourier Transform (FFT) was applied to each segment to analyze the frequency spectrum. The 4.2. Sound source identification system frequency data for each segment was then mapped. Next, The frequency conversion from sound to vibration is we performed Inverse Fourier Transform (IFFT) on the achieved through the following steps: mapped frequency data to reconstruct the signal in the 1. Detect sound using a microphone time domain. Finally, the segments were reassembled to 2. Perform Fourier Transform (FFT) on the detected generate the continuous audio signal after processing. data Figure 5 shows the frequency conversion diagram of the ambulance siren.
We also applied the conversion to human speech (female saying "Konnichiwa")[14]. For human speech, the optimal range for formant analysis is said to be 0.02 to 0.04 seconds. Therefore, for human speech, we performed Fourier Transform at 0.02-second intervals. Figure 6 shows the frequency conversion diagram of the human (female) voice.
In this section, we describe a system that enables both sound localization and sound source identification by building upon the beret-type device for sound localization introduced in Section 4.1 . The system replaces the disk-shaped vibration motors used for sound source identification, discussed in Section 4.2, with vibrators called Haptuators, allowing for simultaneous sound localization and sound source identification.
The components used include two Arduino UNO boards, a sound direction sensor board, six Haptuators, two 8-channel relay modules, a PC (MacBook), an audio amplifier, six speakers, and three AUX cables. An experimental implementation was conducted using 15 types of sound source data.
As the rfist step, we prepared 15 diferent sound source data. These sounds were collected by category from The audio data output from the R (right) side of the amthose we commonly hear in daily life [15]. The categories plifier was directed to any selected speaker using a relay include siren, car engine sound, horn, bicycle, phone module and an Arduino UNO. To enable output to a speringtone, and human voice. These categories are listed cific speaker, a push-button switch was incorporated for in Table 1. switching between speakers. A tactile switch was used
These sounds underwent frequency conversion follow- as the push-button, and it was controlled by the Arduino ing the method described in Section 4.2. UNO.
4.3.3. Implementation of Sound Localization The audio data output from the L (left) side of the amplifier was directed to the Haptuator corresponding to the detected sound direction using a relay module, an Arduino UNO, and a sound direction sensor board.
The mechanism for outputting to the Haptuator follows the same design as the Beret-type device described in Section 4.1.2. The implementation, which combines the methods discussed in Sections 4.3.2 and 4.3.3, is shown in Figure 7.
5.1. Interviews and Surveys at Schools for
the Deaf With the cooperation of the Nara Prefectural School for the Deaf, we conducted a trial session for the prototype device. We received the following feedback:
Regarding the Headphone-Type Device (Figure 3) • “The ear pads are soft and comfortable to wear.” • “Vibrations may become stressful if multiple peo
ple speak at once.” • “Prolonged use could cause discomfort or pain.” • “Walking while wearing headphones may raise
concerns about etiquette.” Regarding the Beret-Type Device (Figure 4) • “It is excellent that sounds from behind can be
detected.” • “The weight of the battery is noticeable.” • “The device might fall of if the head is moved
quickly.” • “It is unclear if the proximity of vibrators allows
for detailed perception.” • “It is easier to use than the headphone-type de
vice.”
Additionally, participants shared a general observation about both devices: apart from the Ontenna, they had not encountered any other devices that allow users to perceive sounds through vibrations. Many expressed interest in using such a device if it could help diferentiate between various sounds and types. 5.2. Exhibits at Visitor Experience-Type
Events Based on feedback from the School for the Deaf, we decided to adopt the beret-type device. We then presented this device at the Innovation Stream KANSAI 6.0, held on February 21–22, 2023. Below are some of the main comments and questions received during the exhibition:
The feedback from this academic exhibition reafirmed the high potential value of sound localization for various applications.
• “As a driver, it is often dificult to interact safely with individuals with hearing impairments, leading to potentially dangerous situations. This de- to participate.
vice could enhance safety for both parties.” The participants included one person with Level 3 • “I have seen visually impaired individuals appear hearing impairment and two with Level 2, totaling three anxious when crossing intersections with sound participants (two males and one female). cues, as they cannot always determine the direction. This device might also be useful in such 6.2.2. Experimental System situations.”
The experiment used the Beret-type device developed in Section 4.3, which can simultaneously perform sound localization and sound source identification. The device
The purpose of this experiment is to verify whether the device improves the ability to localize sound sources and identify sound types.
Hypothesis 1 Sound Localization Accuracy via Vibration: Using this device will allow users to identify the direction of sound sources more accurately compared to without assistance.
Hypothesis 2 Sound Source Identification Accuracy via Vibration: Providing frequency-converted vibrations will enable users to identify sound types more efectively compared to without assistance.
This experiment targeted hearing-impaired individuals aged 20 to 40. The scope of hearing impairment was defined as Levels 2, 3, and 4 under the Act on Welfare of Physically Disabled Persons. Participants were recruited through the Nara Prefecture Hearing Impairment Support Center, with approximately 300 individuals invited presents diferent vibration patterns on the head, depending on the direction and type of the sound source. Sound source directions were output from six directions: 0 degrees (forehead), 60 degrees (right front), 120 degrees (right rear), 180 degrees (back of the head), 240 degrees (left rear), and 300 degrees (left front). Six speakers were used as sound sources, with only one speaker playing sound at a time. The number of vibrators matched the number of speakers.
Fifteen types of sound sources were prepared, as shown in Table 1. These sounds were selected from those commonly encountered in daily life, where sound localization is required, referring to "Nijiiro Hearing Aids – Types of Everyday Sounds" [15].
The evaluation metrics included the accuracy rate of sound localization and the accuracy rate of sound source identification. In addition, a self-report questionnaire (using a 7-point Likert scale) was used to evaluate the user experience of the device.
The experiment was conducted in the 3rd meeting room on the 6th floor of the Nara Prefecture Comprehensive Social Welfare Center. The experimental environment was an indoor space where external sounds were dificult to hear and unlikely to escape. A sign language interpreter was also present to facilitate communication with the participants. The experiment setup is shown in Figure 8.
The sound sources used were the 15 types listed in Table 1, which were played randomly in both directions and types. Using the device developed in Section 4.3, the frequency-converted sound sources were output to the vibrators, while the original sound sources were played through the speakers.
participants could feel the vibration pattern of the siren, and the interpreter conveyed that it was an ambulance siren. In this training, participants experienced all types of sound sources prepared for the experiment. The test consisted of 12 questions per set, and the device was worn or removed between each set. A total of four sets were conducted.
The scores for the number of correct answers for sound source direction and type, with and without the device, are shown in Figures 9 and 10. The number of questions was 24 for both conditions (with and without the device). Assuming a normal distribution of the scores out of 24 points, a t-test was conducted.
The results showed no significant diference in the sound localization task between the two conditions ((2) = − 0.277, = 0.808, = − 0.204).
For the sound source identification task, while a clear significant diference was not observed, a trend toward significance was found ( (2) = 3.90, = 0.0599, = 3.00). However, given the relatively small p-value and the large efect size, it is suggested that increasing the number of participants may yield a significant diference.
We conducted a survey using seven questions (Table 2) on a 7-point Likert scale. The results for Q1 and Q4 were relatively positive, but one participant gave low ratings for Q2 and Q3. This participant also had fewer correct answers for sound source identification compared to the other participants. 7.3. Summary of Experimental Results This study examined whether the device we developed can facilitate sound localization and sound source identification for hearing-impaired individuals in daily life.
The following key points were identified from the results: 8. Conclusion
Regarding the accuracy of sound localization and sound source identification, no significant diference was We proposed a device capable of simultaneously performfound for sound localization between the conditions with ing sound localization and sound source identification, and without the device. However, a trend toward sig- addressing the challenges that hearing-impaired individnificance was observed for sound source identification. uals face in these areas. The results showed no significant Due to the small sample size of three participants, further diference in the sound localization task. However, in the experiments with a larger participant pool are necessary. sound source identification task, a large efect size was Additionally, the results for the sound localization task observed, indicating a trend toward significance even may have been influenced by malfunctions of the sound with a small number of participants. direction sensor and the room’s echo. Future challenges include conducting experiments
Regarding user experience with the device, partici- with a larger participant pool and further miniaturizing pants reported little fatigue during use. However, opin- the device. ions varied concerning the sensation of vibration, confidence in the sound-to-vibration conversion, and willingness to use the device in daily life. Acknowledgments We would like to express our sincere gratitude to the Nara Prefecture Hearing Impairment Support Center, the Nara Prefecture Comprehensive Social Welfare Center, and the sign language interpreters for their invaluable support. Their contributions include providing essential data and information related to this study, assisting with the recruitment of participants, and supporting the experiment. [8] Y. Hiromu, W. Shinji, A design proposal of support product for single sided deafness, focused on dificulties of sound localization, bulletin of Japanese Society for the Science of Design 63 (2016) 21. doi:10.11247/jssd.63.0_21. [9] Ministry of Health, Labour and Welfare, Physical disability certification criteria (in japanese), https: //x.gd/qMcil, 2014. [online: access 2024/10/17]. [10] National Institute of Advanced Industrial [1] Widex, , Widex, 2024. URL: https://www.widex. Science and Technology, Iso 226, the internacom/ja-jp/local/ja-jp/deafness/causes_symptoms/ tional standard for auditory isometric curves, mechanism_type/, [online; access 2024/01/05]. has been completely revised. (in japanese), [2] F. Zamiri Abdollahi, M. Joulaie, A. Darouie, T. Ah- https://www.aist.go.jp/aist_j/press_release/ madi, Consequences of unilateral sensory neu- pr2003/pr20031022/pr20031022.html, 2003. [online: ral hearing loss, Global Journal of Otolaryn- access 2024/01/20]. gology 12 (2017) 555834. URL: https://doi.org/ [11] S. Kazuyoshi, S. Yutaka, M. Kazuyuki, T. Masato, 10.19080/GJO.2017.12.555834. doi:10.19080/GJO. Tremor and vibratory perception in a living body 2017.12.555834. : functional evaluation of mechanical vibration, [3] K. Takaki, E. Nozaki, T. Kanai, A. Hautasaari, Tokyo Denki University Press, 2009. URL: https: A. Kashio, D. Sato, T. Kamogashira, T. Uranaka, //ci.nii.ac.jp/ncid/BA90255727.
S. Urata, H. Koyama, T. Yamasoba, Y. Kawahara, [12] S. J. Lederman, R. L. Klatzky, Haptic perception: A Asears: Designing and evaluating the user experi- tutorial, Attention, Perception, & Psychophysics ence of wearable assistive devices for single-sided 71 (2009) 1439–1459. URL: https://doi.org/10.3758/ deafness, in: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, [13] ASoPuPn.7d1.7E. 1fe4c3ts9. dLoaib:10(.in37J5a8p/aAnPePse.)7,1.a7m.b1u4la3n9c.e CHI ’23, Association for Computing Machinery, siren, https://soundefect-lab.info/sound/machine/ New York, NY, USA, 2023. URL: https://doi.org/ machine2.html, 2024. [online: access 2024/01/20]. 10.1145/3544548.3580840. doi:10.1145/3544548. [14] Sound Efects Lab (in Japanese), Girl’s voice 3580840. line sound efects, https://soundefect-lab.info/ [4] K. Yuko, H. Tsuneo, K. Akiyoshi, <brief notes>the sound/voice/line-girl1.html, 2024. [online: access literature survey of studies on sound localization 2024/01/20]. of persons with hearing impairment, Bulletin of [15] Nijiiro hearing aid (in Japanese), Types of sounds defectology 28 (2004) 123–132. URL: https://cir.nii. of daily life, https://nijiho.com/column/lifesound/, ac.jp/crid/1050282677528928256. 2024. [online: access 2024/10/17]. [5] S. Levänen, D. Hamdorf, Feeling vibrations: enhanced tactile sensitivity in congenitally deaf humans, Neuroscience Letters 301 (2001) 75– 77. URL: https://www.sciencedirect.com/science/ article/pii/S030439400101597X. doi:https://doi.
org/10.1016/S0304-3940(01)01597-X. [6] D. Jain, L. Findlater, J. Gilkeson, B. Holland, R. Duraiswami, D. Zotkin, C. Vogler, J. E. Froehlich, Head-mounted display visualizations to support sound awareness for the deaf and hard of hearing, in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 241–250. URL: https: //doi.org/10.1145/2702123.2702393. doi:10.1145/ 2702123.2702393. [7] Fujitsu, Fujitsu digital transformation news (in japanese)20220/07/19 update, Fujitsu, 2022.
URL: https://www.fujitsu.com/jp/microsite/ fujitsutransformationnews/2022-07-19/01/, [online; access 2024/01/05].