Despite the technological advancement of modern hearing aids (HA), many users abandon their devices due to lack of personalization. This is caused by the limited hearing health care resources resulting in users getting only a default 'one size fits all' setting. However, the emergence of smartphoneconnected HA enables the devices to learn behavioral patterns inferred from user interactions and corresponding soundscape. Such data could enable adaptation of settings to individual user needs dependent on the acoustic environments. In our pilot study, we look into how two test subjects adjust their HA settings, and identify main behavioral patterns that help to explain their needs and preferences in different auditory conditions. Subsequently, we sketch out possibilities and challenges of learning contextual preferences of HA users. Finally, we consider how to encompass these aspects in the design of intelligent interfaces enabling smartphone-connected HA to continuously adapt their settings to context-dependent user needs.
personalization; augmented hearing; intelligent interfaces
Even though hearing loss is one of the leading lifestyle causes
of dementia [
HUMANIZE ’18, March 11, 2018, Tokyo, Japan
the reasons behind the prevalence of non-use of fitted HA is
identified by McCormack et al. [
Earlier research by Dillon et al. [
When interpreting user preferences, one needs to consider how
the brain interprets speech. Auditory streams are bottom-up
processes fused into auditory objects, based on spatial cues
related to binaural intensity and time difference [
Two male participants (from a screened population provided
by Eriksholm Research Centre) volunteered for the study
(Table 1). The participants suffer from a symmetrical hearing
loss, ranging from moderate to moderate-severe as described
by the WHO[
Subject Age group 1 65 2 76
The subjects were fitted with a pair of research prototype HA EVOTION extending Oticon Opn. The subjects used Android 6.0 or iOS 10, connected via Bluetooth. Data was logged using the nRF connect app and shared via Google Drive. 1 2
P1 P2 P3 P4 P1 P2 P3 P4 omnidirectional omnidirectional low noise reduction high noise reduction omnidirectional low noise reduction medium noise reduction high noise reduction
Based on the individual hearing loss, the subjects were fitted with 4 programs as shown in Table 2. For all programs, HA volume could be adjusted to one of the levels from 8 + 4, where 0 is the default volume. The subjects were instructed to explore different settings using HA buttons over a period of 6-7 weeks. In the experimental setup, the HA always start up in the default program and volume. The default program for subject 1 was P2 in the first five weeks which was then switched to P1 for the last two weeks at the subject’s request. Subject 2 used P2 as the default program.
To create an interpretable representation of the auditory features defining the context, we applied k-means clustering to the acoustic context data collected from HA. The values comprise auditory features defining how the HA perceive the acoustic environment: sound pressure level measure of estimated loudness, noise floor tracking the lower bound of the signal, modulation envelope tracking the peaks in the signal, modulation index estimated as difference between modulation envelope and noise floor, signal to noise ratio estimated as difference between sound pressure level and noise floor.
The above parameters are captured as a snapshot across multiple frequency bands once per minute. Additionally, the HA perform a rough classification of the auditory environment and represent it as a categorical variable with one of the following values: ’quiet’, ’noise’, ’speech in quiet’, and ’speech in noise’. These labels are used as ground truth for evaluating the performance of the clustering by means of normalized mutual information (NMI) score. The optimal number of clusters K was estimated to be 4 with NMI = 0:35. Tue 00:00
Wed 00:00
Thu 00:00
Fri 00:00
Sat 00:00
Sun 00:00
P1 P2 P3 P4 OFF The resulting four soundscape clusters were labeled according to the proportion of samples with different ground-truth labels within each cluster ( Figure 1) while ambiguities were solved by examination of the cluster centroids. The first cluster mainly captured the ’quiet’ class which is also validated by the cluster centroid having very low values of sound pressure level and noise floor. Thus, the environments assigned to this cluster will be represented as ’quiet’. The second cluster captured both ’speech in noise’ and ’noise’ classes which suggests that the numerical representations of these environments are similar. For simplicity, we label them as ’speech in noise’. The third and fourth cluster both captured mainly ’speech in quiet’ with a small addition of other classes. As the third cluster captured samples with much higher sound pressure level and signal to noise ratio, it will be labeled as ’clear speech’, while the fourth cluster with attributes of the samples closer to mean will be represented as ’normal speech’.
We refer to the user’s selected volume and program choice as user preferences, and to the corresponding auditory environment as the context. Juxtaposing user preferences and the context allows us to learn which HA settings are selected in specific listening scenarios. To facilitate interpretation we assign each cluster a color from white to green gradient, in which increasing darkness correspond to increased noise in the context (quiet ! clean speech ! normal speech ! speech in noise). Likewise, we assign each program a color from yellow to red gradient. Lighter colors define programs with an omnidirectional focus and added brightness. Darker colors indicate increasing attenuation of noise. This coloring scheme will apply throughout the paper.
Figure 2 shows the user preference and context changes for both subjects, plotted across the hours of the day over the weeks constituting the full experimental period. Subject 1 most frequently selects programs which provide an omnidi0.8 0.6 rectional focus with added brightness (the default program was changed from P2 to P1 after week 43). However, the default program is occasionally complemented with programs suppressing noise. This suggests that the user benefits from changing programs dependent on the context.
Subject 2 mainly selects two programs; P1 offering an omnidirectional focus with added soft gain and brightness, and P2 (default) providing slight attenuation of ambient sounds. Compared to subject 1, this user spends more time in ’quiet’ context. Comparing weekdays to weekends, the latter seem to contain a larger contribution of ’normal speech’ and ’speech in noise’ auditory environments.
Figure 3, illustrates subjects’ average usage of their HA and which programs are used most throughout the day. Days without any HA usage are excluded from the average. The HA usage for subject 1 steadily increases in the morning and early afternoon and peaks at around 4pm. P1 and P2 are the most used programs throughout the day. Interestingly, in the evening, P3 is used more frequently reaching similar usage level as P1 and P2 between 11pm and midnight. P4 is used very rarely yet consistently throughout the day. The HA usage of test subject 2 is shifted towards the morning with peak activity around 2pm. The default P2 is the most commonly used program throughout the whole day. However, during afternoon, P1 seems to be chosen more often.
Figure 4 shows which contexts the subjects use their HA at different times of the day. The HA usage for subject 1 is dominated by speech-related contexts most of the day. Only after 5pm, the context has more ’quiet’ and ’clear speech’ and less ’speech in noise’ contribution. From 9pm, the ’quiet’ context rapidly overtakes context containing speech. Subject 2 appears to be exposed to different contextual patterns. Normal and noisy speech contexts seem to be dominated by ’quiet’ soundscapes in the morning. Subsequently, their contributions increase and peak around 7pm. Afterwards, the ’quiet’ 40 context gradually increases. Both subjects seem exposed to more ’speech in noise’ around midday which is likely due to lunchtime activities.
We quantify the relationship between program/volume interaction and context by assuming that the settings are preferred in the corresponding context only at the time when they are being selected. Under this assumption, we count how often programs are selected in different contexts. Table 3 shows the counts of program changes for both subjects. The total number of changes was 52 and 46 for subject 1 and 2 respectively. Considering the small number of changes, we outline only the most apparent behavioral patterns.
Subject 1 switches to P4 mainly in ’speech in noise’ context (twice as often as in ’normal speech’). The fact that ’speech in noise’ is a less common environment than ’normal speech’ strengthens this behavioral pattern. This suggests that subject 1 seems to cope by suppressing noise in challenging listening scenarios. Examples of this behavioral pattern are illustrated in Figure 5. Likewise, a clear behavioral pattern can be seen for subject 2. P1 is the preferred program in ’speech in noise’ environments. Considering that P1 offers maximum brightness and omnidirectionality with reduced attenuation and noise reduction, this behavioral pattern suggests the user compensates by enhancing high frequency gain as a coping strategy in complex auditory environments (examples in Figure 6). Table 4 shows the number of volume changes for subject 2 (subject 1 rarely changes volume). All increases beyond 21 Oct 16 Nov 22 Nov 12:00 15:00 18:00
05 Nov 12:00 15:00 18:00 21:00 the default volume level (0) were made in ’speech in noise’ context. On the other hand, changes to the default volume were evenly distributed across all contexts. This suggests that increasing the volume is another coping strategy for subject 2 in more challenging listening scenarios.
t QUIET tex CLEAN SPEECH on NORMAL SPEECH C SPEECH IN NOISE
Subject 2 0 +1 2 0 2 0 2 0 2 12 +2 0 0 0 1
Figure 7 shows a behavioral pattern that might be more difficult to interpret based on the auditory context alone. Occasionally, subject 1 selects P3 in a ’quiet’ environment late in the evening. The test subject subsequently reported that these situations occur when going out for a walk and wanting to be immersed in subtle sounds such as rustling leaves or the surf of the ocean. The preference for P3 thus implies both increasing the intensity of soft sounds as well as the perceived brightness.
Empowering users to switch between alternative settings on internet connected HA’s, while simultaneously capturing their auditory context allows us to infer how users cope in real life listening scenarios. To the best of our knowledge, this has not been reported before. 15:00 18:00 21:00 00:00 15:00 18:00 21:00 00:00
Learning the mapping between preferences and context is a non-trivial task, as the chosen settings might not be the optimal ones in the context they appear in. For example, looking into the soundscape data, it is clear that the environment soundscape frequently changes without the user responding with an adjustment of the settings. Conversely, the auditory environment may remain stable whereas the user changes settings. We need to take into consideration not only the auditory environment but also the user’s cognitive state due to fatigue or intents related to a specific task. Essentially, the user cannot be expected to exhibit clear preferences or consistent coping strategies at all times. We hypothesize that many reasons could explain why the user does not select an alternative program although the context changes: being too busy to search for the optimal settings, too high effort is required to change programs manually, accepting the current program as sufficient for the task at hand, cognitive fatigue caused by constantly adapting to different programs.
Similarly, we observe situations in which user changes settings even though the auditory environment remain stable, which could be caused by: the user trying out the benefits of different settings, cognitive fatigue due to prolonged exposure to challenging soundscapes the auditory environment not being classified correctly In our pilot study, the context classification was limited to the auditory features which are used for HA signal processing. However, smartphone connectivity offers almost unlimited possibilities of acquisition of contextual data. Applying machine learning methods such as deep learning might facilitate higher level classification of auditory environments. Different types of listening scenarios might be classified as ’speech in noise’ when limited to parameters such as signal to noise ratio or modulation index. In fact, these could encompass very different listening scenarios such as an office or a party where the user’s intents would presumably not be the same. Here the acoustic scene classification could be supported by motion data, geotagging or activities inferred from the user’s calendar to provide a more accurate understanding of needs and intents. Nevertheless, in some situations as illustrated in Figure 6, the behavioral patterns seem very consistent; the user preferences appear to change simultaneously with the context, remain unchanged as long as the context remains stable, and change back when the context changes again. Identifying such behaviors could allow to reliably detect user preferences with limited amount of user interaction data. Furthermore, time as a parameter also highlights patterns as illustrated in Figure 6 related to activities around lunch time, or late in the evening ( Figure 7), as well as the contrasting behavior in weekends versus specific weekdays.
Even though our study was limited to only two users, we identified evident differences in the HA usage patterns. Subject 1 tends to use the HA mostly in environments involving speech, whereas subject 2 spends substantial amount of time in quiet non-speech environments. This might translate into different expectations among HA users. Furthermore, our analysis suggests that users apply unique coping strategies in different listening scenarios, particularly for complex ’speech in noise’ environments. Subject 1 relies on suppression of background noise to increase the signal to noise ratio in challenging scenarios. Subject 2 responds to speech in noise in a completely different way - he chooses maximum omnidirectionality with added brightness and increased volume to enhance spatial cues to separate sounds. These preferences are not limited to challenging environments but extends to the ambience and overall quality of sound, as subject 1 reported that he enhances brightness and amplification of quiet sounds to feel immersed in the subtle sounds of nature. We find this of particular importance as it indicates that users expect their HA not only to improve speech intelligibility, but in a broader sense to provide aspects of augmented hearing which might even go beyond what is experienced by normal hearing people.
We propose that learning and addressing user needs could be conceptualized as an adaptive augmented hearing interface that incorporates a simplified model reflecting the bottom-up and top-down processes in the auditory system. We believe that such an intelligent auditory interface should: continuously learn and adapt to user preferences, relieve users of manually adjusting the settings by taking over control whenever possible, recommend coping strategies inferred from the preferences of other users, actively assist users in finding the optimal settings based on crowdsourced data, engage the user to be an active part in their hearing care. Such an interface would infer top-down preferences based on the bottom-up defined context and continuously adapt the HA settings accordingly. This would offer immense value to users by providing the optimal settings at the right time, dependent on the dynamically changing context. However, the system should not be limited to passively inferring intents, but rather incorporate a feedback loop providing user input. We see a tremendous potential in conversational audio interfaces as HAs resemble miniature wearable smartspeakers which would allow the user to directly interact with the device, e.g. by means of a chatbot or voice AI. First of all, such an interface might resolve ambiguities in order to interpret behavioral patterns. In a situation when user manually changes the settings in a way that is not recognized by the learned model, the system could ask for a reason in order to update its beliefs. Ideally, questions would be formulated in a way allowing the system to directly learn and update the underlying parameters. This could be accomplished by validating specific hypotheses that refer to the momentary context as well as the characteristics captured in the HA user model, incorporating needs, behavior and intents; e.g.’Did you choose this program because the environment got noisy / you are tired / you are in a train? Secondly, a voice interface could recommend new settings based on collaborative filtering methods. Users typically stick to their preferences and may be reluctant to explore available alternatives although they might provide additional value. Similarly, in the case of HA users, preferred settings might not necessarily be the optimal ones. Applying clustering analysis based on behavioral patterns, we could encourage users to explore the available settings space by proposing preferences inferred on the basis of ’users like me, in soundscapes like this’. For instance, the inteface could say: ’Many users which share your preferences seem to benefit from these settings in a similar context - would you like to try them out?’ This would encourage users to continuously exploit the potential of their HA to the fullest. Additionally, behavioral patterns shared among users, related to demographics (e.g. age, gender) and audiology (e.g. audiogram) data, could alleviate the cold start problem in this recommender system, thus enabling personalisation to kick in earlier even when little or even no HA usage data is available.
Lastly, users should be able to communicate their intents, as the preferences inferred by the system might differ from the actual ones. In such scenarios, users could express their intents along certain rules easily interpreted by the system (e.g. ’I need more brightness.’) or indicate the problem in the given situation (e.g. ’The wind noise bothers me.’). Naturally, translating the user’s descriptive feedback into new settings is more challenging, but could potentially offer huge value by relieving users of the need to understand how multiple underlying audiological parameters influence the perceived outcome.
Combining learned preferences and soundscapes into intelligent augmented hearing interfaces would be a radical paradigm shift in hearing health care. Instead of a single default setting, users may navigate a multidimensional continuum of settings. The system could be optimized in real-time by combining learned preferences with crowdsourced behavioral patterns. With growing numbers of people suffering from hearing loss we need to make users an active part of hearing health care. Conversational augmented hearing interfaces may not only provide a scalable sustainable solution but also actively engage users and thereby improve their quality of life.
This work is supported by the Technical University of Denmark and the Oticon Foundation. Oticon EVOTION HAs are partly funded by European Union’s Horizon 2020 research and innovation programme under Grant Agreement 727521 EVOTION. We would like to thank Eriksholm Research Centre and Oticon A/S for providing hardware, access to test subjects, clinical approval and clinical resources.