In the light of the characteristics of health behavior recommendations, we discuss implications of recommendation uncertainty from a practical and ethical point of view. Considering problem complexity and data structure as well as security and autonomy aspects, we demonstrate the importance of user empowerment, fostering reflection of the recommendations and of integrating the user into the loop. In a preliminary empirical analysis, we show that presenting uncertainty to the user might help the user to reflect the recommendation and integrate him into the loop. Moreover, it might increase trust, perceived transparency, system responsibility, and overall user satisfaction.
• Human-centered computing → User interface design; User studies; Empirical studies in visualization; uncertainty, recommender systems, health, behavior change, reflection
Recommender Systems are very popular in various areas such as e-commerce or entertainment as they can support users in choice processes. Recommender systems are also an emerging topic in the field of health-related systems. However, due to noise in the underlying data, contextual factors that the system does not know, sparse data, and algorithmic errors, a recommendation will always contain uncertainty to a certain extend. We use uncertainty as an umbrella term for error on the one hand and confidence on the other as they are two sides of the same coin.
Even if one might argue, that uncertainty might be negligible for shopping, movie, or music recommendations, it has a more serious impact for health-related recommender systems. In the first International Workshop on Health Recommender Systems, August 2017, Como, Italy © 2017 Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. part of this paper, we will discuss practical and ethical implications of uncertainty in health-related recommender systems. We argue that it is essential that such systems present the uncertainty of the recommendation and its impact in the choice situation. We further present an empirical analysis based on data derived from two studies concerned with recommendations of physical activity goals (one with presentation of uncertainty, the other without). In this analysis, we investigate whether trust, perceived transparency of the system, perceived responsibility of the system, reflection of the recommended goal, and overall user satisfaction are influenced by presenting (vs. not presenting) recommendation uncertainty. 2
While early work in the field of recommender systems focused on
improving algorithm accuracy, in recent years also the impact of
transparency and user integration in the recommendation process
has gained increasing attention. Indeed, it has been shown that
people wish for a more active role in the recommendation process [
Recommendations for health-related behavior, such as recommending an activity goal level, have specific characteristics which lead to specific demands of recommender systems in this domain. They are discussed in the following. 3.1
Health-related recommender systems often deal with complex problems and many influencing factors, which are often very unstable. For instance, recommendations are influenced by the users’ current and prospective health state, current and prospective barriers and other context factors influencing the health-related behavior, and by motivational aspects. Even if trying to consider most of these aspects, some of them will not be measurable and/or predictable. To handle this problem, it seems helpful to integrate the user into the loop. The recommendation should be seen as an anchor instead of a final item. Therefore, we argue that, in this domain, recommendations should be provided as a range with probability information and should be adjustable by the user.
This is possible as the data structure often difers from those known of conventional recommender systems. Conventional recommender systems recommend items. This might also be the case for health-related recommendations, but especially in this domain there is also another type of recommendation, which we call continuous recommendation and which is based on interval-scaled data (e.g. amount of physical activity or amount of sodium intake). Consequently, the user’s decision is no more just dichotomous (select item or not) or ordinal (choose between alternatives) but quantified. This enables system designers to allow users to directly adjust the recommendation (instead the underlying preferences). This is a chance to deal with problems like unknown context factors etc. discussed above. Moreover, because of the quantification, the feedback the system can derive from a user’s decision is more informative than in conventional recommender systems. Thus, we argue that users should be encouraged to adjust the recommendation provided by the system. This might be stimulated by presenting uncertainty, as we will show later. 3.2
Health-related decisions are a very private issue with high personal
importance. Although it is an area in which people are used to get
advice, based on self-determination theory it is important for
motivation that people maintain a high level of autonomy [
Certainly the most important characteristic of recommendations for health-related behavior are the consequences that result from incorrect, inadequate, or misleading recommendations. While in other areas such problems can lead to decreasing satisfaction or acceptance problems, consequences are more serious in health-related recommendations. Taking the example of recommending an activity goal, an inappropriate goal could be physically overburdening and lead to physically dangerous behavior if the user unreflectedly trusts the recommendation.
This is why we emphasize the importance of designing healthrelated recommender systems in a way that they support reflection of the recommendation even if this might decrease trust in the recommendation. 4
We compared data from two studies, we conducted in 2017. The survey period was approximately two weeks in both cases. In the ifrst study, participants were presented an individualized activity goal in an online questionnaire. They were ofered the opportunity to modify the goal. However, we did not present them any information on the uncertainty of the recommendation. Afterwards, they had to rate several items related to aspects of trust, responsibility, transparency, etc. In the second study, participants were presented activity goals in a smartphone application. They were shown four diferent visualizations which also informed the user about uncertainty respectively confidence of the recommendation. Again, the recommended goal could be modified. Afterwards participants answered the same items as mentioned above. The first study was originally conducted to evaluate the used algorithm and the second one to compare diferent visualizations of recommendation uncertainty. However, for a preliminary investigation of the question, whether communication of recommendation uncertainty can influence trust, perceived transparency of the system, perceived responsibility of the system, reflection of the recommended goal, and overall user satisfaction, we decided to compare the participants’ ratings from both studies regarding those aspects. For better comparability, from the second study, we chose the visualization that was most similar to the one in study 1 (see Figure 1 and 2). Both contain a slider to modify the goal. The visualization in study 2 additionally indicates uncertainty by presenting the level of accuracy. Moreover, a variable textual information on the goal’s suitability expresses uncertainty by using an expression like probably. 4.1
In the first study 25 participants (16 female, 9 male) and in the second study 14 participants (10 female, 4 male) took part. Mean age was 34 years in both studies (range study 1: 19-64; range study 2: 21-57).
The Impact of Prediction Uncertainty in Recommendations for Health-Related Behavior 4.2.2 Trust, Perceived Transparency, and Perceived Responsibility. We also asked participants, if the system in general and the recommendation seem trustworthy to them and whether the system seems honest and transparent. Further, we wanted to know whether information about uncertainty increases perceived system responsibility. As can bee seen from the results in Table 3- 7, the presentations with information about uncertainty are significantly superior for trust, perceived honesty, transparency, and responsibility with medium to high efect sizes. In both studies, we asked the users about the above-mentioned aspects. The same items were used in both studies. The scale was a Likert-Scale from 1 (= not agree at all) to 5 (= strongly agree). Normal distribution was just given in about half of the cases. Although t-tests are rather resistant against violation of this precondition, due to unbalanced sample size we report parametric as well as nonparametric tests. All significance tests are conducted on a 5% significance level.
4.2.1 Reflection of the Recommended Goal. To evaluate, if information about uncertainty can foster reflection of the recommendation and integrate the user into the loop, we asked participants if they felt encouraged to reconsider the recommended goal and if they felt encouraged to modify the recommended goal. As the results in Table 1 and 2 show, the ratings for both items were significantly higher when uncertainty information was given. Efect sizes are medium to high.
Overall User Satisfaction. Assessment of the overall user satisfaction consisted of three items. We asked if the participants were satisfied with the recommendation. Further, we asked them if they think the system is exact, in order to see, if information about uncertainty has a negative efect on perceived exactness. Third, we asked the participants if they think, the system does a good job.
For all three items, user ratings were significantly higher when information about uncertainty was provided with a medium to large efect (see Table 8- 10). 5
In the light of the characteristics of health behavior recommendations, we discussed implications of uncertainty from a practical and ethical point of view. Considering problem complexity and data structure as well as security and autonomy aspects, we demonstrated the importance of user empowerment by communicating uncertainty to the users and of integrating them into the loop.
As the main conclusion of our theoretical discussion, we strongly advise to design health-related recommender systems in a way that they support reflection of the recommendation.
In the empirical part of this paper, we showed that presenting uncertainty or confidence to the user might be an appropriate way to increase such kind of reflection. Also user ratings regarding trust, perceived transparency, system responsibility and the overall user satisfaction were superior when presenting uncertainty.
Although the results are very strong with significant diferences in all cases and high efect sizes, it should be considered, that they base on two diferent studies with an unbalanced sample size and some diferences in study design. Moreover, we chose a positive way of indicating uncertainty (presenting the confidence instead of the error). Especially with regard to trust, future research should investigate whether results presenting uncertainty by the showing error are diferent.
In general, future work should take these preliminary results as a starting point to further investigate the issue of uncertainty communication in the area of health-related recommender systems. We want to encourage designers of such systems to integrate uncertainty information and other techniques to foster reflection of the recommendation. Further, we hope that (stimulation of ) recommendation reflection will become an accepted quality criterion for health-related recommender systems.