A multi-criteria rating looks for important dimensions to more extensively capture an individual's opinion about a recommended item. Health Recommender Systems (HRS) is considered to be an emerging domain of recommender systems. In HRS, criteria for a multi-criteria preference elicitation of a recommendation have not yet been fully investigated to the best of our knowledge. In this paper, we investigate the criteria for the rating of a health promotion recommendation using an online survey. Drawing on both the relevant literature and the users' responses, we came up with a list of 33 criteria that users are considering when they rate a health promotion recommendation. However, these criteria are not equally important to users. We discuss which of these criteria are more important in the users' opinions. In short, our results show that users consistently consider efectiveness, emotional gain, and giving a good feeling as the most important criteria. Using the criteria derived from the literature, we came up with a model for the importance of the criteria which has three dimensions: efect, efort, and context. This study is the first step toward enhancing our understanding of HRS and the rating of a health promotion recommendation.
• Information systems → Information systems applications; Recommender systems; Information retrieval; • Human-centered computing → Human computer interaction (HCI); • Applied computing → Health care information systems; Health informatics;
Recommender systems (RS) have used various methods and algorithms that utilize implicit and explicit user feedback in order to HealthRecSys’18, October 6, 2018, Vancouver, BC, Canada © 2018 Copyright for the individual papers remains with the authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors. build up a user profile. While such systems may obtain implicit user feedback via modeling user behavior, explicit user feedback mainly relies on the ratings that a user gives to either the recommended or consumed items. With a user-oriented perspective toward RS, one may consider that in real-life, users may use various dimensions to describe their attitude toward a product or recommendation. For instance, a user may say: I like the durability of the product, but it is not aesthetically beautiful, or she might say: I like how the actor is playing in this movie, but I hate the ending. Therefore, it is important to consider how a rating feedback for a recommended item should be, how it should be interpreted, and how it is reflected in the algorithm and recommendation process.
Researchers have looked into one-dimensional or single-criterion
rating and later also into multi-criteria rating. In single-criterion
rating, user feedback is captured via a single value from a bounded
set of natural numbers. The user feedback here represents either
an overall rating of a recommended item or an alternative question
that one may ask when the rating task is presented to the users.
Even though single-criterion rating has been commonly used, over
the years researchers [
Health recommender system (HRS), as an emerging domain of
RS, represents three classes of solutions: I. the application of RS in
the health (medical) domain, II. the application of RS in the health
promotion and well-being domain, and III. RS that exploit health
information in order to give recommendations in other domains.
The field of HRS, as an interdisciplinary domain, deals with various
challenges [
While one can obtain the criteria for multi-criteria rating from both previous research and the collected datasets in domains such as movies or tourism, in the health domain accessing similar datasets is not easily possible and such criteria, therefore, have not been fully investigated. As a result, criteria for this particular domain remain unclear. Even for those available datasets with a multi-criteria rating, it is up to researchers to investigate how important each dimension is and construct a consistent group of criteria. Such criteria enable the scientific community to come up with suitable algorithms for multi-criteria RS. The present study reports on an exploratory attempt to find out the dimensions or criteria that are influential in the rating of a recommended item in the HRS domain, focusing on the second category of HRS. This paper also reports on which criteria users consider the most important and ultimately, proposes a model for user satisfaction with a recommendation. The result of this attempt can be used to construct a primary set of criteria for health-specific domains and would help the RS community to deepen their understanding of user satisfaction and rating. 2
An exploratory study was conducted using an online survey in order to investigate the criteria. In addition, we sought to investigate the importance of the obtained dimensions or criteria for the overall rating in the users’ opinions. Participants answered the questions regarding various recommendation scenarios related to a smartphone health application. For this study, we considered four aspects: I. the study should be health-domain-dependent II. various scenarios should be considered when asking for a user rating, III. familiar scenarios with mobile app mockups should be presented to the participants, so they can easily rate them, and IV. study design should prevent bias related to the criteria and their importance by ordering questions appropriately and ensuring the sequence of the given criteria changes randomly.
The online study contains four sections with the following order: subject knowledge, scenario, importance ranking, and demographics. While the demographic section asks for standard questions about gender, education, age, country of origin, and native language, subject knowledge section asks participants if they have ever rated a product or wrote a review on a product online. The subject knowledge section also determines to what extent a participant has made decisions on consumption of a recommended item based on its rating.
The scenario section included three scenarios. A recommendation (social engagement, physical activity, and positive psychology) follows each of the scenarios with the aim of mental health promotion and reduction of the negative efects of stress in daily life. The survey shows a mockup containing the recommendation on the smartphone, and then asks the participants to follow the recommendation. They are then asked if they did follow the recommendation and subsequently, they rate the given recommendation overall. After the rating, the participants are asked to write down the criteria and the reasons that they have considered for giving the overall rating in an open-ended question. In another open-ended question, the survey asks participants if the ratings are to be used to provide personalized recommendations for them, which factors they would consider while rating a health promotion recommendation. We designed these open-ended questions, particularly to capture criteria that users consider for giving a rating to a recommendation.
The section then continues by showing the previously shown
scenarios and recommendations again. This time, the participants
are asked to give separate ratings for the recommendations in the
following criteria: location, fitting personal preference, suitable
time, meeting their goals, being interesting, enjoyable, being
effective, suitable time consumption, emotional aspect, easy to do,
suitable cost, and socially acceptable. We obtained this list of
criteria from the literature. For instance, using both Fogg’s behavior
model for persuasive design [
Other factors have their origin in the literature as well. For
example, easy to do refers to a task not being too dificult and not
being too easy for a user inspired from both self-eficacy theory [
The next section of the study, the ranking section, asks participants to rate a set of given criteria (randomly ordered) based on the importance a criterion has in the participant’s overall rating of a health promotion recommendation. We included 19 items (listed in the left column of Table 2) by extending the previously mentioned criteria. For example, we had two items of emotional gain or positivity and good feeling. In addition, to re-check for the most important criteria in the participants’ opinion, participants ranked the eight top criteria in order of importance in another question. The survey was online for the duration of 5 days. 3
Participants’ description. A total of 74 (36 females, 30 males, and 8 prefer not to say) participants completed our study. The age range of the participants is between 20-54 years old with 32.5% between 20-30 and 56.8% between 30-40 years old. Among the participants, 87.8% had university degree education. The participants’ origin is from 14 diferent nationalities. While 79.7% of participants have previously rated a product or service online, only 66.2% have written reviews. Among the participants, 75.6% consider online rating important and very important in their decision making on the consumption or purchasing of a product or service and another 23.0% consider the rating moderately important. Finally, 72.9% of the participants very often and always check online ratings when they are looking for a product or services and another 17.6% sometimes check the ratings.
Descriptive results. Qualitative analysis of 296 open-ended answers in total gives us the insight for criteria that were not included in our 19-item criteria. Interestingly, we were able to easily code the answers into the 19-item criteria described earlier in the method section so that only a few instances of a new criterion were left out. In addition to the 19-item criteria, the participants mentioned the explanation of the recommendation and its origin (n=13) and the recommendation being new or not new (n=26) as criteria for rating a health promotion recommendation. The participants also mentioned the following criteria in less than five instances: empowering, trust, transparency, suitable for my current mood, encouraging, not-human-like (getting advice from a machine), understandable, easy to read, needing it, intrusiveness, funny, and relevance for me. The participants in general mentioned these criteria from the 19-item criteria more than others: efectiveness of a recommendation (n=97), the emotional gain (n=66), fitting one’s preferences (n=47), liking (n=43), at a right time (n=32), in a right location (n=25), and easy to do (n=19). In total, all criteria including repeating similar items create a list of 33-items.
Although in the open-ended questions we observed a diference in the frequency of the mentioned criteria between diferent scenarios, in all three scenarios as well as in the general question, efectiveness of the recommendation is the most frequently mentioned criterion. Emotional gain for the two scenarios of positive psychology and social engagement was the second most frequently mentioned criterion. At the same time for the physical activity scenario, the second most frequently mentioned criterion was at a suitable time. Fitting one’s preferences and liking as well as would do the recommendation criteria were mentioned frequently in the physical activity and social engagement scenarios, however, these criteria were not among five top frequently mentioned criteria for the positive psychology scenario. Instead, participants mentioned not dificult , meeting my goals, and not time-consuming criteria for the positivity psychology scenario.
The overall rating and criteria-based rating of the recommendations, in all scenarios together, show positive correlations (p < .01) with the following criteria: efectiveness ( rs = .68), emotional gain (rs = .61), fitting personal preferences ( rs = .65), being interesting (rs = .62), and enjoyable (rs = .57). Other criteria are also significantly related to the overall rating ( .2 < rs < .44).
Two questions considered the importance rating of the criteria: the ranking question (Q1) and the importance rating (Q2). The ranking1 result shows that the participants as a whole relatively consider three criteria of giving a good feeling (M = 3.41, SD = 2.36), emotional gain or positivity (M = 3.29, SD = 2.36), and efectiveness (M = 3.90, SD = 2.05) to be the most important ones. The analysis of the importance rating question, Q1, with the range of importance score being between (not important at all)1-10 (very important) only 1 Q1: the most important being 1 to 8 confirms top three important criteria, Table 1. Among other criteria, iftting personal preferences , being interesting as well as fulfilling my goals of using the app all had mean values among the top eight mean values in both Q1 and Q2, however, there is a slight diference for the last two sorted items in Q1 and the results of Q2. In Q2, the recommendation at a suitable time and I enjoy the recommendation have better mean values while in Q1, would follow the recommendation and the recommendation is not time-consuming have better mean values.
An exploratory factor analysis (i.e. maximum likelihood using Promax rotation) on 19 criteria from Q2 was conducted. Three factors were extracted that accounts for 58% of the variance overall. The loading of each of the criteria on these factors is presented in Table 2. Based on the intuitive interpretation of the analyzed item, we summarize the factors as the recommendation efect (F1), the recommendation efort (F2), and the recommendation context (F3). These three factors show three important aspects of the rating of a health promotion recommendation. While efect represents the influence of the recommendation on the users including the recommendation efectiveness and its emotional consequences, the recommendation efort represents the challenges that a user may need to overcome in order to follow the recommendation. This aspect may also deal with the motivation and goal setting. Finally the third factor, the context represents the physical context of a user such as their location and time of the recommendation. The internal consistency of the factors was assessed using Cronbach’s α for the factors F1, F2, and F3 and we obtained the following values respectively .947, .811, and .839 that is acceptable. 4
The results of our study in the demographic and subject-knowledge sections reveal that our participants were familiar with RS and ratings as users, and in fact, they were making some of their purchasing and searching decisions by relying heavily on such ratings. The variety of nationalities and ages in our study represents a wider range in comparison to the college-student population. These sections leads to the assumption that the participants were familiar with both giving a rating and the concept of a recommendation based on the ratings. Consequently, we may assume that our participants were able to reflect on the rating criteria in the questions.
The open-ended questions suggest criteria in addition to our 19-item criteria extended from the literature. These results confirm our design of 19-item criteria and add to it. Some criteria mentioned by participants such as trust or transparency are very important particularly in HRS despite being mentioned only in a few instances by the participants. Lack of trust in an HRS domain can lead to disastrous consequences such as harm to individual or leak of sensitive data. Furthermore, users also have a higher expectation of privacy, trust, and transparency for such systems in comparison to other RS domains. However, one may also consider that a criterion such as the recommendation explanation can have an efect on these criteria. It is therefore important for the HRS community to consider the explanation aspect as a criterion alongside designing trust-aware and privacy-aware systems.
A criterion such as being new represents two contrasting groups of participants. Some participants appreciated the familiarity of the recommendation and emphasized that they would like to be reminded of the recommendation that they had prior positive experience with. These participants mentioned that they would follow the recommendation, and they like the recommendation and its efectiveness due to their prior positive experiences. In contrast, some participants gave a low rating for the recommendation despite expecting a positive efect from the recommendations. These participants mentioned that they do not need the recommendation, because they already do the recommendation on a daily basis. Such a diference only highlights the importance of personalization of a recommendation. For example, imagine in specific domains of HRS that a behavior change design requires user engagement in a repetitive behavior, particularly for such a design, the researchers should be aware of the user preferences and their assumptions about giving a rating. Alternatively, one could also consider specific engagement approaches or motivations such as emotional gain to increase the user satisfaction and compliance alongside instructing the users for giving a rating feedback.
From all the mentioned criteria, efectiveness seems to be the most important criteria. However for HRS in health promotion or behavior change, instantaneous efectiveness may not be observable for the users. In fact, the efectiveness depends on the users’ goals of using the HRS application and its design or their prior experiences. A user may need to follow a program to the end to feel a change or improvement, or they could even feel worse right after initiating a behavior change process, for example, in a smoke cessation behavior change. Therefore, a challenge would be to show the users an efect. Diferentiating between long-term and short-term efect and planning for instantaneous efect could be a possible solution. In particular, users do not always consider a final goal as the efectiveness and therefore a design for an instantaneous emotional gain (hope, pleasure, reward, etc.) may give the users the efect they would like to receive.
Emotional gain or positivity and good feeling were also frequently mentioned as important criteria. While these two criteria in our opinion represent the same aspect, i.e. emotional gain, other criteria also may contribute to it. Imagine criteria such as liking, interesting, or enjoying; they all contribute to the users’ final emotional gain or loss feeling. Even motivational criteria such as a task being too dificult or too easy can indirectly lead to the feeling of pleasure. Criteria such as social acceptance can lead to the feeling of shame or fear. Interruption, suitable time or location can also lead to a higher cognitive load or even sometimes turn into a hassle. Accordingly, it is expected that emotional gain criterion is very important in the rating of a recommendation and one should consider that other criteria at the end may influence this criterion.
We intentionally designed diferent scenarios to capture
various criteria. Initially, we thought that these scenarios may lead
to various ratings of criteria. Nevertheless, except for some minor
diferences, the overall result did not suggest a major diference
in the ratings of the most important criteria being efectiveness
and emotional gain, user preferences, interesting, and fulfilling the
goals of using the application. For instance, criteria related to F3: the
context, namely, at a suitable time, not interrupt, not time-consuming,
and right location were expected to show more importance for the
users. One could explain this in two ways. It is plausible that the
context criteria are more important on the decision-making of the
user on following the recommendation versus ignoring it rather
than on its overall rating since these criteria may play a role as a
signal as the trigger in Fogg’s behavior model [
In either possibility, the HRS designers could rely on using more than just explicit ratings to capture the users’ feedback. For instance, one can easily capture the context criteria using user behavior tracking or use just in time interruption techniques to minimize interruptions. Alternatively, one could also ask users if they would follow the recommendations and only capture user feedback for the consumed items. The efectiveness of a recommendation may also be captured using indirect methods. For example in a physical activity recommendation, one can easily capture the changes using the activity trackers or for stress reduction, one can assess before and after level of stress. Overall, the feedback mechanism and the use of rating in HRS depends on the targets of an HRS as well as the study design. Imagine that you want to use a specific behavior change model or a set of intervention that may not focus on the Would engage with the recommendation e m it e l b a it u s a t A n o it a c o l e l b a it u s a n I
Context t p u r r e t n it o n o D e l b a t p e c c a y ll a i c o S g n i m u s n o c e m it t o N y e n o m t s o
C
Effort t r o f f e h c u m o o t t o N lt u c iiff d t o N o d o t y s a E s s e n e v it c e f f
E Ability n i a g l a n o it o m E
Effect s e c n e r e f e r p r e s u it F e c n e i r e p x e e l b a y o j n E g n it s e r e t n
I
Effect (motivation in Fogg's model) n o it c a f s it a S r e s U user’s rating at all. When the design of HRS allows, a design using a multi-criteria rating feedback may allow a better personalization and a higher efectiveness and user satisfaction as a result. 4.1
A basic model for user satisfaction with a
recommendation
We propose a basic model for user satisfaction with a health
promotion recommendation. Considering both the three extracted factors
for importance and the importance rating of the criteria, and taking
inspiration from Fogg’s behavior model that encompasses
motivation, ability, and trigger [
Context. A user would engage with a recommendation if it is suggested in a convenient situation or physical context for the user. A convenient situation means at the right time, in the right location, for the right duration, with no inconvenient interruption, or with the recommendation being suitable considering the presence of others near the user. If the context is not convenient for the recommendation, then regardless of its efect, it could be simply ignored.
Efort. A user would engage with a recommendation if it falls within her capacity or ability to do or cope with the task. If the efort condition is not met, then the recommendation may be ignored or would possibly yield unsatisfactory results. One should, however, consider that these two dimensions of efort and context are preconditions that could determine if the user will follow a recommendation to the end.
Efect. A user accordingly would be satisfied by a recommendation if the recommendation has an efect on her, including either emotional gain, pleasure, or efect, or fulfilling her explicit or implicit goals and motivations. These goals could be a final target or a behavior change, or be as simple as a reward or pleasurable experience. All three components together are required for a recommendation to satisfy a user. First, the user should be able to do the recommendation, or it should be probable that the user can follow the recommendation. Then she should engage with the recommendation to the end, and finally, the recommendation should bring the user the efect .
In this model, any of the context criteria can disrupt fulfillment of the context dimension. For the efort dimension, a combination of the criteria may or may not disrupt its fulfillment, and only the capacity of an individual user would describe this combination. Any of the criteria in the efect dimension may complete its fulfillment and lead to a higher user satisfaction. Efect not only influences user satisfaction, but it can also shape the users’ preferences. For example, if a user has a good prior experience with a recommendation, she might decide to follow the recommendation with an even greater efort. On the contrary, an unsatisfactory efect could possibly lead the user to ignore the recommendation right away. As tagged in figure 1, the context dimension is parallel with the trigger factor in Fogg’s behavior model. The efort dimension is parallel with the ability factor, and the efect dimension is parallel with the motivation factor in Fogg’s model. Nevertheless, Fogg’s model describes a behavior model for a persuasive design and is not a model of user satisfaction from a recommendation. 5
CONCLUSION AND FURTHER WORK Multi-criteria rating provides an opportunity to gain a fuller picture of an individual’s opinion. This paper explored the criteria for the domain of health recommender systems and provided 33 criteria that users are considering when rating a health promotion recommendation. Finding out that these criteria are not equally important for the users, this study determined the criteria that are more important for users than others. Using a 19-item criteria list, this work presented a three-dimensional model that represents the aspects that these criteria refer to. Our work clearly has some limitations. The online survey is limited in its ability to capture real-life user responses and reactions. In addition, criteria such as item explanation and novelty were not included in the factor analysis since they were captured in the same survey. Despite this, we believe that our work could be a starting point for developing multi-criteria recommendation algorithms for HRS. The extracted list of criteria gives an insight into the criteria that are important in the users’ ratings of a health promotion recommendation.