In face-to-face interactions, advice acceptance depends on how it is presented, as well as a number of social factors. For example, some persons are inclined to accept advice from an expert if they possess little domain knowledge. In contrast, if such advice is unsolicited, persons might only accept advice from a trusted source, such as a family member. Whether these mechanisms also play a role in the recommender context is unknown, even though advice solicitation may be particularly important in domains where a recommender user seeks behavioral change (e.g. energy conservation, healthy eating). This study examines the role of advice solicitation (i.e. whether one asks for advice or simply receives it) and advice source (i.e. either explained in social terms or not) in our 'Saving Aid' energy recommender system. Through a web-based user study with 252 participants, we find that allowing users to solicit advice themselves increases their perceived level of trust with our energy recommender system, compared to users that are presented unsolicited advice. In turn, we find that trust positively affects user satisfaction levels, as well as the number of chosen energy-saving measures. We discuss how system designers should consider how advice is presented and in which context.
• Informa on Systems Decision Support Systems
• Human‐centered Compu ng User Studies
Recommender Systems, Behavioral Change, Energy
Conserva on, User Experience, Advice Solicita on
When one gives another person advice, it is often worded in no
uncertain terms what the other person should do, for example by
saying “You should buy this item”. Such advice may be
unsolicited by the other person and, therefore, not accepted. In
other situations, much more precaution is taken by sharing one’s
own behavior: “this is what I would do”, even though this might
not be as persuasive. In addition, whether one’s advice is
accepted might actually depend on who is giving advice: a good
friend or a stranger [
Research has examined the effects of advice form, task
difficulty, and advice source on whether this advice is accepted
by others [
Such mechanisms of advice-taking are given little attention in
an HCI context, but may impact whether a user actually acts on
presented recommendations. Most recommender systems tend to
be prescriptive in presenting their content (“chosen for you”),
and do not consider whether some users prefer to browse for
appropriate content instead. Moreover, although it has been
shown that explaining recommendation in terms of social peers
increases the likelihood that items are chosen [
The interplay between advice solicitation and advice source
might be particularly important in domains where behavioral
change is part of the recommender ecosystem, such as energy
conservation and health [
Furthermore, energy recommender systems need to consider
that each behavior has a different execution difficulty [25, 29].
An effective energy recommender should persuade its users to
choose and perform more energy-efficient behaviors [18, 29, 32].
Whether a user is willing to only make a small change in his
conservation habits (e.g. changing one’s light bulbs), or a large
one (e.g. installing solar PV), may depend on which other users
have already adopted a certain behavior [
IntRS ’19, September 19, 2019, Copenhagen, Denmark
This paper examines whether advice solicitation affects how a recommender system and its advice are evaluated. For example, some users benefit from browsing a recommender’s personalized interface [15], without requiring an explicit description of each item’s fit. Moreover, whether users wish to solicit advice themselves or that unsolicited advice is accepted, may depend on the advice’s source. In this case, we examine whether the advice is explained in social terms or not, coming from either the system itself, a similar user, or an expert.
We posit the following research question: [RQ]: To what extent do advice solicitation and social advice affect a user’s perception and evaluation of an energy recommender system?
In the upcoming section, we discuss which psychological concepts underlie advice-giving and taking, and how advice solicitation and social advice might influence these. In addition, to generate energy recommendations, we present a ‘light personalization’ algorithm using the psychometric Rasch model. In section 3, we present our ‘Saving Aid’ recommender system. 2
When someone receives advice in a face-to-face situation, three
different motives on the advisee’s part are found to be important
[
One’s autonomy in decision-making has traditionally been defined as one’s need to resist influence or coercion, or to strive for independence [22]. Although this begs the question why persons with a high need for autonomy would seek advice from others [20, 22], let alone recommender systems, much of the advice and personalization that one faces in life is unsolicited [12]. Even so, it is arguably inevitable that also a highlyautonomous person requires advice from others for complex decisions where he or she lacks domain knowledge.
How advice is evaluated, appears to depend on the interplay
between how advice representation and the advisee’s autonomy
[
Furthermore, unsolicited advice also tends to be accepted
when it is given by a friend or family member [10, 12]. However,
some users might feel as if the added social sources are ‘butting
in’, an experience that is often associated with unsolicited aid
and which usually leads to poor rates of advice acceptance [
Advice solicitation impacts advice acceptance, regardless of
whether the advice is of high quality or not. According to Fisher
et al. [
In the context of an energy recommender system, the addition of social sources might only work if the user has an option to solicit advice or not, as it might otherwise pose a threat to one’s autonomy and, in turn, decrease the trust and quality of the recommended advice. Hence, we expect that allowing users to solicit advice leads to different evaluations of a recommender system in terms of trust and quality, compared to a system that presents unsolicited advice. Moreover, we expect this effect to depend on how the advice is explained, using either social peers or not.
Our research question is contextualized in the household energy domain. Previous energy recommender studies have shown that a simple personalization algorithm, based on the psychometric Rasch model, can lead to positive changes in user evaluations [29]. This item response theory model assumes that all energysaving measures and persons share a one-dimensional trait for the goal of saving energy [34], which manifests itself as a single measurement scale.
The Rasch model is an operationalization of ‘Campbell’s Paradigm’ [16]. This attitude theory postulates that one’s energy-saving attitude becomes apparent through the behavioral steps that one is willing to take. Put differently, the more measures one takes with the goal of saving energy, the stronger one’s energy-saving attitude is expected to be [16, 29, 34]. In a similar vein, these energy-saving measures are assumed to differ in how difficult they are to perform, which is operationalized as their behavioral costs [16]. Measures that are performed less often are assumed to have higher behavioral costs, and vice versa [16, 34].
The Rasch model is described in Equation 1. A person n with an attitude θn that exceeds the behavioral costs δi of an energysaving measure i, has a high probability P of performing such measure: P {Xni = 1}. In contrast, that probability is expected to be low if that person’s attitude is much lower than the behavioral costs of the measure at hand:
Using the Rasch model, Starke et al. [29] have reliably fitted a one-dimensional scale of 79 energy-saving measures. These are either one-time investments (“insulate the exterior walls”), or frequent curtailment behaviors (“turn off lights after leaving a room”). The construct is used in the current study to present a list of personalized energy-saving measures that strike the right balance between attractiveness (the behavioral costs are not too high for the user’s attitude) and novelty (the user is expected to not already perform each measure), by minimizing the difference between a user’s energy-saving attitude and a measure’s behavioral costs. If attitude and behavioral costs are equal, a measure’s engagement probability is 50% (cf. Equation 1). 3
To investigate how advice solicitation and social advice explanations affect a user’s evaluation of an energy recommender system, we performed a between-subjects web study. To do so, we developed the ‘Saving Aid’ recommender system (cf. Figure 2), a free-to-use ‘web shop’ that presented attitude-tailored energy-saving measures based on a user’s past behavior and the Rasch model. Users could choose any number of measures they wished to perform in the weeks following the study, which would be sent to them by email.
Figure 1 depicts the general procedure of the current recommender study, which was similar to the procedure used in an earlier study of Starke et al. [29]. First, we estimated each user’s energy-saving attitude, by surveying their current energysaving behavior. Similar to other studies [30, 31], we used a short survey of thirteen energy-saving measures. To do so, we divided the Rasch scale in thirteen subsets across its entire behavioral cost range (from δ = −4.41 to δ = 4.42; Mδ = 0.05), and randomly sampled a measure from each subset. For each measure, users had to indicate whether they already performed it (‘yes’ or ‘no’), or that a measure did not apply to their housing situation (e.g. energy-efficient garden lighting did not apply to a user who did not own a garden).
The estimated attitude θ was based on the number of ‘yes’ responses. We used the average behavioral cost level δ of the equivalent Rasch scale subset. For example, if a user had submitted six ‘yes’ responses, the attitude was estimated to be equal to the average behavioral costs of the sixth subset.
Subsequently, users were navigated to the ‘Saving Aid’ interface, which is shown in Figure 2. It presented the Rasch scale of 79 energy-saving measures in ascending order of behavioral costs (from ‘popular’ to ‘challenging’), which users were free to navigate. The name, costs (NL: ‘Kosten’) and kWh savings (NL: ‘Besparing’) of each measure were listed. To personalize the Saving Aid, it first presented the best fitting measures according to the Rasch model (cf. Equation 1), presenting those that were closest to a user’s attitude in terms of their behavioral costs. Users were then asked to choose (NL: ‘Kies’) any number of measures they would like to perform in the weeks following the study. As soon as users had finished navigating the interface and had chosen measures, they were presented a questionnaire on how they perceived the presented recommendations, as well as how they evaluated the system.
How the advice and the recommender interface were presented, was subject to a 3x2 between-subjects research design. For each condition, we changed how advice was presented. On the one hand, we discerned between three different sources of advice: pointing out that specific measures were recommended by either an expert, similar users, or the system itself. These would be added as explanations to the system’s solicitation button (cf. top of Figure 2) or the unsolicited measures (cf. Figure 3).
On the other hand, we differentiated between two levels of advice solicitation. In the unsolicited advice condition, users were immediately presented three different energy-saving measures in the pop-up screen depicted in Figure 3. They were IntRS ’19, September 19, 2019, Copenhagen, Denmark free to choose any of these measures, or could continue to the main interface. In the solicited advice condition, users were only shown a pop-up that explained how the interface worked, after which they could continue to the main Saving Aid interface. However, users could solicit personalized recommendations in the main interface by clicking a button, which is depicted at the top of Figure 2: ‘click here if you want to see what others would recommend to you’.
In total, 260 participants used our Saving Aid recommender system and finished the subsequent questionnaire. Among them, 110 participants were recruited from the Jan Frederik Schouten database of Eindhoven University of Technology, while others were recruited through posts on social media (e.g. Twitter, Facebook, etc.) Each participant entered a raffle in which they had a 20% probability to win €15. We omitted 8 participants from analysis, as they either had finished the study in no more than 2 minutes, or showed no variation in their answers on the evaluation questionnaire.
Eventually, we analyzed a sample of 252 participants (50.8% female), which had a mean age of 31.9 years (SD = 16.0), and who on average chose 8.9 measures (SD = 9.1). Furthermore, approximately half of the sample was still a university student, who typically lived in shared apartments. Most of these students had an income that fell below €1000.
The different recommender conditions (presenting either solicited or unsolicited advice, from either a similar source, an expert, or the system itself) were considered as objective changes in the interface. We tested whether these changes affected how users perceived the presented recommendations and, in turn, evaluated the system. Furthermore, we also examined whether this led to changes in the number of chosen measures.
Upon analysis, we observed that only half of all users in the ‘solicited’ condition had actually clicked to solicit personalized recommendations (cf. Figure 2). Since we expected that either inspecting a recommendation list or not could affect how users perceived and evaluated the system, we decided to discern between three groups of solicitation instead: unsolicited advice (i.e. baseline), no advice solicitation (did not click), and advice solicitation. Since testing a 3x3 design with a sample of 252 participants would only allow us to detect rather large effects, we collapsed the three social conditions (system, similar, or expert) into two: advice from either a social or non-social source. This allowed us to interpret the results with sufficient statistical power. Note that we found no significant differences between similar and expert advice in a separate analysis.
We surveyed users on five subjective constructs: perceived recommendation quality, perceived system trust, threat to negative face (i.e. whether one’s autonomy is affected), system satisfaction, and choice satisfaction. For all these aspects, users were presented survey items on 7-point Likert scales and were asked to indicate to what extent they agreed with each item. The used questionnaire items are described in Table 1.
As prescribed by Knijnenburg and Willemsen [19], we submitted all responses to a confirmatory factor analysis (CFA) using ordinal dependent variables. Table 1 reports only three user experience constructs, as we could not include the threat to face construct in our analysis, for it had high cross loadings with all other aspects in our model. Moreover, we could not discern choice satisfaction from the system satisfaction construct, since it violated divergent validity [19]. The remaining constructs met the guidelines for convergence validity, as the average variance explained (AVE) for each construct was higher than 0.5, and had a good internal consistency (0.8 < α < 0.9) [11].
4
We used Structural Equation Modeling (SEM) to organize all objective and subjective constructs, including relevant interactions, into a path model. As prescribed by Knijnenburg and Willemsen [19], a confirmatory factor analysis was performed first (cf. Table 1), after which we tested a fully saturated model and performed stepwise removal of nonsignificant relations. Figure 4 depicts the final path model, which had an excellent fit: χ2(113) = 131.625, p = 0.11, CFI = 0.995, TLI = 0.996, RMSEA = 0.026, 90%-CI: [0.000,0.042]. Advice solicitation and social peers in energy recommender systems Table 1. Results of the confirmatory factory analysis on user experience. Items without loading were removed from the final model, while the Choice Satisfaction and Threat to Negative Face aspects were excluded as they violated divergent validity. The average variance explained (AVE) and Cronbach’s Alpha of other aspects met the prescribed guidelines [11, 19].
Aspect Choice Satisfaction Perceived Rec. Quality AVE: .68 Alpha: .82 Perceived Sys. Trust AVE: .72 Alpha: .88 System satisfaction AVE: .70 Alpha: .88 Threat to Negative Face
Item I am happy with the measures I’ve chosen I know several measures that are better than the ones I selected I would recommend some of the chosen measures to others I am looking forward to implement the chosen measures The measures I’ve chosen fit me seamlessly I found the recommended measures to be attractive The recommended measures fitted my preferences The recommended measures were relevant to me The Saving Aid proposed too many bad measures I did not like any of the measures I think that the Saving Aid was telling me the truth I expected the Saving Aid to be truthful The Saving Aid was honest The Saving Aid was sufficiently knowledgeable to present advice The Saving Aid had the best intentions The Saving Aid has made me more aware of my energy-saving behavior I would like to use the Saving Aid more often I make better decisions using the Saving Aid The Saving Aid helps me to find appropriate measures The Saving Aid allowed me to choose measures easily The Saving Aid respected my autonomy and the choices I made The Saving Aid did not impose anything on me I was free to choose any measure I did not feel forced to take the Saving Aid’s advice Loa‐ ding .813 .857 .731 .756
Figure 4 shows two effects of advice solicitation on how the recommender system and its advice were perceived. First, users who had the option to solicit advice and actually did so (‘advice solicitation’), reported higher levels of system trust than those who were presented unsolicited advice: β = .414, p < 0.05. This suggested that users who asked for personalized advice perceived the system as more trustworthy, than those who immediately faced unsolicited advice, regardless of the social source. A bootstrapped test of indirect effects towards system satisfaction showed that this effect was significant, mediated by perceived trust: β = .151, 95%-CI: [0.004,0.298], p = .044.
Second, Figure 4 also shows two different effects of users who were in the solicited condition, but who did not solicit advice. If such advice was offered using a socially-laden explanation (either through expert advice or similar peers), it negatively impacted the perceived recommendation quality compared to the unsolicited, non-social baseline: β = −1.18, p < 0.01. In contrast, non-social explanations positively affected the perceived recommendation quality compared to the unsolicited baseline: β = .41, p < 0.05. Since non-soliciting users did not actually see a list of highlighted energy-saving recommendations, it was possible they evaluated the main interface instead, which confounds a clear interpretation of this result.
Figure 4 depicts that users who had chosen a measure from either a solicited or unsolicited recommendation list, perceived the recommendation quality to be higher than those who did not: β = .636, p < 0.001. Furthermore, Figure 4 shows two different pathways in which higher levels of recommendation quality increase other subjective aspects. First, higher perceived recommendation quality positively affected perceived trust and, in turn, system satisfaction. Besides this mediated effect, Figure 4 also shows a positive, direct path from recommendation quality to system satisfaction, as well as a positive effect from users choosing a recommended measure to system satisfaction.
Moreover, a positive evaluation of the system also led users to choose more energy-saving measures: β = .979, p < 0.05. Although it could be possible to reverse this particular causal direction (e.g. more choices led to higher satisfaction levels), the current pathway was consistent with previous energy recommender studies [18, 29], and led to the best model fit statistics. 5
As one of the first to do so, this study has applied the phenomenon of face-to-face advice solicitation (asking for advice or not) to the HCI domain. We have investigated to what extent solicitation and social explanations of advice affect how a user perceives and evaluates an energy recommender system.
To do so, we have developed the ‘Saving Aid’ system, from which users could choose any number of energy-saving measures they would like to perform. It employs a simple personalization algorithm, using the psychometric Rasch model, to generate appropriate attitude-tailored recommendations for each user. In this personalized advice context, we have found small differences in how our energy recommender system is evaluated, based on whether advice is solicited by users or presented without being explicitly solicited.
Our results show that users who have solicited personalized recommendations report higher system trust levels, compared to those who are presented unsolicited advice. It could be that users wish to maintain control over which items they inspect, and that a system that immediately determines which three items are appropriate, is perceived as less trustworthy. The fact that most recommender research does not consider whether some users wish to solicit advice themselves, seems to be a missed opportunity, since higher trust levels in a recommender could have important second-order effects. Indeed, in this study, we find that users who report higher trust levels, also choose more energy-saving measures as a result of a positive user experience.
Furthermore, we find mixed results for perceived recommendation quality across social conditions. Users who have the opportunity to solicit advice but do not do so, report lower levels of recommendation quality if advice is explained in terms of similar peers or experts, compared to users who face unsolicited social advice. This effect reverses for non-social explanations, as users who are presented the opportunity to solicit system advice report higher recommendation quality than those who are presented unsolicited system advice. These findings either suggest that adding social explanations to unsolicited advice could mitigate a user’s feeling that the system is ‘butting in’, or that users who could solicit advice might not be interested in additional social advice after inspecting the main Saving Aid interface. However, these comparisons only apply to those who have not clicked, which has probably increased the differences between conditions.
Our results resonate with earlier findings on inspectability
and control in social recommender systems [17], which has
shown to positively affect system evaluation. Furthermore, it
shows that findings from the advice-giving and taking literature
on face-to-face interactions do translate to the HCI context [e.g.
3, 12]. Even though the interactions in a typical recommender
system are hardly anthropomorphic, the principles of trust,
quality, and possibly autonomy might also translate to
personalized HCI. These factors may be particularly important in
domains where users wish to change their lifestyle, such as
health and energy conservation, which often rely on social proof
to achieve behavioral change [
Unfortunately, we could not relate our path model constructs to choice satisfaction, for it violated divergent validity. However, the strong correlation between both system and choice satisfaction suggests that trust would also have a positive effect on how users evaluate their choices, which has shown to positively affect the probability that chosen measures are actually implemented and a user’s behavior is changed [29].
Our findings are confounded by not all users soliciting advice in the ‘solicitation condition’. This has prevented us from analyzing our original research design, which has now overlooked any differences between specific social explanations, either a similar peer or an expert. Although this should be further investigated in a follow-up study, the discrepancy between social and non-social explanations of advice seems to be an important finding in itself, which is also demonstrated by the interaction effect in our path model.
Furthermore, in studies on face-to-face interactions, advice solicitation is typically related to personality concepts as autonomy and threat to face. Due to a violation of divergent validity in our path model, we have been unable to test whether a violation of autonomy has decreased trust. Nonetheless, our study still shows that changes in the recommender interface, due to advice solicitation and social explanations, lead to changes in system perception and choice behavior, and should be considered in future recommender designs.
We thank Nena van As, Jochem Bek, and Boy Janissen for their time and effort to conduct this study and discuss its implications. This work is part of the Research Talent program with project number 406-14-088, which is financed by the Netherlands Organization for Scientific Research (NWO). It was also made possible by funding from the Niels Stensen Fellowship.
Advice solicitation and social peers in energy recommender systems