The Effect of Sensitivity Analysis on the Usage of Recommender Systems Martina Maida Konradin Maier Nikolaus Obwegeser Vienna University of Vienna University of Vienna University of Economics and Business Economics and Business Economics and Business Augasse 2-6 Augasse 2-6 Augasse 2-6 1090 Vienna 1090 Vienna 1090 Vienna martina.maida@wu.ac.at konradin.maier@wu.ac.at nikolaus.obwegeser@wu.ac.at Volker Stix Vienna University of Economics and Business Augasse 2-6 1090 Vienna volker.stix@wu.ac.at ABSTRACT 1. INTRODUCTION Recommender systems have become a valuable tool for suc- Recommender systems (RS) have become an important cessful e-commerce. The quality of their recommendations tool for successful e-commerce. They help consumers in e- depends heavily on how precisely consumers are able to state commerce settings to overcome the problem of information their preferences. However, empirical evidence has shown overload, which they often face due to the vast amount of that the preference construction process is highly affected available products and of product-related information. From by uncertainties. This has a negative impact on the robust- a consumers-perspective, the main task of RS is to support ness of recommendations. If users perceive a lack of accu- finding the right product. Independent from technical con- racy in the recommendation of recommender systems, this siderations, all RS have in common that they require infor- reduces their confidence in the recommendation generating mation about their users in order to provide personalized process. This in turn negatively influences the adoption of recommendations. This information is basically the con- recommender systems. We argue in this paper that sensi- sumers’ preferences which serve as input for the recommen- tivity analysis is able to overcome this problem. Although dation-generating algorithm [24]. Thus, the users’ prefer- sensitivity analysis has already been well studied, it was ig- ences are clearly of high importance for the quality of the nored to a large extent in the field of recommender systems. RS’ output and the more precise the preferences correspond To close this gap, we propose a research model that shows to the user’s “real” needs, the more accurate will be the rec- how a sensitivity analysis and the presence of uncertainties ommendation of the system. influence decision confidence and the intention to use rec- The problem we want to address here is that the prefer- ommender systems. ences of consumers as well as their measurement are subject to irreducible arbitrariness [12], which potentially has a neg- ative impact on the quality of a RS’s recommendation and Categories and Subject Descriptors on the adoption of RS. To overcome this problem, we pro- H.3.3 [Information Search and Retrieval]; J.4 [Social pose to integrate sensitivity analysis into RS. The remain- and Behavioral Sciences] der of this paper is structured as follows. The next Section describes the uncertainties related to the measurement of General Terms preferences and the implications for RS design. Section 3 provides a short overview of SA methods and possible ways Theory, Human Factors to address uncertainties as well as similar problems of sup- porting consumers via RS. We will propose a research model Keywords in Section 4 and hypothesize how SA and uncertainties in Recommender systems, sensitivity analysis, uncertainties in the process of generating recommendations are related to RS preference construction, technology acceptance usage. The planned methodology for testing our hypotheses is presented in Section 5. Finally, we provide a short discus- sion of our model and present further research opportunities in Section 6. 2. UNCERTAINTY AND RECOMMENDER Paper presented at the 2012 Decisions@RecSys workshop in conjunc- SYSTEMS tion with the 6th ACM conference on Recommender Systems. Copyright Humans often face decisions which have to be made based c 2012 for the individual papers by the papers’ authors. Copying permit- ted for private and academic purposes. This volume is published and copy- on beliefs regarding the likelihood of uncertain events like righted by its editors. future prices of goods or the durability of a product [21]. 15 Here, uncertainty refers to a state of incomplete knowledge, plines, like in chemical engineering, operations research or which is usually rooted in either the individual’s lack of in- management science [20]. According to French [5], a com- formation or in his limited resources to rationally process mon definition of SA involves the variation of input variables the available information [4, 18]. to examine their effect on the output variables. In the case The latter source of uncertainty - limited information pro- of RS, inputs refer to preferences of consumers and out- cessing capabilities - is the rationale underlying the idea to put means the recommendation of the system. Thus, SA support consumers in making their decisions by providing is a valuable tool for detecting uncertainties in inputs, ver- personalized recommendations. In this sense, it is the func- ification and validation of models as well as demonstrating tion of RS to mitigate the information overload which con- the robustness of outputs. Definition and purpose however sumers often face in e-commerce settings [16]. As research vary depending on the field of application [15]. Furthermore, in RS deals with bounded rational consumers, it has to ac- there are different SA methods. They are classified e.g. in knowledge that consumers face uncertainties while making mathematical, statistical and graphical methods [6] or in lo- their purchase decisions, even if they are supported by a cal and global SA methods depending if the input variables RS. The origins of uncertainty in a RS-facilitated purchase are varied over a reduced range of value or over the whole decision can be manifold. For example, a consumer might domain [15]. Both classes allow to vary “one factor at a time” ask himself whether the model underlying the RS is indeed (OAT) or several variables simultaneously (VIC - variation appropriate to support him or whether the complex calcu- in combination). Some researchers (e.g. [17]) argue that a lations underlying a recommendation have been solved ac- variance-based, global SA with VIC is especially useful for curately or in a more heuristic way [4]. Another important comparing input variables and identifying uncertainties. source of uncertainty is the consumer. Often, it is assumed Although SA is in general a well-studied topic, it is ig- that decision makers have stable and coherent preferences nored to a large extent in the field of RS. Papers that treat and sometimes it is even supposed that they accurately know SA as tool for decision support systems are typically from these preferences [9]. However, there is vast empirical evi- the field of multi-criteria decision making. They explain for dence that these assumptions do not model real world deci- instance how SA demonstrates robust solutions or illustrates sion makers very well. For example, it is commonly known the impact of input variations [13]. A reason why SA should that the answers of a decision maker who is requested to be integrated in decision support systems is that it addresses explicitly state his preferences are at least partly dependent certain drawbacks, like a possible lack of transparency. By on the framing of the questions and on what response is considering RS, this would mean that consumers do not re- expected [22]. These and other empirically observed devi- ceive the possibility to understand why a particular product ations from rationality led to the notion that humans do was recommended. Thus, consumers are not able to detect not have well-defined preferences which can be elicited but uncertainties that were introduced during preference elici- that we construct preferences on the spot, usually by apply- tation. As argued by [19, p. 831] “(...) users are not just ing some kind of heuristic information processing strategy. looking for blind recommendations from a system, but are Consequently, our preferences are “labile, inconsistent, sub- also looking for a justification of the system’s choice.”. A ject to factors we are unaware of, and not always in our own possibility to provide justifications are explanation facilities. best interests” [9, p.2]. An approach that was found in literature is to regard SA as For the effort to support consumers with the help of RS being similar to an explanation facility [14]. It facilitates such instable preferences pose a serious problem. RS try to the involvement of users and increases transparency of the support consumers by providing personalized recommenda- recommendation generating process [8]. An integrated SA tions based on the consumer’s preferences. Independent of permits users to interact with the system such that they are how the RS measures the preferences of the consumer (ei- able to explore possible variations of the inputs and see how ther explicitly by asking the consumer or implicitly by ob- their changes influence the robustness of the recommenda- serving his behavior), the ad-hoc construction of preferences tion. A SA is therefore especially important when uncer- implies that RS have to deal with an uncertain information tainties in the inputs are present. In contrast to the various base to make recommendations (cf. [4]), which might lead to types of explanation facilities, it is based on formal sciences inaccurate and therefore unhelpful recommendations. More- and is thus capable of providing objective explanations. over, a consumer who faces a recommendation of a RS might perceive a state of uncertainty regarding the recommenda- tion’s quality because the choice of the recommendation- 4. RESEARCH MODEL AND HYPOTHESES generating algorithm, its inputs (the preferences) as well as DEVELOPMENT its computation are afflicted with uncertainties. The work of Based on the descriptions of the problem of uncertain- Lu et al. [10] shows that a major reason for the rejection of ties and the characteristics of SA we will derive a research decision support technologies is that humans are skeptical model for RS usage in this Section. In order to understand whether the respective technology is indeed able to accu- how SA is related to the adoption of RS, we integrate sensi- rately model their preferences. In other words, the uncer- tivity analysis, perceived uncertainty and decision confidence tainties related to technologically derived recommendations in a common model of RS usage. The definitions of these might hamper the adoption of RS. In order to avoid these concepts are given in Table 1. Our model builds on technol- problems, RS have to address the uncertainties related to ogy acceptance research and its most prominent model, the the generation of recommendations. Here, we propose to technology acceptance model (TAM) [2]. Figure 1 illustrates incorporate SA into RS to overcome this challenge. the proposed model. Sensitivity analysis represents the de- sign feature of interest, decision confidence and perceived 3. SENSITIVITY ANALYSIS uncertainty are used to describe the link between the de- Sensitivity analysis is a widely used tool in various disci- sign feature and RS use in detail. The following paragraphs 16 Table 1: Definitions of Constructs Construct Definition Sensitivity A RS feature which allows a user to Analysis analyze how a recommendation (out- put) changes if the preferences (in- puts) are varied [5] Decision The user’s beliefs that the recommen- Confidence dation matches his preferences [7] Perceived The user’s subjective probability as- Uncertainty sessment of any presence of inaccu- Figure 1: Proposed research model racy in the recommendation generat- ing process [4] separately discuss each proposition of our model. Perceived The user’s perceptions of the utility of Basically, a SA can lead to two different results: Depend- Usefulness the RS [24] ing on inputs and model parameters, it will either confirm Intention to The user’s subjective probability of or disprove the robustness of the recommendations provided Use adopting the RS [3] by the RS. Though we acknowledge that the output of a SA depends on the specific situation and that the concrete outcome of the SA is likely to influence the user’s percep- The relationship between perceived uncertainty and decision tions, we argue that there is also an effect which is indepen- confidence is similar to H4. If users perceive that a recom- dent from such contingencies (see also Section 6). SA helps mendation is based on an uncertain information base or if users to filter out those recommendations which are robust they are not sure about the appropriateness of the recom- to uncertainties and which thereby represent good choices mendation generating algorithm, they are likely not confi- independent from changes in the inputs [12]. Therefore, we dent about the quality of the recommendation. Therefore, hypothesize that we hypothesize that H1: Sensitivity analysis will increase users’ de- cision confidence. H5: Perceived uncertainty will negatively influ- ence decision confidence. The only task which RS perform is to search and suggest decision alternatives on behalf of their users. If a user is not sure whether a RS provides recommendations which match his needs or not, the only reason to use a RS van- 5. PROPOSED METHODOLOGY ishes. Therefore, we hypothesize that We will conduct a laboratory experiment to test our hy- potheses. We will use a 2 x 2 full factorial design with SA H2: Decision confidence will positively affect per- and perceived uncertainty as independent variables. Partic- ceived usefulness of recommender systems. ipants will be asked to use a RS for online shopping which SA is a tool which demonstrates how the output varies when explicitly demands from users to make trade-offs in pref- inputs are changed. This enables user not only to analyze erence construction. They will be randomly assigned to a different scenarios and to search for robust recommendations treatment group and a control group which allows us to ma- but also to learn about the RS and how it generates recom- nipulate SA and perceived uncertainty. We will choose pur- mendations. In this function, SA might be directly related chase decisions with low/high familiarity to induce high/low to perceived usefulness of the RS regardless of its impact on levels of perceived uncertainty. After finishing the shopping decision confidence and independent from whether it con- task, questionnaires will be delivered to the participants to firms the robustness of the recommendation or not. Based assess the proposed relationships. on this argument and on the experiences of Payne et al. [12] Before we are actually able to conduct the experiment, that user perceive SA as a valuable tool, we hypothesize that we will develop new measures for the constructs perceived uncertainty and decision confidence by adopting the method H3: Sensitivity analysis will positively influence of Moore and Benbasat [11] for instrument development. perceived usefulness of recommender systems. The validity and reliability of the items will be tested by We argue that this relationship is moderated by the degree a factor analysis in a pilot test. Items for the remaining of perceived uncertainty: Consider a user who does not per- constructs will be taken from already validated scales, for ceive any uncertainty related to the output of a RS. For such instance from Davis [2] for perceived usefulness. a user a SA is of little to no value. But the more the user per- To test our experimental design, we will conduct a t-test ceives that the recommendation generating process is prone in order to check the manipulation of perceived uncertainty to uncertainties, the more useful is a feature which allows via familiarity of the purchase task. For testing our hypothe- to explore the impact of the uncertainties on the outcomes. ses we will use structural equation modeling (SEM). As our Therefore, we hypothesize that study is the first one regarding the impact of SA and un- certainty on RS usage, it has an exploratory character. To H4: Perceived uncertainty will moderate the in- manage the risks associated with exploratory research, we fluence of sensitivity analysis on perceived use- will keep the sample size rather low (about 10 participants fulness of recommender systems. per indicator [1]). To deal with the small sample size and the 17 exploratory character of our research, we will use a partial [9] S. Lichtenstein and P. Slovic. The construction of least squares approach (component-based SEM) [23]. preference: an overview. In The construction of preference, Lichtenstein, S. and Slovic, P. (Eds.), 6. DISCUSSION AND CONCLUSIONS pages 1–40. Cambridge University Press, 2006. [10] H. Lu, H. Yu, and S. Lu. The effects of cognitive style Based on a literature review, we have argued that the pro- and model type on dss acceptance: An empirical cess of generating recommendations for e-commerce users in- study. European Journal of Operational Research, volves uncertainties, especially regarding the measurement 131(3):649–663, 2001. of preferences, which might lead to users who feel insecure about the quality of a RS’s recommendations. Moreover, we [11] G. Moore and I. Benbasat. Development of an hypothesized that if users do not feel confident about a RS’s instrument to measure the perceptions of adopting an recommendations, they will not perceive RS as useful and information technology innovation. Information thus are less likely to adopt the RS. We proposed to incorpo- systems research, 2(3):192–222, 1991. rate SA into RS to overcome the problems associated with [12] J. Payne, J. Bettman, and D. Schkade. Measuring uncertainties. SA is a tool which enables users to explore constructed preferences: Towards a building code. how changes in the inputs of the recommendation generat- Journal of Risk and Uncertainty, 19(1–3):243–270, ing process (the users’ preferences) are related to changes in 1999. the output of the process (the recommendations). SA can [13] L. Proll, A. Salhi, and D. Rios Insua. Improving an be used to check the robustness of recommendations which optimization-based framework for sensitivity analysis should help users to build confidence in the system’s advice in multi-criteria decision-making. Journal of and the decision. Finally, we proposed a conceptual model Multi-Criteria Decision Analysis, 10(1):1–9, 2001. and corresponding hypotheses of how uncertainties, decision [14] W. Raskob, F. Gering, and V. Bertsch. Approaches to confidence and SA are related to the adoption of RS. visualisation of uncertainties to decision makers in an As outlined in Section 5 our next step is the empirical operational decision support system. In Proceedings of testing of the proposed model by conducting a laboratory the 6th International ISCRAM Conference, page experiment. Further research opportunities include theoret- http://www.iscram.org/ISCRAM2009/papers/, 2009. ical work on how SA can be incorporated into the various [15] A. Saltelli, K. Chan, and E. Scott. Sensitivity analysis. forms of RS, not only on computational level but also on the John Wiley, New York, 2000. level of user interface design and how the outcomes of SA [16] J. Schafer, J. Konstan, and J. Riedl. E-commerce are related to user perceptions. recommendation applications. Data mining and knowledge discovery, 5(1):115–153, 2001. 7. ACKNOWLEDGMENTS [17] K. Siebertz, D. Van Bebber, and T. Hochkirchen. This research has been funded by the Austrian Science Statistische Versuchsplanung: Design of Experiments Fund (FWF): project number TRP 111-G11. (DoE). Springer Verlag, 2010. [18] H. Simon. A behavioural model of rational choice, In: Models of Man: Social and Rational. John Wiley, New 8. REFERENCES York, 1957. [1] J. Christopher Westland. Lower bounds on sample size [19] R. Sinha and K. Swearingen. The role of transparency in structural equation modeling. Electronic Commerce in recommender systems. In CHI’02 extended abstracts Research and Applications, 9(6):476–487, 2010. on Human factors in computing systems, pages [2] F. Davis. Perceived usefulness, perceived ease of use, 830–831. ACM, 2002. and user acceptance of information technology. MIS [20] E. Triantaphyllou and A. Sánchez. A sensitivity Quarterly, 13(3):319–340, 1989. analysis approach for some deterministic multi-criteria [3] M. Fishbein and I. Ajzen. Belief, attitude, intention, decision-making methods*. Decision Sciences, and behavior: an introduction to theory and research. 28(1):151–194, 1997. Addison Wesley Publishing Company, 1975. [21] A. Tversky and D. Kahneman. Judgment under [4] S. French. Uncertainty and imprecision: modelling and uncertainty: Heuristics and biases. science, analysis. The Journal of the Operational Research 185(4157):1124–1131, 1974. Society, 46(1):70–79, 1995. [22] A. Tversky and D. Kahneman. The framing of [5] S. French. Modelling, making inferences and making decisions and the psychology of choice. Science, decisions: the roles of sensitivity analysis. Top, 211(4481):453–458, 1981. 11(2):229–251, 2003. [23] N. Urbach and F. Ahlemann. Structural equation [6] H. Frey and S. Patil. Identification and review of modeling in information systems research using partial sensitivity analysis methods. Risk Analysis, least squares. Journal of Information Technology 22(3):553–578, 2002. Theory and Application, 11(2):5–40, 2010. [7] D. Griffin and A. Tversky. The weighing of evidence [24] B. Xiao and I. Benbasat. E-commerce product and the determinants of confidence. Cognitive recommendation agents: Use, characteristics, and Psychology, 24(3):411–435, 1992. impact. MIS Quarterly, 31(1):137–209, 2007. [8] J. Herlocker, J. Konstan, and J. Riedl. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work, pages 241–250. ACM, 2000. 18