The Effect of Sensitivity Analysis on the Usage of Recommender Systems

Abstract

Recommender systems have become a valuable tool for successful e-commerce. The quality of their recommendations depends heavily on how precisely consumers are able to state their preferences. However, empirical evidence has shown that the preference construction process is highly a ected by uncertainties. This has a negative impact on the robustness of recommendations. If users perceive a lack of accuracy in the recommendation of recommender systems, this reduces their con dence in the recommendation generating process. This in turn negatively in uences the adoption of recommender systems. We argue in this paper that sensitivity analysis is able to overcome this problem. Although sensitivity analysis has already been well studied, it was ignored to a large extent in the eld of recommender systems. To close this gap, we propose a research model that shows how a sensitivity analysis and the presence of uncertainties in uence decision con dence and the intention to use recommender systems.

Keywords

• Recommender systems
• sensitivity analysis
• uncertainties in preference construction
• technology acceptance

Categories and Subject Descriptors

H.3.3 [Information Search and Retrieval]; J.4 [Social and Behavioral Sciences] Paper presented at the 2012 Decisions@RecSys workshop in conjunction with the 6th ACM conference on Recommender Systems. Copyright c 2012 for the individual papers by the papers’ authors. Copying permitted for private and academic purposes. This volume is published and copyrighted by its editors.

1. INTRODUCTION

Recommender systems (RS) have become an important tool for successful e-commerce. They help consumers in ecommerce settings to overcome the problem of information overload, which they often face due to the vast amount of available products and of product-related information. From a consumers-perspective, the main task of RS is to support nding the right product. Independent from technical considerations, all RS have in common that they require information about their users in order to provide personalized recommendations. This information is basically the consumers' preferences which serve as input for the recommendation-generating algorithm [ 24 ]. Thus, the users' preferences are clearly of high importance for the quality of the RS' output and the more precise the preferences correspond to the user's \real" needs, the more accurate will be the recommendation of the system.

The problem we want to address here is that the preferences of consumers as well as their measurement are subject to irreducible arbitrariness [ 12 ], which potentially has a negative impact on the quality of a RS's recommendation and on the adoption of RS. To overcome this problem, we propose to integrate sensitivity analysis into RS. The remainder of this paper is structured as follows. The next Section describes the uncertainties related to the measurement of preferences and the implications for RS design. Section 3 provides a short overview of SA methods and possible ways to address uncertainties as well as similar problems of supporting consumers via RS. We will propose a research model in Section 4 and hypothesize how SA and uncertainties in the process of generating recommendations are related to RS usage. The planned methodology for testing our hypotheses is presented in Section 5. Finally, we provide a short discussion of our model and present further research opportunities in Section 6. 2.

UNCERTAINTY AND RECOMMENDER

SYSTEMS

Humans often face decisions which have to be made based on beliefs regarding the likelihood of uncertain events like future prices of goods or the durability of a product [ 21 ]. Here, uncertainty refers to a state of incomplete knowledge, which is usually rooted in either the individual's lack of information or in his limited resources to rationally process the available information [ 4, 18 ].

The latter source of uncertainty - limited information processing capabilities - is the rationale underlying the idea to support consumers in making their decisions by providing personalized recommendations. In this sense, it is the function of RS to mitigate the information overload which consumers often face in e-commerce settings [ 16 ]. As research in RS deals with bounded rational consumers, it has to acknowledge that consumers face uncertainties while making their purchase decisions, even if they are supported by a RS. The origins of uncertainty in a RS-facilitated purchase decision can be manifold. For example, a consumer might ask himself whether the model underlying the RS is indeed appropriate to support him or whether the complex calculations underlying a recommendation have been solved accurately or in a more heuristic way [ 4 ]. Another important source of uncertainty is the consumer. Often, it is assumed that decision makers have stable and coherent preferences and sometimes it is even supposed that they accurately know these preferences [ 9 ]. However, there is vast empirical evidence that these assumptions do not model real world decision makers very well. For example, it is commonly known that the answers of a decision maker who is requested to explicitly state his preferences are at least partly dependent on the framing of the questions and on what response is expected [ 22 ]. These and other empirically observed deviations from rationality led to the notion that humans do not have well-de ned preferences which can be elicited but that we construct preferences on the spot, usually by applying some kind of heuristic information processing strategy. Consequently, our preferences are \labile, inconsistent, subject to factors we are unaware of, and not always in our own best interests" [9, p.2].

For the e ort to support consumers with the help of RS such instable preferences pose a serious problem. RS try to support consumers by providing personalized recommendations based on the consumer's preferences. Independent of how the RS measures the preferences of the consumer (either explicitly by asking the consumer or implicitly by observing his behavior), the ad-hoc construction of preferences implies that RS have to deal with an uncertain information base to make recommendations (cf. [ 4 ]), which might lead to inaccurate and therefore unhelpful recommendations. Moreover, a consumer who faces a recommendation of a RS might perceive a state of uncertainty regarding the recommendation's quality because the choice of the recommendationgenerating algorithm, its inputs (the preferences) as well as its computation are a icted with uncertainties. The work of Lu et al. [ 10 ] shows that a major reason for the rejection of decision support technologies is that humans are skeptical whether the respective technology is indeed able to accurately model their preferences. In other words, the uncertainties related to technologically derived recommendations might hamper the adoption of RS. In order to avoid these problems, RS have to address the uncertainties related to the generation of recommendations. Here, we propose to incorporate SA into RS to overcome this challenge.

3. SENSITIVITY ANALYSIS

Sensitivity analysis is a widely used tool in various disciplines, like in chemical engineering, operations research or management science [ 20 ]. According to French [ 5 ], a common de nition of SA involves the variation of input variables to examine their e ect on the output variables. In the case of RS, inputs refer to preferences of consumers and output means the recommendation of the system. Thus, SA is a valuable tool for detecting uncertainties in inputs, veri cation and validation of models as well as demonstrating the robustness of outputs. De nition and purpose however vary depending on the eld of application [ 15 ]. Furthermore, there are di erent SA methods. They are classi ed e.g. in mathematical, statistical and graphical methods [ 6 ] or in local and global SA methods depending if the input variables are varied over a reduced range of value or over the whole domain [ 15 ]. Both classes allow to vary \one factor at a time" (OAT) or several variables simultaneously (VIC - variation in combination). Some researchers (e.g. [ 17 ]) argue that a variance-based, global SA with VIC is especially useful for comparing input variables and identifying uncertainties.

Although SA is in general a well-studied topic, it is ignored to a large extent in the eld of RS. Papers that treat SA as tool for decision support systems are typically from the eld of multi-criteria decision making. They explain for instance how SA demonstrates robust solutions or illustrates the impact of input variations [ 13 ]. A reason why SA should be integrated in decision support systems is that it addresses certain drawbacks, like a possible lack of transparency. By considering RS, this would mean that consumers do not receive the possibility to understand why a particular product was recommended. Thus, consumers are not able to detect uncertainties that were introduced during preference elicitation. As argued by [19, p. 831] \(...) users are not just looking for blind recommendations from a system, but are also looking for a justi cation of the system's choice.". A possibility to provide justi cations are explanation facilities. An approach that was found in literature is to regard SA as being similar to an explanation facility [ 14 ]. It facilitates the involvement of users and increases transparency of the recommendation generating process [ 8 ]. An integrated SA permits users to interact with the system such that they are able to explore possible variations of the inputs and see how their changes in uence the robustness of the recommendation. A SA is therefore especially important when uncertainties in the inputs are present. In contrast to the various types of explanation facilities, it is based on formal sciences and is thus capable of providing objective explanations.

RESEARCH MODEL AND HYPOTHESES

DEVELOPMENT

Based on the descriptions of the problem of uncertainties and the characteristics of SA we will derive a research model for RS usage in this Section. In order to understand how SA is related to the adoption of RS, we integrate sensitivity analysis, perceived uncertainty and decision con dence in a common model of RS usage. The de nitions of these concepts are given in Table 1. Our model builds on technology acceptance research and its most prominent model, the technology acceptance model (TAM) [ 2 ]. Figure 1 illustrates the proposed model. Sensitivity analysis represents the design feature of interest, decision con dence and perceived uncertainty are used to describe the link between the design feature and RS use in detail. The following paragraphs separately discuss each proposition of our model.

Basically, a SA can lead to two di erent results: Depending on inputs and model parameters, it will either con rm or disprove the robustness of the recommendations provided by the RS. Though we acknowledge that the output of a SA depends on the speci c situation and that the concrete outcome of the SA is likely to in uence the user's perceptions, we argue that there is also an e ect which is independent from such contingencies (see also Section 6). SA helps users to lter out those recommendations which are robust to uncertainties and which thereby represent good choices independent from changes in the inputs [ 12 ]. Therefore, we hypothesize that

H1: Sensitivity analysis will increase users' decision con dence.

The only task which RS perform is to search and suggest decision alternatives on behalf of their users. If a user is not sure whether a RS provides recommendations which match his needs or not, the only reason to use a RS vanishes. Therefore, we hypothesize that

H2: Decision con dence will positively a ect perceived usefulness of recommender systems.

SA is a tool which demonstrates how the output varies when inputs are changed. This enables user not only to analyze di erent scenarios and to search for robust recommendations but also to learn about the RS and how it generates recommendations. In this function, SA might be directly related to perceived usefulness of the RS regardless of its impact on decision con dence and independent from whether it conrms the robustness of the recommendation or not. Based on this argument and on the experiences of Payne et al. [ 12 ] that user perceive SA as a valuable tool, we hypothesize that

H3: Sensitivity analysis will positively in uence perceived usefulness of recommender systems.

We argue that this relationship is moderated by the degree of perceived uncertainty: Consider a user who does not perceive any uncertainty related to the output of a RS. For such a user a SA is of little to no value. But the more the user perceives that the recommendation generating process is prone to uncertainties, the more useful is a feature which allows to explore the impact of the uncertainties on the outcomes. Therefore, we hypothesize that

H4: Perceived uncertainty will moderate the inuence of sensitivity analysis on perceived usefulness of recommender systems.

The relationship between perceived uncertainty and decision con dence is similar to H4. If users perceive that a recommendation is based on an uncertain information base or if they are not sure about the appropriateness of the recommendation generating algorithm, they are likely not con dent about the quality of the recommendation. Therefore, we hypothesize that

H5: Perceived uncertainty will negatively in uence decision con dence.

5.

PROPOSED METHODOLOGY

We will conduct a laboratory experiment to test our hypotheses. We will use a 2 x 2 full factorial design with SA and perceived uncertainty as independent variables. Participants will be asked to use a RS for online shopping which explicitly demands from users to make trade-o s in preference construction. They will be randomly assigned to a treatment group and a control group which allows us to manipulate SA and perceived uncertainty. We will choose purchase decisions with low/high familiarity to induce high/low levels of perceived uncertainty. After nishing the shopping task, questionnaires will be delivered to the participants to assess the proposed relationships.

Before we are actually able to conduct the experiment, we will develop new measures for the constructs perceived uncertainty and decision con dence by adopting the method of Moore and Benbasat [ 11 ] for instrument development. The validity and reliability of the items will be tested by a factor analysis in a pilot test. Items for the remaining constructs will be taken from already validated scales, for instance from Davis [ 2 ] for perceived usefulness.

To test our experimental design, we will conduct a t-test in order to check the manipulation of perceived uncertainty via familiarity of the purchase task. For testing our hypotheses we will use structural equation modeling (SEM). As our study is the rst one regarding the impact of SA and uncertainty on RS usage, it has an exploratory character. To manage the risks associated with exploratory research, we will keep the sample size rather low (about 10 participants per indicator [ 1 ]). To deal with the small sample size and the exploratory character of our research, we will use a partial least squares approach (component-based SEM) [ 23 ].

DISCUSSION AND CONCLUSIONS

Based on a literature review, we have argued that the process of generating recommendations for e-commerce users involves uncertainties, especially regarding the measurement of preferences, which might lead to users who feel insecure about the quality of a RS's recommendations. Moreover, we hypothesized that if users do not feel con dent about a RS's recommendations, they will not perceive RS as useful and thus are less likely to adopt the RS. We proposed to incorporate SA into RS to overcome the problems associated with uncertainties. SA is a tool which enables users to explore how changes in the inputs of the recommendation generating process (the users' preferences) are related to changes in the output of the process (the recommendations). SA can be used to check the robustness of recommendations which should help users to build con dence in the system's advice and the decision. Finally, we proposed a conceptual model and corresponding hypotheses of how uncertainties, decision con dence and SA are related to the adoption of RS.

As outlined in Section 5 our next step is the empirical testing of the proposed model by conducting a laboratory experiment. Further research opportunities include theoretical work on how SA can be incorporated into the various forms of RS, not only on computational level but also on the level of user interface design and how the outcomes of SA are related to user perceptions.

ACKNOWLEDGMENTS

This research has been funded by the Austrian Science Fund (FWF): project number TRP 111-G11.

References

[1] J. Christopher Westland . Lower bounds on sample size in structural equation modeling . Electronic Commerce Research and Applications , 9 ( 6 ): 476 { 487 , 2010 .

[2] F. Davis . Perceived usefulness, perceived ease of use, and user acceptance of information technology . MIS Quarterly , 13 ( 3 ): 319 { 340 , 1989 .

[3] M. Fishbein and I. Ajzen . Belief, attitude, intention, and behavior: an introduction to theory and research . Addison Wesley Publishing Company, 1975 .

[4] S. French . Uncertainty and imprecision: modelling and analysis . The Journal of the Operational Research Society , 46 ( 1 ): 70 { 79 , 1995 .

[5] S. French . Modelling, making inferences and making decisions: the roles of sensitivity analysis . Top , 11 ( 2 ): 229 { 251 , 2003 .

[6] H. Frey and S. Patil . Identi cation and review of sensitivity analysis methods . Risk Analysis , 22 ( 3 ): 553 { 578 , 2002 .

[7] D. Gri n and A. Tversky . The weighing of evidence and the determinants of con dence . Cognitive Psychology , 24 ( 3 ): 411 { 435 , 1992 .

[8] J. Herlocker , J. Konstan , and J. Riedl . Explaining collaborative ltering recommendations . In Proceedings of the 2000 ACM conference on Computer supported cooperative work , pages 241 { 250 . ACM, 2000 .

[9] S. Lichtenstein and P. Slovic . The construction of preference: an overview. In The construction of preference, Lichtenstein, S. and Slovic , P. (Eds.), pages 1 { 40 . Cambridge University Press, 2006 .

[10] H. Lu , H. Yu , and S. Lu . The e ects of cognitive style and model type on dss acceptance: An empirical study . European Journal of Operational Research , 131 ( 3 ): 649 { 663 , 2001 .

[11] G. Moore and I. Benbasat. Development of an instrument to measure the perceptions of adopting an information technology innovation . Information systems research , 2 ( 3 ): 192 { 222 , 1991 .

[12] J. Payne , J. Bettman , and D. Schkade . Measuring constructed preferences: Towards a building code . Journal of Risk and Uncertainty , 19 ( 1 {3): 243 { 270 , 1999 .

[13] L. Proll , A. Salhi , and D. Rios Insua . Improving an optimization-based framework for sensitivity analysis in multi-criteria decision-making . Journal of Multi-Criteria Decision Analysis , 10 ( 1 ):1{ 9 , 2001 .

[14] W. Raskob , F. Gering , and V. Bertsch . Approaches to visualisation of uncertainties to decision makers in an operational decision support system . In Proceedings of the 6th International ISCRAM Conference , page http://www.iscram.org/ISCRAM2009/papers/, 2009 .

[15] A. Saltelli , K. Chan , and E. Scott . Sensitivity analysis . John Wiley, New York, 2000 .

[16] J. Schafer , J. Konstan , and J. Riedl . E-commerce recommendation applications . Data mining and knowledge discovery , 5 ( 1 ): 115 { 153 , 2001 .

[17] K. Siebertz , D. Van Bebber , and T. Hochkirchen . Statistische Versuchsplanung: Design of Experiments (DoE ). Springer Verlag, 2010 .

[18] H. Simon . A behavioural model of rational choice , In: Models of Man: Social and Rational . John Wiley, New York, 1957 .

[19] R. Sinha and K. Swearingen . The role of transparency in recommender systems . In CHI'02 extended abstracts on Human factors in computing systems , pages 830 { 831 . ACM, 2002 .

[20] E. Triantaphyllou and A. Sanchez . A sensitivity analysis approach for some deterministic multi-criteria decision-making methods* . Decision Sciences , 28 ( 1 ): 151 { 194 , 1997 .

[21] A. Tversky and D. Kahneman . Judgment under uncertainty: Heuristics and biases . science, 185 ( 4157 ): 1124 { 1131 , 1974 .

[22] A. Tversky and D. Kahneman . The framing of decisions and the psychology of choice . Science , 211 ( 4481 ): 453 { 458 , 1981 .

[23] N. Urbach and F. Ahlemann . Structural equation modeling in information systems research using partial least squares . Journal of Information Technology Theory and Application , 11 ( 2 ):5{ 40 , 2010 .

[24] B. Xiao and I. Benbasat . E-commerce product recommendation agents: Use, characteristics, and impact . MIS Quarterly , 31 ( 1 ): 137 { 209 , 2007 .