Human Computation Based Acquisition Of Financial Service Advisory Practices Alexander Felfernig1 and Michael Jeran1 and Martin Stettinger1 and Thomas Absenger1 and Thomas Gruber1 and Sarah Haas1 and Emanuel Kirchengast1 and Michael Schwarz1 and Lukas Skofitsch1 and Thomas Ulz1 Abstract. Knowledge-based recommenders support an easier com- vide knowledge chunks that can be aggregated into a P EOPLE V IEWS prehension of complex item assortments (e.g., financial services recommender knowledge base. and electronic equipment). In this paper we show (1) how such The resulting P EOPLE V IEWS recommenders support customers recommenders can be developed in a Human Computation based (and especially in the financial services domain also sales representa- knowledge acquisition environment (P EOPLE V IEWS) and (2) how tives) in finding products that fit their wishes and needs. Using such a the resulting recommendation knowledge can be exploited in a recommender, items are retrieved within the scope of a dialog (these competition-based e-Learning environment (S TUDY BATTLE). systems are often also denoted as conversational) where users articu- late their requirements and the system tries to identify corresponding solutions. Major advantages of such systems are reduced error rates 1 Introduction in the phase of order acquisition, more time that can be invested in contacting new customers due to fewer errors, more satisfied cus- Knowledge-based recommenders [2] support users on the basis of tomers, and also pre-informed customers due to the fact that recom- semantic knowledge about the item (product) domain.2 One vari- mender applications can be made publicly available. ant of knowledge-based recommenders are constraint-based recom- Knowledge-based recommender systems have been applied in var- menders [8] which exploit explicit constraints (rules) that encode the ious item domains – due to the diversity of applications, we can recommendation knowledge. Another variant are critiquing-based only give some examples of applications of these systems. In the recommenders [4]: new items are presented to the user as long as financial services domain, for example, the following applications of the user is unsatisfied and articulates critiques (e.g., an item should knowledge-based recommendation technologies are reported in the be cheaper). In critiquing-based recommendation, new items are de- literature. Felfernig et al. [11, 12] show an application in the con- termined by similarity functions. For a detailed overview of recom- text of investment decisions where recommenders are provided to mendation approaches we refer to [3, 20]. sales representatives who exploit the recommenders in sales dialogs. In this paper we focus on constraint-based recommenders, i.e., rec- Time savings are reported as one of the major improvements directly ommenders that are based on explicit recommendation rules (con- related to the application of recommendation technologies. Another straints). The development of such recommenders is often a time- application of knowledge-based technologies in financial services is consuming and error-prone process which can be primarily explained presented by Fano and Kurth [7] who introduce a simulation envi- by the knowledge acquisition bottleneck: in the formalization of ronment that can directly visualize the effects of financial decisions product domain and recommendation knowledge, misunderstandings on the financial situation of a family. can occur and as a result knowledge engineers encode this knowledge Felfernig et al. [9] present a digital camera recommender de- in an unintended fashion. The more recommenders have to be devel- ployed on a large Austrian product comparison platform. Peischl oped and maintained the higher the risk that the organization runs et al. [22] show the application of constraint-based recommenda- into a scalability problem where additional resources are needed to tion technologies in the domain of software effort estimation. W EE - be able to perform knowledge engineering and maintenance. V IS[25]3 is a MediaWiki4 based environment for the development An alternative to the hiring of additional staff for development and maintenance of constraint-based recommender applications – and maintenance of recommendation knowledge bases is to change a couple of freely available recommenders have already been de- the underlying knowledge engineering paradigm. The idea of P EO - ployed. Knowledge-based technologies for the recommendation of PLE V IEWS is to engage domain experts more deeply into knowledge business plans are introduced by Jannach and Bundgaard-Joergensen engineering tasks. We do not want to ”convert” them into techni- [19]. The recommendation of equipment configuration in the con- cal experts but to define basic tasks (micro tasks) that are easy to text of smarthomes is introduced by Leitner et al. [21]. Technologies understand and complete even for domain experts without the cor- that recommend changes in software development practices are in- responding technical expertise. Micro tasks completed by users pro- troduced by Pribik and Felfernig [23]. Finally, Burke and Ramezani 1 Applied Software Engineering, Institute for Software Technol- [5] show how to select recommendation algorithms by introducing ogy, Graz University of Technology, Austria, email: {felfernig, rules for recommending recommenders. mjeran, stettinger}@ist.tugraz.at, {thomas.absenger, th.gruber, sarah.haas, emanuel.kirchengast, michael.schwarz, lukas.skofitsch, thomas.ulz}@student.tugraz.at. 3 www.weevis.org. 2 The terms item and product are used synonymously throughout the paper. 4 www.mediawiki.org. In P EOPLE V IEWS, principles of Human Computation [26] are id item name included into the development of knowledge-based recommenders. Φ1 Investment Fund A The idea of Human Computation is to let persons perform tasks in Φ2 Investment Fund B which they are better than computers, for example, the identification Φ3 Building Loan of product properties from a website. In the context of knowledge Φ4 Bond Φ5 Savings Book base development and maintenance the idea is to let domain experts perform tasks they are much better in compared to knowledge engi- neers who typically have less knowledge about the product domain Table 2. Example set of items used in working example. and thus relieve the work of knowledge engineers. M ATCHIN [18] is based on the idea of preference elicitation by asking users what development of the application on the basis of micro tasks. a person would typically prefer when having to choose between al- ternatives. Compared to this work, P EOPLE V IEWS allows to derive user attribute question to user attribute domain constraint-based recommenders which are the basis for intelligent What are your {Studies, Pension, Speculation, goal (gl) user interfaces that support, for example, deep explanations [17] and personal goals? Car, House, World trip, noval} {in 1 year, in 2 years, in 3-5 years, the diagnosis and repair of inconsistent requirements [13, 14]. When is the runtime (rt) in 5-10 years, in 10-20 years, in The major contributions of this paper are the following. First, we money needed? more than 20 years, noval} show how financial service recommender knowledge bases can be Preparedness to risk (ri) {low, medium, high, noval} developed by a community of domain experts. Second, we sketch take risks? how such knowledge bases can also be exploited for teaching advi- sory practices on the basis of games (S TUDY BATTLE environment). Table 3. User attributes u ∈ U of example financial services Third, we provide a discussion of major issues for future research. recommender. The remainder of this paper is organized as follows. In Section 2 we introduce basic concepts of Human Computation based knowl- In the P EOPLE V IEWS recommendation mode, user attributes can edge construction. To give an impression of the P EOPLE V IEWS and be used to specify user (customer) requirements reqi ∈ REQ. In the S TUDY BATTLE user interface, we present example screenshots the modeling mode, user attributes represent a central element of a in Section 3. Preliminary results of empirical evaluations are shortly micro task: given a certain item, users are asked to estimate which discussed in Section 4. In Section 5 we provide an overview of issues values of user attributes are compatible with the item, i.e., are a crite- for future work. We conclude the paper with Section 6. ria for selecting and recommending the item. The evaluation of items with regard to user attributes is the central micro task implemented 2 Developing P EOPLE V IEWS Recommenders in the current P EOPLE V IEWS prototype. A detailed evaluation of the example items (Table 2) regarding the user attributes goal, runtime, The P EOPLE V IEWS environment supports two basic modes of inter- and risk is provided in Table 4. action. First, recommender applications can be created in the mod- Each row of Table 4 specifies a so-called user-specific filter con- eling mode and second, the applications can be executed in the rec- straint [10], i.e., a filter constraint (specified by a user) regarding a ommendation mode. In this section we discuss different tasks to be specific item. For example, user Luc specified Pension and Specu- performed in order to create a P EOPLE V IEWS recommender. Table lation as possible goals that lead to an inclusion of the item Invest- 1 provides an overview of the users of our working example. These ment Fund B into a recommendation. Furthermore, Luc believes that users will jointly develop a P EOPLE V IEWS recommender. a user should have a high preparedness to take risks (attribute risk) and should need the payment in 3-5 years, 5-10 years or 10-20 years user email pwd from now on. Semantically, an item X is selected by a user-specific Andrea andrea@... **** filter constraint if all the preconditions are fulfilled. Mary mary@... ***** Luc luc@... ****** In order to derive recommendation-relevant filter constraints (rec- Torsten torsten@... **** ommendation rules) [10]), user-specific filter constraints have to be aggregated. An example of this aggregation step is depicted in Table 5. For each item all related user-specific filter constraints are inte- Table 1. Example users of P EOPLE V IEWS environment. grated into one constraint. Each row in this table has to be interpreted as a filter constraint for a specific item, for example, the constraint Table 2 contains an overview of items (financial services) that are in the first row of Table 5 is the following. The item Φ1 (Investment used in our working example. The Investment Funds (A and B) have Fund A) is included (recommended) if the user requirements regard- a higher risk of loss and require that customers have a high willing- ing goal (gl), runtime (rt), and risk (ri) are consistent with the condi- ness to take risks, otherwise these services will not be recommended. tion of the recommendation-relevant filter constraint gl ∈ {Studies, Building Loan, Bond, and Savings Book are lower-risk items. In the Pension, Speculation, noval} ∧ rt ∈ {in 5-10 year, in 10-20 years, current version of P EOPLE V IEWS, items can be characterized by ad- noval} ∧ ri ∈ {medium, high, noval} → include(Φ1 ). ditional item attributes, however, these attributes are not used by rec- Table 5 includes the complete set of recommendation-relevant ommendation rules constructed from micro contributions. filter constraints (recommendation rules). Exactly these conditions In P EOPLE V IEWS, user requirements reqi ∈ REQ are specified are applied by P EOPLE V IEWS to determine recommendations for as assignments of user attributes. For our financial services recom- a user. In P EOPLE V IEWS, each item has exactly one related mender we define a set of user attributes which are enumerated in Ta- recommendation-relevant filter constraint; each such filter constraint ble 3. In the current version of the system, user attributes are defined is represented by one row in Table 5. The general logical represen- by the creators of a recommender application, i.e., attribute defini- tation of a recommendation-relevant filter constraint f for an item tions can not be extended by other users who contribute to the further Φ is shown in Formula 1. In this context, values(Φ, u) is the set of user item name (id) goal runtime risk Studies, Pension, in 5-10 years, in 10-20 Andrea Investment Fund A (Φ1 ) high Speculation years in 5-10 years, in 10-20 Luc Investment Fund A (Φ1 ) Pension, Speculation high years in 5-10 years, in 10-20 Mary Investment Fund A (Φ1 ) Pension, Speculation medium, high years in 3-5 years, in 5-10 years, Torsten Investment Fund B (Φ2 ) Pension, Speculation high in 10-20 years in 3-5 years, in 5-10 years, Luc Investment Fund B (Φ2 ) Pension, Speculation high in 10-20 years Studies, Pension, Car, in 5-10 years, in 10-20 Mary Building Loan (Φ3 ) low, medium, high House years Studies, Pension, Car, Andrea Building Loan (Φ3 ) in 5-10 years low, medium House Studies, Pension, Car, Luc Building Loan (Φ3 ) in 5-10 years low, medium House in 2 years, in 3-5 years, in Mary Bond (Φ4 ) Studies, Car, House low, medium 5-10 years Studies, Car, House, World in 1 year, in 2 years, in 3-5 Andrea Savings Book (Φ5 ) low trip years, in 5-10 years in 1 year, in 2 years, in 3-5 Torsten Savings Book (Φ5 ) Studies, House, World trip low years, in 5-10 years Table 4. Example of user-specific filter constraints (= micro contributions). supported domain values of user attribute u ∈ U (see Table 4). The constant noval denotes the fact that no value has been selected for the corresponding user attribute. item name attribute:value support value (id) Investment ^ goal: Studies 0.33 f (Φ) : u ∈ values(Φ, u) ∪ {noval} → include(Φ) (1) Fund A (Φ1 ) goal: Pension, Speculation 1.0 u∈U runtime: in 5-10 years, in 10-20 years 1.0 risk: medium 0.33 For each pair (Φ, val ∈ values(Φ, u)), P EOPLE V IEWS deter- risk: high 1.0 mines a corresponding support value (see Formula 2). In this context, Investment occurrence(Φ, val) denotes the number of times, value val occurs goal: Pension, Speculation 1.0 Fund B (Φ2 ) in a user-specific filter constraint for item Φ and occurrence(Φ) de- runtime: in 3-5 years, in 5-10 years, in 1.0 notes the number of times an item Φ is referred in a user-specific 10-20 years risk:high 1.0 filter constraint. For example, support(Φ1 , Studies) = 13 . Building goal: Studies, Pension, Car, House 1.0 Loan (Φ3 ) occurrence(Φ, val) runtime:in 5-10 years 1.0 support(Φ, val) = (2) occurrence(Φ) runtime:in 10-20 years 0.33 risk:low, medium 1.0 The complete set of support values is depicted in Table 6. In P EO - risk:high 0.33 PLE V IEWS, an item Φ can have an associated rating (rating(Φ)) Bond (Φ4 ) goal: Studies, Car, House 1.0 which represents an item evaluation with regard to quality and related runtime:in 2 years, in 3-5 years, in 5-10 1.0 services. Such a rating can be determined, for example, by calculat- years risk:low, medium 1.0 ing the average of the individual user item ratings.5 For simplicity, we Savings do not take into account user ratings in the utility function discussed goal: Studies, House, World trip 1.0 Book (Φ5 ) below (see Formula 3). goal:Car 0.5 Depending on the requirements articulated by the current user runtime:in 1 year, in 2 years, in 3-5 1.0 (see, e.g., Table 7), P EOPLE V IEWS determines and ranks a set years, in 5-10 years risk:low 1.0 of relevant items as follows. First, recommendation-relevant fil- ter constraints are applied to pre-select items that fulfill the user requirements REQ = {req1 , req2 , ..., reqk }. In our example, the Table 6. Support values (see Formula 2) derived from user-specific filter set {Investment Fund A, Building Loan} would be selected by the constraints (see Table 4). recommendation-relevant filter constraints (see Table 5). 5 Similar to ratings provided by platforms such as amazon.com. item name (id) goal runtime risk Investment Studies, Pension, in 5-10 years, in 10-20 medium, high Fund A (Φ1 ) Speculation years Investment in 3-5 years, in 5-10 years, Pension, Speculation high Fund B (Φ2 ) in 10-20 years Building Loan Studies, Pension, Car, in 5-10 years, in 10-20 low, medium, high (Φ3 ) House years in 2 years, in 2-5 years, in Bond (Φ4 ) Studies, Car, House low, medium 5-10 years Savings Book Studies, Car, House, World in 1 year, in 2 years, in 3-5 low (Φ5 ) trip years, in 5-10 years Table 5. Example of recommendation-relevant filter constraints which are the result of integrating user-specific filter constraints (see Table 4). id requirement If users are logged in, they are allowed to contribute to the de- req1 goal = Studies velopment of P EOPLE V IEWS recommender applications. Only the req2 goal = Pension creators of a recommender application are allowed to define user at- req3 runtime = in 5-10 years tributes. Other users can complete micro tasks in terms of evaluating req4 risk = medium items with regard to a defined set of user attributes. The list of user attributes used in our working example is depicted in Figure 2 (cor- Table 7. Example set of user requirements (reqi ∈ REQ). responds to the entries of Table 3). The determined recommendation set must be ranked before being presented to the user. In P EOPLE V IEWS, item ranking is based on the following utility function (see Formula 3). The utility of each item is derived from the support values of individual requirements (see Formula 2). utility(Φ, REQ) = Σreq∈REQ support(Φ, req) (3) The item ranking of our working example as a result of apply- ing Formula 3 is depicted in Table 8. For example, utility(Φ3 ,REQ = {goal = Studies, goal = Pension, runtime = in 5-10 years, risk = medium}) = support(Φ3 , goal = Studies) + support(Φ3 , goal = Pension) + support(Φ3 , runtime = in 5-10 years) + support(Φ3 , risk = medium) = 1.0 + 1.0 + 1.0 + 1.0 = 4.0. item name (id) utility rank Building Loan (Φ3 ) 4.0 1 Investment Fund A (Φ1 ) 2.66 2 Table 8. Utility-based ranking of items in the recommendation set. 3 User Interface Figure 1. P EOPLE V IEWS homescreen – the current version of the user 3.1 P EOPLE V IEWS interface is provided in German. The homescreen explains the basic functionalities of the system (development, maintenance, and execution of In this section we discuss the P EOPLE V IEWS user interface6 and also recommender applications). show how P EOPLE V IEWS recommendation knowledge can be ex- Logged-in users are also allowed to enter new items to the recom- ploited by the S TUDY BATTLE learning environment. The P EOPLE - mender product catalog. The P EOPLE V IEWS representation of prod- V IEWS homescreen is depicted in Figure 1. For applying P EOPLE - uct catalogs is exemplified in Figure 3 (corresponds to the list of V IEWS recommenders, there is no explicit need for being logged in. items shown in Table 2). Recommenders can be selected and activated directly from the home- The interface for evaluating an item with regard to a set of user screen (see the tag cloud in Figure 1). attributes is depicted in Figure 4. The screenshot depicts the evalu- ation of Building Loan with regard to the user attribute goal. After 6 The user interface is currently only available in German. having completed the definition of a P EOPLE V IEWS recommender, product knowledge and sales practices. Examples of S TUDY BATTLE games are the following. Assign Properties. Figure 6 depicts an example user interface of a S TUDY BATTLE application that implements a quiz related to knowl- edge about the relationship between user attributes and items. In the example, users have the task to assign items on the left hand side to user attribute values on the right hand side where each product has to be assigned to at least one attribute value and vice-versa. Find Items. A different version of the game depicted in Figure 6 is to ask for products that fulfill certain criteria (represented by a combination of user attribute settings). Figure 2. P EOPLE V IEWS: example user attributes. Find Incompatibilities. This game focuses on combinations of user attribute values that do not lead to a solution, i.e., users have to spec- ify combinations of user attribute values from which they think that no corresponding solution could be found. Maximize Requirements. The task is to identify minimal sets of requirements (from a given set of requirements REQ) that have to be deleted from REQ such that the remaining requirements lead to at least one solution. This game type reflects the principles of model- based diagnosis [6, 24], i.e., support users in learning and improving repair behavior in situations where no solution can be identified. Maximize Items. A similar task is focused on the repair of item sets; in this context the task of users is to identify a maximal set of items from a given set of items such that there exists at least one combination of user attribute values that lead to these items (not nec- essarily exclusively). An additional criteria could be that at least n items from the original item list must remain in the result set. Figure 3. P EOPLE V IEWS: example of an item list. the recommender can directly be executed. The user interface of our financial services recommender is depicted in Figure 5. Figure 4. P EOPLE V IEWS: example of an item evaluation user interface Figure 6. S TUDY BATTLE ”Assign Properties” learning application. The (evaluation of item Building Loan with regard to the user attribute goal). task of the user is to relate items with corresponding attribute values. 3.2 S TUDY BATTLE 4 Preliminary Evaluation Results Recommendation-relevant filter constraints can be further exploited Human Computation based Knowledge Acquisition. Applying Hu- for generating different learning applications that are part of the man Computation concepts [26] in the context of recommender ap- S TUDY BATTLE environment. S TUDY BATTLE is a game-based learn- plication development and maintenance has the potential to lift the ing environment which can be utilized as an environment for learning burden of enormous engineering and maintenance efforts from the Figure 5. P EOPLE V IEWS: example of a recommender application (Financial Services). shoulder of knowledge engineers. Micro tasks as sketched in this Weighting of Item Evaluations. In the current P EOPLE V IEWS ver- paper can be structured in a way that they are understandable for sion it is possible to assign user attribute values to items, i.e., to domain experts without a computer science background. Knowledge specify which criteria are relevant for the selection of a certain item. gained from completed micro tasks can be easily integrated into a In future versions of P EOPLE V IEWS it will be possible to integrate corresponding recommender knowledge base. Due to the increas- weights into item evaluations. This maybe does not play a major role ing size and complexity of knowledge bases, the development of in financial service related recommender applications but can be im- such technologies is crucial since they help to tackle scalability is- portant in other domains were nuances and personal tastes play a sues which otherwise could cause a complete failure with regard to a more important role. For example, in the context of recommending company-wide recommender deployment. As such, P EOPLE V IEWS digital cameras, it can be important to specify degrees regarding cer- technologies can be considered as a first step towards more scalable tain camera properties, for example, the degree to which a camera is development methods that will also help to further increase the pop- able to support sports photography. ularity of knowledge-based (recommendation) technologies. Further Micro Tasks. In the current system version, the only mi- Usability. An initial user study has been conducted with an early cro task to be completed is to define the relationship (compatibility version of P EOPLE V IEWS at the Graz University of Technology [10]. properties) between items and corresponding user attribute values. N=161 (15% female and 85% male) students interacted with the sys- In future versions of P EOPLE V IEWS we will extend this list of micro tem with the goal to develop different recommender applications. Af- tasks (see Table 9). ter having completed the development, the study participants had to User Selection for Micro Tasks. An important enhancement will be complete a questionnaire which was based on the system usability the inclusion of methods that automatically select users for a given scale (SUS) [1]. Evaluation results regarding the SUS aspects are set of micro tasks and also take into account fairness in the distribu- summarized in Figure 7. Besides usability questions, further feed- tion of micro tasks. As detected in our initial studies, users are willing back has been provided by the study participants, for example, the to contribute to the further development of P EOPLE V IEWS recom- majority of the participants (69% of all study participants) would menders. An important issue in this context is to find the users with like to further contribute to P EOPLE V IEWS recommenders. 56% out the right expertise for certain tasks and also to not overload users. of those participants who wanted to contribute agreed to contribute Our approach in this context will be to maintain user profiles which within a time frame of less than 30 minutes per week. are derived from observing the activities of a user within P EOPLE - V IEWS. For example, if a user selects a certain item when interact- ing with the financial services recommender, the keywords extracted 5 Future Work from the corresponding item description are stored in the user pro- file. If (in the future) micro tasks related to similar items (items with The major goal of this paper was to provide an overview of the P EO - a similar description) have to be completed, users with expertise re- PLE V IEWS recommendation environment. There are many issues for garding such items will be the preferred contact persons. future work that we want to tackle and integrate corresponding solu- Games. Games will be another mechanism for data collection in tions in upcoming P EOPLE V IEWS versions. Figure 7. Results of a SUS-based usability study [1] of the P EOPLE V IEWS environment. the P EOPLE V IEWS modeling mode. A single user game will be in- cluded that is quiz-based. The overall goal is to guess user attribute settings correctly that best describe a certain item. In a second game two users will jointly try to figure out user attribute values that best describe shown items. The more matching item evaluations exist the better the team performs. Dependencies between User Attributes and Item Attributes. An ex- name description tension of the current P EOPLE V IEWS version will be the possibility check whether a certain item belongs to to identify direct relationships between user attribute values and tech- item quality check a specific recommender (is an existing nical product properties. This is not the case in the current P EOPLE - recommender-related item) V IEWS version since dependencies are only defined between user check whether a certain attribute attribute values and items. belongs to to a specific recommender attribute quality check Recommendation Algorithms. The current version of P EOPLE - (user attribute or item attribute exists in the item domain) V IEWS relies on the discussed recommendation-relevant filter con- check whether a certain value belongs straints – item ranking is based on a utility-based evaluation (see attribute value quality to the domain of an attribute (user Formula 3). In future versions of P EOPLE V IEWS we will extend the check attribute or item attribute) quality of recommendation algorithms by, for example, adapting the check whether a certain figure belongs graphic check to a certain item determination of support values. If, for example, additional infor- evaluate item assign user attribute values to items mation about the performance of a certain user is available (e.g., attribute value utility derive a ranking that shows which items performance with regard to correctly completed micro tasks in the check best support a user attribute value past), this information can be used to increase/decrease the weight of a user when determining support values. Finally, when users are Table 9. Example list of micro tasks to be integrated in P EOPLE V IEWS. specifying their requirements, future versions of P EOPLE V IEWS will allow the specification of preferences (weights) which indicate user preferences regarding certain requirements. This will also include ap- proaches to the learning of weights (users should not have to specify all weights explicitly). Inconsistency Management. Given a set of customer requirements it could be the case that no solution can be presented to the user. In upcoming versions of P EOPLE V IEWS we will focus on integrating state-of-the-art diagnosis algorithms that help to automatically deter- mine repair actions in such inconsistent situations [15]. These repairs will take into account user weights (preferences) and thus minimize [11] A. Felfernig, K. Isak, K. Szabo, and P. Zachar, ‘The VITA Finan- the number of interaction cycles needed to find a reasonable solu- cial Services Sales Support Environment’, pp. 1692–1699, Vancouver, Canada, (2007). tion. In addition to this more intelligent management of inconsistent [12] A. Felfernig and A. Kiener, ‘Knowledge-based Interactive Selling of requirements, we will integrate mechanisms that help to consolidate Financial Services with FSAdvisor’, in 17th Innovative Applications of the set of user-specific filter constraints in order to make the result- Artificial Intelligence Conference (IAAI05), pp. 1475–1482, Pittsburgh, ing recommendation-relevant filter constraints more compact. Con- Pennsylvania, (2005). solidation will be achieved, for example, on the basis of redundancy [13] A. Felfernig, M. Schubert, G. Friedrich, M. Mandl, M. Mairitsch, and E. Teppan, ‘Plausible repairs for inconsistent requirements’, in 21st In- detection algorithms [16]. ternational Joint Conference on Artificial Intelligence (IJCAI’09), pp. Quality Management. The major task of quality management is 791–796, Pasadena, CA, (2009). to assure the quality of the dataset collected on the basis of differ- [14] A. Felfernig, M. Schubert, and S. Reiterer, ‘Personalized diagnosis for ent micro tasks. Quality assurance must be capable of detecting and over-constrained problems’, in 23rd International Conference on Arti- ficial Intelligence (IJCAI 2013), pp. 1990–1996, Peking, China. preventing manipulations of the dataset (also under the assumption [15] A. Felfernig, M. Schubert, and C. Zehentner, ‘An Efficient Diagnosis that anonymous users are allowed to complete micro tasks), it must Algorithm for Inconsistent Constraint Sets’, Artificial Intelligence for also identify changes to the given set of user-specific filter constraints Engineering Design, Analysis, and Manufacturing (AIEDAM), 25(2), that help to improve the prediction quality of recommendation algo- 175–184, (2012). rithms. Quality assurance is also responsible for the generation of [16] A. Felfernig, C. Zehentner, and P. Blazek, ‘Corediag: Eliminating re- dundancy in constraint sets’. micro tasks that need to be completed in order to improve the overall [17] G. Friedrich, ‘Elimination of spurious explanations’, in European Con- quality of the P EOPLE V IEWS datasets. The micro tasks generated by ference on Artificial Intelligence (ECAI 2004), pp. 813–817, Valencia, quality assurance are summarized as an agenda – this agenda is for- Spain, (2004). warded to micro task scheduling that is responsible for distributing [18] S. Hacker and L. VonAhn, ‘Matchin: Eliciting User Preferences with an Online Game’, in CHI’09, pp. 1207–1216, (2009). micro tasks to the P EOPLE V IEWS user community. [19] D. Jannach and U. Bundgaard-Joergensen, ‘SAT: A Web-Based Inter- active Advisor for Investor-Ready Business Plans’, in Intl. Conference 6 Conclusions on e-Business (ICE-B 2007), pp. 99–106, (2007). [20] D. Jannach, M. Zanker, A. Felfernig, and G. Friedrich, Recommender In this paper we gave an overview of the P EOPLE V IEWS recommen- Systems, Cambridge University Press, 2010. dation environment which exploits concepts of Human Computation [21] G. Leitner, A. Fercher, A. Felfernig, and M. Hitz, ‘Reducing the Entry Threshold of AAL Systems: Preliminary Results from Casa Vecchia’, to integrate domain experts more deeply into knowledge base de- in 13th Intl. Conference on Computers Helping People with Special velopment and maintenance processes. P EOPLE V IEWS knowledge Needs, pp. 709–715, (2012). bases can be exploited to generate learning applications which can [22] B. Peischl, M. Zanker, M. Nica, and W. Schmid, ‘Constraint-based be used in the S TUDY BATTLE environment. A major focus of this Recommendation for Software Project Effort Estimation’, Journal of Emerging Technologies in Web Intelligence, 2(4), 282–290, (2010). paper was to show how P EOPLE V IEWS can be applied in the context [23] I. Pribik and A. Felfernig, ‘Towards Persuasive Technology for Soft- of financial service recommendation. The concepts presented in this ware Development Environments: An Empirical Study’, in Persuasive paper have the potential to avoid scalability issues which already ex- Technology Conference (Persuasive 2012), pp. 227–238, (2012). ist in many knowledge-based environments due to the increasing size [24] R. Reiter, ‘A theory of diagnosis from first principles’, AI Journal, and complexity of knowledge bases. 23(1), 57–95, (1987). [25] S. Reiterer, A. Felfernig, P. Blazek, G. Leitner, F. Reinfrank, and G. Ninaus, ‘WeeVis’, in Knowledge-based Configuration – From Re- REFERENCES search to Business Cases, eds., A. Felfernig, L. Hotz, C. Bagley, and J. Tiihonen, chapter 25, 365–376, Morgan Kaufmann Publishers, [1] A. Bangor, P. Kortum, and J. Miller, ‘An Empirical Evaluation of (2013). the System Usability Scale (SUS)’, International Journal of Human- [26] L. VonAhn, ‘Human Computation’, in Technical Report CM-CS-05- Computer Interaction, 24(6), 574–594, (2008). 193, (2005). [2] R. Burke, ‘Knowledge-based recommender systems’, Encyclopedia of Library and Information Systems, 69(32), 180–200, (2000). [3] R. Burke, A. Felfernig, and M. Goeker, ‘Recommender systems: An overview’, AI Magazine, 32(3), 13–18, (2011). [4] R. Burke and K. Hammond, ‘The FindMe Approach to Assisted Brows- ing’, IEEE Expert, 32–40, (1997). [5] R. Burke and M. Ramezani, ‘Matching recommendation technologies and domains’, in Recommender Systems Handbook, 367–386, Springer, (2010). [6] J. de Kleer, A. Mackworth, and R. Reiter, ‘Characterizing diagnoses and systems’, AI Journal, 56(197–222), 57–95, (1992). [7] A. Fano and S. Kurth, ‘Personal Choice Point: Helping Users Visualize What it Means to Buy a BMW’, in International Conference on In- telligent User Interfaces IUI’03, pp. 46–52, Miami, FL, USA, (2003). ACM, New York, USA. [8] A. Felfernig and R. Burke, ‘Constraint-based recommender systems: Technologies and research issues’, in IEEE ICEC’08, pp. 17–26, Inns- bruck, Austria, (2008). [9] A. Felfernig, G. Friedrich, D. Jannach, and M. Zanker, ‘An environment for the development of knowledge-based recommender applications’, International Journal of Electronic Commerce (IJEC), 11(2), 11–34, (2006). [10] A. Felfernig, S. Haas, G. Ninaus, M. Schwarz, T. Ulz, M. Stettinger, K. Isak, M. Jeran, and S. Reiterer, ‘RecTurk: Constraint-based Recom- mendation based on Human Computation’, in RecSys 2014 CrowdRec Workshop, pp. 1–6, Foster City, CA, USA, (2014).