Knowledge-based recommenders support an easier comprehension of complex item assortments (e.g., financial services and electronic equipment). In this paper we show (1) how such recommenders can be developed in a Human Computation based knowledge acquisition environment (PEOPLEVIEWS) and (2) how the resulting recommendation knowledge can be exploited in a competition-based e-Learning environment (STUDYBATTLE).
Knowledge-based recommenders [
In this paper we focus on constraint-based recommenders, i.e., recommenders that are based on explicit recommendation rules (constraints). The development of such recommenders is often a timeconsuming and error-prone process which can be primarily explained by the knowledge acquisition bottleneck: in the formalization of product domain and recommendation knowledge, misunderstandings can occur and as a result knowledge engineers encode this knowledge in an unintended fashion. The more recommenders have to be developed and maintained the higher the risk that the organization runs into a scalability problem where additional resources are needed to be able to perform knowledge engineering and maintenance.
An alternative to the hiring of additional staff for development and maintenance of recommendation knowledge bases is to change the underlying knowledge engineering paradigm. The idea of PEOPLEVIEWS is to engage domain experts more deeply into knowledge engineering tasks. We do not want to ”convert” them into technical experts but to define basic tasks (micro tasks) that are easy to understand and complete even for domain experts without the corresponding technical expertise. Micro tasks completed by users pro1 Applied Software Engineering, Institute for Software Technology, Graz University of Technology, Austria, email: ffelfernig, mjeran, stettingerg@ist.tugraz.at, fthomas.absenger, th.gruber, sarah.haas, emanuel.kirchengast, michael.schwarz, lukas.skofitsch, thomas.ulzg@student.tugraz.at. 2 The terms item and product are used synonymously throughout the paper. vide knowledge chunks that can be aggregated into a PEOPLEVIEWS recommender knowledge base.
The resulting PEOPLEVIEWS recommenders support customers (and especially in the financial services domain also sales representatives) in finding products that fit their wishes and needs. Using such a recommender, items are retrieved within the scope of a dialog (these systems are often also denoted as conversational) where users articulate their requirements and the system tries to identify corresponding solutions. Major advantages of such systems are reduced error rates in the phase of order acquisition, more time that can be invested in contacting new customers due to fewer errors, more satisfied customers, and also pre-informed customers due to the fact that recommender applications can be made publicly available.
Knowledge-based recommender systems have been applied in
various item domains – due to the diversity of applications, we can
only give some examples of applications of these systems. In the
financial services domain, for example, the following applications of
knowledge-based recommendation technologies are reported in the
literature. Felfernig et al. [
Felfernig et al. [
In PEOPLEVIEWS, principles of Human Computation [
The major contributions of this paper are the following. First, we show how financial service recommender knowledge bases can be developed by a community of domain experts. Second, we sketch how such knowledge bases can also be exploited for teaching advisory practices on the basis of games (STUDYBATTLE environment). Third, we provide a discussion of major issues for future research.
The remainder of this paper is organized as follows. In Section 2 we introduce basic concepts of Human Computation based knowledge construction. To give an impression of the PEOPLEVIEWS and the STUDYBATTLE user interface, we present example screenshots in Section 3. Preliminary results of empirical evaluations are shortly discussed in Section 4. In Section 5 we provide an overview of issues for future work. We conclude the paper with Section 6. 2
The PEOPLEVIEWS environment supports two basic modes of interaction. First, recommender applications can be created in the modeling mode and second, the applications can be executed in the recommendation mode. In this section we discuss different tasks to be performed in order to create a PEOPLEVIEWS recommender. Table 1 provides an overview of the users of our working example. These users will jointly develop a PEOPLEVIEWS recommender.
user Andrea Mary
Luc Torsten
pwd **** ***** ****** ****
Table 2 contains an overview of items (financial services) that are used in our working example. The Investment Funds (A and B) have a higher risk of loss and require that customers have a high willingness to take risks, otherwise these services will not be recommended. Building Loan, Bond, and Savings Book are lower-risk items. In the current version of PEOPLEVIEWS, items can be characterized by additional item attributes, however, these attributes are not used by recommendation rules constructed from micro contributions.
In PEOPLEVIEWS, user requirements reqi 2 REQ are specified as assignments of user attributes. For our financial services recommender we define a set of user attributes which are enumerated in Table 3. In the current version of the system, user attributes are defined by the creators of a recommender application, i.e., attribute definitions can not be extended by other users who contribute to the further
In the PEOPLEVIEWS recommendation mode, user attributes can be used to specify user (customer) requirements reqi 2 REQ. In the modeling mode, user attributes represent a central element of a micro task: given a certain item, users are asked to estimate which values of user attributes are compatible with the item, i.e., are a criteria for selecting and recommending the item. The evaluation of items with regard to user attributes is the central micro task implemented in the current PEOPLEVIEWS prototype. A detailed evaluation of the example items (Table 2) regarding the user attributes goal, runtime, and risk is provided in Table 4.
Each row of Table 4 specifies a so-called user-specific filter
constraint [
In order to derive recommendation-relevant filter constraints
(recommendation rules) [
Table 5 includes the complete set of recommendation-relevant filter constraints (recommendation rules). Exactly these conditions are applied by PEOPLEVIEWS to determine recommendations for a user. In PEOPLEVIEWS, each item has exactly one related recommendation-relevant filter constraint; each such filter constraint is represented by one row in Table 5. The general logical representation of a recommendation-relevant filter constraint f for an item is shown in Formula 1. In this context, values( ; u) is the set of
goal in 5-10 years, in 10-20
years in 5-10 years, in 10-20
years in 5-10 years, in 10-20
years in 3-5 years, in 5-10 years,
in 10-20 years in 3-5 years, in 5-10 years,
in 10-20 years in 5-10 years, in 10-20
years in 5-10 years in 5-10 years in 2 years, in 3-5 years, in
5-10 years in 1 year, in 2 years, in 3-5
years, in 5-10 years in 1 year, in 2 years, in 3-5 years, in 5-10 years risk high high high high supported domain values of user attribute u 2 U (see Table 4). The constant noval denotes the fact that no value has been selected for the corresponding user attribute.
f ( ) : ^ u 2 values( ; u) [ fnovalg ! include( ) (1)
For each pair ( ; val 2 values( ; u)), PEOPLEVIEWS determines a corresponding support value (see Formula 2). In this context, occurrence( ; val) denotes the number of times, value val occurs in a user-specific filter constraint for item and occurrence( ) denotes the number of times an item is referred in a user-specific filter constraint. For example, support( 1; Studies) = 13 . support( ; val) = occurrence( ; val) occurrence( ) (2)
The complete set of support values is depicted in Table 6. In PEOPLEVIEWS, an item can have an associated rating (rating( )) which represents an item evaluation with regard to quality and related services. Such a rating can be determined, for example, by calculating the average of the individual user item ratings.5 For simplicity, we do not take into account user ratings in the utility function discussed below (see Formula 3).
Depending on the requirements articulated by the current user (see, e.g., Table 7), PEOPLEVIEWS determines and ranks a set of relevant items as follows. First, recommendation-relevant filter constraints are applied to pre-select items that fulfill the user requirements REQ = freq1; req2; :::; reqkg. In our example, the set fInvestment Fund A, Building Loang would be selected by the recommendation-relevant filter constraints (see Table 5). 5 Similar to ratings provided by platforms such as amazon.com. item name
(id) Investment Fund A ( 1) Investment Fund B ( 2)
Building Loan ( 3) in 5-10 years, in 10-20
years in 3-5 years, in 5-10 years,
in 10-20 years in 5-10 years, in 10-20
years in 2 years, in 2-5 years, in
5-10 years in 1 year, in 2 years, in 3-5 years, in 5-10 years risk high
The determined recommendation set must be ranked before being presented to the user. In PEOPLEVIEWS, item ranking is based on the following utility function (see Formula 3). The utility of each item is derived from the support values of individual requirements (see Formula 2).
utility( ; REQ) = req2REQ support( ; req) (3)
The item ranking of our working example as a result of applying Formula 3 is depicted in Table 8. For example, utility( 3,REQ = fgoal = Studies, goal = Pension, runtime = in 5-10 years, risk = mediumg) = support( 3, goal = Studies) + support( 3, goal = Pension) + support( 3, runtime = in 5-10 years) + support( 3, risk = medium) = 1.0 + 1.0 + 1.0 + 1.0 = 4.0.
In this section we discuss the PEOPLEVIEWS user interface6 and also show how PEOPLEVIEWS recommendation knowledge can be exploited by the STUDYBATTLE learning environment. The PEOPLEVIEWS homescreen is depicted in Figure 1. For applying PEOPLEVIEWS recommenders, there is no explicit need for being logged in. Recommenders can be selected and activated directly from the homescreen (see the tag cloud in Figure 1).
If users are logged in, they are allowed to contribute to the development of PEOPLEVIEWS recommender applications. Only the creators of a recommender application are allowed to define user attributes. Other users can complete micro tasks in terms of evaluating items with regard to a defined set of user attributes. The list of user attributes used in our working example is depicted in Figure 2 (corresponds to the entries of Table 3). the recommender can directly be executed. The user interface of our financial services recommender is depicted in Figure 5. product knowledge and sales practices. Examples of STUDYBATTLE games are the following.
Assign Properties. Figure 6 depicts an example user interface of a STUDYBATTLE application that implements a quiz related to knowledge about the relationship between user attributes and items. In the example, users have the task to assign items on the left hand side to user attribute values on the right hand side where each product has to be assigned to at least one attribute value and vice-versa.
Find Items. A different version of the game depicted in Figure 6 is to ask for products that fulfill certain criteria (represented by a combination of user attribute settings).
Find Incompatibilities. This game focuses on combinations of user attribute values that do not lead to a solution, i.e., users have to specify combinations of user attribute values from which they think that no corresponding solution could be found.
Maximize Requirements. The task is to identify minimal sets of
requirements (from a given set of requirements REQ) that have to
be deleted from REQ such that the remaining requirements lead to
at least one solution. This game type reflects the principles of
modelbased diagnosis [
Maximize Items. A similar task is focused on the repair of item
sets; in this context the task of users is to identify a maximal set of
items from a given set of items such that there exists at least one
combination of user attribute values that lead to these items (not
necessarily exclusively). An additional criteria could be that at least n
items from the original item list must remain in the result set.
Recommendation-relevant filter constraints can be further exploited
for generating different learning applications that are part of the
STUDYBATTLE environment. STUDYBATTLE is a game-based
learning environment which can be utilized as an environment for learning
Human Computation based Knowledge Acquisition. Applying
Human Computation concepts [
Usability. An initial user study has been conducted with an early
version of PEOPLEVIEWS at the Graz University of Technology [
The major goal of this paper was to provide an overview of the PEOPLEVIEWS recommendation environment. There are many issues for future work that we want to tackle and integrate corresponding solutions in upcoming PEOPLEVIEWS versions.
Weighting of Item Evaluations. In the current PEOPLEVIEWS version it is possible to assign user attribute values to items, i.e., to specify which criteria are relevant for the selection of a certain item. In future versions of PEOPLEVIEWS it will be possible to integrate weights into item evaluations. This maybe does not play a major role in financial service related recommender applications but can be important in other domains were nuances and personal tastes play a more important role. For example, in the context of recommending digital cameras, it can be important to specify degrees regarding certain camera properties, for example, the degree to which a camera is able to support sports photography.
Further Micro Tasks. In the current system version, the only micro task to be completed is to define the relationship (compatibility properties) between items and corresponding user attribute values. In future versions of PEOPLEVIEWS we will extend this list of micro tasks (see Table 9).
User Selection for Micro Tasks. An important enhancement will be the inclusion of methods that automatically select users for a given set of micro tasks and also take into account fairness in the distribution of micro tasks. As detected in our initial studies, users are willing to contribute to the further development of PEOPLEVIEWS recommenders. An important issue in this context is to find the users with the right expertise for certain tasks and also to not overload users. Our approach in this context will be to maintain user profiles which are derived from observing the activities of a user within PEOPLEVIEWS. For example, if a user selects a certain item when interacting with the financial services recommender, the keywords extracted from the corresponding item description are stored in the user profile. If (in the future) micro tasks related to similar items (items with a similar description) have to be completed, users with expertise regarding such items will be the preferred contact persons.
Games. Games will be another mechanism for data collection in the PEOPLEVIEWS modeling mode. A single user game will be included that is quiz-based. The overall goal is to guess user attribute settings correctly that best describe a certain item. In a second game two users will jointly try to figure out user attribute values that best describe shown items. The more matching item evaluations exist the better the team performs.
Dependencies between User Attributes and Item Attributes. An extension of the current PEOPLEVIEWS version will be the possibility to identify direct relationships between user attribute values and technical product properties. This is not the case in the current PEOPLEVIEWS version since dependencies are only defined between user attribute values and items.
Recommendation Algorithms. The current version of PEOPLEVIEWS relies on the discussed recommendation-relevant filter constraints – item ranking is based on a utility-based evaluation (see Formula 3). In future versions of PEOPLEVIEWS we will extend the quality of recommendation algorithms by, for example, adapting the determination of support values. If, for example, additional information about the performance of a certain user is available (e.g., performance with regard to correctly completed micro tasks in the past), this information can be used to increase/decrease the weight of a user when determining support values. Finally, when users are specifying their requirements, future versions of PEOPLEVIEWS will allow the specification of preferences (weights) which indicate user preferences regarding certain requirements. This will also include approaches to the learning of weights (users should not have to specify all weights explicitly).
Inconsistency Management. Given a set of customer requirements
it could be the case that no solution can be presented to the user. In
upcoming versions of PEOPLEVIEWS we will focus on integrating
state-of-the-art diagnosis algorithms that help to automatically
determine repair actions in such inconsistent situations [
Quality Management. The major task of quality management is to assure the quality of the dataset collected on the basis of different micro tasks. Quality assurance must be capable of detecting and preventing manipulations of the dataset (also under the assumption that anonymous users are allowed to complete micro tasks), it must also identify changes to the given set of user-specific filter constraints that help to improve the prediction quality of recommendation algorithms. Quality assurance is also responsible for the generation of micro tasks that need to be completed in order to improve the overall quality of the PEOPLEVIEWS datasets. The micro tasks generated by quality assurance are summarized as an agenda – this agenda is forwarded to micro task scheduling that is responsible for distributing micro tasks to the PEOPLEVIEWS user community. 6
In this paper we gave an overview of the PEOPLEVIEWS recommendation environment which exploits concepts of Human Computation to integrate domain experts more deeply into knowledge base development and maintenance processes. PEOPLEVIEWS knowledge bases can be exploited to generate learning applications which can be used in the STUDYBATTLE environment. A major focus of this paper was to show how PEOPLEVIEWS can be applied in the context of financial service recommendation. The concepts presented in this paper have the potential to avoid scalability issues which already exist in many knowledge-based environments due to the increasing size and complexity of knowledge bases.