=Paper=
{{Paper
|id=Vol-1349/paper5
|storemode=property
|title=Human Computation Based Acquisition of Financial Service Advisory Practices
|pdfUrl=https://ceur-ws.org/Vol-1349/paper05.pdf
|volume=Vol-1349
|dblpUrl=https://dblp.org/rec/conf/finrec/FelfernigJSAGHK15
}}
==Human Computation Based Acquisition of Financial Service Advisory Practices==
https://ceur-ws.org/Vol-1349/paper05.pdf
Human Computation Based Acquisition Of
Financial Service Advisory Practices
Alexander Felfernig1 and Michael Jeran1 and Martin Stettinger1 and
Thomas Absenger1 and Thomas Gruber1 and Sarah Haas1 and Emanuel Kirchengast1 and
Michael Schwarz1 and Lukas Skofitsch1 and Thomas Ulz1
Abstract. Knowledge-based recommenders support an easier com- vide knowledge chunks that can be aggregated into a P EOPLE V IEWS
prehension of complex item assortments (e.g., financial services recommender knowledge base.
and electronic equipment). In this paper we show (1) how such The resulting P EOPLE V IEWS recommenders support customers
recommenders can be developed in a Human Computation based (and especially in the financial services domain also sales representa-
knowledge acquisition environment (P EOPLE V IEWS) and (2) how tives) in finding products that fit their wishes and needs. Using such a
the resulting recommendation knowledge can be exploited in a recommender, items are retrieved within the scope of a dialog (these
competition-based e-Learning environment (S TUDY BATTLE). systems are often also denoted as conversational) where users articu-
late their requirements and the system tries to identify corresponding
solutions. Major advantages of such systems are reduced error rates
1 Introduction in the phase of order acquisition, more time that can be invested in
contacting new customers due to fewer errors, more satisfied cus-
Knowledge-based recommenders [2] support users on the basis of
tomers, and also pre-informed customers due to the fact that recom-
semantic knowledge about the item (product) domain.2 One vari-
mender applications can be made publicly available.
ant of knowledge-based recommenders are constraint-based recom-
Knowledge-based recommender systems have been applied in var-
menders [8] which exploit explicit constraints (rules) that encode the
ious item domains – due to the diversity of applications, we can
recommendation knowledge. Another variant are critiquing-based
only give some examples of applications of these systems. In the
recommenders [4]: new items are presented to the user as long as
financial services domain, for example, the following applications of
the user is unsatisfied and articulates critiques (e.g., an item should
knowledge-based recommendation technologies are reported in the
be cheaper). In critiquing-based recommendation, new items are de-
literature. Felfernig et al. [11, 12] show an application in the con-
termined by similarity functions. For a detailed overview of recom-
text of investment decisions where recommenders are provided to
mendation approaches we refer to [3, 20].
sales representatives who exploit the recommenders in sales dialogs.
In this paper we focus on constraint-based recommenders, i.e., rec-
Time savings are reported as one of the major improvements directly
ommenders that are based on explicit recommendation rules (con-
related to the application of recommendation technologies. Another
straints). The development of such recommenders is often a time-
application of knowledge-based technologies in financial services is
consuming and error-prone process which can be primarily explained
presented by Fano and Kurth [7] who introduce a simulation envi-
by the knowledge acquisition bottleneck: in the formalization of
ronment that can directly visualize the effects of financial decisions
product domain and recommendation knowledge, misunderstandings
on the financial situation of a family.
can occur and as a result knowledge engineers encode this knowledge
Felfernig et al. [9] present a digital camera recommender de-
in an unintended fashion. The more recommenders have to be devel-
ployed on a large Austrian product comparison platform. Peischl
oped and maintained the higher the risk that the organization runs
et al. [22] show the application of constraint-based recommenda-
into a scalability problem where additional resources are needed to
tion technologies in the domain of software effort estimation. W EE -
be able to perform knowledge engineering and maintenance.
V IS[25]3 is a MediaWiki4 based environment for the development
An alternative to the hiring of additional staff for development
and maintenance of constraint-based recommender applications –
and maintenance of recommendation knowledge bases is to change
a couple of freely available recommenders have already been de-
the underlying knowledge engineering paradigm. The idea of P EO -
ployed. Knowledge-based technologies for the recommendation of
PLE V IEWS is to engage domain experts more deeply into knowledge
business plans are introduced by Jannach and Bundgaard-Joergensen
engineering tasks. We do not want to ”convert” them into techni-
[19]. The recommendation of equipment configuration in the con-
cal experts but to define basic tasks (micro tasks) that are easy to
text of smarthomes is introduced by Leitner et al. [21]. Technologies
understand and complete even for domain experts without the cor-
that recommend changes in software development practices are in-
responding technical expertise. Micro tasks completed by users pro-
troduced by Pribik and Felfernig [23]. Finally, Burke and Ramezani
1 Applied Software Engineering, Institute for Software Technol- [5] show how to select recommendation algorithms by introducing
ogy, Graz University of Technology, Austria, email: {felfernig, rules for recommending recommenders.
mjeran, stettinger}@ist.tugraz.at, {thomas.absenger, th.gruber,
sarah.haas, emanuel.kirchengast, michael.schwarz, lukas.skofitsch,
thomas.ulz}@student.tugraz.at. 3 www.weevis.org.
2 The terms item and product are used synonymously throughout the paper. 4 www.mediawiki.org.
In P EOPLE V IEWS, principles of Human Computation [26] are id item name
included into the development of knowledge-based recommenders. Φ1 Investment Fund A
The idea of Human Computation is to let persons perform tasks in Φ2 Investment Fund B
which they are better than computers, for example, the identification Φ3 Building Loan
of product properties from a website. In the context of knowledge Φ4 Bond
Φ5 Savings Book
base development and maintenance the idea is to let domain experts
perform tasks they are much better in compared to knowledge engi-
neers who typically have less knowledge about the product domain Table 2. Example set of items used in working example.
and thus relieve the work of knowledge engineers. M ATCHIN [18]
is based on the idea of preference elicitation by asking users what development of the application on the basis of micro tasks.
a person would typically prefer when having to choose between al-
ternatives. Compared to this work, P EOPLE V IEWS allows to derive user attribute question to user attribute domain
constraint-based recommenders which are the basis for intelligent What are your {Studies, Pension, Speculation,
goal (gl)
user interfaces that support, for example, deep explanations [17] and personal goals? Car, House, World trip, noval}
{in 1 year, in 2 years, in 3-5 years,
the diagnosis and repair of inconsistent requirements [13, 14]. When is the
runtime (rt) in 5-10 years, in 10-20 years, in
The major contributions of this paper are the following. First, we money needed?
more than 20 years, noval}
show how financial service recommender knowledge bases can be Preparedness to
risk (ri) {low, medium, high, noval}
developed by a community of domain experts. Second, we sketch take risks?
how such knowledge bases can also be exploited for teaching advi-
sory practices on the basis of games (S TUDY BATTLE environment).
Table 3. User attributes u ∈ U of example financial services
Third, we provide a discussion of major issues for future research. recommender.
The remainder of this paper is organized as follows. In Section 2
we introduce basic concepts of Human Computation based knowl- In the P EOPLE V IEWS recommendation mode, user attributes can
edge construction. To give an impression of the P EOPLE V IEWS and be used to specify user (customer) requirements reqi ∈ REQ. In
the S TUDY BATTLE user interface, we present example screenshots the modeling mode, user attributes represent a central element of a
in Section 3. Preliminary results of empirical evaluations are shortly micro task: given a certain item, users are asked to estimate which
discussed in Section 4. In Section 5 we provide an overview of issues values of user attributes are compatible with the item, i.e., are a crite-
for future work. We conclude the paper with Section 6. ria for selecting and recommending the item. The evaluation of items
with regard to user attributes is the central micro task implemented
2 Developing P EOPLE V IEWS Recommenders in the current P EOPLE V IEWS prototype. A detailed evaluation of the
example items (Table 2) regarding the user attributes goal, runtime,
The P EOPLE V IEWS environment supports two basic modes of inter- and risk is provided in Table 4.
action. First, recommender applications can be created in the mod- Each row of Table 4 specifies a so-called user-specific filter con-
eling mode and second, the applications can be executed in the rec- straint [10], i.e., a filter constraint (specified by a user) regarding a
ommendation mode. In this section we discuss different tasks to be specific item. For example, user Luc specified Pension and Specu-
performed in order to create a P EOPLE V IEWS recommender. Table lation as possible goals that lead to an inclusion of the item Invest-
1 provides an overview of the users of our working example. These ment Fund B into a recommendation. Furthermore, Luc believes that
users will jointly develop a P EOPLE V IEWS recommender. a user should have a high preparedness to take risks (attribute risk)
and should need the payment in 3-5 years, 5-10 years or 10-20 years
user email pwd
from now on. Semantically, an item X is selected by a user-specific
Andrea andrea@... ****
filter constraint if all the preconditions are fulfilled.
Mary mary@... *****
Luc luc@... ******
In order to derive recommendation-relevant filter constraints (rec-
Torsten torsten@... **** ommendation rules) [10]), user-specific filter constraints have to be
aggregated. An example of this aggregation step is depicted in Table
5. For each item all related user-specific filter constraints are inte-
Table 1. Example users of P EOPLE V IEWS environment.
grated into one constraint. Each row in this table has to be interpreted
as a filter constraint for a specific item, for example, the constraint
Table 2 contains an overview of items (financial services) that are in the first row of Table 5 is the following. The item Φ1 (Investment
used in our working example. The Investment Funds (A and B) have Fund A) is included (recommended) if the user requirements regard-
a higher risk of loss and require that customers have a high willing- ing goal (gl), runtime (rt), and risk (ri) are consistent with the condi-
ness to take risks, otherwise these services will not be recommended. tion of the recommendation-relevant filter constraint gl ∈ {Studies,
Building Loan, Bond, and Savings Book are lower-risk items. In the Pension, Speculation, noval} ∧ rt ∈ {in 5-10 year, in 10-20 years,
current version of P EOPLE V IEWS, items can be characterized by ad- noval} ∧ ri ∈ {medium, high, noval} → include(Φ1 ).
ditional item attributes, however, these attributes are not used by rec- Table 5 includes the complete set of recommendation-relevant
ommendation rules constructed from micro contributions. filter constraints (recommendation rules). Exactly these conditions
In P EOPLE V IEWS, user requirements reqi ∈ REQ are specified are applied by P EOPLE V IEWS to determine recommendations for
as assignments of user attributes. For our financial services recom- a user. In P EOPLE V IEWS, each item has exactly one related
mender we define a set of user attributes which are enumerated in Ta- recommendation-relevant filter constraint; each such filter constraint
ble 3. In the current version of the system, user attributes are defined is represented by one row in Table 5. The general logical represen-
by the creators of a recommender application, i.e., attribute defini- tation of a recommendation-relevant filter constraint f for an item
tions can not be extended by other users who contribute to the further Φ is shown in Formula 1. In this context, values(Φ, u) is the set of
user item name (id) goal runtime risk
Studies, Pension, in 5-10 years, in 10-20
Andrea Investment Fund A (Φ1 ) high
Speculation years
in 5-10 years, in 10-20
Luc Investment Fund A (Φ1 ) Pension, Speculation high
years
in 5-10 years, in 10-20
Mary Investment Fund A (Φ1 ) Pension, Speculation medium, high
years
in 3-5 years, in 5-10 years,
Torsten Investment Fund B (Φ2 ) Pension, Speculation high
in 10-20 years
in 3-5 years, in 5-10 years,
Luc Investment Fund B (Φ2 ) Pension, Speculation high
in 10-20 years
Studies, Pension, Car, in 5-10 years, in 10-20
Mary Building Loan (Φ3 ) low, medium, high
House years
Studies, Pension, Car,
Andrea Building Loan (Φ3 ) in 5-10 years low, medium
House
Studies, Pension, Car,
Luc Building Loan (Φ3 ) in 5-10 years low, medium
House
in 2 years, in 3-5 years, in
Mary Bond (Φ4 ) Studies, Car, House low, medium
5-10 years
Studies, Car, House, World in 1 year, in 2 years, in 3-5
Andrea Savings Book (Φ5 ) low
trip years, in 5-10 years
in 1 year, in 2 years, in 3-5
Torsten Savings Book (Φ5 ) Studies, House, World trip low
years, in 5-10 years
Table 4. Example of user-specific filter constraints (= micro contributions).
supported domain values of user attribute u ∈ U (see Table 4). The
constant noval denotes the fact that no value has been selected for
the corresponding user attribute. item name
attribute:value support value
(id)
Investment
^ goal: Studies 0.33
f (Φ) : u ∈ values(Φ, u) ∪ {noval} → include(Φ) (1) Fund A (Φ1 )
goal: Pension, Speculation 1.0
u∈U
runtime: in 5-10 years, in 10-20 years 1.0
risk: medium 0.33
For each pair (Φ, val ∈ values(Φ, u)), P EOPLE V IEWS deter- risk: high 1.0
mines a corresponding support value (see Formula 2). In this context, Investment
occurrence(Φ, val) denotes the number of times, value val occurs goal: Pension, Speculation 1.0
Fund B (Φ2 )
in a user-specific filter constraint for item Φ and occurrence(Φ) de- runtime: in 3-5 years, in 5-10 years, in
1.0
notes the number of times an item Φ is referred in a user-specific 10-20 years
risk:high 1.0
filter constraint. For example, support(Φ1 , Studies) = 13 . Building
goal: Studies, Pension, Car, House 1.0
Loan (Φ3 )
occurrence(Φ, val) runtime:in 5-10 years 1.0
support(Φ, val) = (2)
occurrence(Φ) runtime:in 10-20 years 0.33
risk:low, medium 1.0
The complete set of support values is depicted in Table 6. In P EO - risk:high 0.33
PLE V IEWS, an item Φ can have an associated rating (rating(Φ)) Bond (Φ4 ) goal: Studies, Car, House 1.0
which represents an item evaluation with regard to quality and related runtime:in 2 years, in 3-5 years, in 5-10
1.0
services. Such a rating can be determined, for example, by calculat- years
risk:low, medium 1.0
ing the average of the individual user item ratings.5 For simplicity, we
Savings
do not take into account user ratings in the utility function discussed goal: Studies, House, World trip 1.0
Book (Φ5 )
below (see Formula 3). goal:Car 0.5
Depending on the requirements articulated by the current user runtime:in 1 year, in 2 years, in 3-5
1.0
(see, e.g., Table 7), P EOPLE V IEWS determines and ranks a set years, in 5-10 years
risk:low 1.0
of relevant items as follows. First, recommendation-relevant fil-
ter constraints are applied to pre-select items that fulfill the user
requirements REQ = {req1 , req2 , ..., reqk }. In our example, the Table 6. Support values (see Formula 2) derived from user-specific filter
set {Investment Fund A, Building Loan} would be selected by the constraints (see Table 4).
recommendation-relevant filter constraints (see Table 5).
5 Similar to ratings provided by platforms such as amazon.com.
item name (id) goal runtime risk
Investment Studies, Pension, in 5-10 years, in 10-20
medium, high
Fund A (Φ1 ) Speculation years
Investment in 3-5 years, in 5-10 years,
Pension, Speculation high
Fund B (Φ2 ) in 10-20 years
Building Loan Studies, Pension, Car, in 5-10 years, in 10-20
low, medium, high
(Φ3 ) House years
in 2 years, in 2-5 years, in
Bond (Φ4 ) Studies, Car, House low, medium
5-10 years
Savings Book Studies, Car, House, World in 1 year, in 2 years, in 3-5
low
(Φ5 ) trip years, in 5-10 years
Table 5. Example of recommendation-relevant filter constraints which are the result of integrating user-specific filter constraints (see Table 4).
id requirement If users are logged in, they are allowed to contribute to the de-
req1 goal = Studies velopment of P EOPLE V IEWS recommender applications. Only the
req2 goal = Pension creators of a recommender application are allowed to define user at-
req3 runtime = in 5-10 years tributes. Other users can complete micro tasks in terms of evaluating
req4 risk = medium items with regard to a defined set of user attributes. The list of user
attributes used in our working example is depicted in Figure 2 (cor-
Table 7. Example set of user requirements (reqi ∈ REQ). responds to the entries of Table 3).
The determined recommendation set must be ranked before being
presented to the user. In P EOPLE V IEWS, item ranking is based on
the following utility function (see Formula 3). The utility of each
item is derived from the support values of individual requirements
(see Formula 2).
utility(Φ, REQ) = Σreq∈REQ support(Φ, req) (3)
The item ranking of our working example as a result of apply-
ing Formula 3 is depicted in Table 8. For example, utility(Φ3 ,REQ
= {goal = Studies, goal = Pension, runtime = in 5-10 years, risk =
medium}) = support(Φ3 , goal = Studies) + support(Φ3 , goal =
Pension) + support(Φ3 , runtime = in 5-10 years) + support(Φ3 ,
risk = medium) = 1.0 + 1.0 + 1.0 + 1.0 = 4.0.
item name (id) utility rank
Building Loan (Φ3 ) 4.0 1
Investment Fund A (Φ1 ) 2.66 2
Table 8. Utility-based ranking of items in the recommendation set.
3 User Interface
Figure 1. P EOPLE V IEWS homescreen – the current version of the user
3.1 P EOPLE V IEWS interface is provided in German. The homescreen explains the basic
functionalities of the system (development, maintenance, and execution of
In this section we discuss the P EOPLE V IEWS user interface6 and also recommender applications).
show how P EOPLE V IEWS recommendation knowledge can be ex- Logged-in users are also allowed to enter new items to the recom-
ploited by the S TUDY BATTLE learning environment. The P EOPLE - mender product catalog. The P EOPLE V IEWS representation of prod-
V IEWS homescreen is depicted in Figure 1. For applying P EOPLE - uct catalogs is exemplified in Figure 3 (corresponds to the list of
V IEWS recommenders, there is no explicit need for being logged in. items shown in Table 2).
Recommenders can be selected and activated directly from the home- The interface for evaluating an item with regard to a set of user
screen (see the tag cloud in Figure 1). attributes is depicted in Figure 4. The screenshot depicts the evalu-
ation of Building Loan with regard to the user attribute goal. After
6 The user interface is currently only available in German.
having completed the definition of a P EOPLE V IEWS recommender,
product knowledge and sales practices. Examples of S TUDY BATTLE
games are the following.
Assign Properties. Figure 6 depicts an example user interface of a
S TUDY BATTLE application that implements a quiz related to knowl-
edge about the relationship between user attributes and items. In the
example, users have the task to assign items on the left hand side to
user attribute values on the right hand side where each product has to
be assigned to at least one attribute value and vice-versa.
Find Items. A different version of the game depicted in Figure 6
is to ask for products that fulfill certain criteria (represented by a
combination of user attribute settings).
Figure 2. P EOPLE V IEWS: example user attributes. Find Incompatibilities. This game focuses on combinations of user
attribute values that do not lead to a solution, i.e., users have to spec-
ify combinations of user attribute values from which they think that
no corresponding solution could be found.
Maximize Requirements. The task is to identify minimal sets of
requirements (from a given set of requirements REQ) that have to
be deleted from REQ such that the remaining requirements lead to
at least one solution. This game type reflects the principles of model-
based diagnosis [6, 24], i.e., support users in learning and improving
repair behavior in situations where no solution can be identified.
Maximize Items. A similar task is focused on the repair of item
sets; in this context the task of users is to identify a maximal set of
items from a given set of items such that there exists at least one
combination of user attribute values that lead to these items (not nec-
essarily exclusively). An additional criteria could be that at least n
items from the original item list must remain in the result set.
Figure 3. P EOPLE V IEWS: example of an item list.
the recommender can directly be executed. The user interface of our
financial services recommender is depicted in Figure 5.
Figure 4. P EOPLE V IEWS: example of an item evaluation user interface Figure 6. S TUDY BATTLE ”Assign Properties” learning application. The
(evaluation of item Building Loan with regard to the user attribute goal). task of the user is to relate items with corresponding attribute values.
3.2 S TUDY BATTLE 4 Preliminary Evaluation Results
Recommendation-relevant filter constraints can be further exploited Human Computation based Knowledge Acquisition. Applying Hu-
for generating different learning applications that are part of the man Computation concepts [26] in the context of recommender ap-
S TUDY BATTLE environment. S TUDY BATTLE is a game-based learn- plication development and maintenance has the potential to lift the
ing environment which can be utilized as an environment for learning burden of enormous engineering and maintenance efforts from the
Figure 5. P EOPLE V IEWS: example of a recommender application (Financial Services).
shoulder of knowledge engineers. Micro tasks as sketched in this Weighting of Item Evaluations. In the current P EOPLE V IEWS ver-
paper can be structured in a way that they are understandable for sion it is possible to assign user attribute values to items, i.e., to
domain experts without a computer science background. Knowledge specify which criteria are relevant for the selection of a certain item.
gained from completed micro tasks can be easily integrated into a In future versions of P EOPLE V IEWS it will be possible to integrate
corresponding recommender knowledge base. Due to the increas- weights into item evaluations. This maybe does not play a major role
ing size and complexity of knowledge bases, the development of in financial service related recommender applications but can be im-
such technologies is crucial since they help to tackle scalability is- portant in other domains were nuances and personal tastes play a
sues which otherwise could cause a complete failure with regard to a more important role. For example, in the context of recommending
company-wide recommender deployment. As such, P EOPLE V IEWS digital cameras, it can be important to specify degrees regarding cer-
technologies can be considered as a first step towards more scalable tain camera properties, for example, the degree to which a camera is
development methods that will also help to further increase the pop- able to support sports photography.
ularity of knowledge-based (recommendation) technologies. Further Micro Tasks. In the current system version, the only mi-
Usability. An initial user study has been conducted with an early cro task to be completed is to define the relationship (compatibility
version of P EOPLE V IEWS at the Graz University of Technology [10]. properties) between items and corresponding user attribute values.
N=161 (15% female and 85% male) students interacted with the sys- In future versions of P EOPLE V IEWS we will extend this list of micro
tem with the goal to develop different recommender applications. Af- tasks (see Table 9).
ter having completed the development, the study participants had to User Selection for Micro Tasks. An important enhancement will be
complete a questionnaire which was based on the system usability the inclusion of methods that automatically select users for a given
scale (SUS) [1]. Evaluation results regarding the SUS aspects are set of micro tasks and also take into account fairness in the distribu-
summarized in Figure 7. Besides usability questions, further feed- tion of micro tasks. As detected in our initial studies, users are willing
back has been provided by the study participants, for example, the to contribute to the further development of P EOPLE V IEWS recom-
majority of the participants (69% of all study participants) would menders. An important issue in this context is to find the users with
like to further contribute to P EOPLE V IEWS recommenders. 56% out the right expertise for certain tasks and also to not overload users.
of those participants who wanted to contribute agreed to contribute Our approach in this context will be to maintain user profiles which
within a time frame of less than 30 minutes per week. are derived from observing the activities of a user within P EOPLE -
V IEWS. For example, if a user selects a certain item when interact-
ing with the financial services recommender, the keywords extracted
5 Future Work from the corresponding item description are stored in the user pro-
file. If (in the future) micro tasks related to similar items (items with
The major goal of this paper was to provide an overview of the P EO - a similar description) have to be completed, users with expertise re-
PLE V IEWS recommendation environment. There are many issues for garding such items will be the preferred contact persons.
future work that we want to tackle and integrate corresponding solu- Games. Games will be another mechanism for data collection in
tions in upcoming P EOPLE V IEWS versions.
Figure 7. Results of a SUS-based usability study [1] of the P EOPLE V IEWS environment.
the P EOPLE V IEWS modeling mode. A single user game will be in-
cluded that is quiz-based. The overall goal is to guess user attribute
settings correctly that best describe a certain item. In a second game
two users will jointly try to figure out user attribute values that best
describe shown items. The more matching item evaluations exist the
better the team performs.
Dependencies between User Attributes and Item Attributes. An ex-
name description tension of the current P EOPLE V IEWS version will be the possibility
check whether a certain item belongs to to identify direct relationships between user attribute values and tech-
item quality check a specific recommender (is an existing nical product properties. This is not the case in the current P EOPLE -
recommender-related item) V IEWS version since dependencies are only defined between user
check whether a certain attribute attribute values and items.
belongs to to a specific recommender
attribute quality check Recommendation Algorithms. The current version of P EOPLE -
(user attribute or item attribute exists in
the item domain) V IEWS relies on the discussed recommendation-relevant filter con-
check whether a certain value belongs straints – item ranking is based on a utility-based evaluation (see
attribute value quality
to the domain of an attribute (user Formula 3). In future versions of P EOPLE V IEWS we will extend the
check
attribute or item attribute) quality of recommendation algorithms by, for example, adapting the
check whether a certain figure belongs
graphic check
to a certain item
determination of support values. If, for example, additional infor-
evaluate item assign user attribute values to items mation about the performance of a certain user is available (e.g.,
attribute value utility derive a ranking that shows which items performance with regard to correctly completed micro tasks in the
check best support a user attribute value past), this information can be used to increase/decrease the weight
of a user when determining support values. Finally, when users are
Table 9. Example list of micro tasks to be integrated in P EOPLE V IEWS. specifying their requirements, future versions of P EOPLE V IEWS will
allow the specification of preferences (weights) which indicate user
preferences regarding certain requirements. This will also include ap-
proaches to the learning of weights (users should not have to specify
all weights explicitly).
Inconsistency Management. Given a set of customer requirements
it could be the case that no solution can be presented to the user. In
upcoming versions of P EOPLE V IEWS we will focus on integrating
state-of-the-art diagnosis algorithms that help to automatically deter-
mine repair actions in such inconsistent situations [15]. These repairs
will take into account user weights (preferences) and thus minimize [11] A. Felfernig, K. Isak, K. Szabo, and P. Zachar, ‘The VITA Finan-
the number of interaction cycles needed to find a reasonable solu- cial Services Sales Support Environment’, pp. 1692–1699, Vancouver,
Canada, (2007).
tion. In addition to this more intelligent management of inconsistent [12] A. Felfernig and A. Kiener, ‘Knowledge-based Interactive Selling of
requirements, we will integrate mechanisms that help to consolidate Financial Services with FSAdvisor’, in 17th Innovative Applications of
the set of user-specific filter constraints in order to make the result- Artificial Intelligence Conference (IAAI05), pp. 1475–1482, Pittsburgh,
ing recommendation-relevant filter constraints more compact. Con- Pennsylvania, (2005).
solidation will be achieved, for example, on the basis of redundancy [13] A. Felfernig, M. Schubert, G. Friedrich, M. Mandl, M. Mairitsch, and
E. Teppan, ‘Plausible repairs for inconsistent requirements’, in 21st In-
detection algorithms [16]. ternational Joint Conference on Artificial Intelligence (IJCAI’09), pp.
Quality Management. The major task of quality management is 791–796, Pasadena, CA, (2009).
to assure the quality of the dataset collected on the basis of differ- [14] A. Felfernig, M. Schubert, and S. Reiterer, ‘Personalized diagnosis for
ent micro tasks. Quality assurance must be capable of detecting and over-constrained problems’, in 23rd International Conference on Arti-
ficial Intelligence (IJCAI 2013), pp. 1990–1996, Peking, China.
preventing manipulations of the dataset (also under the assumption [15] A. Felfernig, M. Schubert, and C. Zehentner, ‘An Efficient Diagnosis
that anonymous users are allowed to complete micro tasks), it must Algorithm for Inconsistent Constraint Sets’, Artificial Intelligence for
also identify changes to the given set of user-specific filter constraints Engineering Design, Analysis, and Manufacturing (AIEDAM), 25(2),
that help to improve the prediction quality of recommendation algo- 175–184, (2012).
rithms. Quality assurance is also responsible for the generation of [16] A. Felfernig, C. Zehentner, and P. Blazek, ‘Corediag: Eliminating re-
dundancy in constraint sets’.
micro tasks that need to be completed in order to improve the overall [17] G. Friedrich, ‘Elimination of spurious explanations’, in European Con-
quality of the P EOPLE V IEWS datasets. The micro tasks generated by ference on Artificial Intelligence (ECAI 2004), pp. 813–817, Valencia,
quality assurance are summarized as an agenda – this agenda is for- Spain, (2004).
warded to micro task scheduling that is responsible for distributing [18] S. Hacker and L. VonAhn, ‘Matchin: Eliciting User Preferences with
an Online Game’, in CHI’09, pp. 1207–1216, (2009).
micro tasks to the P EOPLE V IEWS user community. [19] D. Jannach and U. Bundgaard-Joergensen, ‘SAT: A Web-Based Inter-
active Advisor for Investor-Ready Business Plans’, in Intl. Conference
6 Conclusions on e-Business (ICE-B 2007), pp. 99–106, (2007).
[20] D. Jannach, M. Zanker, A. Felfernig, and G. Friedrich, Recommender
In this paper we gave an overview of the P EOPLE V IEWS recommen- Systems, Cambridge University Press, 2010.
dation environment which exploits concepts of Human Computation [21] G. Leitner, A. Fercher, A. Felfernig, and M. Hitz, ‘Reducing the Entry
Threshold of AAL Systems: Preliminary Results from Casa Vecchia’,
to integrate domain experts more deeply into knowledge base de- in 13th Intl. Conference on Computers Helping People with Special
velopment and maintenance processes. P EOPLE V IEWS knowledge Needs, pp. 709–715, (2012).
bases can be exploited to generate learning applications which can [22] B. Peischl, M. Zanker, M. Nica, and W. Schmid, ‘Constraint-based
be used in the S TUDY BATTLE environment. A major focus of this Recommendation for Software Project Effort Estimation’, Journal of
Emerging Technologies in Web Intelligence, 2(4), 282–290, (2010).
paper was to show how P EOPLE V IEWS can be applied in the context
[23] I. Pribik and A. Felfernig, ‘Towards Persuasive Technology for Soft-
of financial service recommendation. The concepts presented in this ware Development Environments: An Empirical Study’, in Persuasive
paper have the potential to avoid scalability issues which already ex- Technology Conference (Persuasive 2012), pp. 227–238, (2012).
ist in many knowledge-based environments due to the increasing size [24] R. Reiter, ‘A theory of diagnosis from first principles’, AI Journal,
and complexity of knowledge bases. 23(1), 57–95, (1987).
[25] S. Reiterer, A. Felfernig, P. Blazek, G. Leitner, F. Reinfrank, and
G. Ninaus, ‘WeeVis’, in Knowledge-based Configuration – From Re-
REFERENCES search to Business Cases, eds., A. Felfernig, L. Hotz, C. Bagley,
and J. Tiihonen, chapter 25, 365–376, Morgan Kaufmann Publishers,
[1] A. Bangor, P. Kortum, and J. Miller, ‘An Empirical Evaluation of (2013).
the System Usability Scale (SUS)’, International Journal of Human- [26] L. VonAhn, ‘Human Computation’, in Technical Report CM-CS-05-
Computer Interaction, 24(6), 574–594, (2008). 193, (2005).
[2] R. Burke, ‘Knowledge-based recommender systems’, Encyclopedia of
Library and Information Systems, 69(32), 180–200, (2000).
[3] R. Burke, A. Felfernig, and M. Goeker, ‘Recommender systems: An
overview’, AI Magazine, 32(3), 13–18, (2011).
[4] R. Burke and K. Hammond, ‘The FindMe Approach to Assisted Brows-
ing’, IEEE Expert, 32–40, (1997).
[5] R. Burke and M. Ramezani, ‘Matching recommendation technologies
and domains’, in Recommender Systems Handbook, 367–386, Springer,
(2010).
[6] J. de Kleer, A. Mackworth, and R. Reiter, ‘Characterizing diagnoses
and systems’, AI Journal, 56(197–222), 57–95, (1992).
[7] A. Fano and S. Kurth, ‘Personal Choice Point: Helping Users Visualize
What it Means to Buy a BMW’, in International Conference on In-
telligent User Interfaces IUI’03, pp. 46–52, Miami, FL, USA, (2003).
ACM, New York, USA.
[8] A. Felfernig and R. Burke, ‘Constraint-based recommender systems:
Technologies and research issues’, in IEEE ICEC’08, pp. 17–26, Inns-
bruck, Austria, (2008).
[9] A. Felfernig, G. Friedrich, D. Jannach, and M. Zanker, ‘An environment
for the development of knowledge-based recommender applications’,
International Journal of Electronic Commerce (IJEC), 11(2), 11–34,
(2006).
[10] A. Felfernig, S. Haas, G. Ninaus, M. Schwarz, T. Ulz, M. Stettinger,
K. Isak, M. Jeran, and S. Reiterer, ‘RecTurk: Constraint-based Recom-
mendation based on Human Computation’, in RecSys 2014 CrowdRec
Workshop, pp. 1–6, Foster City, CA, USA, (2014).