=Paper=
{{Paper
|id=Vol-1349/paper3
|storemode=property
|title=An Integrated Knowledge Engineering Environment for Constraint-based Recommender Systems
|pdfUrl=https://ceur-ws.org/Vol-1349/paper03.pdf
|volume=Vol-1349
|dblpUrl=https://dblp.org/rec/conf/finrec/Reiterer15
}}
==An Integrated Knowledge Engineering Environment for Constraint-based Recommender Systems==
https://ceur-ws.org/Vol-1349/paper03.pdf
An Integrated Knowledge Engineering Environment for
Constraint-based Recommender Systems
Stefan Reiterer1
Abstract. Constraint-based recommenders support customers in The user interface of the W EE V IS environment provides intel-
identifying relevant items from complex item assortments. In this pa- ligent mechanisms that help to make development and mainte-
per we present a constraint-based environment already deployed in nance operations easier. Based on model-based diagnosis techniques
real-world scenarios that supports knowledge acquisition for recom- [12, 17, 26], the environment supports users in the following situa-
mender applications in a MediaWiki-based context. This technology tions: (1) if no solution could be found for a set of user requirements,
provides the opportunity do directly integrate informal Wiki content the system proposes repair actions that help to find a way out from
with complementary formalized recommendation knowledge which the ”no solution could be found” dilemma; (2) if the constraints in
makes information retrieval for users (readers) easier and less time- the recommender knowledge base are inconsistent with a set of test
consuming. The user interface supports recommender development cases (situation detected within the scope of regression testing of the
on the basis of intelligent debugging and redundancy detection. The knowledge base), those constraints are shown to the users (knowl-
results of a user study show the need of automated debugging and edge engineers) who are responsible for the faulty behavior of the
redundancy detection even for small-sized knowledge bases. knowledge base; (3) if the recommender knowledge base includes
redundant constraints, i.e., constraints that – if removed from the
knowledge base – logically follow from the remaining constraints,
1 Introduction these constraints are also determined in an automated fashion and
Constraint-based recommenders support the identification of relevant shown to knowledge engineers.
items from large and often complex assortments on the basis of an ex- The major contributions of this paper are the following. (1) on the
plicitly defined set of recommendation rules [3]. Example item do- basis of a working example from the domain of financial services,
mains are digital cameras and financial services [5, 8, 9]. For a long we provide an overview of the diagnosis and redundancy detection
period of time the engineering of recommender knowledge bases (for techniques integrated in the W EE V IS environment. (2) we report the
constraint-based recommenders) required that knowledge engineers results of an empirical study which analyzed the usability of W EE -
are technical experts (in the majority of the cases computer scien- V IS functionalities.
tists) with the needed technical capabilities [14]. Developments in The remainder of this paper is organized as follows. In Section
the field moved one step further and provided graphical engineering 2 we discuss related work. In Section 3 we present an overview of
environments [5], which improve the accessibility and maintainabil- the recommendation environment W EE V IS and discuss the included
ity of recommender knowledge bases. However, users still have to knowledge engineering support mechanisms. In Section 4 we present
deal with additional tools and technologies which is in many cases a results of an empirical study that show the need of intelligent diagno-
reason for not applying constraint-based environments. sis and redundancy detection support. In Section 5 we discuss issues
Similar to the idea of Wikipedia to allow user communities to de- for future work, with Section 6 we conclude the paper.
velop and maintain Wiki pages in a cooperative fashion, we intro-
duce the W EE V IS2 environment, which supports the community- 2 Related Work
based development of constraint-based recommender applications
within a Wiki environment. W EE V IS has been implemented on the Based on original static Constraint Satisfaction Problem (CSP) rep-
basis of MediaWiki3 , which is an established standard Wiki platform. resenations [15, 20, 29], many different types of constraint-based
Compared to other types of recommender systems such as collabo- knowledge representations have been developed. Mittal and Falken-
rative filtering [19] and content-based filtering [25], constraint-based hainer [22] introduced dynamic constraint satisfaction problems
recommender systems are based on an underlying recommendation where variables have an activity status and only active variables
knowledge base, i.e., recommendation knowledge is defined explic- are taken into account by the search process. Stumptner et al. [28]
itly. W EE V IS is already applied by four Austrian universities (within introduced the concept of generative constraint satisfaction where
the scope of recommender systems courses) and two companies for variables can be generated on demand within the scope of solution
the purpose of prototyping recommender applications in the financial search. Compared to existing work, W EE V IS supports the solving of
services domain. static CSPs on the basis of conjunctive queries where each solution
1 corresponds to a result of querying a relational database. Addition-
SelectionArts Intelligent Decision Technologies GmbH, Austria,
email:stefan.reiterer@selectionarts.com ally, W EE V IS includes diagnosis functionalities that help to auto-
2 www.weevis.org. matically determine repair proposals in situations where no solution
3 www.mediawiki.org.
could be found [12].
A graphical recommender development environment for single to the knowledge can be immediately experienced by switching from
users is introduced in [5]. This Java-based environment supports the the view source to the read mode). In the read mode, knowledge
development of constraint-based recommender applications for on- bases can as well be tested and in the case of inconsistencies (some
line selling platforms. Compared to Felfernig et al. [5], W EE V IS test cases were not fulfilled within the scope of regression testing)
provides a wiki-based user interface that allows user communities to corresponding diagnoses are shown to the user.
develop recommender applications. Furthermore, W EE V IS includes
efficient diagnosis [12] and redundancy detection [13] mechanisms
that allow the support of interactive knowledge base development. 3.1 Overview
A Semantic Wiki-based approach to knowledge acquisition for
The website www.weevis.org provides a selection of different rec-
collaborative ontology development is introduced in [2]. Compared
ommender applications (full list, list of most popular recommenders,
to Baumeister et al. [2], W EE V IS is based on a recommendation do-
and recommenders that have been defined previously) that can be
main specific knowledge representation (in contrast to ontology rep-
tested and extended. Most of these applications have been developed
resentation languages) which makes the definition of domain knowl-
within the scope of university courses on recommender systems (con-
edge more accessible also for domain experts. Furthermore, W EE -
ducted at four Austrian universities). W EE V IS recommenders can be
V IS includes intelligent debugging and redundancy detection mech-
integrated seamlessly into standard Wiki pages, i.e., informally de-
anisms which make development and maintenance operations more
fined knowledge can be complemented or even substituted with for-
efficient. We want to emphasize that intended redundancies can ex-
mal definitions.
ist, for example, for the purpose of better understandability of the
In the following we will present the concepts integrated in the
knowledge base. If such constraints are part of a knowledge base,
W EE V IS environment on the basis of a working example from the
these should be left out from the redundancy detection process.
domain of financial services. In such a recommendation scenario,
A first approach to a conflict-directed search for hitting sets in in-
a user has to specify his/her requirements regarding, for example,
consistent CSP definitions was introduced by Bakker et al. [1]. In
the expected capital guarantee level of the financial product or the
this work, minimal sets of faulty constraints in inconsistent CSP def-
amount of money he or she wants to invest. A corresponding W EE -
initions were identified on the basis of the concepts of model-based
V IS user interface is depicted in Figure 1 where requirements are
diagnosis [26]. In the line of Bakker et al. [1], Felfernig et al. [4]
specified on the left hand side and the corresponding recommenda-
introduced concepts that allow the exploitation of the concepts of
tions are displayed in the right hand side.
model-based diagnosis in the context of knowledge base testing and
Each recommendation (item) has a corresponding support value
debugging. Compared to earlier work [4, 24], W EE V IS provides an
that indicates the share of requirements that are currently supported
environment for development, testing, debugging, and application of
by the item. A support value of 100% indicates that each requirement
recommender systems. With regard to diagnosis techniques, W EE -
is satisfied by the corresponding item. If the support value is below
V IS is based on more efficient debugging and redundancy detection
100%, corresponding repair alternatives are shown to the user, i.e.,
techniques that make the environment applicable in interactive set-
alternative answers to questions that guarantee the recommendation
tings [12, 16, 21].
of at least one item (with 100% support).
Since W EE V IS is a MediaWiki-based environment, the definition
3 The W EE V IS Environment of a recommender knowledge base is supported in a textual fashion
on the basis of a syntax similar to MediaWiki. An example of the def-
In it’s current version, W EE V IS supports scenarios where user re-
inition of a (simplified) financial services recommender knowledge
quirements can be defined in terms of functional requirements [23].
base is depicted in Figure 2. Basic syntactical elements provided in
The corresponding recommendations (solutions) are retrieved from
W EE V IS will be introduced in the next subsection.
a predefined set of alternatives (also denoted as item set or product
catalog). Requirements are checked with regard to their consistency
with the underlying item set (consistency is given if at least one so- 3.2 W EE V IS Syntax
lution could be identified). If no solution could be found, W EE V IS
repair alternatives are determined on the basis of direct diagnosis al- Constraint-based recommendation requires the explicit definition of
gorithms [12]. This way, W EE V IS does not only support item se- questions and possible answers, items and their properties, and con-
lection but also consistency maintenance processes on the basis of straints (see Figure 2).
intelligent repair mechanisms [6]. In W EE V IS the tag &QUESTIONS enumerates the set of user re-
W EE V IS is based on the idea that a community of users coop- quirements where, for example, pension specifies whether the user
eratively contributes to the development of a recommender knowl- wants a financial product to support his private pension plan [yes, no]
edge base. The environment supports knowledge acquisition pro- and maxinvestment specifies the amout of money the user wants to
cesses on the basis of tags that can be used for defining and test- invest. Furthermore, payment represents the frequency in which the
ing recommendation knowledge bases. Using W EE V IS, standard payment should be done [once, periodical], payout specifies the fre-
Wikipedia pages can be extended with recommendation knowledge quency the customer gets a payout from the financial product (out of
that helps to represent domain knowledge in a more accessible and [once,monthly]), and guarantee the expected capital guarantee [low,
understandable fashion. The same principles used for the developing high].
Wikipedia pages can also be used for the development and mainte- An item assortment can be specified in W EE V IS using the
nance of recommender knowledge bases, i.e., in the read mode rec- &PRODUCTS tag (see Figure 2). In our example, the item (prod-
ommenders can be executed and in the view source mode recommen- uct) assortment is specified by values related to the attributes name;
dation knowledge can be defined and adapted. This way, rapid pro- guaranteep, the capital guarantee the product provides; payoutp, the
totyping processes can be supported in an intuitive fashion (changes payout frequency of the product; mininvestp the minimal amount of
Figure 1. A simple financial service recommender (W EE V IS read mode).
money for the financial service. Three items are specified: SecureFin, COM P and F ILT . On the basis of such a definition, W EE V IS is
BonusFin, and DynamicFin. able to calculate recommendations that take into account a specified
Incompatibility constraints describe incompatible combinations of set of requirements. Such requirements are represented as unary con-
requirements. Using the &INCOMPATIBLE keyword, we are able to straints (in our case R = {r1 , r2 , ..., rk }).
describe an incompatibility between the variables pension and guar- If requirements ri ∈ R are inconsistent with the constraints in
antee. For example, financial services with low guarantee must not be C, we are interested in a subset of these requirements that should
recommended to users interested in a product that supports their pri- be adapted in order to be able to restore consistency. On a formal
vate pension plan. Filter constraints describe relationships between level we define a requirements diagnosis task and a corresponding
requirements and items, for example, maxinvest ≥ mininvestp, i.e., diagnosis (see Definition 1).
the amount of money the user is willing to invest must exceed the Definition 1 (Requirements Diagnosis Task). Given a set of re-
minimal payment necessary for the financial product. quirements R and a set of constraints C (the recommendation knowl-
In addition the recommendation knowledge base itself, W EE V IS edge base), the requirements diagnosis task is to identify a minimal
supports the specification of test cases that can be used for the pur- set ∆ of constraints (the diagnosis) that has to be removed from R
poses of regression testing (see also Section 3.4). After changes to such that R − ∆ ∪ C is consistent.
the knowledge base, regression tests can be triggered by setting the An example of a set of requirements inconsistent with the defined
—show— tag, that specifies whether the recommender system user recommendation knowledge is R = {r1 : pension = yes, r2 :
interface should show the status of the test case (satisfied or not). maxinvest = 13500, r3 : payment = periodical, r4 : payout =
once, r5 : guarantee = high}. The recommendation knowledge
base induces two minimal conflict sets (CS) [18] in R which are
3.3 Recommender Knowledge Base CS1 : {r1 , r5 } and CS2 : {r1 , r4 }. For these conflict sets we have
Recommendation knowledge can be represented as a CSP [20] with two diagnoses: ∆1 : {r4 , r5 } and ∆2 : {r1 }. The pragmatics, for
the variables V (V = U ∪ P ) and the constraints C = COM P ∪ example, of ∆1 is that at least r4 and r5 have to be adapted in order
P ROD ∪ F ILT where ui ∈ U are variables describing possible to be able to find a solution. How to determine such diagnoses on the
user requirements (e.g., pension) and pi ∈ P are describing item basis of a HSDAG (hitting set directed acyclic graph) is shown, for
properties (e.g., payoutp). Furthermore, COM P represents incom- example, in [4].
patibility constraints of the form ¬X ∨ ¬Y , P ROD the products In interactive settings, where diagnoses should be determined in
with their attributes in disjunctive normal form (each product is de- an efficient fashion [12], hitting set based approaches tend to become
scribed as a conjunction of individual product properties), and F ILT too inefficient. The reason for this is that conflict sets [18] have to be
the given filter constraints of the form X → Y . determined as an input for the diagnosis process. This was the ma-
The knowledge base specified in Figure 2 can be translated into jor motivation for developing and integrating FAST D IAG [12] into
a corresponding CSP where &QUESTIONS represents U , &PROD- the W EE V IS environment. Analogous to Q UICK XP LAIN [18], this
UCTS represents P and P ROD, and &CONSTRAINTS represents algorithm is based on a divide-and-conquer based approach that en-
Figure 2. Financial services knowledge base (view source (edit) mode).
ables the determination of minimal diagnoses without the determi- and thus have a higher probability of being part of a diagnosis. In our
nation of conflict sets. A minimal diagnosis ∆ can be used as basis working example ∆1 = {r4 , r5 }. The corresponding repair actions
for determining repair actions, i.e., concrete measures to change user (solutions for R − ∆1 ∪ C) is A = {r40 : payout = monthly, r50 :
requirements in R such that the resulting R0 is consistent with C. guarantee = low}, i.e., {r1 , r2 , r3 , r4 , r5 } − {r4 , r5 } ∪ {r40 , r50 } is
consistent. The item that satisfies R − ∆1 ∪ A is {DynamicF in}
(see in Figure 2). The identified items (p) are ranked according to
3.4 Diagnosis and Repair of Requirements their support value (see Formula 1).
Definition 2 (Repair Task). Given a set of requirements R = #adaptions in A
{r1 , r2 , ..., rk } inconsistent with the constraints in C and a corre- support(p) = (1)
#requirements in R
sponding diagnosis ∆ ⊆ R (∆ = {rl , ..., ro }), the corresponding
repair task is to determine an adaption A = {rl0 , ..., ro0 } such that
R − ∆ ∪ A is consistent with C. 3.5 Regression Testing
In W EE V IS, repair actions are determined conform to Definition
2. For each diagnosis ∆ determined by FAST D IAG (currently, the W EE V IS supports regression testing processes by the definition and
first n=3 leading diagnoses are determined), the corresponding solu- execution of (positive) test cases which specify the intended behavior
tion search for R − ∆ ∪ C returns a set of alternative repair actions of the knowledge base. If some of the test cases are not accepted by
(represented as adaptation A). In the following, all products that sat- the knowledge base (are inconsistent with the knowledge base), the
isfy R − ∆ ∪ A are shown to the user (see the right hand side of causes of this unintended behavior have to be identified. On a formal
Figure 1). level a recommender knowledge base (RKB) diagnosis task can be
Diagnosis determination in FAST D IAG is based on a total lexico- defined as follows (see Definition 3).
graphical ordering of the customer requirements [12]. This ordering Definition 3 (RKB Diagnosis Task). Given a set C (recommender
is derived from the order in which a user has entered his/her require- knowledge base) and a set T = {t1 , t2 , ..., tq } of test cases ti , the di-
ments. For example, if r1 : pension = yes has been entered before agnosis task is to identify a minimal set ∆ of constraints (the diagno-
r4 : payout = once and r5 : guarantee = high then the underly- sis) that have to be removed from C such that ∀ti ∈ T : C −∆∪{ti }
ing assumption is that r4 and r5 are of lower importance for the user is consistent.
Figure 3. W EE V IS maintenance support: diagnosis and redundancy detection.
An example test case inducing an inconsistency with C is t : redundancies. Consequently, the corresponding set of constraints C
pension = yes and guarantee = high and payout = once does not represent a minimal core. Taking a closer look at the knowl-
(see Figure 2). In this context, t induces two conflicts in C which edge base it appears that two individual filter constraints are redun-
are CS1 : ¬(pension = yes ∧ guarantee = high) and CS2 : dant with each other. More precisely, either the constraint &IF guar-
¬(pension = yes ∧ payout = once). In order to make C consis- antee? = high &THEN guaranteep = high or the constraint &IF
tent with t, both incompatibility constraints have to be deleted from guarantee? = high &THEN guaranteep <> low can be removed
C, i.e., are part of the diagnosis ∆ (see Figure 3). from the knowledge base (in our example, the latter is proposed as
In contrast to the hitting set based approach [4], W EE V IS includes redundant by C ORE D IAG – see Figure 3). In the general case, higher
a FAST D IAG based approach for knowledge base debugging which cardinality constraint sets can be removed, not only cardinality-1 sets
is more efficient and can therefore be applied in interactive settings as in our example [13].
[12]. In this context, diagnoses are searched in C (the test cases used Similar to the diagnosis of inconsistent requirements the C ORE -
for regression testing are assumed to be correct). In the case of re- D IAG algorithm is based on the principle of divide-and-conquer:
quirements diagnosis, the total ordering of the requirements is related whenever a set S which is a subset of C is inconsistent with C, it
to user preferences. In the case of knowledge base diagnosis [4, 16], is or contains a minimal core, i.e., a set of constraints which pre-
the ordering is currently derived from the ordering of the constraints serve the semantics of C. C ORE D IAG is based on the principle of
in the knowledge base. Q UICK XP LAIN [18]. As a consequence a minimal core (minimal set
of constraints that preserve the semantics of C ) can be interpreted as
a minimal conflict, i.e., a minimal set of constraints that are incon-
3.6 Identifying Redundancies sistent with C. Based on the assumption of a strict lexicographical
ordering [12] of the constraints in C, C ORE D IAG determines pre-
To support users in identifying redundant constraints in recom-
ferred minimal cores.
mender knowledge bases, the C ORE D IAG [13] algorithm has been
integrated into the W EE V IS environment. C ORE D IAG relies on
Q UICK XP LAIN [18] and is used for the determination of minimal 4 Empirical Study
cores (minimal non-redundant constraint sets). On a formal level a 4.1 Study Design
recommendation knowledge base (RKB) redundancy detection task
can be defined as follows (see Definition 4). We conducted an experiment to highlight potential reductions of de-
Definition 4 (RKB Redundancy Detection Task). Let ca be a con- velopment and maintenance efforts facilitated by the W EE V IS de-
straint of C (the recommendation knowledge base) and C the logical bugging and redundancy detection support. For this study we defined
negation (the complement or inversion) of C. Redundancy can be an- four knowledge bases that differed with regard to the number of con-
alyzed by checking C − {ca } ∪ C for consistency - if consistency straints, variables, faulty constraints, and redundancies (see Table 1).
is given, ca is non-redundant. If this condition is not fulfilled, ca is Based on these example knowledge bases, the participants had to find
said to be redundant. By iterating over each constraint of C, execut- solutions for the following two types of tasks:
ing the non-redundancy check C − {ca } ∪ C, and deleting redundant 1. Diagnosis task: The participants had to answer the question which
constraints from C results in a set of non-redundant constraints (the minimal set ∆ of faulty constraints has to be removed from C
minimal core). (C = COM P ∪F ILT ) such that there exists at least one solution
As an example, the knowledge base shown in Figure 2 contains for ( (C − ∆) ∪ P ROD).
2. Redundancy detection task: The participants had to answer the groupB groupA
question which constraints in C = COM P ∪ F ILT are redun- (kb2 ) (kb4 )
dant (if C − {ca } ∪ C is inconsistent then the constraint ca is average time (sec.) 281.3 497.5
redundant). correct (%) 50.0 10.0
incorrect (%) 50.0 90.0
knowledge base number of constraints
/variables /faulty Table 3. Time efforts and error rates related to the completion of diagnosis
constraints /test cases tasks.
/redundancies
kb1 (redundant) 5/5/0/0/2
kb2 (inconsistent) 5/5/1/2/0
The second goal of our experiment was to analyze time efforts
kb3 (redundant) 10/10/0/0/4 and error rates related to the identification of redundant constraints
kb4 (inconsistent) 10/10/2/4/0 in recommender knowledge bases. The second hypothesis tested in
our experiment was the following:
Table 1. Knowledge bases used in the empirical study. Hypothesis 2: Even low-complexity knowledge bases
trigger the faulty identification of redundant constraints.
The average time for identifying redundant constraints in knowl-
edge base kb1 was 189.2 seconds, for kb3 337.4 seconds were
The participants (subjects N=20) of our experiment were separated needed. The results show a significantly higher error rate when the
into two groups (groups A and B). All subjects were students of Com- participants had to identify redundant constraints in the more com-
puter Science (20% female, 80% male) who successfully completed plex knowledge base (see Table 4). Hypothesis 2 can be confirmed
a course on constraint technologies and recommender systems. Each since even for low complexity knowledge bases error rates related to
subject had to complete the assigned tasks on his/her own on a sheet redundancy detection tasks are high. With the automated redundancy
of paper and they had to track the time for each task. In our exper- detection mechanisms integrated in W EE V IS, reductions of related
iment we randomly assigned the participants to one of the two test error rates and time efforts can be expected.
groups shown in Table 2. This way we were able to compare the time
efforts of identifying faulty constraints and redundancies in knowl- groupA groupB
edge bases as well as to estimate error rates related to the given tasks. (kb1 ) (kb3 )
average time (sec.) 189.2 337.4
testgroup 1st knowledge 2nd knowledge correct (%) 40.0 0.0
base base incorrect (%) 60.0 100.0
A (n = 10) kb1 (redundancy detection) kb4 (diagnosis)
B (n = 10) kb2 (diagnosis) kb3 (redundancy detection) Table 4. Time efforts and error rates related to the completion of
redundancy detection tasks.
Table 2. Each subject had to complete one diagnosis and one redundancy
detection task. Members of group A had a redundancy detection task of
lower complexity and a higher complexity diagnosis detection task
(randomized order). Vice-versa members of group B had to solve a higher
complexity redundancy detection and a lower complexity diagnosis task. 5 Future Work
There are a couple of issues for future work. The current W EE -
V IS version does not include functionalities that allow the learn-
ing/prediction of user preferences. The importance of individual user
4.2 Study Results
requirements is based on the assumption that the earlier a require-
The first goal of our experiment was to analyze time efforts and er- ment has been specified the more important it is. In future versions
ror rates related to the identification of faulty constraints in recom- we want to make the modeling of preferences more intelligent by in-
mender knowledge bases. The first hypothesis tested in our experi- tegrating, for example, learning mechanisms that derive requirements
ment was the following: importance distributions on the basis of analyzing already completed
Hypothesis 1: Even low-complexity knowledge bases recommendation sessions.
trigger the identification of faulty diagnoses (note that all Diagnoses and redundancies are currently implemented on the
knowledge bases used in the experiment can be interpreted level of constraints, i.e., intra-constraint diagnoses and redundancies
as low-complexity knowledge bases [13]). are not supported. In future W EE V IS versions we want to integrate
The average time effort for identifying minimal diagnoses in fine-granular analysis methods that will help to make analysis and
knowledge base kb2 was 281.3 seconds, the average time needed to repair of constraints even more efficient. A major research challenge
identify diagnoses in kb4 was 497.5 seconds. The results show a sig- in this context is to integrate intelligent mechanisms for diagnosis
nificantly higher error rate when the participants had to identify the discrimination [27] since in many scenarios quite a huge number
faulty constraints in the more complex knowledge base (see Table 3). of alternative diagnoses exists. In such scenarios it is important for
Hypothesis 1 can be confirmed by the results in Table 3 that show that knowledge engineers to receive recommendations of diagnoses that
even simple knowledge bases trigger high error rates and increasing are reasonable. This challenge has already been tackled in the context
time efforts. With the automated diagnosis detection mechanisms in- of diagnosing inconsistent user requirements (see, e.g., [6]), however,
tegrated in W EE V IS, reductions of related error rates and time efforts heuristics with high prediction quality for knowledge bases have not
can be expected. been developed up to now [10, 11].
A major issue for future work is to integrate alternative mech- [8] A. Felfernig, K. Isak, K. Szabo, and P. Zachar, ‘The VITA Finan-
anisms for knowledge base development and maintenance. The cial Services Sales Support Environment’, pp. 1692–1699, Vancouver,
Canada, (2007).
knowledge engineer centered approach to knowledge base construc- [9] A. Felfernig and A. Kiener, ‘Knowledge-based Interactive Selling of
tion leads to scalability problems in the long run, i.e., knowledge Financial Services with FSAdvisor’, in 17th Innovative Applications of
engineers are not able to keep up with the speed of knowledge base Artificial Intelligence Conference (IAAI05), pp. 1475–1482, Pittsburgh,
related change and extension requests. An alternative approach to Pennsylvania, (2005).
knowledge base development and maintenance is the inclusion of [10] A. Felfernig, S. Reiterer, M. Stettinger, and J. Tiihonen, ‘Intelligent
Techniques for Configuration Knowledge Evolution’, in VAMOS Work-
concepts of Human Computation [7, 30] which allow a more deep shop 2015, pp. 51–60, Hildesheim, Germany, (2015).
integration of domain experts into knowledge engineering processes [11] A. Felfernig, S. Reiterer, M. Stettinger, and J. Tiihonen, ‘Towards Un-
on the basis of simple micro tasks. Resulting micro contributions can derstanding Cognitive Aspects of Configuration Knowledge Formaliza-
be automatically integrated into constraints part of the recommenda- tion’, in VAMOS Workshop 2015, pp. 117–124, Hildesheim, Germany,
(2015).
tion knowledge base. [12] A. Felfernig, M. Schubert, and C. Zehentner, ‘An Efficient Diagnosis
Finally, we are interested in a better understanding of the key fac- Algorithm for Inconsistent Constraint Sets’, Artificial Intelligence for
tors that make knowledge bases understandable. More insights and Engineering Design, Analysis and Manufacturing, 26, 53–62, (2012).
answers related to this question will help us to better identify prob- [13] A. Felfernig, C. Zehentner, and P. Blazek, ‘COREDIAG: Eliminating
lematic areas in a knowledge base which could cause maintenance Redundancy in Constraint Sets’, International Workshop on Principles
of Diagnosis (DX’11), 219–224, (2011).
efforts above average. A first step in this context will be to analyze [14] G. Fleischanderl, G. Friedrich, A. Haselböck, H. Schreiner, and
existing practices in knowledge base development and maintenance M. Stumptner, ‘Configuring Large Systems Using Generative Con-
with the goal to figure out major reasons for the knowledge acquisi- straint Satisfaction’, IEEE Intelligent Systems, 13(4), 59–68, (1998).
tion bottleneck and how this can be avoided in the future. [15] E. Freuder, ‘In Pursuit of the Holy Grail’, Constraints, 2(1), 57–61,
(1997).
[16] G. Friedrich, ‘Interactive Debugging of Knowledge Bases’, in Interna-
6 Conclusion tional Workshop on Principles of Diagnosis (DX’14), pp. 1–4, Graz,
Austria, (2014).
In this paper we presented W EE V IS which is an open constraint- [17] Russell Greiner, Barbara A. Smith, and Ralph W. Wilkerson, ‘A Cor-
rection to the Algorithm in Reiter’s Theory of Diagnosis’, Artificial In-
based recommendation environment. By exploiting the advantages telligence, 41(1), 79–88, (1989).
of Mediawiki, W EE V IS provides an intuitive basis for the devel- [18] U. Junker, ‘QUICKXPLAIN: Preferred Explanations and Relaxations
opment and maintenance of constraint-based recommender appli- for Over-constrained Problems’, in AAAI, volume 4, pp. 167–172,
cations. W EE V IS is already applied by four Austrian universities (2004).
within the scope of recommender systems courses and also applied [19] J. Konstan, B. Miller, D. Maltz, J. Herlocker, L. Gordon, and J. Riedl,
‘GroupLens: Applying Collaborative Filtering to Usenet News’, Com-
by companies for the purpose of prototyping recommender appli- munications of the ACM, 40(3), 77–87, (1997).
cations. The results of our empirical study indicate the potential of [20] A. Mackworth, ‘Consistency in Networks of Relations’, Artificial Intel-
reductions of error rates and time efforts related to diagnosis and re- ligence, 8(1), 99–118, (1977).
dundancy detection. In industrial scenarios, W EE V IS can improve [21] J. Marques-Silva, F. Heras, M. Janota, A. Previti, and A. Belov, ‘On
Computing Minimal Correction Subsets’, in IJCAI 2013, pp. 615–622,
the quality of knowledge representations, for example, documenta- Peking, China, (2013).
tions can at least partially be formalized which makes knowledge [22] S. Mittal and B. Falkenhainer, ‘Dynamic Constraint Satisfaction’, in
more accessible – instead of reading a complete documentation, the National Conference on Artificial Intelligence, pp. 25–32, (1990).
required knowledge chucks can be identified easier. [23] S. Mittal and F. Frayman, ‘Towards a Generic Model of Configuraton
Tasks’, in IJCAI, volume 89, pp. 1395–1401, (1989).
[24] B. O’Sullivan, A. Papadopoulos, B. Faltings, and P. Pu, ‘Representa-
REFERENCES tive Explanations for Over-constrained Problems’, in Twenty-Second
AAAI Conference on Artificial Intelligence (AAAI-07), eds., R. Holte
[1] R. Bakker, F. Dikker, F. Tempelman, and P. Wogmim, ‘Diagnosing and and A. Howe, pp. 323–328, Vancouver, Canada, (2007). AAAI Press.
Solving Over-determined Constraint Satisfaction Problems’, in 13th In- [25] M. Pazzani and D. Billsus, ‘Learning and Revising User Profiles: The
ternational Joint Conference on Artificial Intelligence, pp. 276–281, Identification of Interesting Web Sites’, Machine learning, 27(3), 313–
Chambery, France, (1993). 331, (1997).
[2] J. Baumeister, J. Reutelshoefer, and F. Puppe, ‘KnowWE: a Semantic [26] R. Reiter, ‘A Theory of Diagnosis From First Principles’, Artificial in-
Wiki for Knowledge Engineering’, Applied Intelligence, 35(3), 323– telligence, 32(1), 57–95, (1987).
344, (2011). [27] K. Shchekotykhin and G. Friedrich, ‘Diagnosis discrimination for on-
[3] A. Felfernig and R. Burke, ‘Constraint-based Recommender Systems: tology debugging’, in ECAI 2010, pp. 991–992, (2010).
Technologies and Research Issues’, in 10th International Conference [28] M. Stumptner, G. Friedrich, and A. Haselböck, ‘Generative Constraint-
on Electronic Commerce, p. 3. ACM, (2008). based Configuration of Large Technical Systems’, AI EDAM, 12(04),
[4] A. Felfernig, G. Friedrich, D. Jannach, and M. Stumptner, 307–320, (1998).
‘Consistency-based Diagnosis of Configuration Knowledge Bases’, Ar- [29] E. Tsang, Foundations of Constraint Satisfaction, volume 289, Aca-
tificial Intelligence, 152(2), 213–234, (2004). demic press London, 1993.
[5] A. Felfernig, G. Friedrich, D. Jannach, and M. Zanker, ‘An Integrated [30] L. VonAhn, ‘Human Computation’, in Technical Report CM-CS-05-
Environment for the Development of Knowledge-based Recommender 193, (2005).
Applications’, International Journal of Electronic Commerce, 11(2),
11–34, (2006).
[6] A. Felfernig, G. Friedrich, M. Schubert, M. Mandl, M. Mairitsch, and
E. Teppan, ‘Plausible Repairs for Inconsistent Requirements’, in IJCAI,
volume 9, pp. 791–796, (2009).
[7] A. Felfernig, S. Haas, G. Ninaus, M. Schwarz, T. Ulz, M. Stettinger,
K. Isak, M. Jeran, and S. Reiterer, ‘RecTurk: Constraint-based Recom-
mendation based on Human Computation’, in RecSys 2014 CrowdRec
Workshop, pp. 1–6, Foster City, CA, USA, (2014).