The concepts and semantics of constraint solving and configuration need to be understood in order to be able to develop one's own configuration knowledge bases. Developing a related basic understanding is in many cases quite challenging. Consequently, further support is needed that makes the learning of configuration knowledge representation practices and semantics less efortful. In this paper, we provide a short overview of ConGuess which is a game-based learning environment for constraint-based configuration tasks. In this context, we report the results of a user study which focused on an analysis of the perceived complexity of diferent constraint types and on a corresponding usability analysis.
1. Introduction
Assuring the correct understanding of configuration
knowledge representations and corresponding semantics
is an important issue specifically in industrial
configuration settings. Such an understanding can be regarded
as a precondition for successful configurator
development and maintenance [
The overall idea of ConGuess is to pre-generate conifguration tasks (represented as CSPs) and let users (game players) try to figure out correct solutions for the defined tasks. With this, ConGuess follows the idea of earlier related work focusing on the learning of graphical conifguration constraints (specifically, incompatibility constraints) and the concepts of hitting sets in model-based diagnosis (specifically, minimal food item sets that cover all relevant vitamins) [7, 8, 9].
Also in this line of research, Jeferson et al. [ 10] present the application Combination which supports the learning of configuring color ray emitting wooden pieces such that no color array hits a wooden piece of diferent color.
Compared to related work, ConGuess extends the expressivity of constraint representations and also includes a gamification-based approach that can help to increase user engagement.
The contributions of this paper are the following: (1) we provide a short introduction to the ConGuess gaming app, (2) we report the results of an initial complexity and usability analysis that has been conducted in an Artificial Intelligence university course, and (3) we discuss diferent open issues for further related research.
The remainder of this paper is organized as follows. In
Section 2, we provide a short overview of the ConGuess
app specifically introducing the major idea behind.
Thereafter, in Section 3, we discuss first insights regarding the
perceived complexity of diferent constraint types. In
Section 4, we report results regarding the usability of
ConGuess. In Section 5, we discuss potential threats
to validity. The paper is concluded with a discussion of
open research issues in Section 6.
In the line of related research (e.g., [7]), ConGuess is
provided as Android app1 which includes mechanisms for
automated CSP generation and evaluation of solutions.2
In ConGuess, players have to solve pre-generated CSPs
[
(b) A more complex configuration task. which increases their overall game score. If a player proposes a configuration inconsistent with the given set of constraints, his/her score is not reduced and further tries are possible. With an increasing number of unsuccessful tries, the number of points that can be received for a correct solution gets decreased. Finally, the game provides a global highscore ranking which helps to further motivate users to improve their personal highscore. 3. Complexity of Constraint Types Our goal was to better understand in which way diferent types of constraints are understood by players. In order to achieve this goal, we performed a user study with 150 bachelor students engaged in an Artificial Intelligence course at the Graz University of Technology.
Best-performing students had the chance to achieve additional bonus points considered then as a part of the overall evaluation. In total, 780 game sessions have been completed within the scope of the user study resulting in an average number of 5.2 gaming sessions per study participant (with an average of 6 levels per session).
A screenshot of ConGuess in action is provided in Figure 1 which depicts two diferent configuration tasks (a more simple one on the left hand side and a more complex one on the right hand side). The value of each corresponding variable has to be specified individually indicated by the select button. A major objective of the In each ConGuess session, correct and wrong guesses app is to make the configuration task representation as were tracked in combination with the corresponding conunderstandable as possible. For this reason, the overall figuration task shown to the player. The error rate rule of the app in terms of information visualization is of specific configuration tasks (CSPs) was tracked followthat each configuration task fits into the screen without ing the metric shown in Formula 1. In this context, the need of scrolling. is the total amount of guesses and is the amount of wrong guesses for a CSP.
As mentioned, constraint satisfaction problems (CSPs)
in ConGuess are pre-generated. The consistency of in- Within the scope of our study, we compared diferent
dividual configuration tasks (CSPs) is checked with the configuration task types with regard to their
understandChoco constraint solver. If a generated configuration ability: configuration tasks (1) consisting of equality and
task (variables and corresponding constraints) is consis- inequality constraints, (2) consisting of constraints
intent, the corresponding setting is stored for further usage cluding a range restriction, i.e., <, >, ≤ , ≥ , (3) with
impli(as configuration task given to players). Player-proposed cations (requires) and equivalences, and (4) with diferent
solutions as well as generated configuration tasks are numbers of constraints and variables.
checked for consistency using Choco.3
=
(1)
3Details on the ConGuess constraint solving and configuration task
generation approach can be found in Hofbauer and Felfernig [
In a first step, we focused on the analysis of singleconstraint configuration tasks, i.e., configuration tasks with only one constraint included (|| = 1). Tables 1–5 include example constraints which represent a corresponding analysis class, for example, the constraint 1 = 2 in Table 1 represents a configuration task with a singleton constraint of type equality constraint.
Similarly, the first constraint in Table 2 represents a conifguration task with a single constraint of type <.
Equality and Inequality Constraints. First, we have analyzed player failure rates when being confronted with singleton equality and inequality constraints. The corresponding rates are depicted in Table 1. As can be immediately seen, the error rates for such constraints are rather low (on an average, below 5%) indicating a high degree of understandability in the reported basic setting.
()
avg. in % X1 = X2
X3 != X2
Number of Constraints. When using {→, ←} instead of ↔ for expressing equivalence knowledge, we need twice the amount of constraints. As could be observed in our analysis, an increasing number of constraints leads to increasing error rates due to a lower understandability of the configuration task (see Table 4).
() avg. in %
Range Restriction Constraints. In the next step, we 17.12 analyzed the understandability of range restriction constraints (<, >, ≤ , ≥ ) (see Table 2). Compared to settings including the < and > operators, error rates significantly 36.87 increase with settings including ≤ and ≥ operators. One way to explain this significant diference is the increased complexity of {≤ , ≥} due to the fact that both dimen- Table 4 sions, inequality and equality have to be taken into ac- Error rates with an increasing number of constraints. count at the same time.
Requires and Equivalence Constraints. In this context, we have compared the understandability of individual implications (requires) and equivalences (see Table 3). We can see that equivalence constraints have slightly lower error-rates than requires constraints. This is a result that has also been confirmed by a previous study of Felfernig et al [3]. One way to explain this diference is a potentially higher overhead induced by the analysis of implications since equivalences can be reduced to settings where both sides of the logical operator must have the same logical value. Further related work on model understandability can be found in Sepasi et al. [11] where cognitive complexity is also measured on the basis of eye tracking technologies.
N-ary Constraints. The highest error-rates in our study were encountered with constraints involving more than 2 variables. As we can see in Table 5, even configuration tasks with || = 1 are already dificult to solve for players, if the number of included variables is greater than 2. Furthermore, by increasing the number of constraints in a configuration task, it becomes extremely challenging for players to find a consistent solution.
() c1: X3 ≤ (X1 + 3) = (X2 - 2) c1: X3 ≤ (X4 + 4) c2: X4 < X1 ≤ X3 c1: X3 ≥ X1 ≤ X2 c2: (X1 · 2) > X2 c3: X3 < X2 c1: X4 != (X3 + 3) < X1 c2: (X4 - 2) ≥ X1 c3: X1 < (X3 · 3) c4: X4 != (X1 + 3) we will try to adapt the complexity measures currently integrated into the ConGuess configuration task generTo evaluate the usability of ConGuess, we have per- ation. Improved complexity measures could result in a formed a usability study with 10 participants (computer more user-centered increase of configuration task comscience students on the bachelor level). For this purpose, plexity and with this potentially also to a corresponding we have used the System Usability Scale (SUS) [12]. improved learning experience.
SUS is a widely used tool for evaluating the usability of systems and software applications and it consists of a questionnaire with 10 items. After using ConGuess References (without further explanations), the participants had to rate their perception of the software on a 5-point Likert scale. Following the SUS calculation scheme resulted in an overall evaluation of 89.5 out of 100 which is an excellent SUS rating expressing clear understandability and high willingness to use the system.