-Goal models have been suggested to be an effective way to support decision making in early requirements engineering. Such models are capable of representing a large number of alternative ways to solve stakeholder problems and comparing them against each other with respect to higher level objectives. Core to the realization of such analysis is the concept of the contribution link that represents how satisfaction of one goal affects satisfaction of another. Many ways for representing and assigning precise meaning to contribution links have been proposed, each with different properties and advantages. But which one agrees more with user preferences on how such links should be used? In this paper, we present an experimental design for comparing two ways for representing contribution links, symbolic versus numeric, with respect to how accurately and quickly users identify optimal decisions using each representation format. Apart from comparing the two representation techniques and advising the modeling practice accordingly, the study aims at showing how a quality construct we call intuitiveness can be added to the range of criteria a modeling language designer has at her disposal for evaluating her language design decisions.
Goal models [
Many representational and semantic frameworks have been
proposed within the goal modeling community to allow such
analysis [
However, limited work has been done in terms of how users of the models perceive what the symbols and/or numbers mean and how they expect to use them in order to make inferences pertinent to decision making. It is particularly useful to understand how users intuitively assign meaning to signifiers within the language, when no prior training and/or experience with the language can be assumed for them. Knowing what untrained user’s intuition is, language designers can settle for representations and semantics that are closer to the user’s expectations and, as such, easier to learn and more accurate to use.
In this paper we present an experimental design aimed at comparing the intuitiveness of qualitative versus quantitative contribution labels in goal models, having assumed specific semantics for each. Our design aims at showing which of the two visualization-meaning pairs leads to more accurate decisions in the least amount of time.
The rest of the paper is organized as follows. In Section II we offer some background on goal models, contribution links and their semantics. In Section III we describe the experimental design and in Section IV we summarize and review some of the related work.
The goal models we consider in this study look like the ones in Figure 1. The nodes (ovals and clouds) are goals that describe states of the world that the actor in question (circular shape) has within their scope (large shaded dashed circle) and want to achieve or maintain. The ovals describe hard-goals, which are goals that come with a clear way to decide when they are satisfied, while soft-goals (the clouds) are goals for which this is not the case.
Goal modeling languages define a variety of relationships
between goals and allow for great structural freedom [
A contribution link shows in what way satisfaction (or not) of the origin of the link affects satisfaction (or not) of the destination of the link. This way of affecting the other goal is described through the label of the contribution link. Typically the label will show whether the effect is positive or negative and/or how large it is. Nevertheless, there are more than one ways to represent contribution labels and, for each, multiple ways to define their semantics.
The original and seemingly most popular approach to
modeling contribution labels is through symbols (diagram
on the left in Figure 1). Thus “+”, “++”, “ ” and “ ”
denote respectively positive (“helps”) very positive (“makes”),
negative (“hurts”) and very negative contribution (“breaks”).
Alternatively numbers can be used to convey this information
(diagram on the right in Figure 1). Two distinct numeric
approaches have been introduced in the literature. The
approach by Giorgini et al. [
Informal descriptions such as the above about the meaning of the contribution link allow a model reader/user (henceforth simply user) perform some very basic inferences by looking at the model. For example, she can compare two contributions with respect to which one is larger or she can even choose between alternatives in the hard-goal decomposition with respect to a soft-goal of interest. For example, in the symbolic model on the left side of Figure 1, if to Reduce Scheduling Effort is an important soft-goal, then we know that (Choose Schedule) Automatically is preferable than doing so Manually, by simply looking at the contribution labels and without knowing precisely what they mean. However, more detailed semantics need to be given in order to perform more complex inferences such as deciding on the satisfaction status of a goal that receives multiple incoming contribution links, 1Giorgini et al.’s expressive framework also includes a subscript representing what is being contributed between satisfaction and denial; both their quantitative and qualitative version includes this dimension. Presentation of this dimension is outside our scope.
Label ++ +
Effect FS ! FS PS ! PS PD ! PD FD ! FD FS ! PS PS ! PS PD ! PD FD ! PD
Label
Effect FS ! FD PS ! PD PD ! PS FD ! FS FS ! PD PS ! PD PD ! PS
FD ! PS or, as we will see below, deciding the optimal alternative by considering all contribution links in the structure.
Giorigini et al. have developed the most expressive
semantics for both symbolic and numeric links [
In the quantitative (numeric) framework the rules are replaced by algebraic formulae. The researchers allude to three possible ways by which this formula can be structured, seen in the top three rows of Table II; in practice their framework is open to the adoption of many other ways. Given a set of goals g0 2 Og , each with satisfaction value s(g0) 2 [0:0; 1:0] targeting goal g with contribution links weighted as w(g0; g), the satisfaction value of goal g is expected to be s(g) as defined in each of the formulae. In all the proposed formulae (“Bayesian”, “Min-Max” and “Serial-Parallel”) aggregation is implemented through maximization. Note that in this semantic framework, users are supposed to understand the numbers of the contribution links as absolute contribution values potentially elicited and understood in isolation from the other ones.
A different interpretation of numeric contributions, which is
of particular interest here, is the de-facto approach followed
by URN [
While the linear interpretation is arguably less expressive
and imposes structural limitations to the models (the
softgoal sub-graph must be acyclic) they have been found [
standing the use of contribution labels for inferring satisfaction propagation, it is natural to ask which one is more “friendly” to users of the models. One aspect of “friendliness” is the level by which the intended meaning and use of the contribution aligns with the users’ intuition.
We use the (working) theoretical construct “intuitiveness”
of a model construct to describe the ability of untrained users
of a conceptual model to readily understand what the construct
means and how it should be used to make inferences in the
model. The concept is analogous to the idea of an intuitive
human-machine interface: the more intuitive an interface is,
the more readily first-time users can use it without the need
to resort to help, a manual etc. The term is akin to that of
learnability which is a quality of an interface that allows
users to learn how to use it easily and quickly [
With this user-machine interface analogy in mind, we can
reasonably claim that conceptual models are also artifacts to be
efficiently used by people, where “use” here is “understanding
and communication” [
Intuitiveness, as we conceptualize and apply it here, measures the entire package of a concept and its visualization: the visualization evokes a meaning, which, in turn, is used to make inferences. When a user is exposed to a visualization and ends up performing an inference that is not intended by the designers, a sub-optimal decision may be claimed at any of the levels: either the users did not map the visualization to the right concept (e.g. confused a “goal” for an “event”, both otherwise being clearly understood concepts), or they did so correctly but did not understand the concept as the language designers intended them to (e.g., they correctly mapped a symbol to an “upper-goal” but didn’t know what to do with the latter). While training may arguably establish correct bridging between visualization and inference in the long term, intuitiveness is exhibited when limited such training is necessary.
In the context of contribution links in goal models, the inference we are interested in is how users assign satisfaction to goals given satisfaction of other goals based on their own interpretation of what contribution labels seem to mean. Reversely, their observed inferences reveal their perceived meaning of the links, and, as such, the former can be used to develop empirical operationalizations of the latter. In the experiment we describe below, we ask the users to make decisions using goal models. To do so, they need to adopt a way of using the contribution link and, implicitly, a semantics for those links. The alignment of the semantics implied by how users use the models with the designed semantics (i.e. the semantics intended by the designers), as exhibited by whether the results of the inference match, is, we claim, a possible indication of the intuitiveness of the designed semantics.
and assigning meaning to contribution links and compare them with regards to measures of intuitiveness and efficiency.
We specifically compare the symbolic against the numeric approach, the latter under the linear interpretation. There is one main research question we wish to address:
RQ. Which of the two methods for modeling contribution links is the most (a) intuitive and (b) efficient for the task of identifying optimal alternatives in goal models?
We address the above through a controlled experiment with human participants.
B. Experimental Tasks and Measurements
1) Measures: The two constructs we are considering are intuitiveness and efficiency. We theoretically defined intuitiveness as the degree by which untrained users can make accurate inferences with models they are exposed to. Operationally, we will measure intuitiveness by exposing the experimental participants to a sample of goal models and asking them to perform an inference, which we then compare with the “correct” inference as dictated by the adopted contribution modeling approach. Perception of intuitiveness is also included as a possible measure via self-reporting of participants’ confidence about the aforementioned inferences they perform.
Efficiency, will be, in this context, measured as the total time it takes for participants to perform this inference, independent of correctness.
2) Experimental Units: We develop a number of goal models such as those in Figure 1. We specifically develop two (2) sets of models: qualitative, in which contribution lables are symbolic, and quantitative, where contribution labels are numeric following the “linear” semantics. All models contain one OR-decomposition of hard-goals (so one decision) together with a hierarchy of soft-goals that are used as criteria for choosing the optimal alternative within the decomposition.
By having a unique root goal in the soft-goal hierarchy the goal model implies that, generally, one of the depicted alternatives is optimal compared to the others.
To show how this is possible let us go back to Figure 1 and consider the decomposition Manually versus Automatically.
We can assume that whenever we pick one of the alternatives the corresponding hard-goal is assigned maximum satisfaction and, if applicable minimum denial value. Thus, to choose the alternative Manually we assign to it maximum satisfaction values fFS, NDg (qualitative case) or s(Manually) = 1 (quantitative case), and to all other alternatives (in our case only Automatically) values fNS, NDg or s( ) = 0. We then perform recursive bottom-up application of the propagation rules of Tables I and II (depending on case), in order to calculate the satisfaction of the root goal Overall Scheduling Quality.
For the quantitative models specifically we follow the linear interpretation of the last row of Table II. Different choices of alternative will result in different satisfaction level for the root goal. The alternative that results to the highest satisfaction value for the root goal is the optimal.
In the quantitative case, satisfaction is a unique value and the comparison straightforward. In the example of Figure 1 (model on the right), Manually causes satisfaction of Overall Scheduling Quality by approx. 0.6 compared to approx. 0.4 implied by selection of Automatically. Thus, Manually is the optimal alternative2.
In the qualitative case, calculation is less straightforward
in that there are two variables to consider, satisfaction and
denial. To make different satisfaction levels comparable we
aggregate the two values into one, the aggregated satisfaction
value. To calculate the aggregated satisfaction values, we
firstly associate qualitative satisfaction labels fN, P, Fg with
numeric values 0,1,2, respectively. We denote the resulting
numeric satisfaction and denial of a goal g as sat(g) and
den(g), respectively. The aggregated satisfaction value is then
sat(g) den(g) which results to an integer in [
2To simulate the experience of our experimental participants the reader can look at the diagram and verify if the assertion that Manually is optimal can be inferred intuitively, by roughly comparing the numbers and without performing precise calculations. model on the left of Figure 1, it can be verified that Overall Scheduling Quality is fPS, PDg for Manually and also fPS, PDg for Automatically. Hence, both alternatives lead to the same aggregated satisfaction value for the root goal, that is 0, and as such they are equally optimal.
3) Model Sampling: To develop the samples of goal models that we need, we pick a goal structure (more below) and randomly choose contribution link labels, such that the distance in satisfaction value of the best alternative (i.e., optimal with respect to the root soft-goal) compared to the second best alternative is controlled to not exceed or be less than a fixed value. Thus, we ensure that the distance is neither too large so that the task of identifying the optimal alternative is trivial in all cases, nor too small to constitute an unimportant distance in terms of decision making and also be impossible to detect even by some of the participants.
Specifically, in qualitative models contribution labels are assigned randomly one of the labels “++”, “+”, “ ”, and “ ”, such that the first alternative has a distance from the second alternative of two (2) levels of satisfaction, based on the aggregated satisfaction value of the root soft-goal that each alternative results in.
Thus, a goal model in which the best alternative, when chosen, makes the root goal fFS, NDg, hence aggregated value 2 0 = 2 and the second best makes the root goal fPS, PDg, hence aggregated value 1 1 = 0, qualifies for inclusion to our sample as the distance of the two top alternatives is 2. A goal model, on the other hand, in which the top two alternatives are both fFS, PDg have both an aggregated value of 1 and hence distance of zero; so they do not qualify.
In quantitative models we also randomly sample while ensuring that the first alternative has a distance of 0.4 from the second; again, in terms of the satisfaction they imply for the root goal. For example a set of weights that gives satisfaction value 0.7 to the first alternative and 0.3 to the second qualifies for inclusion to our sample. The model of Figure 1 (right), focussing on the Choose Schedule decision, does not qualify as the distance is 0:6 0:4 = 0:2
The choice of 0.4 is made to match the corresponding choice in the qualitative models. Observe that in qualitative models the maximum distance between alternatives is 4 (fFS, NDg versus fNS, FDg so 2 - (-2)). The distance we demand is 2, thus half of this space. Respectively in the quantitative models the maximum theoretical distance is 1.0, so half the space would be 0.5. However we end-up to 0.4 – biasing slightly against numeric models – as for some of our structures there does not seem to exist combinations of numeric labels that yield a distance of exactly 0.5.
To remain consistent with the claim that linear interpretation is chosen due to the systematic elicitation approach that is afforded by it, namely AHP, all numeric sampling is done through simulated AHP pair comparison processes and subsequent profile calculations, such that the consistency ratio (CR) is less than 0.1.
It is worthwhile to note that, while it is understood that the two representation approaches have different precision levels, the numeric one being understood as more precise, this difference does not seem to threaten our comparison effort but rather offer us a possible explanatory view to a potential result. If, for example, a difference is discovered in favor of the numeric format, it might be due to a number of reasons including precision but also, e.g., familiarity of the participants with numerical reasoning and assessment of proportions. Identifying those precise reasons – assuming the effect is eventually observed – is a matter for future investigation.
4) Instrument and Tasks: Using the sampling procedure described above we develop a total of twelve (12) quantitative models. The goal structures refer to three (3) different domains describing intentional structures in the context of decisions: Choosing an Apartment, Choosing a Course, and Choosing a Means of Transportation. We develop the models based on specific domains, rather than using dummy names (A, B, C etc.) for the purpose of making the tasks more realistic.
This introduces the threat that participants may use their own opinion of how goals are related to each other ignoring the information provided in the contribution link. To avoid this bias, participants are told that the structures represent decision problems of a third party and that their task is to help that party make the decision based on the priorities of that party as these priorities are represented in the goal structure.
For each domain we develop two (2) structures (one with two and one with three alternatives) and for each structure we sample two (2) labels-sets (i.e., sets of labels for the contributions) sampled as described above. To produce qualitative counterparts we simply copy the twelve (12) quantitative structures and replace the numbers with randomly sampled symbolic labels – again, as described above.
We then present the resulting twelve (12) models of each type (qualitative and quantitative) to the participants one after the other asking each time what they believe the optimal alternative is. Domains are presented in random order and models within the domains in random order as well. Three video presentations precede these tasks: one describes decision problems in general, another introduces goal models and a third one introduces the three domains. The second video specifically, describes the intuition behind the contribution links of each type carefully without getting into the mechanics of satisfaction propagation. The videos are scripted and are the same in the two cases (qualitative and quantitative) except obviously for the places where the numbers or symbols are presented.
The videos are chosen as the instruction method for three reasons (a) allow for repeatability of the procedure, (b) control for biases in training, and (c) allow for remote administration or administration by non-experts.
A simple demographics questionnaire (age, sex, education, prior knowledge of goal models) precedes the main test.
Participants are unlikely to be familiar with goal models, and input coming from those who actually are will be discarded.
However, if familiarity to goal modes turns out to be more prevalent, treating familiarity as a covariant is another option.
5) Participant Sample: We plan to consider the University option of measuring perceived intuitiveness via self-reporting
student pool as the population to opportunistically sample the confidence of participants on their inferences. Our design
from, specifically intermediate/senior students from various relies on random sampling of a number of goal structures
disciplines. We claim that this does not harm the generalizabil- depicting a decision problem and asking participants to choose
ity of the particular study. Firstly, having a valid noteworthy the optimal of the available choices, thereby making intuitive
intuition about how the particular conceptual modeling con- inferences about contribution links. The decision problems are
struct works does not seem to require experience and skill in carefully sampled to allow for a controlled distance between
any specific field: goal models refer to concepts (goals and the optimal and second optimal choice.
their fulfillment) that should be accessible to anyone who has Empirically evaluating the effectiveness of diagrammatic
successfully entered post-secondary education – compared to, notations has been widely studied in the literature. Much of the
for instance, component diagrams describing software designs. research in the field has been dedicated towards understanding
Secondly, it seems to be the implicit ambition of goal modeling the comprehensibility of (various aspects of) UML and ER
language designers that goal models are artifacts that not only diagrams – e.g., [
6) Variables and Analysis Approach: It becomes obvious Work that relates to understanding the comprehensibility
from the above that the experiment is a simple compari- of goal models specifically is more limited. Horkoff et al.
son between two levels (qualitative vs. quantitative) of one evaluate an interactive evaluation technique for goal models
independent variable/factor (contribution link representation [
It is also possible to measure confidence in each partic- of the language’s primitives. Elsewhere, Hadar et al. [
Simple comparisons between means appear to be sufficient intuitiveness of the rules in Table I. In that experiment, we
as a statistical procedure, with the expected deviation from presented to experimental participants a series of contribution
normality kept in view – the scale [
IV. SUMMARY AND RELATED WORK “hunch” is with respect to the satisfaction of the destination
We presented an experimental design for comparing the goal and comparing their input to the authoritative one of Table
intuitiveness of symbolic versus numeric contribution links in I. Among our findings were that rules involving positive labels
goal models. We use intuitiveness as our main comparison and goal satisfaction are more intuitive to ones with negative
construct, defined as the ability of novice users of the notation labels and goal denial.
to correctly understand how they can use it. We operationalize We also endeavored to compare the quantitative rules of
intuitiveness by measuring agreement between authoritative Table II [
Finally, in a different effort [
Exactly as we propose here, we presented users with decision problems and asked them to pick the optimal alternative using each of the visualizations under comparison. We found that the combination of pie-charts and bar-charts lead to more accurate identification of the optimal alternative and that diagrams were not better in none of the tests or measures.
The difference of the above effort [
The result can thus be interpretable at either level. For the future, we are interested in exploring theoretical and methodological approaches through which these two aspects can be separately evaluated. The endeavour is not a simple one, as understanding of any communication of a concept can be argued to be affected by the way it is communicated – through words, visualizations or other methods. Thus it may prove difficult to measure comprehension of a concept as a “pure” abstraction. Such a problematic demonstrates how empirical investigation, even at the conceptualization stage, forces us to think more deeply into the substance of the process of conceptual modeling and the nature of its artifacts.