One of the most useful features of goal models of the i* family is their ability to represent and reason about satisfaction in uence of one goal to another. This is done through contribution links, which represent how satisfaction or denial of the origin of the link constitutes evidence of satisfaction/denial of its destination. Typically in the i* family, the nature and level of contribution is represented through qualitative labels (\+", \ ", \++" etc.), with the possibility of alternatively using numeric values, as per various proposals in the literature. Obviously, our intuition seems to suggest, labels are easier to comprehend and to come up with, while the use of numbers raises the question of where they come from and what they mean, adds unwarranted precision and overwhelms readers. But are such claims fair? Based on some early experimental results, we make the case for more empirical work on the matter in order to better clarify the di erences and understand how to use contribution representations more e ectively.
Goal models of the i* family [
Which type of label should we then use? Our intuition suggests that
qualitative labels are easier to comprehend and to come up with, while the use of
numbers automatically raises the question of where they come from [
The rest of the presentation is organized as follows. In Section 2 we discuss qualitative versus quantitative contribution in goal models. In Section 3 we describe our experiments. Then in Sections 4 and 5 we o er conclusions and our plans for the future. 2
Modeling and reasoning about contribution in i* languages is generally
understood as something to be done in a qualitative fashion. The use of qualitative
labels is rooted on the idea that Non-Functional Requirements (NFRs), the
matter modeled through e.g. i* soft-goals, need to be dealt with in a way that does
not necessitate availability of accurate empirical data and complex and hard
calculations of global optima [
However, Giogrini et al. show that numbers can be used as well [
We have recently shown, however, that numeric representations of
contributions have their merits [
What is perplexing, though, is concern (b) above: the use of the numbers not to just make the AHP decisions but also as contribution labels on the goal model in order to convey information to readers. Moreover, the question seems to extend to both qualitative and quantitative approaches. The issue seems to be the di culty to de ne what exactly this information is, i.e. what exactly the reader is supposed to learn or understand by looking at models such as those of Fig. 1. Moreover, if we explicate the objective of the representation, it is logical to subsequently ask whether there are ways to read it (i.e. ways to understand labels and how they combine) that are more natural and e ective than others. Below we describe our attempts to understand this problem better via conducting two small experimental studies. 3
The information contained in the contribution structure (i.e. a portion of a goal model containing contribution links) amounts to a set of binary relations between goals showing how one contributes to the other. Of course, this information may as well be expressed in text or a catalogue of separate logical formulae. But we choose to put all these individual pieces in a graph, because we apparently aim at presenting a whole that emerges by combining them. In particular, if we consider contribution structures to be visual representations of a decisionmaking problem, we seem to be hoping that using a graph facilitates ad-hoc detection of good decisions. In other words, by just looking at such diagrams some readers must be able to intuitively combine individual contribution links and detect optimal solutions. Moreover, it may be fair to even assume that the detection process is natural and does not assume prior training to the language.
We investigated whether the current representations of contribution indeed
have such properties and whether the choice of contribution labels
(qualitative vs. numbers) has any in uence [
Inspired be these early indications, we recently performed a second study. This time we presented 10 models, ve (5) quantitative and ve (5) qualitative to another eight (8) participants. Each goal model of one group matches a model of the other group in all aspects except for the contribution labels, as seen in Fig. 1. For the labels, we randomly assign contribution symbols and values, to produce qualitative and quantitative models respectively. Overall, the models contain 4 to 7 soft-goals and 6 to 14 contribution links in various organizations and two alternatives to choose from. We present the models separately in a random order to the participants and we give them sixty (60) seconds to select the optimal alternative for each. The optimal solution in the qualitative models is this time based on the standard label propagation algorithm. Of the forty (40) answers received for each type of model, qualitative and quantitative, sixteen (16) and thirty (30) agree respectively with our calculation of the optimal. The binomial test shows that in the latter case (the quantitative responses), the result is signi cantly unlikely to be random. Further, while in the qualitative case the successful responses uctuate with respect to model size (in a noncharacterizable fashion), the successful responses in the quantitative case are una ected by model size (6 out of 8 persons get it right for all models).
To sum up, considering contribution structures as visual instruments for adhoc detection of optimal solutions, these small studies aim at understanding whether there is an a-priori way by which uninitiated readers expect that such instruments work. The preliminary evidence appears to support that some participants may have some prior inclination to deal with numbers in a speci c way. This, we conjecture, might be due to the fact that dealing with weights, percentages and proportions is much more common in education and daily life than the use of custom symbols like qualitative labels. Regardless, we believe that until we have conclusive results we must resist the temptation to dismiss numerical representations of contribution as confusing or di cult to comprehend. 4
The possibility of qualitative satisfaction analysis is one of the major advantages of using goal modeling notations of the i* family. However, when one considers goal models as visual instruments for making quick assessments about optimal decisions, the question of naturalness and e ciency of the representation arises. Our preliminary trials on readers without prior training seem to suggest that numeric representations evoke a way of visual reasoning that is consistent with simple aggregation arithmetic over contribution labels.
If further study con rms this or other similar results, the emerging research
problem is how we incorporate them in the language itself, also in a way that
both the good properties of qualitative reasoning are preserved and a
systematic elicitation approach remains available. For example, designs that depart
from static visual representations, such as interactive evaluation procedures [
The speci c empirical goal we have set, i.e. that of understanding the comprehensibility of di erent labelling and aggregation strategies for contributions, requires us to consider many more experimental trials before getting a clear picture. Sizes, structures of models and domains they represent, numerical precision levels or even font sizes and shapes are examples of simple variables that need to be controlled for.
Furthermore, depending on what training and background assumptions one makes for the use of i* representations in practice there are at least two ways to organize the investigation. One possibility is to keep looking for natural a-priori visual reasoning procedures, formed through other experiences of the reader or potentially utilized through analogies. In that case the educational, professional or other background of the reader becomes an important factor. We plan to run the same experimental procedures with student participants from nonmathematical disciplines such as humanities and from random samples from the general population. We plan to also add a qualitative component in order to elicit how participants exactly think whenever they try to reason about contribution structures. Are for example any analogies utilized, i.e. does the representation remind them of something familiar to which they refer in applying a reasoning strategy? A second possibility is to assume that contribution labelling and aggregation models are a subject to be trained at beforehand. In that case, the learnability of these models needs to be tested, such as how soon and with what accuracy trainees are able to comprehend a contribution structure.
Finally, in all cases, comprehensibility criteria can be de ned in di erent ways. For us it is the ad-hoc detection of optimal solutions. Alternative criteria include how quickly and accurately the user can read and remember individual contributions, successfully explain a given satisfaction degree or, say, detect con icting nodes. By thinking of concrete criteria we actually force ourselves to think, decide and propose what uses of goal models are important and why. This call for re ection is, we nd, one of the greatest bene ts of experimental work.