i* is a well known goal modeling framework, developed by a large and geographically dispersed research community. Currently, i* users tend to ascribe different and conflicting meanings to its constructs, leading to a nonuniform use of the language, and consequently undermining its adoption. In previous works, we proposed ontological guidelines to support the creation of i* models, in an attempt to provide a solution to this problem. In this paper, we present an empirical study, to evaluate these ontological guidelines. Results show that for more experienced conceptual modelers, the ontological guidelines indeed support i* modeling. However, results are not as positive for nonexperienced conceptual modelers.
1. Introduction
We believe this problem can be solved with the use of a foundational ontology
to interpret the semantics of the i* concepts. A foundational ontology is a formal system
of domainindependent categories that can be used to characterize the most general
aspects of concepts and entities that belong to different domains in reality
Nowadays, empirical studies are considered appropriate means to prove the
effectiveness of a new approach. For
Having this in mind, we decided to conduct an experiment to confirm our intuitions that the use of ontological guidelines lead to i* models of better quality. The experiment was conducted in two colleges, having fiftyfive subjects in total. The subjects were students of a Systems Analysis and Development course and the PhD and Master program in Computer Science. The main goal of the experiment was to verify if the ontological guidelines cited above are useful or not in the development of i* models. For that, the subjects participated in modeling activities with and without the use of the guidelines and, then, the results were compared. In the experiment applied with PhD and master students, the results show that the ontological guidelines are useful for the development of i* models. Among the population of the second experiment application, composed of less experienced conceptual modelers, the experiment results were not so positive.
The remainder of this article is organized as follows: Section 2 presents information on the i* framework and its variants; Section 3 describes the UFO fragment applied in this work; Section 4 presents some of the proposed ontological guidelines; Section 5 describes the empirical study; and, finally, Section 6 concludes the paper. 2.
The original i* framework is described in (Yu, 1995). Since then, several variants have
been proposed, for instance GRL and Tropos, see
One of the most controversial constructs in the language is the meansend link.
In the original i* (Yu, 1995), this link is used to connect a goal or a task to softgoals. In
GRL, this link is applied to connect a task to a goal, a task to a task and a resource to a
task. However, in the i* wiki, one of the major sources of material about the language,
this link is only used to connect a task to a goal.
These different interpretations and uses make a new i* learner confuse. She may ask herself: when can I use a meansend link after all? Why is it used this way? Why can't I use a meansend link between a resource and a goal, for example? We argue that the best way to respond to these questions is to understand the ontological semantics behind the constructs of the language. By understanding their ontological nature, we may provide good reasons why a concept or a link may or may not be used in a particular way. 3.
Here we briefly present the UFO concepts that are used in this paper provide an
interpretation to i*. To facilitate reading we use a different font to highlight the UFO
concepts. For a fuller presentation on UFO, the reader should refer to
In UFO, a stakeholder is represented by the Agent concept, defined as a concrete Endurant (i.e. an entity that endures in time while maintaining its identity) which can bear certain Intentional States. These intentional states include Beliefs, Desires and Intentions. Intentions are mental states of Agents which refer to (are about) certain Situations in reality. Situation are snapshots of reality. The propositional-content (i.e., proposition) of an Intention is termed a Goal.
In contrast to Endurants, Events are perduring entities, i.e., entities that occur in time accumulating their temporal parts. Events are triggeredby certain Situations in reality (termed their presituations) and they change the world by producing a different postsituation. Action are deliberate Events, i.e., Events deliberately performed by Agents in order to fulfill their Intentions. An Action achieves a Goal if the Action brings about a Situation in the world that satisfies that Goal.
In contrast with an Agent, an Object is a concrete Endurant that does not bear intentional states or perform actions. An Object participating in an Action is termed a Resource.
In this section, we describe some of the proposed ontological guidelines. For lack of
space, we are not able to present them all and refer to
First, it is important to point out that we interpret i* goals, tasks, resources and agents as their counterparts in UFO (with Action as task). Having that in mind, let us try to interpret the i* decomposition relation. Since goals are propositions, due to its ontological nature, it is impossible for a goal to be decomposed into tasks or resources. Thus, goals can only be decomposed into subgoals. Consequently, when decomposing goals, an i* anddecomposition is interpreted as a conjunction of subgoals, while an i* ordecomposition is interpreted as a disjunction of subgoals. Similarly, softgoals, tasks and resources can only be decomposed into softgoal, tasks and resources, respectively. This originates the ontological guideline describe in the first line of Table 1.
In i*, a meansend link is applied to connect a means to an end. For example, a
task T (means) to a goal G (end), meaning that the execution of T leads to the
achievement of G. Here, we adopt the conceptual modeling evaluation method proposed
in
In i*, a makecontribution is applied between a task T and a goal G, meaning that if T is executed, then G is fully achieved. But if this is so, how can one differentiate between meansend and makecontribution? Using UFO, we differentiate this by looking at the intention behind the execution of T. To understand this, let us consider the i* model depicted in Figure 1, which exemplifies the use of the meansend and the make contribution links.
In Figure 1, a Car Passenger1 agent executes the Take a car sick pill task in order to prevent himself from being sick during the journey he is making (meansend link to Car sickness prevented goal). As a side effect of this medication, the Car Passenger also goes to sleep (makecontribution link to Asleep fallen goal).
As result of the mapping from i* tasks into UFO actions, every task is associated with a motivating intention whose propositional content is a goal. In other words, we 1 From now on, we use a different font for the names of the instances of the i* actors and intentional concepts, such as goals, tasks, and resources. execute a particular task in order to accomplish a specific goal. In i*, the association between the task and the goal in this case is made by a meansend link (e.g. Take a car sick pill task as means to Car sickness prevented goal). On the other hand, this same task can also generate some other goals to be accomplished, without however, being intended be the choice of this particular task. In this case, a makecontribution link is established (e.g. Take a car sick pill task contributing to asleep fallen goal). In other words, the meansend link or the makecontribution link should be applied according to the ontological guideline described in the third line of Table1.
Table 1. Some of the proposed i* ontological guidelines Ontological Guidelines 1. A decomposition link can only be applied between elements of the same kind. E.g. goal>goal, task>task. 2. A meansend link can only be applied between elements of different kinds. E.g. task>goal, resource>task. 3. Taking task T and goal G, if the intention behind the execution of task T is to accom plish G, T and G should be related via meansend link. On the other hand, if by execut ing T, G is unintentionally achieved (i.e., as a sideeffect of the execution of T), then T and G should be related via makecontribution.
In this section, we describe the empirical study we conducted to evaluate the use of the
ontological guidelines. The hypothesis of the study is "the ontological guidelines
enhance the capability of the subjects to create i* models." The experiment was
conducted in a controlled environment and is based on a quantitative strategy, in which
the data is analyzed using statistical and descriptive methods. For the experimental
design, we followed the framework presented in
The experiment has as object of study two i* models (here referred to as Case 1 and Case 2), representing two different situations. Each participant had to complete the models, by filling in the blanks with the correct element or link to be used in each question. Figure 2 illustrates part of one model. For each blank, there are two and more possibilities, having as alternatives constructs of i* whose use normally generates confusion or doubts. For example, in Question 2 (refer to Figure 2), the participants should indicate if “Provide gift wrapping solution” is a goal or a plan. In Question 5, the participants should indicate if “Provide gift wrapping solution” and the two tasks “Organize wrapping stand” and “Allow vendors to wrap gifts” should be linked via OR meansend or via ORdecomposition. The idea is to verify if the participants can select them intuitively (pretest) or if the use of ontological guidelines (posttest) effectively helps the selection of the correct construct.
The experiment was divided in two steps: pretest and posttest. In the pretest, all participants performed the first activity, i.e. filling in the blanks, using Case 1. Then, in a separate form, they justified their choices for each blank. During this activity, all participants had a printout of some slides containing basic information about i* (the i* wiki guidelines), as well as the description of Case 1. No information about the guidelines is given in this first step.
After the pretest activity, the students were randomly divided into two groups: group A (control group) and group B (experimental group). After the division, the participants of group A moved to another room to perform the posttest activity. Both groups had to perform a second activity of filling in the blanks, now using Case 2. However, in this part, only group B received information about the ontological guidelines. Both groups had the description of Case 2 and group B also had a printout of some slides containing the ontological guidelines. In the posttest, the participants of both groups were also asked to fill in a separate form justifying their choices for each blank.
To capture the impression of the participants about the guidelines, the participants were also asked to respond some questions regarding their opinion about the i* wiki guidelines and the ontological guidelines.
The data was collected through questionnaires. Before the experiment activities, we applied a questionnaire to capture the participants’ profile. We applied the experiment twice, with two different populations. We will here refer to these applications as application 1 and application 2. In application 1, there were 24 participants: 16 of them were undergraduate students of Computer Science or Computer Engineering, 7 of them were master students in Computer Science, and 1 of them was a PhD student in Computer Science. The participants were assigned into two groups of 12 participants, which were balanced in terms of educational level and modeling experience. In both groups, there was one participant with 13 years of experience in goal modeling and i*, while the others declared not having experience in this area. In application 2, there were 30 participants, all of them in the final year of an undergraduate course in Information Systems Analysis and Development. Each group had 15 participants. None of the participants indicated having experience in goal modeling or i*.
Both in the pretest and in the posttest, the same activities and questionnaires were used in applications 1 and 2. The graphs of Figures 3 and 4 show the results for the first and the second application, respectively. When the participant fills in the blank correctly, we say that he has a hit.
Tables 2 and 3 present data regarding the number of hits per participant in the first and second application, respectively. The columns present data on average, median, highest and lowest value of number hits per participants.
By looking at Figure 4, we see that in the pretest of the second application, groups A and B showed a great balance in realizing the activities; both groups scored the same number of hits and errors. In the posttest, group B achieved a significantly higher number of hits in relation to group A, as seen in Figure 4. Again, this result favors our hypothesis, supporting the idea that the ontological guidelines effectively help the creation of i* models.
The descriptive analysis we presented so far is able to provide us with some evidence supporting the hypothesis, We can quantify this support by a statistical test. Thus, we also applied the WilcoxonMannWhitney statistical test, with a significance level of 5%, to compare the hits for each participant between the experimental (group B) and control (group A) groups, in both experiment applications. This statistical method is a nonparametric method recommended for small samples or groups with less than 20 participants (Robson 2002). In the first application, the calculated U value is 23 and the critical U value from the MannWhitney index is 37. Since the calculated U is lower than the critical U, then we may conclude that the values are significantly different between the groups, which supports our hypothesis. In the second application, the calculated U value is 65 and the critical U value from the MannWhitney index is 64. Since in this case, the calculated U is not lower than the critical U, then we cannot confirm our hypothesis.
Given the results of the MannWhitney test, we cannot conclude that the ontological guidelines are always helpful. We attribute this difference to the divergence in profiles in the two experiment applications. The participants of the first application have a higher graduation level than the participants of the second application, and thus are, in general, more experienced in conceptual modeling. Thus, we claim that the ontological guidelines are helpful for more mature conceptual modelers. New empirical studies should be conducted to confirm this hypothesis.
Regarding the qualitative evaluation of the ontological guidelines, we have the following results. In the first application, 7 out of 12 participants considered that the ontological guidelines are better than the i* wiki guidelines. The other 5 participants considered that the ontological guidelines and the i* wiki guidelines have the same quality. When asked about the usefulness of the ontological guidelines, 8 participants considered them very useful, 2 participants found them not very useful and 2 participants found them indifferent. In the second application, 13 out of 15 participants considered that the ontological guidelines are better than the i* wiki guidelines, while 2 participants considered that the ontological guidelines and the i* wiki guidelines have the same quality. Regarding the usefulness of the ontological guidelines, 10 participants found them very useful, 3 participants found them not so useful and 2 found them indifferent. We find these results positive, as most of the participants had a good perception regarding the ontological guidelines.
Let us now analyze which questions were more difficult, i.e. led to more errors in both experiment applications. This will allow us to find out which ontological guidelines are not clear and should be improved. In the first application, the questions that led to more errors were questions 8 and 10. In the second application, the questions that led to more errors were questions 7, 9 and 14. Questions 8, 9, 10 and 14 regard the use of the meansend, makecontribution and helpcontribution links. We conclude that the participants in both experiment applications could not understand well the ontological difference between these three links. Thus, the ontological guidelines concerning this differentiation should be improved. Question 7 regards the differentiation among AND and OR decomposition. We conclude that in the second application, the participants also had doubts regarding the use of decomposition. Thus, the guidelines concerning these links should also be improved.
The following factors are considered the main threats to the validity of this empirical a) the heterogeneity of the participants of the first application, since they had different academics degrees. To mitigate this risk, we collected information about the academic degree of the participants in the profile questionnaire and took this into account in our experiment design; b) the possibility that the participants had previous knowledge of the ontological guidelines. To remediate this risk, we asked in the experiment questionnaire if the participant had had previous contact with the guidelines. This information was taken into account in our analysis; c) the chance that the participants had low interest in the experiment results, carelessly performing the experiment activities. To mitigate this risk, we tried to motivate the participants, showing the importance of the results of the experiment. Moreover, the experiment was designed to be as short as possible, so as to prevent tiredness and disinterest; d) the possibility that the researcher conducting the experiment influenced the experiment results. To remediate this risk, the researcher conducting the experiment tried to be as objective and unbiased as possible during the experiment activities; e) the possibility that the subjects had a positive opinion about the guidelines, because they knew we were the ones who formulated them. To remediate this risk, we did not tell them we were the authors of the guidelines.
This article presented an empirical study with the objective to evaluate the use of ontological guidelines to create i* models. For that, the experiment was conducted in two steps (pre and posttest), in which the participants performed modeling activities without (pretest) and with (posttest) the use of ontological guidelines. To analyze the results, we performed the MannWhitney statistical test. The outcome supports our hypothesis that states that the guidelines are useful, and does not provide evidence against it. Moreover, most participants stated that they found the ontological guidelines useful to support them in the creation of i* models.
Given the results of this experiment, we intend to develop an i* modeling tool that uses the ontological guidelines as support for the model designer. For that, we aim at proposing a metamodel that is compatible with these guidelines, to serve as basis for the development of the tool.
For the future, we also intend to perform new experiments to collect more data regarding the use of the ontological guidelines to create i* models. In order to confirm our hypothesis, we must repeat the designed experiment, taking populations of different profiles. We aim, for example, to conduct the experiment with professional modelers. Moreover, we intend to perform different experiments. For instance, we would like to conduct an experiment in which the participants are asked to create i* models from scratch, with and without the use of the ontological guidelines. Then, based on some preestablished criteria collected from i* experts, we will be able to analyze if the models created with the use of ontological guidelines have higher quality than the ones created without them. Acknowledgement. This work is partially supported by CAPES/CNPq (grant number 402991/20125), CNPq (grant numbers 461777/20142 and 485368/20137), and the Spanish project EOSSAC, ref. TIN201344641P.