Applying systematic analysis procedures to early requirements models can test the satisfaction of stakeholder goals and facilitate an evaluation of design alternatives. We introduce a qualitative and interactive model evaluation procedure for goal and agent-oriented models. In applying the procedure to a variety of case studies we found that the interactive nature of the procedure prompts model iteration, producing higher quality models.
Introduction
Goal- and agent-oriented modeling frameworks, such as i* [
The informal and incomplete nature of goal models calls for analysis procedures
which are interactive, qualitative, and simple to apply. We introduce such a
procedure for goal- and agent-oriented models. We describe the specifics of the
procedure in terms of the i* Framework, although the procedure can potentially be
applied to other similar models. The procedure is adapted from an evaluation
procedure originating from the NFR Framework [
Giorgini et al. have introduced qualitative and quantitative procedures for goal
model analysis which separately propagate negative and positive evidence, are fully
automated, and work in a forwards and backwards direction [
For later stages of system analysis, where quantitative information is known and
where models are relatively stable, fully automated and quantitative evaluation can be
appropriate. Example methods more appropriate for this later-stage evaluation
include evaluation in the KAOS Framework [
The Qualitative, Interactive Evaluation Procedure
The procedure is designed to be applied either manually or semi-automatically. Here,
we focus on describing the procedure so that it can be applied manually. In order to
concretely describe the procedure, we apply it to the i* Framework [
Procedure Overview: The proposed procedure starts with an analysis question such as “How effective is this design option with respect to the desired goals?” The procedure makes use of a set of qualitative evaluation labels, assigned to elements to express their degree of satisfaction or denial. The process starts by assigning initial label values to model elements representing the analysis question. These values are propagated through the model links using defined rules. Human judgment is needed when multiple conflicting or partial values must be combined to determine the satisfaction or denial of a softgoal. The final satisfaction and denial values for the elements of each actor are analyzed in light of the original question. An assessment is made as to whether the design choice is satisficed (“good enough”), likely stimulating further analysis and potential model refinement. More detail can be found in [10].
Detailed Steps: We first provide the steps of the evaluation procedure, followed by detailed explanation of the concepts. 1. Initiation: The evaluator decides on an analysis question and applies the initial evaluation labels to the model. The initial values are added to a label queue.
Iteratively, until the label queue is empty or a cycle is found: 2. Propagation: The evaluation labels in the label queue are propagated through all outgoing adjacent model links. Resulting labels propagated through non-contribution links are placed in the label queue. Results propagated through contribution links are placed into a “label bag” for that element. 3. Softgoal Resolution: Label bags are manually resolved, producing a single result label which is added to the label queue. 4. Analysis: The final results are examined to find an answer to the analysis questions. Issues with the model can be discovered, prompting further analysis.
Model Syntax: The procedure assumes that models are well-formed. However, as propagation is dependent on link and not element type, most models can be evaluated.
Qualitative Evaluation Labels: We adopt the qualitative labels used in NFR evaluation (Table 1). The (Partially) Satisfied label represents the presence of evidence which is (insufficient) sufficient to satisfy an element. Partially denied and denied have the same definition with respect to negative evidence. Conflict indicates the presence of both positive and negative evidence of roughly the same strength. Unknown represents the situation where there is evidence, but its effect is unknown. We introduce the “None” label to indicate a lack of any label. We adopt the use of partial labels for non-softgoals to allow for greater expressiveness.
Initial Evaluation Values: In order to start an evaluation of a model, a set of initial evaluation values must be placed on the model, reflecting a particular analysis question, comprising Step 1 of the procedure. Often, initial values are placed on “leaf” elements in the model, elements that do not receive input from other elements. However, initial values can also be placed on non-leaf elements. In this case, we avoid overriding the initial labels with subsequent propagation.
Evaluation Propagation Rules: We define rules in order to facilitate a standard propagation of values given a link type and contributing label in Step 2 of the procedure. Here, we must define how evaluation values should be propagated through link types that are in i* but not in the NFR framework, namely, Means-Ends, Decomposition, and Dependency links. The nature of a Dependency indicates that if the dependee is satisfied then the dependum will be satisfied, and so will the depender. Decomposition links depict the elements necessary to accomplish a task, indicating the use of an AND relationship, selecting the "minimum" value amongst all of the values. Similarly, the Means-Ends link depicts the alternative tasks which are able to satisfy a goal, indicating an OR relationship, taking the maximum values of
Contribution Link Type Make Help Some+ Break
elements in the relation. To increase flexibility, the OR is interpreted to be inclusive. We expand the order of the values presented in the NFR Framework to allow for partial values, producing: None < < < < < <
We adopt the contribution links propagation rules from the NFR procedure. These rules intuitively reflect the semantics of contribution links. Note that the “None” label is not propagated or placed in the label queue.
Resolving Multiple Contributions: Softgoals are often the recipient of multiple contribution links. We adopt the notion of a “Label Bag” from the NFR Framework, used to store all incoming labels for a particular softgoal. Labels in the bag are combined into a single label in Step 3, either by identifying specific cases where the label can be determined without judgment (Table 2), or by human judgment. Label Bag Contents Resulting Label 1. The bag has only one label : {<v, es>} the label: v 2. The bag has multiple full labels of the same polarity, and no the full label: other labels. Ex: { , , } or { , } 3. All labels in the bag are of the same polarity, and a full label is the full label: present. Ex: { , , } or { , } 4. The previous human judgment produced or , and a new the full label: contribution is of the same polarity or or or
Human Judgment in Evaluation: Human judgment is used to decide on a label for softgoals in Step 3, for the cases not covered in Table 2. Human judgment may be as simple as promoting partial values to a full value, or may involve combining many sources of conflicting evidence. When making judgments, domain knowledge related to the destination and source elements should be used.
Combinations of Links: Elements in i* are often the destination of multiple types of links. “Hard” links (Decomposition, Means-Ends and Dependency) are combined using an AND of the final results of each link type. If Contribution and Dependency links share the same destination, the result of the Dependency links are treated as a Make contribution, considered with the other contributions in the label bag.
Incomplete Labels: In the procedure, information present in each step is propagated, even if this information in incomplete, i.e., other incoming contributions are missing. As a result, the same element may receive multiple evaluation labels in pl eH PC Products Allow Peer
to-Peer
one evaluation, and the same softgoal may require human judgment multiple times.
Detecting Cycles: Goal models often contain cycles, values which indirectly contribute to themselves and may cause fluctuating values. Experience has shown that the presence of cycles becomes apparent to the evaluator after a few iterations.
Example: We provide a simplified example from the Trusted Computing (TC) Case Study [10] in Fig. 1. This model depicts a simplistic view of the TC domain, showing the intentions of the PC User, the PC Product Provider and the Data Pirate. In our example evaluation, we ask: “If the PC User Obtains PC Products from the Data Pirate, how does this affect the PC Product Provider’s ability to Sell PC Products for Profit?” Initial values are circled and human judgment is annotated in the model.
In this example, when PC Products are Obtained from the Data Pirate, PC Products are Obtained Affordably, but the PC Product Provider does not Sell PC Products for Profit. Further rounds of evaluation and model iteration are needed. In this simple model, analysis results may be apparent without applying explicit procedures. However, in larger goal models results are not apparent and are difficult to derive consistently.
D PCbyULsiecresnAsibnigdeD
ProPdCuct Sell PC Provider Products for
by Licensing
Benefits of applying the procedure described in this work include both the ability to answer strategic questions and the means to iterate upon and improve the quality of the model and subsequent domain understanding. For example, in a case study involving a large social service organization [12], the procedure was applied manually to large models to evaluate the effectiveness of various technologies. Results showed that a wiki was not effective in satisfying the goals of the organization, while a discussion forum showed more promise. In another example, when applying the procedure to the TC case study, the model appeared to be sufficiently complete; however, analysis of the TC Opponent point of view revealed that PC Users would not buy TC Technology. However, the makers of TC Technology must have envisioned some way in which users would accept their product. This result prompted model changes, adding factors such as product lock-in.
In this work, we have introduced a relatively simple analysis procedure which builds on the NFR procedure, providing specific instructions for manual application, and expanding the algorithm to deal with i*-specific constructs. We have highlighted the benefits of such a procedure, including analysis, and model iteration.
This work can be expanded in several ways. For example, evaluating the
satisfaction of actors, as in [
Acknowledgements: Financial support has been provided by Bell University Laboratories and the Ontario Graduate Scholarship Program.