Among the core skills employed during the phase of requirements gathering and elicitation, is that of being able to identify and model the basic concepts of the application domain upon which the software system will be built. Such activity has been named conceptual modelling, and it serves the purpose of glueing together the requests by the customers, and the expertise of designers and developers, providing a platform to ease communication, meet users expectations and distribute knowledge [1]. Two techniques have recently been considered and compared for the modeling of such concepts, one based on scenarios of how the system is going to behave (or how the users will interact with it (e.g., the UML approach [2]); the other expressing what are the needs that the built system will fulfill, relating the business goals of the stakeholders with the functional and non-functional requirements of such system. The latter has been termed goal-based approach, and the agent-oriented software engineering (AOSE) methods have been one the more developed branches of such approach in the requirements elicitation.
Among the goal-based approaches, Tropos is an AOSE methodology based on two key ideas: agents and their interactions within the system environment. The main aim of Tropos is to produce a better understanding of the application domain where a system will operate, and of the kind of interactions that should occur between such system and the human agents. Within Tropos, the notion of agent, together with their goals and plans, are used since the early analysis of requirements elicitation: in the early phase of such analysis, the organizational setting is studied for the purpose of better understanding the scenario. In the late phase of requirements gathering and elicitation, the system is also inserted in the operational environment as one actor: the dependencies with the other actors represent the system's functional and non-functional requirements.
In both phases, the actor and goal diagrams are produced as outcomes, with the system being inserted in the diagrams in the late phase, but not in the early phase. The actor diagram represents the overall view of all the actors with their high-level dependencies to other actors, while the goal diagram is a refinement of the former with emphasis on the goals of a single actor (see Figure 1).
The focus of this work is on the early and late phases of requirements elicitation covered by the Tropos methodology, where the business entities are identified as actors, their goals assessed, and their inter-dependencies defined. In the UML notation instead, as summarised in Table I, these two phases correspond to the production of a model in the problem space (MOPS [3]). Such a model comprises of a set of use cases and business class diagrams (i.e., diagrams documenting business entities, their attributes and behaviors). When the business entities are converted into implementable entities, the UML notation produces the Model of Solution Space (MOSS) with the aim of feeding such model to the design phase.
The aim of this paper is to compare the UML outcomes from the MOPS phase (use cases and business class diagrams) as produced by undergraduate and postgraduate students, when combining (or not) the Tropos methodology as a "treatment". The rationale of such experiments is to determine through evaluation whether the joint use of This paper details one experiment where BSc students at the university of East London, UK, produced both Tropos and UML diagrams towards the assessment of a scenario where a software system has to be built. The UML and Tropos diagrams were assessed against the benchmark produced as a marking scheme, and it is questioned whether the presence of the Tropos methodology has helped in the completeness of the resulting UML diagrams. This paper is the first of two experiments, where the Tropos methodology is used to inform the UML design: we plan to replicate this experiment in the semester starting in February 2011, without the Tropos "treatment": students will be required to work on the same scenario, but no Tropos diagrams will be required (or taught), therefore allowing for the comparison of two different sets of UML diagrams. This will provide the basis for comparing the effectiveness (or not) of the two combined approaches.
This paper builds upon the scenario-based and the goalbased approaches as two viable tools in the requirements elicitation phase and for validation purposes. As a practical exemplification of the scenario-based requirement engineering method, we have used the Jacobson's Use Case technique, which has been lately incorporated into the UML notation language [2]. Such a model is based on the notion of "scenario" which is a sequence of interaction events between a system and its environment in the restricted context of achieving some implicit purposes [4], [5].
On the other hand, this paper relies on the concepts of agents and the agent-oriented paradigm (AOSE), as one example of goal-oriented approach [6], [7], [8]. This second approach is based on agents interacting as a group within a system, not just reacting to stimuli, but also communicating, coordinating, and cooperating as an autonomous and social entity that can to achieve individual and organizational goals.
The main notations of UML (as a scenario-based methodology) and Tropos (as goal-based) are summarised in Figure 1 (taken from [5]). Specifically for the Tropos notation, every system can be thought of several actors, having goals to fulfill with the use of such system. Such goals could be "hard" or "soft", depending on whether it is clear what actions and plans (or resources) should be performed (or used) in order to achieve such goals. A Tropos "actor diagram" details the overall connections between all the actors in the scenario, where a dependee (e.g. actor 3 in Figure 1) fulfills the goal(s) of a depender (e.g., actor 1 in Figure 1). A Tropos "goal diagram" focuses more precisely on one actor, and tries and elaborates on what plans, actions and resources should be performed to achieve each goal, and which actors are needed to fulfill these goals.
In the literature, the effectiveness of goal-oriented and scenario-based approaches is analyzed in several works illustrating the application of different methods to case studies (e.g., [9], [10], [11] or comparing the strengths and limitations of the approaches according to different criteria (e.g., [12], [13]). However, to the best of our knowledge, experimental comparisons of these requirements modelling paradigms using different visualization methods are rare [5]. Such comparisons may raise insights and help decide which modelling paradigm to adopt for a given software development project. The "quality" of UML models, comprised guidelines for the aesthetic quality, have also been evaluated [14].
One important factor for comparison or evaluation is the immediacy in understanding the respective models by projects stakeholders, for instance by requirements analysts [15], who have to understand a given model during analysis and refinement tasks to accommodate new or changed requirements.
This section introduces the definitions used in the following empirical study and presents the general objective of this work, and it does that in the formal way proposed by the Goal-Question-Metric (GQM) framework [16]. The GQM approach evaluates whether a goal has been reached, by associating that goal with questions that explain it from an operational point of view, and providing the basis for applying metrics to answer these questions. This study follows this approach by developing, from the wider goal of this research, the necessary questions to address the goal and then determining the metrics necessary for answering the questions.
Goal: The long term goal of this research is to evaluate whether the Tropos methodology (as an experimental "treatment"), jointly with the UML MOSS notation, produce higher standards of conceptual modelling than the UML notation alone.
Question: In this paper, and considering a given scenario as a case study, the following research questions will be evaluated:
1) Are the models produced by the students with the Tropos notation "complete" against a given benchmark? Rationale: the aim of this question is to check whether the diagrams produced with the Tropos notation are compliant with a minimum list of actors and goals directly derived from the scenario. Such list of actors and goals should be considered as the "absolutely mandatory" in a typical requirements elicitation and gathering phase. 2) Are the models produced with UML complete against a given benchmark? Rationale: similarly to the question above, the aim of this question is to check whether the diagrams produced with a UML editor (Rational Rose, ArgoUML, etc) can be mapped to a minimum list of use case diagrams, necessary to describe the how the users of the system interact with its functionalities.
Rationale: the aim of this question is to evaluate whether the use of "goals" and "actors" can help in focusing on the main functionalities of the system, expressed as UML use cases. Given the set of Tropos diagrams produced by any group of students, and a benchmark mapping of "Goals-to-use-cases" (see last column of Table III), it will be evaluated how the Tropos diagrams have informed the specified group of students in the creation of use cases. Metrics: The Tropos actor and goal diagrams for this scenario have been listed in their minimum form, i.e., the minimum number of functionalities that are expected for (and from) this system, corresponding to both functional and non-functional requirements (see Table II). Also, the minimum set of UML use cases has been developed and it served as a benchmark to evaluate how students assessed the scenario (see Table III). Each group coursework was evaluated against these two lists, and the number of correct diagrams produced by the students evaluated against these baselines.
The first part of the experiment was set up at the University of East London, during the Level 3 module "Advanced Information Systems Development". The experiment population comprised some 65 students, divided in 17 groups of 3 to 4 members 1 . Each group was in charge of producing two sets of diagrams: the Tropos goal and actor diagrams (for both the early and late phase of the requirements); and the UML use cases and class diagrams. All the students in the module had already studied the basic UML concepts in a previous module, while the Tropos concepts were introduced during several lectures, and their practical implementation was assisted in the lab sessions. The scenario was distributed to students on week 4 (out of 12 weeks in the module), and it represents the coursework needed to pass the module, together with the final exam. The students had 9 weeks to complete the task.
In order to produce the Tropos diagrams, the OME tool, implementing the i* notation 2 , was taught and demonstrated during the lab sessions. In order to produce the UML class and use case diagrams, students could select the UML editor of their choice (e.g., the IBM Rational Rose toolkit, or the Open Source ArgoUML tool 3 , etc.).
The following problem statement was provided to the students, with the request for modeling such scenario via a scenario-based approach and a goal-based approach. This is based on a previous job placement where a student effectively designed and developed the system outlined below.
The company is seeking to streamline and automatise its systems for record keeping, therefore enabling the business to offer their clients a better service. The aim of this project would be to develop a system allowing data collection during site visits to be entered onto an online application, that sits on the web: the employee visiting the site's premises would input the data to a specified form (which can be extended by a System Administrator to contain more fields and input data, it could be reused from existing form, and new forms can be created ad-hoc). The data once collected would be synchronized with the companys database, but during the initial trial period, the paper-based system, and the on-line system, would need to run together, and be synchronised. The data collected would be used to keep the clients informed of the results of the employee's visits and the next visit's date. This upgrade project would be expected to cover the following areas: data acquisition using online, secure systems; synchronizing of data; a database to store the data of clients; and a PC based management tool for the data-captured database.
2 OME3, available online at http://www.cs.toronto.edu/km/ome/ 3 ArgoUML, freely available at http://argoUML.tigris.org/
In order to assess the courseworks produced by the students, a list of "model solutions" was produced, and checked against the delivered set of diagrams. In particular, a minimum list of the Tropos actors present in the scenario was produced and their main goals were identified: the following Table II was therefore used as the baseline for marking the assignment. These goals and actors were prepared by one of the authors (running the module) and the assistant, a PhD student whose focus is on the secure aspects of Tropos.
Three main actors (Client, Company and Employee) were identified as expressing goals within the interaction with the system, while other two (the System, and the HM Revenue and Customs agency -HMRC) are also present, acting as dependees in one or more of those goals by the three main actors.
The marks available for the completion of such task were 25 out of 50.
The following Table III summarises instead the list of UML use cases that were set up as a baseline for marking the scenario-based part of the assignments: three main UML actors were expected to be interacting with the system, with increasing amount and type of provileges: the clients of the Tax Revenue company (c i in Table III, i = 1..5), its employees (e i in the same Table, with i = 1..7) and the system administrator (s i in the same Table, with i = 1..6).
The UML use cases listed, and intended as a "model solution", are a subset of what was documented during a business consultancy,where the described system was actually implemented by a student in a job placement. The listed UML use cases should be inferred by reading the problem statement of the scenario, and they should also become clearer after working on the Tropos goals and actors. Albeit more specified UML actors could be identified (e.g., the ISP administrator, the project manager in charge of delivering the requested system, the Tax Revenue company owner, etc.), the above three provide the minimum set of scenarios that fulfill most of the functional and nonfunctional requirements of the scenario. In some of these, one UML use case could be the extension, or being included in some other use case (for instance, the "log-in" use case is typically included in any interaction with the system, independently from the privileges).
The marks available for the completion of the UML task were also 25 out of 50. This was decided to balance the relative importance of both Tropos and UML tasks.
As said above, the results obtained from the marking of the presented coursework represent the first part of this study: the second part will be based on assessing the diagrams produced by the students when the "treatment" Tropos is not taught or requested.
Table IV shows how each group coursework (G1 to G17) was evaluated against the list of Tropos goals and UML use cases, gathered around the main actors expressing their requirements, either in a goal-based approach, or a scenariobased approach.
At a first glance, the results found in the table show that the students found easier to assess the Tropos actors and goals, rather than producing the relative UML diagrams to describe how the actors are interacting with the system. Even when breaking down the aggregated results in the main components and actors, it is visible that some actors were assessed better than others: the Tropos models for the Company providing the Tax Revenue service are more complete than other actors (as visible in Table V where on average 70% of the groups assessed the benchmark goals from the Company actor).
The striking difference with such a finding is visible by observing the results of the UML cases, summarised in Table VI, where on average, only 38% of the groups assessed the "Employee" use cases, and only 43% delivered the "administrator" cases. As a grand average, some 64% of groups successfully assessed the set of Tropos goals proposed as a baseline, while only 38% of students assessed the set of UML cases of the benchmark. • with respect to the UML cases, 1 out of 5 cases were identified for the client (average 1.35 cases); 2 out of 7 cases were identified for the employee (average 2.70 cases); and 2 out of 6 cases were assessed for the administrator of the system (average 2.88 cases) Relatively to the experiment performed with the students of the University of East London, we can conclude that the use of the Tropos approach was not effective to inform the UML conceptual model.
Like any other empirical study, the validity of ours is subject to several threats. In the following, threats to internal (whether confounding factors can influence your findings), external (whether results can be generalized), and construct validity (relationship between theory and observation) are illustrated.
• Internal validity -the terminology "quality of UML models" was used to define whether "better" models could be obtained with the use of the additional Tropos analysis. Obviously the quality of UML diagrams is a multi-faceted dimension of several possible: aesthetic aspects could be considered, but also others based on design metrics of UML diagrams, as coupling, complexity, etc).
• External validity -the following threats to external validity have been identified: 1) these findings cannot be generalised by one scenario, distributed to some 70 students, and based on one observation only. Replications are needed not only regarding the presence or absence of the Tropos "treatment", but also with more students involved. 2) Despite the initial results, a stronger statistical formalism cannot be used for investigating the research questions: this is because since there is no comparison with a null hypothesis, such analysis cannot be properly performed. The results will become much more reliable when the second part of the experiment will be carried out.
• Construct validity: the minimum set of actor and goal diagrams, and the minimum set of UML use cases derived for the construction of the benchmark could play an important part in the outcomes of this experiment. The reason of choosing these use cases, and the relative Tropos actors and goals, are of a practical nature: the proposed one is a real scenario of a past job placement, where a student designed and implemented the system to be delivered: the "model answers" are a subset of the diagrams implemented for the deployment of such system.
The authors would like to thank Dr H. Mouratidis and Michalis Pavlidis for the extensive comments, and the help in formulating the Tropos goals and actors. The authors would also like to thank the anonymous reviewers, since many improvements were added to the text, based on their suggestions.
The usage of visual modelling tools has become a common support for the design of a software system's capabilities; the use of such tools has become more valuable in the early phase of requirement gathering, where the interaction with non-technical stake-holders requires jargon-free and easily usable approaches. Among these techniques, this paper has considered the goal-based (Tropos) and the scenariobased (UML) methodologies, trying to assess whether the use of the first could be useful to inform the definition of more complete use cases.
A set of "model solutions" was prepared for a given scenario, that was handed out as part of a coursework at the University of East London, UK. A baseline set of actors was prepared for the Tropos approach, and one for the UML use cases. Each coursework was assessed against these two baselines. Contrarily to what was expected, a larger number of students correctly assessed a larger amount of Tropos goals, whereas the UML cases were delivered less often, and more erroneously. Although the correct UML cases were assessed where the relevant Tropos actors were identified, this was not always the case: students found it difficult to connect the two approaches, and synchronise the actors and goals with how the system was supposed to perform.
These results are interesting, but we need to produce a similar set of observations when removing the Tropos approach from the experiment: we plan to replicate this experiment in a course starting in February 2011, where the same scenario will be provided, and where only the UML use cases will be requested. This will help in assessing whether the use of the Tropos approach can be considered to play a difference in the requirements gathering phase, when coupled to the UML notation.