Comparing Value-Driven Methods: an experiment design Eric Souza Silvia Abrahão Ana Moreira NOVA LINCS, Departamento de Sistemas NOVA LINCS, Departamento de Informática Informaticos y Computacion, Departamento de Informática Faculdade de Ciências e Tecnologia, Universitat Politecnica de Valencia, Faculdade de Ciências e Tecnologia, Universidade Nova de Lisboa, España Universidade Nova de Lisboa, Portugal sabrahao@dsic.upv.es Portugal er.souza@campus.fct.unl.pt amm@fct.unl.pt João Araújo Emilio Insfran NOVA LINCS, Departamento de Informática Departamento de Sistemas Informaticos y Faculdade de Ciências e Tecnologia, Computacion, Universidade Nova de Lisboa, Portugal Universitat Politecnica de Valencia, España joao.araujo@fct.unl.pt einsfran@dsic.upv.es ABSTRACT inspired in business science, requirements engineering, and A business model is a representation of an organization with a conceptual modeling techniques [5], whose aim is to show how particular point of view. It is common to find different types of economic values are created and exchanged in an inter- models to describe the business. However, methods to create organizational network [6]. business models representing an economic point of view have Value means the relationship between satisfying needs and only emerged over the last few years in the scientific community. expectations and the resources required to achieve them [7]. Value Such methods aid business specialists improving the economics is the reason for companies and people to trade with each other, understanding of the business, helping both defining more offering money to get something in return. Therefore, a value efficient business strategies and better aligning the information model represents a business model from an economic perspective, technology systems with the business. This paper aims at and must determine the economic value exchanged and their describing the design of an experiment to compare two methods to intervenients [8]. This understanding facilitates the aligning of the specify economic values (e3value and value-driven development). software requirements specifications with the value exchanges [9]. Our experiment design allows predicting the acceptance of a There is a number of approaches to represent value. Kundisch and particular method in practice, based on the effort of applying the John [10] classify 12 different business model representations: method, the quality of the artifacts produced, and the user activity system map, business model ontology, causal loop perceptions with regard to the quality of the method. diagram, value map, value net, strategic business model ontology CCS Concepts (SBMO), and business model ontology (BMO), among others. However, there is no empirical evidence about which of these • Information systems → Language models methods is more effective under what circumstances. For Keywords practitioners to consider adopting a given value-driven method, value model; value-driven; experiment design they must know its effectiveness and how it compares with others. The goal of this paper is therefore to describe the design of an 1.   INTRODUCTION experiment aimed at comparing two methods to express economic Models to describe the daily behavior of the business, or what the values: e3value [11], which is a widely established and applied Object Management Group1 calls “business in motion” [1], are business model representation, and our own Value-Driven common. This business behavior, in general, is represented by Development method (VDD), which has been proposed recently. using some well-known description languages such as workflow This design is also valuable because it can be replicated for other diagrams [2], UML activity diagrams [3], and BPMN [4]. methods. However, methods to create business models representing an The remainder of this paper is organized as follows. Section 2 economic point of view have emerged over the last years in the introduces e3value and VDD. Section 3 presents the design of a scientific community. In principle, the reason for this is that an controlled experiment aimed at comparing the effectiveness, organization needs to make money to keep going in a competitive efficiency, perceived ease of use, perceived usefulness, and market. In addition, business specialists need to improve the intention to use of several groups of users employing both economics understanding to define more efficient business methods for creating a value model. Finally, Section 4 concludes strategies and provide a better alignment of the information this paper and summarizes directions for further work. technology with the business. These models can then be used to define the requirements of the underlying information systems. In 2.   METHODS TO BE COMPARED general, these methods are conceptual requirements approaches, This section summarizes the e3value [12] and VDD, comparing them with respect to their concepts and processes. 1 The Object Management Group is an international technology standards consortium.   2.1   e3value method The start stimulus represents customer needs, that is, the 3 The e value method offers modeling constructs for representing beginning of a value scenario, and the stop stimulus represents the graphically and analyzing business requirements from an end of a value scenario. A connection element links a start-stop economic point of view. It is composed of fifty concepts [12], stimulus to a value interface or links value interfaces of the same whose main ones are: elementary actor, composite actor, market actor internally. As a lot of value scenarios are represented in a segment, value interface, value transfer, value port, value object, unique e3value model, AND and OR elements are used to split or value exchange, value transaction, value activity, start stimulus collapse paths of value scenarios, reusing start and stop stimulus (customer needs), stop stimulus (scenario boundary), AND elements. element, OR element, and connect element. Figure 1 presents the e3value metamodel (note that some of these concepts are not present [5], [11]), and Figure 2 exemplifies an e3value model. Actors are environment entities economically independent, which can be specialized as composite or elementary. The difference is that the composite actor is a group actor with value interfaces of the inner elementary actors. Thus, value interfaces allow accessibility to the constituent elementary actors. Value interfaces group value ports. Value ports provide or request value objects to or from actors or market segment. Actors only Figure 2. e3value example extracted from [14]. offer objects to others if they receive adequate compensation in return. Value objects are money, goods, services or information, which are of economic value for the actors. A market segment, on the other hand, is a group of actors in a business segment that 2.2   Value-Driven Development method share common properties. The set of value objects exchanged by VDD is an approach to derive software architecture aligned with actors is defined as a value exchange. Value transfers are used to business economic values supported by model-driven techniques. link two value ports with each other. Value transactions are To improve the understanding of this method, we divided it in groups of value transfers. For a value exchange to happen, actors, three different phases: business analysis, requirements or market segment, must perform a set of operational activities. specification, and software architecture derivation. The business The collection of these activities is called as value activities. analysis is an early requirements phase whose goal is to analyze and represent the economic values exchange through a model called Dynamic Value Description (DVD). From the DVD model, both business analysts and requirements engineers specify information system requirements by using a cognitive requirements approach [15]. The cognitive requirements approach improves the domain understanding because it provides an environment wherein all the stakeholders could share their views and abstractions in a semi- structured mind map model3. Finally, from these requirements specifications, the software architect generates a high-level software architecture by using model-driven techniques. In the context of this paper, we address only the business analysis phase. So, we analyze the DVD model and its creation process. Figure 3 depicts the DVD metamodel and Figure 4 presents a DVD model example (instance from metamodel). As we can see in the metamodel, DVD is composed of eight main concepts: main actor, environment actor, value exchange, who starts the value exchange, value port, value element, value level agreement, and priority. Similarly to the e3value model, actors are Figure 1. e3value metamodel extracted from [12]. environment entities economically independent in the DVD . model. However, each time, the business analyst focuses the analysis on the main actor and represents its relationship with In order to represent value exchange scenarios, the e3value model others environment actors, producing an inter-organizational inherited the start stimulus, the stop stimulus, the AND element, network. As the focus changes, the actor playing the role of “main the OR element, and the connect element from Use Case Maps2 actor” also changes. With this change in focus, new actors and (UCM) [13]. Although these elements are contained in e3value value exchanges may appear. model (see Figure 2), they are absent in the metamodel (see Figure 1), showing that the e3value metamodel is incomplete. 2 Use Case Maps is a requirements language which have the 3 notion of path to show how a particular scenario works. Mind map is considered a simple and accessible model [16]. 2.3   Comparing e3value and VDD This section compares the e3value and the value-driven development methods. Table 1 presents a mapping between e3value and VDD concepts. Table 1. Concepts mapping. # e3value concepts DVD Concepts 1 Elementary actor Main actor or actor 2 Composite actor Main actor or actor 3 Market segment Main actor or actor 4 Value interface Aggregate in value exchange 5 Value transfer Aggregate in value exchange 6 Value port Value port 7 Value object Value element 8 Value exchange Value exchange 9 Value transaction Aggregate in value exchange 10 Value activity - 11 Start stimulus Who starts 12 Stop stimulus Who starts Figure 3. Dynamic Value Description metamodel. 13 AND element Logical operators in exchange element 14 OR element Logical operators in exchange element 15 Connect element - From the actors relationship, a value exchange is performed. It 16 - Value level agreement shows economic reciprocity through two value ports (arrows 17 - Priority connected to value exchange), one for entry and one for exit, which point to value elements (money, goods, services or We observed that the DVD model (from the VDD method) information). If there are many value objects in the same value describes two new concepts in relation to the e3value: value level port, the business analyst must use logical operators (“AND”, agreement (VLA) and priority. In contrast, the e3value model has “OR”, and “XOR”) to detail the relationships among them. the value activity concepts, not offered by the DVD model. The VLA defines the business constraints based on the business In addition, the business analyst also defines who starts the value strategies. For example, a company of the feeding segment exchanges through a configuration of arrows between the main provides food fresher than its competitors, as a business strategy. actor and the environment actor. It is important to notice that Thus, to provide fresher food, it is essential that its suppliers also during the DVD modeling, the business analyst is able to focus on deliver fresh ingredients. Therefore, the business analyst can each actor individually in order to give more attention and details specify a VLA by defining the acceptable time of receipt of these to the actor which must be analyzed. Thus, the analyst sets who is ingredients. Regarding the information system development, the the main actor and a given support tool will display it as the complexity of a software system is determined by its functionality central node of the model, dynamically. Each value exchange (i.e., what the system does) and by global requirements on its needs a level of agreement between the ones involved. This level development, such as operational costs, performance, reliability, of agreement is a particular business aspect that must be maintainability, portability, robustness [17]. These global minimally agreed among the actors in order to enable the value requirements are known as Non-functional Requirements (NFR) exchanges. and they typically refer to the operational quality of a system, as Finally, the DVD model describes a prioritization of value well as the constraints imposed on a solution [18]. Thus, we can exchanges using colors. The red color means high priority, yellow define a VLA as an NFR at the business abstraction level. color means medium priority, and blue color means low priority. In addition, as information systems are usually developed using These priorities are set by business analyst according to the return iterative and incremental processes, the value exchanges of investment of the value exchanges in the business. prioritization may facilitate the scope definition of each iteration, aligning the system development with the business needs and the time to market. Despite having fewer concepts, the DVD model represents several e3value' concepts but some of these concepts are represented in a partial way or with a different meaning (e.g., UCMs elements). However, for the various case studies developed, the concepts offered by DVD have been proved sufficient. 3.   EXPERIMENT DESIGN This section presents the design of a controlled experiment aimed at comparing the value-driven development method against the e3value method. We followed the guidelines proposed by Wohlin et al. [19]. Figure 4. Dynamic Value Description model example. 3.1   Experimental Planning attend the “Empirical Software Engineering” course from September to November 2016. 3.1.1   Experiment Goal •   Undergraduate students, all Computer Science students from According to the Goal-Question Metric (GQM) approach [20], the the Software Engineering intensification at the Universitat goal of this experiment is to analyze VDD and e3value for the Politècnica de València. These students will attended the purpose of comparing them with respect to their effectiveness, “Software Quality” course from February to June 2017. One efficiency, perceived ease of use, perceived usefulness and of the main topics of this course is to evaluate the quality of intention to use in order to obtain high-quality value models from models obtained through the software development process. the point of view of novice business analysts and software A teaching unit on the evaluation of value-driven engineers, in the context of undergraduate and postgraduate development methods will be added to the course program. students in Business Management and Computer Science. •   Master’s students, enrolled on the Master’s Degree in Software Engineering at the Universidade Nova de Lisboa. The broad research questions addressed by the experiment are: These students will attend the “Software Engineering” course •   RQ1: Is the actual efficacy of VDD higher than the actual from September 2016 to January 2017. efficacy of e3value? •   Undergraduate students, all business management students at •   RQ2: Is the perceived efficacy and intention to use of the Universidade Federal de Pernambuco. These students participants applying VDD higher than that of e3value? will attend the “Information System” course from January to July 2017. The context of the experiment is the creation of a business value model for specific software systems. This context is determined These courses were selected because the preparation and training by the product to be developed and the subjects’ selection. and the experimental task itself fit their scope. We take a convenience sample (i.e., all the students available in the class). 3.1.2   Experimental Objects The original experiment will be conducted in the Requirements The software systems to be developed were selected from the Engineering course and the other groups will be exact and/or literature [21], [22]. Two experimental objects were selected from differentiated replications. This will allow us to build a body of the requirements specifications of the following two systems: knowledge about these value-driven development methods. As Basili et al. [24] suggested, relevant and credible results can only •   Waste management (O1): It describes the business where be obtained by replicating the experiments since single studies waste is traded between an exporter and an importer. In the rarely provide definitive answers. majority of cases the exporter has to pay the importer for the waste handling. However, there are some cases where the 3.1.4   Selection of Variables waste can be traded like a regular good, for example, when The independent variable of interest is the use of each value- the waste is recycled. driven method with nominal values: VDD and e3value. Hence, the •   Wireless access provisioning (O2): It describes the business experiment use two treatments: the creation of a value model for where a hotel would like to offer wireless connectivity to two software systems using VDD and the creation of a value businessmen as an additional service. model for the same systems using e3value. The experimental data collected allows comparing the effects of both treatments. To assess the complexity of the models used and to identify possible mistakes, we plan to carry out a pilot experiment with a There are two types of dependent variables in which the small group of PhD students at UPV. treatments are compared: performance-based and perception- based variables. Performance-based variables assess how well the 3.1.3   Participant’s Selection participants perform the experimental task. They are used to The context of this experiment is the evaluation of value-driven evaluate the actual efficacy of the value-driven development development methods from the perspective of novice modelers. methods. Perception-based variables assess the participants’ Although experienced modelers and practitioners are desired, we perceptions of their performance and their subsequent intention to focus on the profile of novice modelers since one of our goals is use VDD or e3value. These variables are used to evaluate the to provide a value-driven development method that will help less perceived efficacy of these methods, as well as their likely experienced modelers to specify value models. In addition, adoption in practice. according to the Technology Transfer Model proposed by There are two performance-based variables: Gorschek et al. [23], it is recommended to first perform initial evaluations in lab environments before the realization of realistic Effectiveness: It is calculated with the Jaccard index (see formula evaluations in industrial environments. (1)) that measures similarity between sample sets and is defined as the size of the intersection divided by the size of the union of the Value models can be produced by business analysts or software sample sets. Given two models, A and B, the Jaccard index engineers. The following groups of participants are therefore measures the overlap that A and B share with their elements. In identified in order to facilitate the generalization of results: our case, we will calculate the Jaccard index between an agreed •   Undergraduate students, all Computer Science students from solution among experts (A) (for the value models obtained with the Software Engineering intensification at the Universitat VDD and e3value) and the solution given by each participant (B). Politècnica de València. These students will attend the A Jaccard index of 0 represents no overlap between the solutions, “Requirements Engineering” course from September 2016 to while 1 indicates that they contain the same results. January 2017, during this time they will have 8h of lectures on business modeling and value-driven development. (1) •   Master’s students, enrolled on the Master’s Degree in Engineering and Technology of Software Systems at the Universitat Politècnica de València. These students will Efficiency: It is the time required to apply the method. There are also three perception-based variables, which are based System Domain. The complexity of the software requirements on the Technology Acceptance Model (TAM) [25], since TAM is considered in the tasks may have a confounding effect on the one of the most widely applied theoretical models when analyzing results. The application domain of the tasks could also be a user acceptance and usage behavior of emerging information confounding factor that could affect the subjects’ comprehension. technologies, and has empirical support through validations and Order of Methods. The order in which the subjects apply the replications [26]. This model has been also applied previously to methods may produce learning effects, which may bias the results. evaluate requirements modeling methods [27]. The perceived efficacy [25] of the method can be broken down into the 3.1.7   Design of the experiment following subjective dependent variables: The experiment is planned as a balanced within-participant design Perceived Ease of Use (PEOU): It refers to the degree to which a with a confounding effect, signifying that the same participants person believes that learning and using a particular value-driven apply both methods with both experimental objects in a different method would be free of effort. order. We plan to establish four groups (each of which will apply one method to one experimental object) and the participants will Perceived Usefulness (PU): It refers to the degree to which a be randomly assigned to each group. Table 3 summarizes the person believes that using a specific method will increase his or design of the experiment. her job performance within an organizational context. The within-participant experimental design is intended to Intention to Use (ITU): It refers to the extent to which a person minimize the impact of learning effects on the results, since none intend to use a particular method. This last variable represents a of the participants repeat any treatment or experimental object perceptual judgment of the method’s efficacy – that is, whether it during the execution. Other factors which may also be present is cost-effective and is commonly used to predict the likelihood of need to be controlled, since they might influence the results, i.e., acceptance of a method in practice. the complexity of experimental objects. The comprehension of the These three subjective variables will be measured by using a software systems requirements to be modeled may also affect the Likert scale questionnaire with a set of 12 closed-questions: 3 application of both methods. We attempted to alleviate the questions for perceived ease of use (PEOU), 6 questions for influence of this factor by selecting two representative software perceived usefulness (PU) and 3 for intention to use (ITU). The systems with software system requirements of a reasonable questionnaire can be found here: http://bit.ly/2ak1wLS. The complexity. The complexity of the software systems selected closed-questions were formulated by using a 5-point Likert scale, made them suitable for application in the time slot available for using the opposing statement question format. In other words, the execution of the experiments (2 hour sessions). each question contains two opposite statements representing the Table 3. Experiment Design maximum and minimum possible values (5 and 1), where the value 3 is considered to be a neutral perception. The aggregated 1st Session value of each subjective variable will be calculated as the Detailed training with VDD and e3value Sessions 2nd Session 3rd Session arithmetical mean of the answers to the questions associated with VDD and e3value quick training each perception-based variable. G1: G2: G2: Group of G1: e3value VDD in VDD in e3value in 3.1.5   Hypotheses Formulation participants O1 O2 in O2 O1 We formulated several null hypotheses, which were formulated in (sample size = 4n G3: G4: G3: G4: a one-tailed manner, since we want to analyze the effect of the use 3 3 e value in e value in VDD in VDD in participants) of VDD on the variables. Each null hypothesis and its alternative O2 O1 O1 O2 are presented as follows: Post-experimental Post-experimental questionnaire questionnaire •   H10: There is no significant difference between the effectiveness of VDD and e3value / H1a: VDD is 3.1.8   Instrumentation significantly more effective than e3value. The experimental task was structured to allow the comparison of •   H20: There is no significant difference between the both methods. Depending on the method, the task was composed efficiency of VDD and e3value / H2a: VDD is significantly of the method activities that help to achieve its purpose. After more efficient than e3value. applying the method, the participants have to fill in a post- •   H30: There is no significant difference between the perceived experimental questionnaire with subjective questions regarding ease of use of evaluators applying VDD and e3value / H3a: the method. VDD is perceived as easier to use than e3value. •   H40: There is no significant difference between the perceived We have defined only one experimental task (create the value usefulness of VDD and e3value / H4a: VDD is perceived as model) of which its steps vary according to the value model that more useful than e3value. the subjects will create. We will offer a training session to explain the concepts of value models and how they are created. During the •   H50: There is no significant difference between the intention experimentation, we will offer the requirements specifications to to use of VDD and e3value / H5a: VDD is perceived as more the subjects (see Section 3.1.2). The requirements specifications likely to be used than e3value. describe how the business works. With these specifications in 3.1.6   Factors to be Controlled their hands, the participants will create a particular value model Although Method is the only factor of interest in this empirical (DVD or e3value) and will register the start time and the end time investigation, other factors may affect the participants’ for each step performed. performance in an undesirable way, thus confounding the Method Figure 5 presents the steps to create the e3value model. In this effect. These factors have to be controlled so that only the effect case, the steps are: of the Method factor, if there is any, is observable: Step 1 - Identify scenarios: Scenarios are short textual sentences, The reason for this is that there is a lack of economics results at meaning the product, service, or experience expected by a this moment of the analysis. customer. Therefore, the goal of this step is that the participants write a scenarios list. Once the value model is created, the participants will answer the post-experimental questionnaire. Hence, we will be able to Step 2 - Identify actors: The participants will identify who offers evaluate the performance-based variables (effectiveness and and who receives the product, service or experience expected efficiency) by comparing the value model they created against the from the scenarios list and they will create a list of actors. value model created by experts4 and by analyzing the time registered to perform each experimental step. In addition, we will Step 3 - Create value model: With the scenarios list and the actors evaluate the perception-based variables (perceived ease of use, list in their hands, the subjects will create the initial e3value model perceived usefulness, and intention to use) from the responses by using the products and services mentioned in the scenarios list received in the post-experimental questionnaire. and the actors described in actors list. The experimental material is composed of a set of documents Step 4 - Identify UCMs: The participants will insert the UCM's required to support the experimental tasks and the training elements, representing the paths of all scenarios in the e3value sessions, along with the post-experimental questionnaire. The model. In other words, they will insert the start stimulus, stop training materials include: i) a set of slides containing an stimulus, AND element, OR element, and connect element in the introduction to business modeling and value-driven development; e3value model. ii) a set of slides describing the VDD method, along with an example of its application; and iii) a set of slides describing the e3value method, with an example of its application. The documents supporting the experimental tasks include: •   Two kinds of booklets covering the two possible combinations of both the value-driven development method and the experimental objects (VDD-O1, VDD-O2, e3value- Figure 5. e3value process. O1, e3value-O2). The purpose of these booklets is i) to describe the experimental tasks to be performed; ii) to In the case of creating the DVD model, the participants will describe the software system requirements; and iii) to gather follow the steps from the VDD process (see Figure 6). These steps the data from the experimental task. are: •   An appendix containing a guideline to help the participants Step 1 - Specify actors: Participants start the DVD model by to apply the value-driven development method. describing the main actor (the focus of their analysis) and their related environment actors. Thus, the participants will create a The post-experimental questionnaire contains a set of closed- DVD model like a mind map, where the main actor is the central questions that allows participants to express their opinion on the node and the environment actors are the leaf nodes. Due to this ease of use, usefulness, and their intention to use of the method in “main actor” focus, the DVD model shows only the environment the future. We also include two open questions to obtain the actors who directly interact with it. Thus, the participants will be participants’ feedback regarding the changes they would make to required to create as many DVD models as necessary to represent improve the method and their reasons for using a given method in the whole business. the future (if any). This questionnaire will be online, using Google Forms and the data collected will be kept anonymously. All the Step 2 - Set value exchanges: Participants will update the model experimental material will be created in Spanish and Portuguese, by adding the value exchanges. During this activity, participants since these are the participants’ native languages. define the value element related to each value port. Step 3 - Set who starts each value exchange: Participants will 3.1.9   Threats to Validity define which actor starts the value exchange. Here, it is important We must consider certain issues which may threaten the validity to check if the value elements are specified in the correct value of this experiment. With regard to internal validity, the main port. threats are: learning effect, participant experience, information exchange among participants, and understandability of the Step 4 - Set value level agreement: Participants define the criteria documents. required for value exchanges to be perform. This step is very important that participants understand the business constraints The learning effect is alleviated by ensuring that groups of participants will apply the two methods to different experimental related to each value exchange. objects in a different order. We also plan to assess the effect of Step 5 - Prioritize the value exchanges: Participants prioritize order of system domain and order of methods by using statistical each value exchange according to the expected return of tests. Participants’ experience is alleviated as none of the investment (ROI). This is a subjective prioritization as participants have any experience in value-driven development. participants will set the value exchanges priority in relation to We plan to confirm this fact by asking the participants about their other value exchanges without the use of any mathematical model. experience with value-driven development methods. To minimize the information exchange among participants, they will be monitored by the experimenters to avoid communication 4 In the case of e3value, the value models which will be used in Figure 6. Dynamic Value Description process. comparison were found in literature. biases while performing the tasks. However, this might affect the With regard to the operation of the experiment, the experiment is results since the experiment will take place over more than one planned to be conducted in three sessions (Table 3 shows the day, and it is difficult to be certain whether or not the participants details for each session). On the first session, the participants will will exchange any information. To alleviate this situation, at least be given a detailed training on the methods to be applied and also to some extent, participants will be asked to return all the material on the tasks to be performed in the execution of the experiment. In at the end of each task. Finally, understandability of the material this session, they will perform a practical session in which they will be alleviated by performing a pilot study. In addition, we will will specify a value model using both methods. clear up all the misunderstandings that may appear in each experimental session. On the second and third sessions, the participants will be given an overview of the training before applying each value-driven With regard to external validity, the main threats are: development method to the experimental objects (O1 or O2). We representativeness of the results and the size and complexity of will establish a slot of 90 min with no time limit for any of the the tasks. The representativeness of the results may be affected by methods to be applied. In addition, we will allow the participants the software systems used and the participant’s context selected. to continue the experiment even though these 90 min is not With regard to the selection of software systems, we attempted to enough in order to avoid a possible ceiling effect. alleviate this by considering a set of artifacts with similar size and With regard to the experiment execution, the experiment will take complexity, and which contains representative artifacts of an place in a single room, and no interaction among participants will existing value-driven development method (i.e., e3value). be allowed. The experimenter will clarify possible questions that Despite the fact that the planned experiments will be performed in may arise during the sessions. an academic context (undergraduate and Master’s students), the participants’ performance can be considered to be representative With regard to the data validation, we plan to verify that the of novice modelers since the kinds of students involved will be participants complete the two experimental sessions. Data points containing only one session will be discarded. If this occurs, other soon integrated into the industry’s market. As further work, we plan to conduct more experiments involving practitioners in order data points may also be discarded in order to maintain the to assess how the experience level would impact on the obtained balanced design shown in Table 3 (i.e., having exactly the same results. Also, since only internal replications will be conducted, number of participants in each group). more external replications need to be conducted by other 3.3   Data Analysis & Interpretation experimenters in other settings to confirm these results. In order to This section introduces the statistical tests that will be used to address the aforementioned limitations, these external replications analyze the data collected: the influence of the method on the will involve participants from different contexts and also with dependent variables and the effect of system domain and order of different levels of experience in value-driven development. method. These tests have been chosen because they are very The size and complexity of the tasks may also affect the external robust and sensitive, and have been used in previous experiments validity. We use relatively small tasks that would be applied in a similar to ours, e.g., [29], [30]. As usual, in all the tests we have few representative software artifacts since a controlled experiment decided to accept a probability of 5% of committing a Type-I- requires participants to complete the assigned tasks in a limited Error [19], i.e., of rejecting the null hypothesis when it is actually amount of time. true. With regard to construct validity, the main threats are: the 3.3.1   Influence of Method measures that will be applied in the data analysis and the We plan to use boxplots and statistical tests to analyze the data reliability of the questionnaire. We attempt to alleviate this threat collected. In particular, we will test the normality of the data by using measures that are commonly applied in other software distribution by applying the Shapiro–Wilk test. The results of the engineering experiments. In particular, the Effectiveness was normality test will allow us to select the proper significance test in measured using the Jaccard index, also known as the Jaccard order to test our hypotheses. When data is assumed to be normally similarity coefficient, which has commonly been used to measure distributed (p-value≥0.05), we will apply the parametric one- the similarity and diversity of sample sets. The subjective tailed t-test for independent samples [31]. However, when data variables are based on the Technology Acceptance Method could not be assumed to be normally distributed (p-value <0.05), (TAM), a well-known and empirically validated model for the we will apply the non-parametric Mann–Whitney test [32]. evaluation of information technologies [25], [26]. The reliability of the questionnaire will be tested by applying the Cronbach test. 3.3.2   Influence of Order of System Domain and With regard to conclusion validity, the main threats are: the data Order of Method collection and the validity of the statistical tests applied. With To test the influence of order of system domain and order of regard to the data collection, we plan to apply the same procedure method (both independent variables), we plan to use a method in each individual experiment in order to extract the data, and similar to that proposed by Briand et al. [30]. We will use the Diff ensure that each dependent variable is calculated by applying the function: same formula. With regard to the validity of the statistical tests Diffx = observationx(A) - observationx(B) (2) proposed, we chose the most common tests that are employed in the empirical software engineering field due to their robustness where x denotes a particular participant, and A, B are the two and sensitivity [28]. possible nominal values of an independent variable. We plan to 3.2   Operation and Execution create Diff variables from each dependent variable e.g., This section describes the experimental operation, including the Effectiveness_Diff(VDD) will represent the difference in preparation, execution, data recording and data validation. effectiveness of the subjects who used VDD first and e3value second. On the other hand, Effectiveness_Diff(e3value) will represent the difference in effectiveness of the participants who used e3value first and VDD second. The aim is to verify that there [12] J. Gordijn, “Value-based Requirements Engineering”, Phd are no significant differences between Diff functions since that Thesis, 2002. would signify that there is no influence in the order of the [13] R. J. A. Buhr and R. S. Casselman, Use case maps for independent variables. We also plan to apply the Shapiro-Wilk object-oriented systems. Prentice-Hall, Inc., 1995. test to prove the normality of the Diff functions. The hypotheses [14] V. Pijpers, P. de Leenheer, J. Gordijn, and H. Akkermans, related to the Diff functions are two-sided since we do not make “Using conceptual models to explore business-ICT any assumption about whether one specific order would be more alignment in networked value constellations”, RE, vol. 17, influential than another. We plan to verify these hypotheses by no. 3, pp. 203–226, Oct. 2011. applying the parametric two-tailed t-test for independent samples [15] F. Wanderley, A. Silva, and J. Araújo, “Evaluation of or the non-parametric Mann-Whitney test depending on the results BehaviorMap: A user-centered behavior language”, in 9th of the normality test. RCIS, 2015, pp. 309–320. [16] S. Ambler, Agile modeling: effective practices for extreme 4.   CONCLUSIONS AND FURTHER WORK programming and the unified process. John Wiley & Sons, We have presented two early requirements modeling methods to Inc., New York, 2002. represent a business from an economic point of view: value- [17] L. Chung, B. A. Nixon, E. S.-K. Yu, and J. Mylopoulos, driven development method and e3value. Moreover, we also have Non-Functional Requirements in Software Engineering. presented an experiment design aimed at comparing these two Boston, MA: Kluwer Academic Publishers, 1999. methods. Our experiment design allows predicting the acceptance [18] I. Jacobson, J. Rumbaugh, and G. Booch, The Unified of a particular method in practice, based on the effort of applying Modeling Language Reference Manual. Reading, MA: the method, the quality of the artifacts produced, and the user Addison-Wesley, 1999. perceptions with regard to the quality of the method. In future [19] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. work, we plan to perform a family of experiments by using the Regnell, and A. Wesslén, Experimentation in Software proposed experiment design. Engineering. Springer Berlin Heidelberg, 2012. [20] V. R. Basili and H. D. Rombach, “The Tame Project - 5.   ACKNOWLEDGMENTS Towards Improvement-Oriented Software Environments”, This research is supported by the Value@Cloud project IEEE Trans. Software Eng., vol. 14, no. 6, 1988. (MINECO TIN2013-46300-R), NOVA LINCS Research [21] C. Huemer, A. Schmidt, and H. Werthner, “A UML profile Laboratory (Ref. UID/CEC/04516/2013), and programa Ciência for the e3-value e-business model ontology”, in BUSITAL sem Fronteiras - CAPES. held in conjunction with CAiSE”08 Conference, 2008. [22] Z. Derzsi and J. Gordijn, “A Framework for Business/IT 6.   REFERENCES Alignment in Networked Value Constellations”, in [1] R. Veryard, Six Viewpoints of Business Architecture. BUSITAL, 2006. Leanpub, 2015. [23] T. Gorschek, C. Wohlin, P. Garre, and S. Larsson, “A model [2] R. Passonneau, K. Kukich, and J. Robin, “Generating for technology transfer in practice”, IEEE Software, vol. 23, summaries of workflow diagrams”, in International no. 6, pp. 88–95, 2006. Conference on Natural Language Processing and Industrial [24] V. R. Basili, F. Shull, and F. Lanubile, “Building knowledge Applications, Moncton, Canada, 1996. through families of experiments”, IEEE Trans. Software [3] M. Dumas and A. H. M. ter Hofstede, “UML Activity Eng., vol. 25, no. 4, pp. 456–473, 1999. Diagrams as a Workflow Specification Language”, in UML: [25] F. D. Davis, “Perceived usefulness, perceived ease of use, Modeling Languages, Concepts, and Tools, vol. 2185, no. 7, and user acceptance of information technology”, MIS Toronto, Canada, 2001, pp. 76–90. quarterly, vol. 13, no. 3 , 1989. [4] S. A. White, “Introduction to BPMN”, IBM, 2004. [26] W. R. King and J. He, “A meta-analysis of the technology [5] A. Rasiwasia, “Meta Model for Business Model Design: acceptance model”, Information & Management, vol. 43, Designing a Meta model for E3 value model based on no. 6, pp. 740–755, Sep. 2006. MOF”, Stockholm, Sweden, 2013. [27] S. Abrahao, E. Insfran, J. A. Carsí, and M. Genero, [6] J. Gordijn and H. Akkermans, “Designing and evaluating e- “Evaluating requirements modeling methods based on user business models”, IEEE Intell. Syst., vol. 16, no. 4, 2001. perceptions: A family of experiments”, Information [7] The Institute of Value Management, “What is Value?.” Sciences, vol. 181, no. 16, pp. 3356–3378, Aug. 2011. [Online]. Available: https://ivm.org.uk/what-is-value- [28] K. Maxwell, Applied statistics for software managers. management. [Accessed: 21-Jul-2016]. Software Quality Institute Series, Prentice Hall, 2002. [8] J. Gordijn and J. M. Akkermans, “Value-based requirements [29] S. Abrahao, C. Gravino, E. Insfran, G. Scanniello, and G. engineering: exploring innovative e-commerce ideas”, RE, Tortora, “Assessing the Effectiveness of Sequence vol. 8, no. 2, pp. 114–134, Jul. 2003. Diagrams in the Comprehension of Functional [9] V. Kartseva, J. Gordijn, and Y.-H. Tan, “Inter- Requirements: Results from a Family of Five Experiments”, organisational Controls as Value Objects in Network IEEE Trans. Software Eng., vol. 39, no. 3, 2013. Organisations”, in Experiences with Model Reuse: Non- [30] L. C. Briand, Y. Labiche, M. Di Penta, and H. Yan-Bondoc, Functional Requirements Catalogues for Ubiquitous “An experimental investigation of formality in UML-based Systems, vol. 4001, no. 23, 2006, pp. 336–350. development”, IEEE Trans. Software Eng., vol. 31, no. 10, [10] D. Kundisch and T. John, “Business Model Representation pp. 833–849, Nov. 2005. Incorporating Real Options: An Extension of e3-Value”, in [31] N. Juristo and A. M. Moreno, Basics of Software 45th HICSS, 2012, pp. 4456–4465. Engineering Experimentation, 1st edition. Springer [11] J. Gordijn, “E3-value in a Nutshell “, International Publishing Company, Incorporated, 2010. Workshop on e-Business Modeling, 2002. [32] W. J. Conover, Practical Nonparametric Statistics, 3rd ed. Wiley India Pvt. Limited, 2006.