A business model is a representation of an organization with a particular point of view. It is common to find different types of models to describe the business. However, methods to create business models representing an economic point of view have only emerged over the last few years in the scientific community. Such methods aid business specialists improving the economics understanding of the business, helping both defining more efficient business strategies and better aligning the information technology systems with the business. This paper aims at describing the design of an experiment to compare two methods to specify economic values (e3value and value-driven development). Our experiment design allows predicting the acceptance of a particular method in practice, based on the effort of applying the method, the quality of the artifacts produced, and the user perceptions with regard to the quality of the method.
• Information systems → Language models value model; value-driven; experiment design
Models to describe the daily behavior of the business, or what the
Object Management Group1 calls “business in motion” [
This section summarizes the e3value [
The e3value method offers modeling constructs for representing
graphically and analyzing business requirements from an
economic point of view. It is composed of fifty concepts [
Value interfaces group value ports. Value ports provide or request value objects to or from actors or market segment. Actors only offer objects to others if they receive adequate compensation in return. Value objects are money, goods, services or information, which are of economic value for the actors. A market segment, on the other hand, is a group of actors in a business segment that share common properties. The set of value objects exchanged by actors is defined as a value exchange. Value transfers are used to link two value ports with each other. Value transactions are groups of value transfers. For a value exchange to happen, actors, or market segment, must perform a set of operational activities. The collection of these activities is called as value activities.
.
In order to represent value exchange scenarios, the e3value model
inherited the start stimulus, the stop stimulus, the AND element,
the OR element, and the connect element from Use Case Maps2
(UCM) [
VDD is an approach to derive software architecture aligned with
business economic values supported by model-driven techniques.
To improve the understanding of this method, we divided it in
three different phases: business analysis, requirements
specification, and software architecture derivation. The business
analysis is an early requirements phase whose goal is to analyze
and represent the economic values exchange through a model
called Dynamic Value Description (DVD). From the DVD model,
both business analysts and requirements engineers specify
information system requirements by using a cognitive
requirements approach [
The cognitive requirements approach improves the domain understanding because it provides an environment wherein all the stakeholders could share their views and abstractions in a semistructured mind map model3. Finally, from these requirements specifications, the software architect generates a high-level software architecture by using model-driven techniques. In the context of this paper, we address only the business analysis phase. So, we analyze the DVD model and its creation process. Figure 3 depicts the DVD metamodel and Figure 4 presents a DVD model example (instance from metamodel).
As we can see in the metamodel, DVD is composed of eight main
concepts: main actor, environment actor, value exchange, who
starts the value exchange, value port, value element, value level
agreement, and priority. Similarly to the e3value model, actors are
environment entities economically independent in the DVD
model. However, each time, the business analyst focuses the
analysis on the main actor and represents its relationship with
others environment actors, producing an inter-organizational
network. As the focus changes, the actor playing the role of “main
actor” also changes. With this change in focus, new actors and
value exchanges may appear.
2 Use Case Maps is a requirements language which have the
notion of path to show how a particular scenario works.
3 Mind map is considered a simple and accessible model [
From the actors relationship, a value exchange is performed. It shows economic reciprocity through two value ports (arrows connected to value exchange), one for entry and one for exit, which point to value elements (money, goods, services or information). If there are many value objects in the same value port, the business analyst must use logical operators (“AND”, “OR”, and “XOR”) to detail the relationships among them. In addition, the business analyst also defines who starts the value exchanges through a configuration of arrows between the main actor and the environment actor. It is important to notice that during the DVD modeling, the business analyst is able to focus on each actor individually in order to give more attention and details to the actor which must be analyzed. Thus, the analyst sets who is the main actor and a given support tool will display it as the central node of the model, dynamically. Each value exchange needs a level of agreement between the ones involved. This level of agreement is a particular business aspect that must be minimally agreed among the actors in order to enable the value exchanges.
Finally, the DVD model describes a prioritization of value exchanges using colors. The red color means high priority, yellow color means medium priority, and blue color means low priority. These priorities are set by business analyst according to the return of investment of the value exchanges in the business.
This section compares the e3value and the value-driven
development methods. Table 1 presents a mapping between
e3value and VDD concepts.
We observed that the DVD model (from the VDD method)
describes two new concepts in relation to the e3value: value level
agreement (VLA) and priority. In contrast, the e3value model has
the value activity concepts, not offered by the DVD model. The
VLA defines the business constraints based on the business
strategies. For example, a company of the feeding segment
provides food fresher than its competitors, as a business strategy.
Thus, to provide fresher food, it is essential that its suppliers also
deliver fresh ingredients. Therefore, the business analyst can
specify a VLA by defining the acceptable time of receipt of these
ingredients. Regarding the information system development, the
complexity of a software system is determined by its functionality
(i.e., what the system does) and by global requirements on its
development, such as operational costs, performance, reliability,
maintainability, portability, robustness [
This section presents the design of a controlled experiment aimed
at comparing the value-driven development method against the
e3value method. We followed the guidelines proposed by Wohlin
et al. [
According to the Goal-Question Metric (GQM) approach [
The software systems to be developed were selected from the
literature [
Wireless access provisioning (O2): It describes the business where a hotel would like to offer wireless connectivity to businessmen as an additional service.
To assess the complexity of the models used and to identify possible mistakes, we plan to carry out a pilot experiment with a small group of PhD students at UPV.
The context of this experiment is the evaluation of value-driven
development methods from the perspective of novice modelers.
Although experienced modelers and practitioners are desired, we
focus on the profile of novice modelers since one of our goals is
to provide a value-driven development method that will help less
experienced modelers to specify value models. In addition,
according to the Technology Transfer Model proposed by
Gorschek et al. [
Value models can be produced by business analysts or software engineers. The following groups of participants are therefore identified in order to facilitate the generalization of results: Undergraduate students, all Computer Science students from the Software Engineering intensification at the Universitat Politècnica de València. These students will attend the “Requirements Engineering” course from September 2016 to January 2017, during this time they will have 8h of lectures on business modeling and value-driven development.
Master’s students, enrolled on the Master’s Degree in Engineering and Technology of Software Systems at the Universitat Politècnica de València. These students will • • • attend the “Empirical Software Engineering” course from September to November 2016.
Undergraduate students, all Computer Science students from the Software Engineering intensification at the Universitat Politècnica de València. These students will attended the “Software Quality” course from February to June 2017. One of the main topics of this course is to evaluate the quality of models obtained through the software development process. A teaching unit on the evaluation of value-driven development methods will be added to the course program. Master’s students, enrolled on the Master’s Degree in Software Engineering at the Universidade Nova de Lisboa. These students will attend the “Software Engineering” course from September 2016 to January 2017.
Undergraduate students, all business management students at the Universidade Federal de Pernambuco. These students will attend the “Information System” course from January to July 2017.
These courses were selected because the preparation and training
and the experimental task itself fit their scope. We take a
convenience sample (i.e., all the students available in the class).
The original experiment will be conducted in the Requirements
Engineering course and the other groups will be exact and/or
differentiated replications. This will allow us to build a body of
knowledge about these value-driven development methods. As
Basili et al. [
The independent variable of interest is the use of each valuedriven method with nominal values: VDD and e3value. Hence, the experiment use two treatments: the creation of a value model for two software systems using VDD and the creation of a value model for the same systems using e3value. The experimental data collected allows comparing the effects of both treatments. There are two types of dependent variables in which the treatments are compared: performance-based and perceptionbased variables. Performance-based variables assess how well the participants perform the experimental task. They are used to evaluate the actual efficacy of the value-driven development methods. Perception-based variables assess the participants’ perceptions of their performance and their subsequent intention to use VDD or e3value. These variables are used to evaluate the perceived efficacy of these methods, as well as their likely adoption in practice.
There are two performance-based variables:
Effectiveness: It is calculated with the Jaccard index (see formula
(1)) that measures similarity between sample sets and is defined as
the size of the intersection divided by the size of the union of the
sample sets. Given two models, A and B, the Jaccard index
measures the overlap that A and B share with their elements. In
our case, we will calculate the Jaccard index between an agreed
solution among experts (A) (for the value models obtained with
VDD and e3value) and the solution given by each participant (B).
A Jaccard index of 0 represents no overlap between the solutions,
while 1 indicates that they contain the same results.
(1)
Efficiency: It is the time required to apply the method.
There are also three perception-based variables, which are based
on the Technology Acceptance Model (TAM) [
Perceived Usefulness (PU): It refers to the degree to which a person believes that using a specific method will increase his or her job performance within an organizational context. Intention to Use (ITU): It refers to the extent to which a person intend to use a particular method. This last variable represents a perceptual judgment of the method’s efficacy – that is, whether it is cost-effective and is commonly used to predict the likelihood of acceptance of a method in practice.
These three subjective variables will be measured by using a Likert scale questionnaire with a set of 12 closed-questions: 3 questions for perceived ease of use (PEOU), 6 questions for perceived usefulness (PU) and 3 for intention to use (ITU). The questionnaire can be found here: http://bit.ly/2ak1wLS. The closed-questions were formulated by using a 5-point Likert scale, using the opposing statement question format. In other words, each question contains two opposite statements representing the maximum and minimum possible values (5 and 1), where the value 3 is considered to be a neutral perception. The aggregated value of each subjective variable will be calculated as the arithmetical mean of the answers to the questions associated with each perception-based variable.
We formulated several null hypotheses, which were formulated in a one-tailed manner, since we want to analyze the effect of the use of VDD on the variables. Each null hypothesis and its alternative are presented as follows: • • • • •
H10: There is no significant difference between the effectiveness of VDD and e3value / H1a: VDD is significantly more effective than e3value.
H20: There is no significant difference between the efficiency of VDD and e3value / H2a: VDD is significantly more efficient than e3value.
H30: There is no significant difference between the perceived ease of use of evaluators applying VDD and e3value / H3a: VDD is perceived as easier to use than e3value.
H40: There is no significant difference between the perceived usefulness of VDD and e3value / H4a: VDD is perceived as more useful than e3value.
H50: There is no significant difference between the intention to use of VDD and e3value / H5a: VDD is perceived as more likely to be used than e3value.
Although Method is the only factor of interest in this empirical investigation, other factors may affect the participants’ performance in an undesirable way, thus confounding the Method effect. These factors have to be controlled so that only the effect of the Method factor, if there is any, is observable: System Domain. The complexity of the software requirements considered in the tasks may have a confounding effect on the results. The application domain of the tasks could also be a confounding factor that could affect the subjects’ comprehension. Order of Methods. The order in which the subjects apply the methods may produce learning effects, which may bias the results.
The experiment is planned as a balanced within-participant design with a confounding effect, signifying that the same participants apply both methods with both experimental objects in a different order. We plan to establish four groups (each of which will apply one method to one experimental object) and the participants will be randomly assigned to each group. Table 3 summarizes the design of the experiment.
The within-participant experimental design is intended to minimize the impact of learning effects on the results, since none of the participants repeat any treatment or experimental object during the execution. Other factors which may also be present need to be controlled, since they might influence the results, i.e., the complexity of experimental objects. The comprehension of the software systems requirements to be modeled may also affect the application of both methods. We attempted to alleviate the influence of this factor by selecting two representative software systems with software system requirements of a reasonable complexity. The complexity of the software systems selected made them suitable for application in the time slot available for the execution of the experiments (2 hour sessions).
Sessions Group of participants (sample size
= 4n participants)
We have defined only one experimental task (create the value model) of which its steps vary according to the value model that the subjects will create. We will offer a training session to explain the concepts of value models and how they are created. During the experimentation, we will offer the requirements specifications to the subjects (see Section 3.1.2). The requirements specifications describe how the business works. With these specifications in their hands, the participants will create a particular value model (DVD or e3value) and will register the start time and the end time for each step performed. Step 1 - Identify scenarios: Scenarios are short textual sentences, meaning the product, service, or experience expected by a customer. Therefore, the goal of this step is that the participants write a scenarios list.
Step 2 - Identify actors: The participants will identify who offers and who receives the product, service or experience expected from the scenarios list and they will create a list of actors. Step 3 - Create value model: With the scenarios list and the actors list in their hands, the subjects will create the initial e3value model by using the products and services mentioned in the scenarios list and the actors described in actors list.
Step 4 - Identify UCMs: The participants will insert the UCM's elements, representing the paths of all scenarios in the e3value model. In other words, they will insert the start stimulus, stop stimulus, AND element, OR element, and connect element in the e3value model.
In the case of creating the DVD model, the participants will follow the steps from the VDD process (see Figure 6). These steps are: Step 1 - Specify actors: Participants start the DVD model by describing the main actor (the focus of their analysis) and their related environment actors. Thus, the participants will create a DVD model like a mind map, where the main actor is the central node and the environment actors are the leaf nodes. Due to this “main actor” focus, the DVD model shows only the environment actors who directly interact with it. Thus, the participants will be required to create as many DVD models as necessary to represent the whole business.
Step 2 - Set value exchanges: Participants will update the model by adding the value exchanges. During this activity, participants define the value element related to each value port.
Step 3 - Set who starts each value exchange: Participants will define which actor starts the value exchange. Here, it is important to check if the value elements are specified in the correct value port.
Step 4 - Set value level agreement: Participants define the criteria required for value exchanges to be perform. This step is very important that participants understand the business constraints related to each value exchange.
Step 5 - Prioritize the value exchanges: Participants prioritize each value exchange according to the expected return of investment (ROI). This is a subjective prioritization as participants will set the value exchanges priority in relation to other value exchanges without the use of any mathematical model.
The reason for this is that there is a lack of economics results at this moment of the analysis.
Once the value model is created, the participants will answer the post-experimental questionnaire. Hence, we will be able to evaluate the performance-based variables (effectiveness and efficiency) by comparing the value model they created against the value model created by experts4 and by analyzing the time registered to perform each experimental step. In addition, we will evaluate the perception-based variables (perceived ease of use, perceived usefulness, and intention to use) from the responses received in the post-experimental questionnaire.
The experimental material is composed of a set of documents required to support the experimental tasks and the training sessions, along with the post-experimental questionnaire. The training materials include: i) a set of slides containing an introduction to business modeling and value-driven development; ii) a set of slides describing the VDD method, along with an example of its application; and iii) a set of slides describing the e3value method, with an example of its application.
The documents supporting the experimental tasks include: • Two kinds of booklets covering the two possible combinations of both the value-driven development method and the experimental objects (VDD-O1, VDD-O2, e3valueO1, e3value-O2). The purpose of these booklets is i) to describe the experimental tasks to be performed; ii) to describe the software system requirements; and iii) to gather the data from the experimental task. •
An appendix containing a guideline to help the participants to apply the value-driven development method.
The post-experimental questionnaire contains a set of closedquestions that allows participants to express their opinion on the ease of use, usefulness, and their intention to use of the method in the future. We also include two open questions to obtain the participants’ feedback regarding the changes they would make to improve the method and their reasons for using a given method in the future (if any). This questionnaire will be online, using Google Forms and the data collected will be kept anonymously. All the experimental material will be created in Spanish and Portuguese, since these are the participants’ native languages.
We must consider certain issues which may threaten the validity of this experiment. With regard to internal validity, the main threats are: learning effect, participant experience, information exchange among participants, and understandability of the documents.
The learning effect is alleviated by ensuring that groups of participants will apply the two methods to different experimental objects in a different order. We also plan to assess the effect of order of system domain and order of methods by using statistical tests. Participants’ experience is alleviated as none of the participants have any experience in value-driven development. We plan to confirm this fact by asking the participants about their experience with value-driven development methods.
To minimize the information exchange among participants, they will be monitored by the experimenters to avoid communication 4 In the case of e3value, the value models which will be used in comparison were found in literature. biases while performing the tasks. However, this might affect the results since the experiment will take place over more than one day, and it is difficult to be certain whether or not the participants will exchange any information. To alleviate this situation, at least to some extent, participants will be asked to return all the material at the end of each task. Finally, understandability of the material will be alleviated by performing a pilot study. In addition, we will clear up all the misunderstandings that may appear in each experimental session.
With regard to external validity, the main threats are: representativeness of the results and the size and complexity of the tasks. The representativeness of the results may be affected by the software systems used and the participant’s context selected. With regard to the selection of software systems, we attempted to alleviate this by considering a set of artifacts with similar size and complexity, and which contains representative artifacts of an existing value-driven development method (i.e., e3value). Despite the fact that the planned experiments will be performed in an academic context (undergraduate and Master’s students), the participants’ performance can be considered to be representative of novice modelers since the kinds of students involved will be soon integrated into the industry’s market. As further work, we plan to conduct more experiments involving practitioners in order to assess how the experience level would impact on the obtained results. Also, since only internal replications will be conducted, more external replications need to be conducted by other experimenters in other settings to confirm these results. In order to address the aforementioned limitations, these external replications will involve participants from different contexts and also with different levels of experience in value-driven development. The size and complexity of the tasks may also affect the external validity. We use relatively small tasks that would be applied in a few representative software artifacts since a controlled experiment requires participants to complete the assigned tasks in a limited amount of time.
With regard to construct validity, the main threats are: the
measures that will be applied in the data analysis and the
reliability of the questionnaire. We attempt to alleviate this threat
by using measures that are commonly applied in other software
engineering experiments. In particular, the Effectiveness was
measured using the Jaccard index, also known as the Jaccard
similarity coefficient, which has commonly been used to measure
the similarity and diversity of sample sets. The subjective
variables are based on the Technology Acceptance Method
(TAM), a well-known and empirically validated model for the
evaluation of information technologies [
This section describes the experimental operation, including the preparation, execution, data recording and data validation. With regard to the operation of the experiment, the experiment is planned to be conducted in three sessions (Table 3 shows the details for each session). On the first session, the participants will be given a detailed training on the methods to be applied and also on the tasks to be performed in the execution of the experiment. In this session, they will perform a practical session in which they will specify a value model using both methods.
On the second and third sessions, the participants will be given an overview of the training before applying each value-driven development method to the experimental objects (O1 or O2). We will establish a slot of 90 min with no time limit for any of the methods to be applied. In addition, we will allow the participants to continue the experiment even though these 90 min is not enough in order to avoid a possible ceiling effect.
With regard to the experiment execution, the experiment will take place in a single room, and no interaction among participants will be allowed. The experimenter will clarify possible questions that may arise during the sessions.
With regard to the data validation, we plan to verify that the participants complete the two experimental sessions. Data points containing only one session will be discarded. If this occurs, other data points may also be discarded in order to maintain the balanced design shown in Table 3 (i.e., having exactly the same number of participants in each group).
This section introduces the statistical tests that will be used to
analyze the data collected: the influence of the method on the
dependent variables and the effect of system domain and order of
method. These tests have been chosen because they are very
robust and sensitive, and have been used in previous experiments
similar to ours, e.g., [
We plan to use boxplots and statistical tests to analyze the data
collected. In particular, we will test the normality of the data
distribution by applying the Shapiro–Wilk test. The results of the
normality test will allow us to select the proper significance test in
order to test our hypotheses. When data is assumed to be normally
distributed (p-value≥0.05), we will apply the parametric
onetailed t-test for independent samples [
To test the influence of order of system domain and order of
method (both independent variables), we plan to use a method
similar to that proposed by Briand et al. [
Diffx = observationx(A) - observationx(B) (2) where x denotes a particular participant, and A, B are the two possible nominal values of an independent variable. We plan to create Diff variables from each dependent variable e.g., Effectiveness_Diff(VDD) will represent the difference in effectiveness of the subjects who used VDD first and e3value second. On the other hand, Effectiveness_Diff(e3value) will represent the difference in effectiveness of the participants who used e3value first and VDD second. The aim is to verify that there are no significant differences between Diff functions since that would signify that there is no influence in the order of the independent variables. We also plan to apply the Shapiro-Wilk test to prove the normality of the Diff functions. The hypotheses related to the Diff functions are two-sided since we do not make any assumption about whether one specific order would be more influential than another. We plan to verify these hypotheses by applying the parametric two-tailed t-test for independent samples or the non-parametric Mann-Whitney test depending on the results of the normality test.
We have presented two early requirements modeling methods to represent a business from an economic point of view: valuedriven development method and e3value. Moreover, we also have presented an experiment design aimed at comparing these two methods. Our experiment design allows predicting the acceptance of a particular method in practice, based on the effort of applying the method, the quality of the artifacts produced, and the user perceptions with regard to the quality of the method. In future work, we plan to perform a family of experiments by using the proposed experiment design.
This research is supported by the Value@Cloud project (MINECO TIN2013-46300-R), NOVA LINCS Research Laboratory (Ref. UID/CEC/04516/2013), and programa Ciência sem Fronteiras - CAPES.