=Paper=
{{Paper
|id=Vol-1805/Souza2016HuFaMo
|storemode=property
|title=Comparing Value-Driven Methods: an Experiment Design
|pdfUrl=https://ceur-ws.org/Vol-1805/Souza2016HuFaMo.pdf
|volume=Vol-1805
|authors=Eric Souza,Silvia Abrahão,Ana Moreira,João Araújo,Emilio Insfran
|dblpUrl=https://dblp.org/rec/conf/models/SouzaAM0I16
}}
==Comparing Value-Driven Methods: an Experiment Design==
https://ceur-ws.org/Vol-1805/Souza2016HuFaMo.pdf
Comparing Value-Driven Methods: an experiment design
Eric Souza Silvia Abrahão Ana Moreira
NOVA LINCS, Departamento de Sistemas NOVA LINCS,
Departamento de Informática Informaticos y Computacion, Departamento de Informática
Faculdade de Ciências e Tecnologia, Universitat Politecnica de Valencia, Faculdade de Ciências e Tecnologia,
Universidade Nova de Lisboa, España Universidade Nova de Lisboa,
Portugal sabrahao@dsic.upv.es Portugal
er.souza@campus.fct.unl.pt amm@fct.unl.pt
João Araújo Emilio Insfran
NOVA LINCS, Departamento de Informática Departamento de Sistemas Informaticos y
Faculdade de Ciências e Tecnologia, Computacion,
Universidade Nova de Lisboa, Portugal Universitat Politecnica de Valencia, España
joao.araujo@fct.unl.pt einsfran@dsic.upv.es
ABSTRACT inspired in business science, requirements engineering, and
A business model is a representation of an organization with a conceptual modeling techniques [5], whose aim is to show how
particular point of view. It is common to find different types of economic values are created and exchanged in an inter-
models to describe the business. However, methods to create organizational network [6].
business models representing an economic point of view have Value means the relationship between satisfying needs and
only emerged over the last few years in the scientific community. expectations and the resources required to achieve them [7]. Value
Such methods aid business specialists improving the economics is the reason for companies and people to trade with each other,
understanding of the business, helping both defining more offering money to get something in return. Therefore, a value
efficient business strategies and better aligning the information model represents a business model from an economic perspective,
technology systems with the business. This paper aims at and must determine the economic value exchanged and their
describing the design of an experiment to compare two methods to intervenients [8]. This understanding facilitates the aligning of the
specify economic values (e3value and value-driven development). software requirements specifications with the value exchanges [9].
Our experiment design allows predicting the acceptance of a
There is a number of approaches to represent value. Kundisch and
particular method in practice, based on the effort of applying the
John [10] classify 12 different business model representations:
method, the quality of the artifacts produced, and the user activity system map, business model ontology, causal loop
perceptions with regard to the quality of the method.
diagram, value map, value net, strategic business model ontology
CCS Concepts (SBMO), and business model ontology (BMO), among others.
However, there is no empirical evidence about which of these
• Information systems → Language models
methods is more effective under what circumstances. For
Keywords practitioners to consider adopting a given value-driven method,
value model; value-driven; experiment design they must know its effectiveness and how it compares with others.
The goal of this paper is therefore to describe the design of an
1. INTRODUCTION experiment aimed at comparing two methods to express economic
Models to describe the daily behavior of the business, or what the values: e3value [11], which is a widely established and applied
Object Management Group1 calls “business in motion” [1], are business model representation, and our own Value-Driven
common. This business behavior, in general, is represented by Development method (VDD), which has been proposed recently.
using some well-known description languages such as workflow This design is also valuable because it can be replicated for other
diagrams [2], UML activity diagrams [3], and BPMN [4]. methods.
However, methods to create business models representing an
The remainder of this paper is organized as follows. Section 2
economic point of view have emerged over the last years in the
introduces e3value and VDD. Section 3 presents the design of a
scientific community. In principle, the reason for this is that an
controlled experiment aimed at comparing the effectiveness,
organization needs to make money to keep going in a competitive
efficiency, perceived ease of use, perceived usefulness, and
market. In addition, business specialists need to improve the
intention to use of several groups of users employing both
economics understanding to define more efficient business
methods for creating a value model. Finally, Section 4 concludes
strategies and provide a better alignment of the information
this paper and summarizes directions for further work.
technology with the business. These models can then be used to
define the requirements of the underlying information systems. In 2. METHODS TO BE COMPARED
general, these methods are conceptual requirements approaches, This section summarizes the e3value [12] and VDD, comparing
them with respect to their concepts and processes.
1
The Object Management Group is an international technology standards
consortium.
2.1 e3value method The start stimulus represents customer needs, that is, the
3
The e value method offers modeling constructs for representing beginning of a value scenario, and the stop stimulus represents the
graphically and analyzing business requirements from an end of a value scenario. A connection element links a start-stop
economic point of view. It is composed of fifty concepts [12], stimulus to a value interface or links value interfaces of the same
whose main ones are: elementary actor, composite actor, market actor internally. As a lot of value scenarios are represented in a
segment, value interface, value transfer, value port, value object, unique e3value model, AND and OR elements are used to split or
value exchange, value transaction, value activity, start stimulus collapse paths of value scenarios, reusing start and stop stimulus
(customer needs), stop stimulus (scenario boundary), AND elements.
element, OR element, and connect element. Figure 1 presents the
e3value metamodel (note that some of these concepts are not
present [5], [11]), and Figure 2 exemplifies an e3value model.
Actors are environment entities economically independent, which
can be specialized as composite or elementary. The difference is
that the composite actor is a group actor with value interfaces of
the inner elementary actors. Thus, value interfaces allow
accessibility to the constituent elementary actors.
Value interfaces group value ports. Value ports provide or request
value objects to or from actors or market segment. Actors only Figure 2. e3value example extracted from [14].
offer objects to others if they receive adequate compensation in
return. Value objects are money, goods, services or information,
which are of economic value for the actors. A market segment, on
the other hand, is a group of actors in a business segment that 2.2 Value-Driven Development method
share common properties. The set of value objects exchanged by VDD is an approach to derive software architecture aligned with
actors is defined as a value exchange. Value transfers are used to business economic values supported by model-driven techniques.
link two value ports with each other. Value transactions are To improve the understanding of this method, we divided it in
groups of value transfers. For a value exchange to happen, actors, three different phases: business analysis, requirements
or market segment, must perform a set of operational activities. specification, and software architecture derivation. The business
The collection of these activities is called as value activities. analysis is an early requirements phase whose goal is to analyze
and represent the economic values exchange through a model
called Dynamic Value Description (DVD). From the DVD model,
both business analysts and requirements engineers specify
information system requirements by using a cognitive
requirements approach [15].
The cognitive requirements approach improves the domain
understanding because it provides an environment wherein all the
stakeholders could share their views and abstractions in a semi-
structured mind map model3. Finally, from these requirements
specifications, the software architect generates a high-level
software architecture by using model-driven techniques. In the
context of this paper, we address only the business analysis phase.
So, we analyze the DVD model and its creation process. Figure 3
depicts the DVD metamodel and Figure 4 presents a DVD model
example (instance from metamodel).
As we can see in the metamodel, DVD is composed of eight main
concepts: main actor, environment actor, value exchange, who
starts the value exchange, value port, value element, value level
agreement, and priority. Similarly to the e3value model, actors are
Figure 1. e3value metamodel extracted from [12]. environment entities economically independent in the DVD
. model. However, each time, the business analyst focuses the
analysis on the main actor and represents its relationship with
In order to represent value exchange scenarios, the e3value model others environment actors, producing an inter-organizational
inherited the start stimulus, the stop stimulus, the AND element, network. As the focus changes, the actor playing the role of “main
the OR element, and the connect element from Use Case Maps2 actor” also changes. With this change in focus, new actors and
(UCM) [13]. Although these elements are contained in e3value value exchanges may appear.
model (see Figure 2), they are absent in the metamodel (see
Figure 1), showing that the e3value metamodel is incomplete.
2
Use Case Maps is a requirements language which have the
3
notion of path to show how a particular scenario works. Mind map is considered a simple and accessible model [16].
2.3 Comparing e3value and VDD
This section compares the e3value and the value-driven
development methods. Table 1 presents a mapping between
e3value and VDD concepts.
Table 1. Concepts mapping.
# e3value concepts DVD Concepts
1 Elementary actor Main actor or actor
2 Composite actor Main actor or actor
3 Market segment Main actor or actor
4 Value interface Aggregate in value exchange
5 Value transfer Aggregate in value exchange
6 Value port Value port
7 Value object Value element
8 Value exchange Value exchange
9 Value transaction Aggregate in value exchange
10 Value activity -
11 Start stimulus Who starts
12 Stop stimulus Who starts
Figure 3. Dynamic Value Description metamodel. 13 AND element Logical operators in exchange element
14 OR element Logical operators in exchange element
15 Connect element -
From the actors relationship, a value exchange is performed. It 16 - Value level agreement
shows economic reciprocity through two value ports (arrows 17 - Priority
connected to value exchange), one for entry and one for exit,
which point to value elements (money, goods, services or We observed that the DVD model (from the VDD method)
information). If there are many value objects in the same value describes two new concepts in relation to the e3value: value level
port, the business analyst must use logical operators (“AND”, agreement (VLA) and priority. In contrast, the e3value model has
“OR”, and “XOR”) to detail the relationships among them. the value activity concepts, not offered by the DVD model. The
VLA defines the business constraints based on the business
In addition, the business analyst also defines who starts the value strategies. For example, a company of the feeding segment
exchanges through a configuration of arrows between the main provides food fresher than its competitors, as a business strategy.
actor and the environment actor. It is important to notice that Thus, to provide fresher food, it is essential that its suppliers also
during the DVD modeling, the business analyst is able to focus on deliver fresh ingredients. Therefore, the business analyst can
each actor individually in order to give more attention and details specify a VLA by defining the acceptable time of receipt of these
to the actor which must be analyzed. Thus, the analyst sets who is ingredients. Regarding the information system development, the
the main actor and a given support tool will display it as the complexity of a software system is determined by its functionality
central node of the model, dynamically. Each value exchange (i.e., what the system does) and by global requirements on its
needs a level of agreement between the ones involved. This level development, such as operational costs, performance, reliability,
of agreement is a particular business aspect that must be maintainability, portability, robustness [17]. These global
minimally agreed among the actors in order to enable the value requirements are known as Non-functional Requirements (NFR)
exchanges. and they typically refer to the operational quality of a system, as
Finally, the DVD model describes a prioritization of value well as the constraints imposed on a solution [18]. Thus, we can
exchanges using colors. The red color means high priority, yellow define a VLA as an NFR at the business abstraction level.
color means medium priority, and blue color means low priority. In addition, as information systems are usually developed using
These priorities are set by business analyst according to the return iterative and incremental processes, the value exchanges
of investment of the value exchanges in the business. prioritization may facilitate the scope definition of each iteration,
aligning the system development with the business needs and the
time to market. Despite having fewer concepts, the DVD model
represents several e3value' concepts but some of these concepts
are represented in a partial way or with a different meaning (e.g.,
UCMs elements). However, for the various case studies
developed, the concepts offered by DVD have been proved
sufficient.
3. EXPERIMENT DESIGN
This section presents the design of a controlled experiment aimed
at comparing the value-driven development method against the
e3value method. We followed the guidelines proposed by Wohlin
et al. [19].
Figure 4. Dynamic Value Description model example.
3.1 Experimental Planning attend the “Empirical Software Engineering” course from
September to November 2016.
3.1.1 Experiment Goal • Undergraduate students, all Computer Science students from
According to the Goal-Question Metric (GQM) approach [20], the the Software Engineering intensification at the Universitat
goal of this experiment is to analyze VDD and e3value for the Politècnica de València. These students will attended the
purpose of comparing them with respect to their effectiveness, “Software Quality” course from February to June 2017. One
efficiency, perceived ease of use, perceived usefulness and of the main topics of this course is to evaluate the quality of
intention to use in order to obtain high-quality value models from models obtained through the software development process.
the point of view of novice business analysts and software A teaching unit on the evaluation of value-driven
engineers, in the context of undergraduate and postgraduate development methods will be added to the course program.
students in Business Management and Computer Science. • Master’s students, enrolled on the Master’s Degree in
Software Engineering at the Universidade Nova de Lisboa.
The broad research questions addressed by the experiment are: These students will attend the “Software Engineering” course
• RQ1: Is the actual efficacy of VDD higher than the actual from September 2016 to January 2017.
efficacy of e3value? • Undergraduate students, all business management students at
• RQ2: Is the perceived efficacy and intention to use of the Universidade Federal de Pernambuco. These students
participants applying VDD higher than that of e3value? will attend the “Information System” course from January to
July 2017.
The context of the experiment is the creation of a business value
model for specific software systems. This context is determined These courses were selected because the preparation and training
by the product to be developed and the subjects’ selection. and the experimental task itself fit their scope. We take a
convenience sample (i.e., all the students available in the class).
3.1.2 Experimental Objects The original experiment will be conducted in the Requirements
The software systems to be developed were selected from the Engineering course and the other groups will be exact and/or
literature [21], [22]. Two experimental objects were selected from differentiated replications. This will allow us to build a body of
the requirements specifications of the following two systems: knowledge about these value-driven development methods. As
Basili et al. [24] suggested, relevant and credible results can only
• Waste management (O1): It describes the business where be obtained by replicating the experiments since single studies
waste is traded between an exporter and an importer. In the rarely provide definitive answers.
majority of cases the exporter has to pay the importer for the
waste handling. However, there are some cases where the 3.1.4 Selection of Variables
waste can be traded like a regular good, for example, when The independent variable of interest is the use of each value-
the waste is recycled. driven method with nominal values: VDD and e3value. Hence, the
• Wireless access provisioning (O2): It describes the business experiment use two treatments: the creation of a value model for
where a hotel would like to offer wireless connectivity to two software systems using VDD and the creation of a value
businessmen as an additional service. model for the same systems using e3value. The experimental data
collected allows comparing the effects of both treatments.
To assess the complexity of the models used and to identify
possible mistakes, we plan to carry out a pilot experiment with a There are two types of dependent variables in which the
small group of PhD students at UPV. treatments are compared: performance-based and perception-
based variables. Performance-based variables assess how well the
3.1.3 Participant’s Selection participants perform the experimental task. They are used to
The context of this experiment is the evaluation of value-driven evaluate the actual efficacy of the value-driven development
development methods from the perspective of novice modelers. methods. Perception-based variables assess the participants’
Although experienced modelers and practitioners are desired, we perceptions of their performance and their subsequent intention to
focus on the profile of novice modelers since one of our goals is use VDD or e3value. These variables are used to evaluate the
to provide a value-driven development method that will help less perceived efficacy of these methods, as well as their likely
experienced modelers to specify value models. In addition, adoption in practice.
according to the Technology Transfer Model proposed by There are two performance-based variables:
Gorschek et al. [23], it is recommended to first perform initial
evaluations in lab environments before the realization of realistic Effectiveness: It is calculated with the Jaccard index (see formula
evaluations in industrial environments. (1)) that measures similarity between sample sets and is defined as
the size of the intersection divided by the size of the union of the
Value models can be produced by business analysts or software
sample sets. Given two models, A and B, the Jaccard index
engineers. The following groups of participants are therefore
measures the overlap that A and B share with their elements. In
identified in order to facilitate the generalization of results:
our case, we will calculate the Jaccard index between an agreed
• Undergraduate students, all Computer Science students from solution among experts (A) (for the value models obtained with
the Software Engineering intensification at the Universitat VDD and e3value) and the solution given by each participant (B).
Politècnica de València. These students will attend the A Jaccard index of 0 represents no overlap between the solutions,
“Requirements Engineering” course from September 2016 to while 1 indicates that they contain the same results.
January 2017, during this time they will have 8h of lectures
on business modeling and value-driven development. (1)
• Master’s students, enrolled on the Master’s Degree in
Engineering and Technology of Software Systems at the
Universitat Politècnica de València. These students will Efficiency: It is the time required to apply the method.
There are also three perception-based variables, which are based System Domain. The complexity of the software requirements
on the Technology Acceptance Model (TAM) [25], since TAM is considered in the tasks may have a confounding effect on the
one of the most widely applied theoretical models when analyzing results. The application domain of the tasks could also be a
user acceptance and usage behavior of emerging information confounding factor that could affect the subjects’ comprehension.
technologies, and has empirical support through validations and Order of Methods. The order in which the subjects apply the
replications [26]. This model has been also applied previously to methods may produce learning effects, which may bias the results.
evaluate requirements modeling methods [27]. The perceived
efficacy [25] of the method can be broken down into the 3.1.7 Design of the experiment
following subjective dependent variables: The experiment is planned as a balanced within-participant design
Perceived Ease of Use (PEOU): It refers to the degree to which a with a confounding effect, signifying that the same participants
person believes that learning and using a particular value-driven apply both methods with both experimental objects in a different
method would be free of effort. order. We plan to establish four groups (each of which will apply
one method to one experimental object) and the participants will
Perceived Usefulness (PU): It refers to the degree to which a be randomly assigned to each group. Table 3 summarizes the
person believes that using a specific method will increase his or design of the experiment.
her job performance within an organizational context.
The within-participant experimental design is intended to
Intention to Use (ITU): It refers to the extent to which a person minimize the impact of learning effects on the results, since none
intend to use a particular method. This last variable represents a of the participants repeat any treatment or experimental object
perceptual judgment of the method’s efficacy – that is, whether it during the execution. Other factors which may also be present
is cost-effective and is commonly used to predict the likelihood of need to be controlled, since they might influence the results, i.e.,
acceptance of a method in practice.
the complexity of experimental objects. The comprehension of the
These three subjective variables will be measured by using a software systems requirements to be modeled may also affect the
Likert scale questionnaire with a set of 12 closed-questions: 3 application of both methods. We attempted to alleviate the
questions for perceived ease of use (PEOU), 6 questions for influence of this factor by selecting two representative software
perceived usefulness (PU) and 3 for intention to use (ITU). The systems with software system requirements of a reasonable
questionnaire can be found here: http://bit.ly/2ak1wLS. The complexity. The complexity of the software systems selected
closed-questions were formulated by using a 5-point Likert scale, made them suitable for application in the time slot available for
using the opposing statement question format. In other words, the execution of the experiments (2 hour sessions).
each question contains two opposite statements representing the Table 3. Experiment Design
maximum and minimum possible values (5 and 1), where the
value 3 is considered to be a neutral perception. The aggregated 1st Session
value of each subjective variable will be calculated as the Detailed training with VDD and e3value
Sessions
2nd Session 3rd Session
arithmetical mean of the answers to the questions associated with
VDD and e3value quick training
each perception-based variable. G1: G2: G2:
Group of G1: e3value
VDD in VDD in e3value in
3.1.5 Hypotheses Formulation participants
O1 O2
in O2
O1
We formulated several null hypotheses, which were formulated in (sample size
= 4n G3: G4: G3: G4:
a one-tailed manner, since we want to analyze the effect of the use 3 3
e value in e value in VDD in VDD in
participants)
of VDD on the variables. Each null hypothesis and its alternative O2 O1 O1 O2
are presented as follows: Post-experimental Post-experimental
questionnaire questionnaire
• H10: There is no significant difference between the
effectiveness of VDD and e3value / H1a: VDD is 3.1.8 Instrumentation
significantly more effective than e3value. The experimental task was structured to allow the comparison of
• H20: There is no significant difference between the both methods. Depending on the method, the task was composed
efficiency of VDD and e3value / H2a: VDD is significantly of the method activities that help to achieve its purpose. After
more efficient than e3value. applying the method, the participants have to fill in a post-
• H30: There is no significant difference between the perceived experimental questionnaire with subjective questions regarding
ease of use of evaluators applying VDD and e3value / H3a: the method.
VDD is perceived as easier to use than e3value.
• H40: There is no significant difference between the perceived We have defined only one experimental task (create the value
usefulness of VDD and e3value / H4a: VDD is perceived as model) of which its steps vary according to the value model that
more useful than e3value. the subjects will create. We will offer a training session to explain
the concepts of value models and how they are created. During the
• H50: There is no significant difference between the intention
experimentation, we will offer the requirements specifications to
to use of VDD and e3value / H5a: VDD is perceived as more
the subjects (see Section 3.1.2). The requirements specifications
likely to be used than e3value.
describe how the business works. With these specifications in
3.1.6 Factors to be Controlled their hands, the participants will create a particular value model
Although Method is the only factor of interest in this empirical (DVD or e3value) and will register the start time and the end time
investigation, other factors may affect the participants’ for each step performed.
performance in an undesirable way, thus confounding the Method Figure 5 presents the steps to create the e3value model. In this
effect. These factors have to be controlled so that only the effect case, the steps are:
of the Method factor, if there is any, is observable:
Step 1 - Identify scenarios: Scenarios are short textual sentences, The reason for this is that there is a lack of economics results at
meaning the product, service, or experience expected by a this moment of the analysis.
customer. Therefore, the goal of this step is that the participants
write a scenarios list. Once the value model is created, the participants will answer the
post-experimental questionnaire. Hence, we will be able to
Step 2 - Identify actors: The participants will identify who offers evaluate the performance-based variables (effectiveness and
and who receives the product, service or experience expected efficiency) by comparing the value model they created against the
from the scenarios list and they will create a list of actors. value model created by experts4 and by analyzing the time
registered to perform each experimental step. In addition, we will
Step 3 - Create value model: With the scenarios list and the actors evaluate the perception-based variables (perceived ease of use,
list in their hands, the subjects will create the initial e3value model perceived usefulness, and intention to use) from the responses
by using the products and services mentioned in the scenarios list received in the post-experimental questionnaire.
and the actors described in actors list.
The experimental material is composed of a set of documents
Step 4 - Identify UCMs: The participants will insert the UCM's required to support the experimental tasks and the training
elements, representing the paths of all scenarios in the e3value
sessions, along with the post-experimental questionnaire. The
model. In other words, they will insert the start stimulus, stop training materials include: i) a set of slides containing an
stimulus, AND element, OR element, and connect element in the introduction to business modeling and value-driven development;
e3value model. ii) a set of slides describing the VDD method, along with an
example of its application; and iii) a set of slides describing the
e3value method, with an example of its application.
The documents supporting the experimental tasks include:
• Two kinds of booklets covering the two possible
combinations of both the value-driven development method
and the experimental objects (VDD-O1, VDD-O2, e3value-
Figure 5. e3value process. O1, e3value-O2). The purpose of these booklets is i) to
describe the experimental tasks to be performed; ii) to
In the case of creating the DVD model, the participants will
describe the software system requirements; and iii) to gather
follow the steps from the VDD process (see Figure 6). These steps
the data from the experimental task.
are:
• An appendix containing a guideline to help the participants
Step 1 - Specify actors: Participants start the DVD model by
to apply the value-driven development method.
describing the main actor (the focus of their analysis) and their
related environment actors. Thus, the participants will create a The post-experimental questionnaire contains a set of closed-
DVD model like a mind map, where the main actor is the central questions that allows participants to express their opinion on the
node and the environment actors are the leaf nodes. Due to this ease of use, usefulness, and their intention to use of the method in
“main actor” focus, the DVD model shows only the environment the future. We also include two open questions to obtain the
actors who directly interact with it. Thus, the participants will be participants’ feedback regarding the changes they would make to
required to create as many DVD models as necessary to represent improve the method and their reasons for using a given method in
the whole business. the future (if any). This questionnaire will be online, using Google
Forms and the data collected will be kept anonymously. All the
Step 2 - Set value exchanges: Participants will update the model
experimental material will be created in Spanish and Portuguese,
by adding the value exchanges. During this activity, participants
since these are the participants’ native languages.
define the value element related to each value port.
Step 3 - Set who starts each value exchange: Participants will 3.1.9 Threats to Validity
define which actor starts the value exchange. Here, it is important We must consider certain issues which may threaten the validity
to check if the value elements are specified in the correct value of this experiment. With regard to internal validity, the main
port. threats are: learning effect, participant experience, information
exchange among participants, and understandability of the
Step 4 - Set value level agreement: Participants define the criteria documents.
required for value exchanges to be perform. This step is very
important that participants understand the business constraints The learning effect is alleviated by ensuring that groups of
participants will apply the two methods to different experimental
related to each value exchange.
objects in a different order. We also plan to assess the effect of
Step 5 - Prioritize the value exchanges: Participants prioritize order of system domain and order of methods by using statistical
each value exchange according to the expected return of tests. Participants’ experience is alleviated as none of the
investment (ROI). This is a subjective prioritization as participants have any experience in value-driven development.
participants will set the value exchanges priority in relation to We plan to confirm this fact by asking the participants about their
other value exchanges without the use of any mathematical model. experience with value-driven development methods.
To minimize the information exchange among participants, they
will be monitored by the experimenters to avoid communication
4
In the case of e3value, the value models which will be used in
Figure 6. Dynamic Value Description process. comparison were found in literature.
biases while performing the tasks. However, this might affect the With regard to the operation of the experiment, the experiment is
results since the experiment will take place over more than one planned to be conducted in three sessions (Table 3 shows the
day, and it is difficult to be certain whether or not the participants details for each session). On the first session, the participants will
will exchange any information. To alleviate this situation, at least be given a detailed training on the methods to be applied and also
to some extent, participants will be asked to return all the material on the tasks to be performed in the execution of the experiment. In
at the end of each task. Finally, understandability of the material this session, they will perform a practical session in which they
will be alleviated by performing a pilot study. In addition, we will will specify a value model using both methods.
clear up all the misunderstandings that may appear in each
experimental session. On the second and third sessions, the participants will be given an
overview of the training before applying each value-driven
With regard to external validity, the main threats are: development method to the experimental objects (O1 or O2). We
representativeness of the results and the size and complexity of will establish a slot of 90 min with no time limit for any of the
the tasks. The representativeness of the results may be affected by methods to be applied. In addition, we will allow the participants
the software systems used and the participant’s context selected. to continue the experiment even though these 90 min is not
With regard to the selection of software systems, we attempted to enough in order to avoid a possible ceiling effect.
alleviate this by considering a set of artifacts with similar size and With regard to the experiment execution, the experiment will take
complexity, and which contains representative artifacts of an place in a single room, and no interaction among participants will
existing value-driven development method (i.e., e3value). be allowed. The experimenter will clarify possible questions that
Despite the fact that the planned experiments will be performed in may arise during the sessions.
an academic context (undergraduate and Master’s students), the
participants’ performance can be considered to be representative With regard to the data validation, we plan to verify that the
of novice modelers since the kinds of students involved will be participants complete the two experimental sessions. Data points
containing only one session will be discarded. If this occurs, other
soon integrated into the industry’s market. As further work, we
plan to conduct more experiments involving practitioners in order data points may also be discarded in order to maintain the
to assess how the experience level would impact on the obtained balanced design shown in Table 3 (i.e., having exactly the same
results. Also, since only internal replications will be conducted, number of participants in each group).
more external replications need to be conducted by other 3.3 Data Analysis & Interpretation
experimenters in other settings to confirm these results. In order to
This section introduces the statistical tests that will be used to
address the aforementioned limitations, these external replications
analyze the data collected: the influence of the method on the
will involve participants from different contexts and also with
dependent variables and the effect of system domain and order of
different levels of experience in value-driven development.
method. These tests have been chosen because they are very
The size and complexity of the tasks may also affect the external robust and sensitive, and have been used in previous experiments
validity. We use relatively small tasks that would be applied in a similar to ours, e.g., [29], [30]. As usual, in all the tests we have
few representative software artifacts since a controlled experiment decided to accept a probability of 5% of committing a Type-I-
requires participants to complete the assigned tasks in a limited Error [19], i.e., of rejecting the null hypothesis when it is actually
amount of time. true.
With regard to construct validity, the main threats are: the
3.3.1 Influence of Method
measures that will be applied in the data analysis and the
We plan to use boxplots and statistical tests to analyze the data
reliability of the questionnaire. We attempt to alleviate this threat
collected. In particular, we will test the normality of the data
by using measures that are commonly applied in other software
distribution by applying the Shapiro–Wilk test. The results of the
engineering experiments. In particular, the Effectiveness was
normality test will allow us to select the proper significance test in
measured using the Jaccard index, also known as the Jaccard
order to test our hypotheses. When data is assumed to be normally
similarity coefficient, which has commonly been used to measure
distributed (p-value≥0.05), we will apply the parametric one-
the similarity and diversity of sample sets. The subjective
tailed t-test for independent samples [31]. However, when data
variables are based on the Technology Acceptance Method
could not be assumed to be normally distributed (p-value <0.05),
(TAM), a well-known and empirically validated model for the
we will apply the non-parametric Mann–Whitney test [32].
evaluation of information technologies [25], [26]. The reliability
of the questionnaire will be tested by applying the Cronbach test. 3.3.2 Influence of Order of System Domain and
With regard to conclusion validity, the main threats are: the data Order of Method
collection and the validity of the statistical tests applied. With To test the influence of order of system domain and order of
regard to the data collection, we plan to apply the same procedure method (both independent variables), we plan to use a method
in each individual experiment in order to extract the data, and similar to that proposed by Briand et al. [30]. We will use the Diff
ensure that each dependent variable is calculated by applying the function:
same formula. With regard to the validity of the statistical tests
Diffx = observationx(A) - observationx(B) (2)
proposed, we chose the most common tests that are employed in
the empirical software engineering field due to their robustness where x denotes a particular participant, and A, B are the two
and sensitivity [28]. possible nominal values of an independent variable. We plan to
3.2 Operation and Execution create Diff variables from each dependent variable e.g.,
This section describes the experimental operation, including the Effectiveness_Diff(VDD) will represent the difference in
preparation, execution, data recording and data validation. effectiveness of the subjects who used VDD first and e3value
second. On the other hand, Effectiveness_Diff(e3value) will
represent the difference in effectiveness of the participants who
used e3value first and VDD second. The aim is to verify that there [12] J. Gordijn, “Value-based Requirements Engineering”, Phd
are no significant differences between Diff functions since that Thesis, 2002.
would signify that there is no influence in the order of the [13] R. J. A. Buhr and R. S. Casselman, Use case maps for
independent variables. We also plan to apply the Shapiro-Wilk object-oriented systems. Prentice-Hall, Inc., 1995.
test to prove the normality of the Diff functions. The hypotheses [14] V. Pijpers, P. de Leenheer, J. Gordijn, and H. Akkermans,
related to the Diff functions are two-sided since we do not make “Using conceptual models to explore business-ICT
any assumption about whether one specific order would be more alignment in networked value constellations”, RE, vol. 17,
influential than another. We plan to verify these hypotheses by no. 3, pp. 203–226, Oct. 2011.
applying the parametric two-tailed t-test for independent samples [15] F. Wanderley, A. Silva, and J. Araújo, “Evaluation of
or the non-parametric Mann-Whitney test depending on the results BehaviorMap: A user-centered behavior language”, in 9th
of the normality test. RCIS, 2015, pp. 309–320.
[16] S. Ambler, Agile modeling: effective practices for extreme
4. CONCLUSIONS AND FURTHER WORK programming and the unified process. John Wiley & Sons,
We have presented two early requirements modeling methods to Inc., New York, 2002.
represent a business from an economic point of view: value- [17] L. Chung, B. A. Nixon, E. S.-K. Yu, and J. Mylopoulos,
driven development method and e3value. Moreover, we also have Non-Functional Requirements in Software Engineering.
presented an experiment design aimed at comparing these two Boston, MA: Kluwer Academic Publishers, 1999.
methods. Our experiment design allows predicting the acceptance [18] I. Jacobson, J. Rumbaugh, and G. Booch, The Unified
of a particular method in practice, based on the effort of applying Modeling Language Reference Manual. Reading, MA:
the method, the quality of the artifacts produced, and the user Addison-Wesley, 1999.
perceptions with regard to the quality of the method. In future [19] C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B.
work, we plan to perform a family of experiments by using the Regnell, and A. Wesslén, Experimentation in Software
proposed experiment design. Engineering. Springer Berlin Heidelberg, 2012.
[20] V. R. Basili and H. D. Rombach, “The Tame Project -
5. ACKNOWLEDGMENTS Towards Improvement-Oriented Software Environments”,
This research is supported by the Value@Cloud project IEEE Trans. Software Eng., vol. 14, no. 6, 1988.
(MINECO TIN2013-46300-R), NOVA LINCS Research [21] C. Huemer, A. Schmidt, and H. Werthner, “A UML profile
Laboratory (Ref. UID/CEC/04516/2013), and programa Ciência for the e3-value e-business model ontology”, in BUSITAL
sem Fronteiras - CAPES. held in conjunction with CAiSE”08 Conference, 2008.
[22] Z. Derzsi and J. Gordijn, “A Framework for Business/IT
6. REFERENCES Alignment in Networked Value Constellations”, in
[1] R. Veryard, Six Viewpoints of Business Architecture. BUSITAL, 2006.
Leanpub, 2015. [23] T. Gorschek, C. Wohlin, P. Garre, and S. Larsson, “A model
[2] R. Passonneau, K. Kukich, and J. Robin, “Generating for technology transfer in practice”, IEEE Software, vol. 23,
summaries of workflow diagrams”, in International no. 6, pp. 88–95, 2006.
Conference on Natural Language Processing and Industrial [24] V. R. Basili, F. Shull, and F. Lanubile, “Building knowledge
Applications, Moncton, Canada, 1996. through families of experiments”, IEEE Trans. Software
[3] M. Dumas and A. H. M. ter Hofstede, “UML Activity Eng., vol. 25, no. 4, pp. 456–473, 1999.
Diagrams as a Workflow Specification Language”, in UML: [25] F. D. Davis, “Perceived usefulness, perceived ease of use,
Modeling Languages, Concepts, and Tools, vol. 2185, no. 7, and user acceptance of information technology”, MIS
Toronto, Canada, 2001, pp. 76–90. quarterly, vol. 13, no. 3 , 1989.
[4] S. A. White, “Introduction to BPMN”, IBM, 2004. [26] W. R. King and J. He, “A meta-analysis of the technology
[5] A. Rasiwasia, “Meta Model for Business Model Design: acceptance model”, Information & Management, vol. 43,
Designing a Meta model for E3 value model based on no. 6, pp. 740–755, Sep. 2006.
MOF”, Stockholm, Sweden, 2013. [27] S. Abrahao, E. Insfran, J. A. Carsí, and M. Genero,
[6] J. Gordijn and H. Akkermans, “Designing and evaluating e- “Evaluating requirements modeling methods based on user
business models”, IEEE Intell. Syst., vol. 16, no. 4, 2001. perceptions: A family of experiments”, Information
[7] The Institute of Value Management, “What is Value?.” Sciences, vol. 181, no. 16, pp. 3356–3378, Aug. 2011.
[Online]. Available: https://ivm.org.uk/what-is-value- [28] K. Maxwell, Applied statistics for software managers.
management. [Accessed: 21-Jul-2016]. Software Quality Institute Series, Prentice Hall, 2002.
[8] J. Gordijn and J. M. Akkermans, “Value-based requirements [29] S. Abrahao, C. Gravino, E. Insfran, G. Scanniello, and G.
engineering: exploring innovative e-commerce ideas”, RE, Tortora, “Assessing the Effectiveness of Sequence
vol. 8, no. 2, pp. 114–134, Jul. 2003. Diagrams in the Comprehension of Functional
[9] V. Kartseva, J. Gordijn, and Y.-H. Tan, “Inter- Requirements: Results from a Family of Five Experiments”,
organisational Controls as Value Objects in Network IEEE Trans. Software Eng., vol. 39, no. 3, 2013.
Organisations”, in Experiences with Model Reuse: Non- [30] L. C. Briand, Y. Labiche, M. Di Penta, and H. Yan-Bondoc,
Functional Requirements Catalogues for Ubiquitous “An experimental investigation of formality in UML-based
Systems, vol. 4001, no. 23, 2006, pp. 336–350. development”, IEEE Trans. Software Eng., vol. 31, no. 10,
[10] D. Kundisch and T. John, “Business Model Representation pp. 833–849, Nov. 2005.
Incorporating Real Options: An Extension of e3-Value”, in [31] N. Juristo and A. M. Moreno, Basics of Software
45th HICSS, 2012, pp. 4456–4465. Engineering Experimentation, 1st edition. Springer
[11] J. Gordijn, “E3-value in a Nutshell “, International Publishing Company, Incorporated, 2010.
Workshop on e-Business Modeling, 2002. [32] W. J. Conover, Practical Nonparametric Statistics, 3rd ed.
Wiley India Pvt. Limited, 2006.