=Paper=
{{Paper
|id=Vol-2248/paper7
|storemode=property
|title=On the Importance of Simulation in Enabling Continuous Delivery and Evaluating Deployment Pipeline Performance
|pdfUrl=https://ceur-ws.org/Vol-2248/paper7.pdf
|volume=Vol-2248
|authors=Andrea D'Ambrogio,Alberto Falcone,Alfredo Garro,Andrea Giglio
|dblpUrl=https://dblp.org/rec/conf/ciise/DAmbrogioFGG18
}}
==On the Importance of Simulation in Enabling Continuous Delivery and Evaluating Deployment Pipeline Performance==
https://ceur-ws.org/Vol-2248/paper7.pdf
On the Importance of Simulation in Enabling
Continuous Delivery and Evaluating Deployment
Pipeline Performance
Andrea D’Ambrogio Alberto Falcone, Alfredo Garro Andrea Giglio
Dept. of Enterprise Engineering Dept. of Informatics, Modeling, Dept. of Innovation and
University of Rome Electronics and Systems Engineering Information Engineering
Tor Vergata University of Calabria Guglielmo Marconi University
Via del Politecnico 1 Via P. Bucci 41C Via Plinio 44
00133, Rome, Italy 87036, Rende, Italy 00193, Rome, Italy
dambro@uniroma2.it {alberto.falcone, alfredo.garro}@dimes.unical.it a.giglio@unimarconi.it
Abstract—In modern Software Engineering, Continuous De- Furthermore, the dramatic impact of the Agile Manifesto on
livery (CD) is a development approach in which a software is the way software production is managed, has led in recent
iteratively developed in short cycles ensuring, for each cycle, years to the widespread adoption of new emerging approaches
that the new features are available to end users as soon as they
are implemented and tested. CD aims at defining, building and in software engineering. The first of the twelve fundamental
releasing software with greater speed and frequency through the principles defined in the aforementioned manifesto states that
deployment pipeline resulting in three major benefits, visibility, “Our highest priority is to satisfy the customer through early
feedback and continuous deployment respectively, enabling the and continuous delivery of valuable software” [20].
software functional items to efficiently flow from development
to production. In this domain, the need for evaluating the Despite the fact that continuous delivery was explicitly
performance of the deployment pipeline emerges, since the con- mentioned in this principle, its widespread adoption is the
ventional metrics available in the software engineering discipline result of an evolution of different approaches through the
are not suited to handle all the involved aspects. In this paper, years. Starting from continuous integration (CI) shifting to-
the metrics suited for supporting CD are introduced and an
integration with Modeling and Simulation (M&S) techniques is
wards modern continuous deployment pipelines, releasing new
discussed, based on the Business Process Model and Notation software versions early and often has become mainstream, and
(BPMN) standard, which could represent a valid support offering now represents a concrete option also for an ever growing
a graphical notation to easily specify the deployment pipeline steps number of companies and practitioners. Let us summarize
as a standard and repeatable process. The main objective is to some baseline definitions:
identify feasible perspectives in which simulation methods and
principles can be exploited, thus evaluating the effectiveness of • Continuous Integration. CI is the process of ensuring that
M&S to support performance analysis of a deployment pipeline,
seen as a predictable process. Specifically, M&S can be seen as an a software build is in a correct working state, guaranteeing
enabling tool for the evaluation and comparison of different CD developers that the release is suitable for production. In
choices against requirements through an effective implementation this approach, members of a team are pushed to integrate
of simulation techniques and virtual testing. their work frequently, typically each person integrates
Index Terms—Continuous Delivery, Business Process Model at least daily leading to several integrations per day.
and Notation (BPMN), Modeling and Simulation
It is worth noting that each integration is verified by
I. INTRODUCTION an automated build and test procedure with the aim of
detecting integration errors as soon as possible.
Nowadays, in the software engineering and software devel-
• Continuous Delivery. Deriving from CI, CD ensures that
opment domains, many influencing factors play a fundamental
a build is always in a releasable state. Going hand to hand
role in making software production activities increasingly
with a DevOps approach, where the team is responsible
challenging. Fast-changing customer requirements, as well as
for all aspects of software delivery, after integration and
unpredictability of market and the need of an ever shorter time
testing activities the software is actually provisioned to a
to-market are non-negligible aspects that have to be taken into
full, production-like stack, and delivered to some set of
account, since software development has become a demanding
end users.
area of business [8]. First approaches, strongly based on attrac-
• Continuous Deployment. When the entire process, from
tive lean principles proposed the adoption of strategies such
check-in to production, is fully automated, with no human
as “Decide as late as possible” [30], configured themselves
intervention, continuous deployment is implemented.
as early steps towards a Continuous Delivery (CD) approach.
• Deployment Pipeline. The foundation of continuous deliv-
Copyright c held by the author ery is the deployment pipeline, which can be seen the path
that a code change takes from check-in (i.e. the commit simplifying and automating the whole delivery process for an
operation) to production. application after the source code has been committed into a
CD is a software engineering approach that represents a repository. CD is centered around the collaboration aspects of
huge step forward in productivity and quality for companies teams involved in the software production such as, developers,
and organizations that adopt it. CD is gaining attention since it project managers and users, as well as automating the software
provides a well-established process that allows team members delivery steps so as to ensure that a software is always ready
to work together to define and create large and complex to be deploy and then delivered to end-users.
software with a higher level of control. CI claims to enable Basically, the deployment pipeline is an automated process
companies to release new features, configuration properties, for getting software from the repository into the hands of end-
bug fixes and tests, to customers safely and quickly in a users. CD is preceded by the Continuous Integration (CI) in
sustainable way. This means that it mainly focuses on asserting which the team members integrates the source code [31], [34].
the correct compilation of the source code and that it passes This step leads to a faster feedback cycle and to benefits such
a chain of test units. as improved productivity and increased communication [31].
However, measuring the performance of a CD deployment Figure 1 shows the deployment pipeline.
pipeline is a challenging task that requires a considerable effort
in terms of both time and cost. Many companies including Source Code
Automatic
Google, Facebook and IBM are developing their software Manual
products by using CD [8] and are trying to find out viable Commit
Commit
solutions to measure the CD performance. To this purpose,
significant benefits can derive from the possibility to perform
Build
Modeling and Simulation (M&S) activities on the CD deploy- Compile and
Build
ment pipeline in order to understand, measure and optimize
the behavior of systems on which to perform experiments and Test
Environment
theoretical analyses. tests
CD Pipeline
In this context, the paper introduces metrics that are suited to
evaluate the performance of a CD deployment pipeline through Functional
tests
M&S techniques based on the Business Process Model and
Notation (BPMN) standard, in the purpose of defining the Non-Functional
pipeline as a standard process and also of analyzing it as a tests
repeatable and predictable process.
Performance
The ultimate aim is to provide an exploratory analysis of the tests
available approaches to minimize the cycle-time in software
production (i.e. the period it takes from an idea to provide Stability
actual business value), and also to investigate the different tests
domains in which simulation techniques can be fruitfully
Manual
exploited to enforce continuous delivery activities through a tests
deployment pipeline optimization.
The rest of this paper is organized as follows. Section II Deploy
Production
tests
provides an introduction to the essential concepts and back-
ground knowledge on the research domain. The Continuous
Delivery (CD) methodology and the typical architecture of Fig. 1. Overview of the CD deployment pipeline.
a CD deployment pipeline are presented along with some
related works available in literature. In Section III, the CD The CD deployment pipeline starts when developers commit
deployment pipeline is defined through the concepts provided the source code changes into the version control system. At
by the Business Process Model and Notation (BPMN) standard this point, the CI Management System (CI-MS) triggers a
with particular focus on how to make available Modeling and new instance of the deployment pipeline and compiles the
Simulation (M&S) practices during the pipeline flow in order source code. If the compile procedure successfully completed
to evaluate, predict and optimize its behavior. Finally, Section a transition to the test phase is performed; otherwise the
IV discusses the main objectives of the work and presents procedure terminates with an error. In the test phase, the
some future works. CI-MS runs a set of unit tests, performs code analysis, and
builds the installer package. In this step, some manual tests
II. BACKGROUND AND R ELATED WORK needs to be performed in order to evaluate domain-dependent
Continuous Delivery (CD) is a concept which has been functional and non-functional requirements. If all the tests
taken as a pillar of modern Agile software engineering [11] pass, the executable code is used to generate the binaries that
because its enables higher software development velocity will be stored in an artifact repository [21].
and productivity. The concept of continuous delivery means The related works available in the literature are vast and
extend through empirical and modeling and simulation ap- III. T HE P IPELINE AS A S TANDARD AND P REDICTABLE
proaches to studying processes in software development. P ROCESS
In [33] Van Der Hoek et al. identified the problems of A. The Pipeline
releasing component-based software and presented a flexible
software release management tool that acts as a bridge to Under the assumption that both a business process collabo-
reduces the gap between the development process and deploy- ration and a CD deployment pipeline implement an exchange
ment process. of information between logical processes, in this paper the
Lahtela et al. in [24] presented nine significant challenges adoption of BPMN as a notation to define a deployment
regarding the delivery process of software. The authors, also pipeline is proposed. This allows to perform simulation test
presented solutions to overcome these challenges through well- units in order to evaluate the performance of a pipeline through
defined guidelines on how to build an ITIL-based release a set of well-known metrics such as, those presented in [25].
management process and how it should be performed. The deployment pipeline is the workflow, with related activ-
Kajko-Mattson and Fan in [22] outlined a model of the ities, that a source code follows from commit to production.
release management process integrating both the vendor and Each commit, made by the development team, generates a
acceptor sides. release candidate of the software which flows through the
pipeline. If everything goes well, which means that all steps
Krishnan in [23] presented an economic model to capture
of the pipeline are correctly executed, the software is ready
and analyze a set of tredoffs involved in the software release
for release. Figure 2 shows a general CD deployment pipeline
decisions and discussed a method to optimize the delivery
formalized in BPMN notation [17].
process that takes into account crucial factors such as the
With reference to the BPMN/CD deployment pipeline de-
software development cycle and the changes in market needs.
picted in Figure 2, the pipeline starts in the Development
The above presented works do not cover all the aspects
swing-lane where the source code is ready to be committed
engaged in the continuous delivery process and do not provide
into a repository. A transition to the Continuous Integration
an adequate view on the issues that can emerge in performing
swing-lane happens and during the activity transition a connec-
the release management for IT services [3]. Moreover, they
tion to the repository is performed. At this point, the commit
are based on empirical results and not on simulation ones that
activity is performed in order to save the changes to both the
can help to better understand, predict and optimize the delivery
local and remote repository. In the Build activity, the source
process of software.
code is built and the resulting executable file is generated.
Through the following related works, an insight into the After that, a transition to the Simulation activity is done.
main issues related to the use of modeling and simulation Throughout this stage, the new software version is rigorously
techniques in delivering software, is given. tested through simulation techniques to evaluate, predict and
Dlugi et al. in [10] highlighted the difficulties to evaluate the optimize its behavior. It is important that all the requirements
performance of a software release, mainly related to the lack of aspects whether functional, non-functional, performance, se-
a test environment that is comparable to a production system. curity and compliance are verified. For each simulation test,
The authors introduced a model-based performance change some software metrics related to various constructs like class,
detection process that uses modeling and simulation methods cohesion and inheritance have been evaluated. To evaluate
for evaluating performance metrics of different software ver- the complexity of the source code, three standard metrics,
sions. Also, they developed a a plug-in for a CD deployment which are proposed by various researchers, can considered
pipeline based on Jenkins. [36]: (i) CCM (Cyclomatic Complexity Metric), which is a
In [35], Vöst and Wagner investigated the CD pipeline metric based on graph theory that represents the number
in the automotive domain. They evaluated the effect of of linearly independent paths through a program’s code; (ii)
various functional and non-functional requirements on the HCM (Halstead Complexity Metric), which measures the logic
development life cycle. They presented a typical delivery volume of the code. It is calculated on the count of the
pipeline in automotive software that adopts Hardware-in-the- operators and operands; (iii) (v) NF (number of function),
Loop-Simulation approaches to evaluate the software with the which represents the total number of functions. Typically,
operating environment. the lower are the values of these metrics the lower is the
Zia et al. in [1] used the Business Process Model and complexity of the source code and thus higher should be the
Notation (BPMN) standard to evaluate DevOps alternatives. code compactness, readability and reliability.
The effectiveness of BPMN has been demonstrated in other In the last activity, named Test units, a set of parametrized
domain [15], [16], [17], and this can represent a viable solution test units are executed. This allows to measure the progress
to model and simulate CD pipelines. of the software and detect side effects. A transition to the
This paper has various purposes in common with the pre- Production swing-lane happens if all the activities have been
sented works but unlike them it introduces metrics to evaluate successfully completed. Otherwise, if at least one activity
the performance of a CD deployment pipeline through M&S failed, a state transition to the source code activity is per-
techniques based on BPMN so as to define a standard CD formed, since the source code needs to be revised, and the
pipeline on which to perform analysis. pipelines terminates.
Development
source code
Continuous Integration
Commit
commit Build Simulation Test units
repository
requirements
Production
Configuration
Environment Source code manual test
configuration configuration units
Production
Manual fix Error trace
Developer
error
repository
error
Release
Software Push repository
Images Infrastructure
Fig. 2. Overview of the CD deployment pipeline defined in BPMN.
The Production swing-lane, the parameters related to the tar- to identify process improvement potential.
get environment (e.g. AWS Amazon Cloud, Microsoft Azure, On the other hand, we find those metrics that refer to the
Google Cloud, etc.) are configured in the Environment con- amount of work done in a certain time, for example the number
figuration activity. Thereafter, the Source code configuration of users (or jobs) that complete the whole process during an
activity is performed, in which the software is configured to observation time. In this case, the abovementioned metrics are
run into the target environment. Finally, additional manual test related to the concept of throughput.
units, which take into account the production environment, are In the context of continuous delivery and deployment, the
executed in the manual test units activity. concepts of “user” or “job” can be effectively intended as
A transition to the Release swing-lane happens if everything “features” flowing throughout the pipeline, from the initial
goes well. Otherwise, the issues occurred during the produc- commit to the final deployment in production.
tion activities are collected and stored into the error repository Since from a business analyst perspective, the cycle time
in order to be evaluated and fixed by developers. In the Release is the period it takes from an idea to provide actual business
swing-lane, the image of the software release is created and value, it is easy to understand that most profits could be gained
deployed in the environment. from that idea if it can be implemented quickly. If the cycle
B. Relevant Metrics time is too high, competitors might be first or the idea might
Generally speaking, in the study of business processes not be relevant anymore.
performance, metrics of particular interest fall under two main Under this perspective, it becomes clear how, to that op-
categories. timizing the cycle time requires an efficient process which
On the one hand we find those metrics that are directly usually spans multiple departments and involves a lot of people
related to time, for example the time in which a user (or job) (i.e., a complex process).
crosses the entire process starting from the initial state arriving In order to optimize a complex process, we need an efficient
at the final state, executing different tasks in its path. In this way to specify and analyze each task that is involved in it. As
case, this metric is defined as cycle time. aforementioned we propose BPMN to describe the deployment
Since cycle time includes both value-adding and non value- pipeline process, and propose the use of simulation techniques
adding activity times (e.g., waiting times), it is a powerful tool to enact analysis activities.
A similar approach enabling performance predictions over • Releases Per Month (RPM), measures the number of
a business process has been introduced and discussed in [5], releases that have been completed during a single month.
[9] It is worth noting how changes in the long term of this
Benefits of analyzing the process of such a pipeline include metric suggest information on changes in the release
finding and removing bottlenecks (e.g., a specific activity that cycle. Data for obtaining such a measure can be retrieved
is too expensive in terms of time, due to some inefficiency), from the version control system or from the integration
shortening the feedback loop (i.e., measuring activities in the server logs, for instance.
middle of the pipeline can be useful to provide an early • Fastest Possible Feature Lead Time, measures a sort of
feedback to end users), automating as much as possible (i.e., best case scenario in which, without delays and latencies
an accurate analysis can lead to the identification and the (e.g., caused by the use of feature branches or by separate
reduction of expensive manual activities and eliminate any build processes), the feature under study spends time only
error-prone manual tasks). In other words, the ultimate purpose in the build and test phase on the pipeline.
is to optimize and visualize what’s going on to improve the Also monitoring Mean Time Between Failures (MTBF) and
flow. Mean Time To Repair (MTTR) and their balancing can be an
In the continuous delivery and deployment domain, it is effective way to optimize the deployment pipeline.
easy to understand how the throughput of the pipeline used to Mean time between failures reminds the team to to avoid
deliver new features to production (i.e., to the end user) is a easy failures. However, core point of software development is
relevant metric. to provide new value to users, and only looking at MTBF can
Referring to the lean world and to kanban practices, a metric result in teams becoming overly cautious and never releasing
that completely captures the concept of the pipeline is certainly anything new.
flow efficiency, which is calculated based on the actual work In order to avoid such a drawback, a focus on mean time
time (i.e., the actual time spent working on a feature) measured to recover (i.e., a metric that measures the ability to rollback
against the total wait time (e.g., data transfer, hardware or in case of a mistake) can be a key counterbalance.
software provisioning, environment setting, etc.), presented in
• Mean Time Between Failures (MTTF), is the predicted
[26].
elapsed time between inherent failures of a system, during
The calculation of flow efficiency is as follows: normal system operation. In the deployment pipeline can
Work Time be a measure of the mean time between a feature not
Flow E f f iciency in % = passing some test, for example. MTBF can be calculated
Work Time + Wait Time
as the arithmetic mean (average) time between failures
Where:
of a system (i.e., a feature scoring a KO in some test).
• Work Time is the time in which the development of the Achieving a good MTBF in a pipeline involves getting
feature under study can be seen as in progress. feedback early on and making sure that thorough val-
• Wait Time is the time in which the development of the idation occurs in testing environments. Moreover, such
feature under study can be seen as blocked or waiting. validations should be run on environments that are similar
Other interesting metrics have been introduced, which can to production, and with realistic data.
be fruitfully taken into account for analyzing a deployment • Mean Time To Repair (MTTR), represents the average
pipeline, as Features Per Month (FPM), Releases Per Month time required to repair a failed component or device (i.e.,
(RPM), or Fastest Possible Feature Lead Time [25]. the failure of a feature in the pipeline). Assuming failure
is inevitable, it’s important to ensure that mean time to
TABLE I recover is as fast as possible. In other words, it measures
S UMMARY OF R ELEVANT M ETRICS . the time that is needed to “rollback” to a previous build
after a release failure. In this scenario it is clear how
Metric Unit of Measure
robust monitoring of production is essential. In order to
Cycle Time Time make this approach more effective, teams should learn
Throughput Frequency
Flow Efficiency % about failures through monitoring and alerts, not through
Features Per Month (FPM) Features / Time customer complaints.
Releases Per Month (RPM) Releases / Time
Fastest Possible Feature Lead Time Time Table I summarizes relevant metrics of interest and their units
MTBF Time of measure.
MTTR Time
C. Simulation Benefits
• Features Per Month (FPM), identifies the number of new The ever-growing complexity of modern software, which
features that have crossed the entire pipeline during a also involves physical and/or virtual platforms, requires the
month. The metric is based on Day-by-the-Hour (DbtH), adoption of effective analysis techniques to support the design
which measures quantity produced over hours worked and operation. Using M&S techniques is it possible to generate
[29]. the real-world scenarios that would be hard to get in the
real world, especially in distributed environments [13], [14], In this paper, we have discussed the adoption of a well-
[18], [27], [12], [28]. Moreover, moving from monolith to known standard to represent the deployment pipeline as a
Microservice architecture, the involved parts have their own process (i.e., BPMN), in order to make it repeatable and clearly
lifecycle [32]. specified.
The solution introduced in section III (see Figure 2) regards Moreover, we have discussed a set of candidate metrics
the availability of a Simulation activity that combines the static which can be suitable to analyze the performance of such
test units with simulation ones. The benefit of the Simulation process, most of them referable to the concepts of throughput
activity is that it continuously urges the software services, and cycle-time.
creating a constant and uniform load on them. This allows to Finally, we have discussed the benefits of simulation ap-
monitor the CD development pipeline and notify developers if proaches and techniques to carry on performance predictions
any activity failed. over a deployment pipeline.
Moreover, through the use of the Simulation activity is it More specifically, the support provided from simulation can
possible to evaluate the source code, which is managed by a be twofold. On the one hand, end-to-end performance of the
CD development pipeline, on specific embedded platforms and whole pipeline can be analyzed over different configurations,
then overcome typical issues involved in the hardware-based and, on the other hand, particular hardware or software com-
setups, such as: ponents can be simulated (e.g., using contracts in case of
• Hardware Checking. Managing simulation tests is much microservices or using a virtual hardware component).
easier respect to use hardware, especially when complete As a further step, work is ongoing to implement a full stack
and actual tests are too expensive to be performed in framework to enact modeling and simulation of a deployment
terms of cost, time and other resources. Though simula- pipeline, leveraging on previous experiences in a cloud-based
tion is also easier to handle multiple hardware configura- environment [4], [6], [7], [2], [19].
tions, since this means to change configuration parameters
in the tests. R EFERENCES
• Test Software Reflecting the Hardware. Generally, Hard-
[1] Zia Babar, Alexei Lapouchnian, and Eric Yu. Modeling devops deploy-
ware platforms have limited amounts of computational ment choices using process architecture design dimensions. In IFIP
power and memory space. Through simulation is it pos- Working Conference on The Practice of Enterprise Modeling, pages
sible to evaluate the maximum workload. 322–337. Springer, 2015.
[2] P. Bocciarelli, A. D’Ambrogio, E. Paglia, T. Panetti, and A. Giglio. A
• Configuration Reuse. With a simulation is very easy to
cloud-based service-oriented architecture for business process modeling
create, save and reuse hardware/software configurations. and simulation. In CEUR Workshop Proceedings – INCOSE Italia
• Error Handling. The part of the source code that is Conference on Systems Engineering. CEUR-WS, CEUR-WS, 2017.
responsible for handling faults and errors conditions can [3] Paolo Bocciarelli, Andrea D’Ambrogio, Alberto Falcone, Alfredo Garro,
and Andrea Giglio. A model-driven approach to enable the dis-
be difficult to test on hardware. Through a simulator, tributed simulation of complex systems. In Complex Systems Design
inject errors/faults is easy since any part of the simulator & Management, Proceedings of the Sixth International Conference on
code can be accessed and modified. Complex Systems Design & Management, CSD&M 2015, Paris, France,
November 23-25, 2015, pages 171–183. Springer-Verlag, 2015.
• Environmental aspects. By using simulation is it possible [4] Paolo Bocciarelli, Andrea D’Ambrogio, Andrea Giglio, and Antonio
to imitate environmental conditions in order to evaluate Mastromattei. Automated development of web-based modeling services
how the system responds to environmental stresses. Such for msaas platforms. In International Workshop on Model-Driven
Approaches for Simulation Engineering (Mod4Sim 2017), part of the
tests are very hard to perform using the real system. 2017 Spring Simulation Multiconference (SpringSim 2017), Virginia
Beach, VA, USA, 2017.
[5] Paolo Bocciarelli, Andrea D’Ambrogio, Andrea Giglio, and Emiliano
Paglia. A bpmn extension to enable the explicit modeling of complex
IV. C ONCLUSIONS task resources. In Proceedings of the INCOSE Italy Conference on
Systems Engineering, 2016.
In the last decades software production activities have [6] Paolo Bocciarelli, Andrea D’Ambrogio, Antonio Mastromattei, Emiliano
become increasingly challenging and software development Paglia, and Andrea Giglio. Business process modeling and simulation:
has become a demanding area of business. In order to reduce State of the art and msaas opportunities. In Proceedings of the Summer
Simulation Multi-Conference, SummerSim ’17, pages 1–12, San Diego,
the time to market of new product features, following the CA, USA, 2017. Society for Computer Simulation International.
Agile and DevOps principles, new approaches as continuous [7] Paolo Bocciarelli, Andrea D’Ambrogio, Emiliano Paglia, and Andrea
delivery and continuous integration are now widely adopted, Giglio. A service-in-the-loop approach for business process simulation
based on microservices. In Proceedings of the 50th Computer Simulation
enabling the software functional items to efficiently flow from Conference, SummerSim 2018, Bordeaux, France, July 09-12, 2018,
development to production. pages 24:1–24:12, 2018.
Such an approach leverages on methods and tools as the [8] Gerry Gerard Claps, Richard Berntsson Svensson, and Aybke Au-
rum. On the journey to continuous deployment: Technical and social
deployment pipeline, which represents a series of tasks to be challenges along the way. Information and Software Technology,
executed in order to deliver a software product or feature to 57(Complete):21–31, 2015.
end users, starting from a commit stage. [9] Andrea D’Ambrogio, Emiliano Paglia, Paolo Bocciarelli, and Andrea
Giglio. Towards performance-oriented perfective evolution of bpmn
From this perspective, the need for analyzing and optimizing models. In 6th International Workshop on Model-Driven Approaches
such activities emerges. for Simulation Engineering, 2016.
[10] Markus Dlugi, Andreas Brunnert, and Helmut Krcmar. Model-based per- simulations in the space domain: The SISO space reference FOM
formance evaluations in continuous delivery pipelines. In Proceedings initiative. In 20th IEEE/ACM International Symposium on Distributed
of the 1st International Workshop on Quality-Aware DevOps, QUDOS Simulation and Real Time Applications, DS-RT 2016, London, United
2015, Bergamo, Italy, September 1, 2015, pages 25–26, 2015. Kingdom, September 21-23, 2016, pages 100–107. Institute of Electrical
[11] Christof Ebert, Gorka Gallardo, Josune Hernantes, and Nicolas Serrano. and Electronics Engineers Inc., 2016.
Devops. IEEE Software, 33(3):94–100, 2016. [28] Björn Möller, Alfredo Garro, Alberto Falcone, Edwin Z. Crues, and
[12] Alberto Falcone and Alfredo Garro. The SEE HLA starter kit: enabling Daniel E. Dexter. On the execution control of HLA federations using
the rapid prototyping of hla-based simulations for space exploration. the SISO space reference FOM. In 21st IEEE/ACM International
In Modeling and Simulation of Complexity in Intelligent, Adaptive and Symposium on Distributed Simulation and Real Time Applications, DS-
Autonomous Systems 2016, MSCIAAS 2016, and Space Simulation for RT 2017, Rome, Italy, October 18-20, 2017, pages 75–82. Institute of
Planetary Space Exploration, SPACE 2016, part of the 2016 Spring Sim- Electrical and Electronics Engineers Inc., 2017.
ulation Multiconference, SpringSim 2016, Pasadena, CA, USA, April 3- [29] Kai Petersen and Claes Wohlin. Measuring the flow in lean software
6, 2016, page 1. The Society for Modeling and Simulation International development. Softw., Pract. Exper., 41(9):975–996, 2011.
Inc., 2016. [30] Mary Poppendieck and Tom Poppendieck. Lean Software Development:
[13] Alberto Falcone and Alfredo Garro. A java library for easing the dis- An Agile Toolkit. Addison-Wesley Longman Publishing Co., Inc.,
tributed simulation of space systems. In 16th International Conference Boston, MA, USA, 2003.
on Modeling and Applied Simulation, MAS 2017, Held at the Interna- [31] Daniel Ståhl and Jan Bosch. Modeling continuous integration practice
tional Multidisciplinary Modeling and Simulation Multiconference, I3M differences in industry software development. Journal of Systems and
2017, Barcelona, Spain, September 18-20, 2017, pages 6–13. CAL-TEK Software, 87:48–59, 2014.
S.r.l., 2017. [32] Guido Steinacker. Scaling with microservices and vertical decomposi-
[14] Alberto Falcone, Alfredo Garro, Anastasia Anagnostou, and Simon J. E. tion. dev. otto. de, 2014.
Taylor. An introduction to developing federations with the high level [33] André van der Hoek and Alexander L. Wolf. Software release manage-
architecture (HLA). In 2017 Winter Simulation Conference, WSC 2017, ment for component-based software. Softw., Pract. Exper., 33(1):77–98,
Las Vegas, NV, USA, December 3-6, 2017, pages 617–631. Institute of 2003.
Electrical and Electronics Engineers Inc., 2017. [34] Manish Virmani. Understanding devops & bridging the gap from
[15] Alberto Falcone, Alfredo Garro, Andrea D’Ambrogio, and Andrea continuous integration to continuous delivery. In Innovative Computing
Giglio. Engineering systems by combining bpmn and hla-based dis- Technology (INTECH), 2015 Fifth International Conference on, pages
tributed simulation. In 2017 IEEE International Conference on Systems 78–82. IEEE, 2015.
Engineering Symposium, ISSE 2017, Vienna, Austria, October 11-13, [35] Sebastian Vöst and Stefan Wagner. Towards continuous integration
2017, pages 1–6. Institute of Electrical and Electronics Engineers Inc., and continuous delivery in the automotive industry. arXiv preprint
2017. arXiv:1612.04139, 2016.
[16] Alberto Falcone, Alfredo Garro, Andrea D’Ambrogio, and Andrea [36] Sheng Yu and Shijie Zhou. A survey on metric of software complexity.
Giglio. A model-driven method to allow the distributed simulation of In Information Management and Engineering (ICIME), 2010 The 2nd
bpmn models. In 27th IEEE International Conference on Enabling IEEE International Conference on, pages 352–356. IEEE, 2010.
Technologies: Infrastructure for Collaborative Enterprises, WETICE
2018, Paris, France, June 27-29, 2018, pages 0–0. Institute of Electrical
and Electronics Engineers Inc., 2018.
[17] Alberto Falcone, Alfredo Garro, Andrea D’Ambrogio, and Andrea
Giglio. Using bpmn and hla for engineering sos : lessons learned and
future directions. In 2018 IEEE International Conference on Systems
Engineering Symposium, ISSE 2018, Rome, Italy, October 1-3, 2018.
Institute of Electrical and Electronics Engineers Inc., 2018.
[18] Alberto Falcone, Alfredo Garro, Simon J. E. Taylor, Anastasia Anag-
nostou, Nauman R. Chaudhry, and Omar-Alfred Salah. Experiences in
simplifying distributed simulation: The HLA development kit frame-
work. Journal of Simulation, 11(3):208–227, 2017.
[19] Angelo Furfaro, Teresa Gallo, Alfredo Garro, Domenico Saccà, and
Andrea Tundis. Requirements specification of a cloud service for cyber
security compliance analysis. In Cloud Computing Technologies and
Applications (CloudTech), 2016 2nd International Conference on, pages
205–212. IEEE, 2016.
[20] Jim Highsmith and Martin Fowler. The agile manifesto. Software
Development Magazine, 9(8):29–30, 2001.
[21] Jez Humble and David Farley. Continuous delivery: reliable software
releases through build, test, and deployment automation. Addison-
Wesley Boston, 2011.
[22] Mira Kajko-Mattsson and YuLong Fan. Outlining a model of a release
management process. Transactions of the SDPS, 9(4):13–25, 2005.
[23] Mayuram S. Krishnan. Software release management: a business
perspective. In Proceedings of the 1994 Conference of the Centre for
Advanced Studies on Collaborative Research, October 31 - November
3, 1994, Toronto, Ontario, Canada, page 36, 1994.
[24] A. Lahtela and M. Jntti. Challenges and problems in release management
process: A case study. In 2011 IEEE 2nd International Conference on
Software Engineering and Service Science, pages 10–13, July 2011.
[25] Timo Lehtonen, Sampo Suonsyrjä, Terhi Kilamo, and Tommi Mikkonen.
Defining metrics for continuous delivery and deployment pipeline. In
Proceedings of the 14th Symposium on Programming Languages and
Software Tools (SPLST’15), Tampere, Finland, October 9-10, 2015.,
pages 16–30, 2015.
[26] N. Modig. This is Lean: Resolving the Efficiency Paradox. Rheologica,
2012.
[27] Björn Möller, Alfredo Garro, Alberto Falcone, Edwin Z. Crues, and
Daniel E. Dexter. Promoting a-priori interoperability of hla-based