-In modern Software Engineering, Continuous Delivery (CD) is a development approach in which a software is iteratively developed in short cycles ensuring, for each cycle, that the new features are available to end users as soon as they are implemented and tested. CD aims at defining, building and releasing software with greater speed and frequency through the deployment pipeline resulting in three major benefits, visibility, feedback and continuous deployment respectively, enabling the software functional items to efficiently flow from development to production. In this domain, the need for evaluating the performance of the deployment pipeline emerges, since the conventional metrics available in the software engineering discipline are not suited to handle all the involved aspects. In this paper, the metrics suited for supporting CD are introduced and an integration with Modeling and Simulation (M&S) techniques is discussed, based on the Business Process Model and Notation (BPMN) standard, which could represent a valid support offering a graphical notation to easily specify the deployment pipeline steps as a standard and repeatable process. The main objective is to identify feasible perspectives in which simulation methods and principles can be exploited, thus evaluating the effectiveness of M&S to support performance analysis of a deployment pipeline, seen as a predictable process. Specifically, M&S can be seen as an enabling tool for the evaluation and comparison of different CD choices against requirements through an effective implementation of simulation techniques and virtual testing. Index Terms-Continuous Delivery, Business Process Model and Notation (BPMN), Modeling and Simulation
Nowadays, in the software engineering and software
development domains, many influencing factors play a fundamental
role in making software production activities increasingly
challenging. Fast-changing customer requirements, as well as
unpredictability of market and the need of an ever shorter time
to-market are non-negligible aspects that have to be taken into
account, since software development has become a demanding
area of business [
Copyright c held by the author
Furthermore, the dramatic impact of the Agile Manifesto on
the way software production is managed, has led in recent
years to the widespread adoption of new emerging approaches
in software engineering. The first of the twelve fundamental
principles defined in the aforementioned manifesto states that
“Our highest priority is to satisfy the customer through early
and continuous delivery of valuable software” [
Despite the fact that continuous delivery was explicitly mentioned in this principle, its widespread adoption is the result of an evolution of different approaches through the years. Starting from continuous integration (CI) shifting towards modern continuous deployment pipelines, releasing new software versions early and often has become mainstream, and now represents a concrete option also for an ever growing number of companies and practitioners. Let us summarize some baseline definitions:
Continuous Integration. CI is the process of ensuring that a software build is in a correct working state, guaranteeing developers that the release is suitable for production. In this approach, members of a team are pushed to integrate their work frequently, typically each person integrates at least daily leading to several integrations per day. It is worth noting that each integration is verified by an automated build and test procedure with the aim of detecting integration errors as soon as possible.
Continuous Delivery. Deriving from CI, CD ensures that a build is always in a releasable state. Going hand to hand with a DevOps approach, where the team is responsible for all aspects of software delivery, after integration and testing activities the software is actually provisioned to a full, production-like stack, and delivered to some set of end users.
Continuous Deployment. When the entire process, from check-in to production, is fully automated, with no human intervention, continuous deployment is implemented. Deployment Pipeline. The foundation of continuous delivery is the deployment pipeline, which can be seen the path that a code change takes from check-in (i.e. the commit operation) to production.
CD is a software engineering approach that represents a huge step forward in productivity and quality for companies and organizations that adopt it. CD is gaining attention since it provides a well-established process that allows team members to work together to define and create large and complex software with a higher level of control. CI claims to enable companies to release new features, configuration properties, bug fixes and tests, to customers safely and quickly in a sustainable way. This means that it mainly focuses on asserting the correct compilation of the source code and that it passes a chain of test units.
However, measuring the performance of a CD deployment
pipeline is a challenging task that requires a considerable effort
in terms of both time and cost. Many companies including
Google, Facebook and IBM are developing their software
products by using CD [
In this context, the paper introduces metrics that are suited to evaluate the performance of a CD deployment pipeline through M&S techniques based on the Business Process Model and Notation (BPMN) standard, in the purpose of defining the pipeline as a standard process and also of analyzing it as a repeatable and predictable process.
The ultimate aim is to provide an exploratory analysis of the available approaches to minimize the cycle-time in software production (i.e. the period it takes from an idea to provide actual business value), and also to investigate the different domains in which simulation techniques can be fruitfully exploited to enforce continuous delivery activities through a deployment pipeline optimization.
The rest of this paper is organized as follows. Section II provides an introduction to the essential concepts and background knowledge on the research domain. The Continuous Delivery (CD) methodology and the typical architecture of a CD deployment pipeline are presented along with some related works available in literature. In Section III, the CD deployment pipeline is defined through the concepts provided by the Business Process Model and Notation (BPMN) standard with particular focus on how to make available Modeling and Simulation (M&S) practices during the pipeline flow in order to evaluate, predict and optimize its behavior. Finally, Section IV discusses the main objectives of the work and presents some future works.
Continuous Delivery (CD) is a concept which has been
taken as a pillar of modern Agile software engineering [
Basically, the deployment pipeline is an automated process
for getting software from the repository into the hands of
endusers. CD is preceded by the Continuous Integration (CI) in
which the team members integrates the source code [
Commit Build Test Deploy
Automatic Manual e n il e p i P D C
The CD deployment pipeline starts when developers commit
the source code changes into the version control system. At
this point, the CI Management System (CI-MS) triggers a
new instance of the deployment pipeline and compiles the
source code. If the compile procedure successfully completed
a transition to the test phase is performed; otherwise the
procedure terminates with an error. In the test phase, the
CI-MS runs a set of unit tests, performs code analysis, and
builds the installer package. In this step, some manual tests
needs to be performed in order to evaluate domain-dependent
functional and non-functional requirements. If all the tests
pass, the executable code is used to generate the binaries that
will be stored in an artifact repository [
The related works available in the literature are vast and extend through empirical and modeling and simulation approaches to studying processes in software development.
In [
Lahtela et al. in [
Kajko-Mattson and Fan in [
Krishnan in [
The above presented works do not cover all the aspects
engaged in the continuous delivery process and do not provide
an adequate view on the issues that can emerge in performing
the release management for IT services [
Through the following related works, an insight into the main issues related to the use of modeling and simulation techniques in delivering software, is given.
Dlugi et al. in [
In [
Zia et al. in [
This paper has various purposes in common with the presented works but unlike them it introduces metrics to evaluate the performance of a CD deployment pipeline through M&S techniques based on BPMN so as to define a standard CD pipeline on which to perform analysis.
III. THE PIPELINE AS A STANDARD AND PREDICTABLE
PROCESS
Under the assumption that both a business process
collaboration and a CD deployment pipeline implement an exchange
of information between logical processes, in this paper the
adoption of BPMN as a notation to define a deployment
pipeline is proposed. This allows to perform simulation test
units in order to evaluate the performance of a pipeline through
a set of well-known metrics such as, those presented in [
The deployment pipeline is the workflow, with related activities, that a source code follows from commit to production.
Each commit, made by the development team, generates a
release candidate of the software which flows through the
pipeline. If everything goes well, which means that all steps
of the pipeline are correctly executed, the software is ready
for release. Figure 2 shows a general CD deployment pipeline
formalized in BPMN notation [
With reference to the BPMN/CD deployment pipeline depicted in Figure 2, the pipeline starts in the Development swing-lane where the source code is ready to be committed into a repository. A transition to the Continuous Integration swing-lane happens and during the activity transition a connection to the repository is performed. At this point, the commit activity is performed in order to save the changes to both the local and remote repository. In the Build activity, the source code is built and the resulting executable file is generated.
After that, a transition to the Simulation activity is done.
Throughout this stage, the new software version is rigorously
tested through simulation techniques to evaluate, predict and
optimize its behavior. It is important that all the requirements
aspects whether functional, non-functional, performance,
security and compliance are verified. For each simulation test,
some software metrics related to various constructs like class,
cohesion and inheritance have been evaluated. To evaluate
the complexity of the source code, three standard metrics,
which are proposed by various researchers, can considered
[
In the last activity, named Test units, a set of parametrized test units are executed. This allows to measure the progress of the software and detect side effects. A transition to the Production swing-lane happens if all the activities have been successfully completed. Otherwise, if at least one activity failed, a state transition to the source code activity is performed, since the source code needs to be revised, and the pipelines terminates. t n e m p o l e v e D n o itr a g e It n s u o u n it n o C n o it c u d o r P e s a e l e R
source code
The Production swing-lane, the parameters related to the tar- to identify process improvement potential. get environment (e.g. AWS Amazon Cloud, Microsoft Azure, On the other hand, we find those metrics that refer to the Google Cloud, etc.) are configured in the Environment con- amount of work done in a certain time, for example the number figuration activity. Thereafter, the Source code configuration of users (or jobs) that complete the whole process during an activity is performed, in which the software is configured to observation time. In this case, the abovementioned metrics are run into the target environment. Finally, additional manual test related to the concept of throughput. units, which take into account the production environment, are In the context of continuous delivery and deployment, the executed in the manual test units activity. concepts of “user” or “job” can be effectively intended as
A transition to the Release swing-lane happens if everything “features” flowing throughout the pipeline, from the initial goes well. Otherwise, the issues occurred during the produc- commit to the final deployment in production. tion activities are collected and stored into the error repository Since from a business analyst perspective, the cycle time in order to be evaluated and fixed by developers. In the Release is the period it takes from an idea to provide actual business swing-lane, the image of the software release is created and value, it is easy to understand that most profits could be gained deployed in the environment. from that idea if it can be implemented quickly. If the cycle B. Relevant Metrics time is too high, competitors might be first or the idea might
Generally speaking, in the study of business processes not be relevant anymore. performance, metrics of particular interest fall under two main Under this perspective, it becomes clear how, to that opcategories. timizing the cycle time requires an efficient process which
On the one hand we find those metrics that are directly usually spans multiple departments and involves a lot of people related to time, for example the time in which a user (or job) (i.e., a complex process). crosses the entire process starting from the initial state arriving In order to optimize a complex process, we need an efficient at the final state, executing different tasks in its path. In this way to specify and analyze each task that is involved in it. As case, this metric is defined as cycle time. aforementioned we propose BPMN to describe the deployment
Since cycle time includes both value-adding and non value- pipeline process, and propose the use of simulation techniques adding activity times (e.g., waiting times), it is a powerful tool to enact analysis activities.
A similar approach enabling performance predictions over
a business process has been introduced and discussed in [
Benefits of analyzing the process of such a pipeline include finding and removing bottlenecks (e.g., a specific activity that is too expensive in terms of time, due to some inefficiency), shortening the feedback loop (i.e., measuring activities in the middle of the pipeline can be useful to provide an early feedback to end users), automating as much as possible (i.e., an accurate analysis can lead to the identification and the reduction of expensive manual activities and eliminate any error-prone manual tasks). In other words, the ultimate purpose is to optimize and visualize what’s going on to improve the flow.
In the continuous delivery and deployment domain, it is easy to understand how the throughput of the pipeline used to deliver new features to production (i.e., to the end user) is a relevant metric.
Referring to the lean world and to kanban practices, a metric
that completely captures the concept of the pipeline is certainly
flow efficiency, which is calculated based on the actual work
time (i.e., the actual time spent working on a feature) measured
against the total wait time (e.g., data transfer, hardware or
software provisioning, environment setting, etc.), presented in
[
The calculation of flow efficiency is as follows:
Flow E f f iciency in % =
Work Time + Wait Time Work Time is the time in which the development of the feature under study can be seen as in progress.
Wait Time is the time in which the development of the feature under study can be seen as blocked or waiting.
Other interesting metrics have been introduced, which can
be fruitfully taken into account for analyzing a deployment
pipeline, as Features Per Month (FPM), Releases Per Month
(RPM), or Fastest Possible Feature Lead Time [
Releases Per Month (RPM), measures the number of releases that have been completed during a single month. It is worth noting how changes in the long term of this metric suggest information on changes in the release cycle. Data for obtaining such a measure can be retrieved from the version control system or from the integration server logs, for instance.
Fastest Possible Feature Lead Time, measures a sort of best case scenario in which, without delays and latencies (e.g., caused by the use of feature branches or by separate build processes), the feature under study spends time only in the build and test phase on the pipeline.
Also monitoring Mean Time Between Failures (MTBF) and Mean Time To Repair (MTTR) and their balancing can be an effective way to optimize the deployment pipeline.
Mean time between failures reminds the team to to avoid easy failures. However, core point of software development is to provide new value to users, and only looking at MTBF can result in teams becoming overly cautious and never releasing anything new.
In order to avoid such a drawback, a focus on mean time to recover (i.e., a metric that measures the ability to rollback in case of a mistake) can be a key counterbalance.
Mean Time Between Failures (MTTF), is the predicted elapsed time between inherent failures of a system, during normal system operation. In the deployment pipeline can be a measure of the mean time between a feature not passing some test, for example. MTBF can be calculated as the arithmetic mean (average) time between failures of a system (i.e., a feature scoring a KO in some test). Achieving a good MTBF in a pipeline involves getting feedback early on and making sure that thorough validation occurs in testing environments. Moreover, such validations should be run on environments that are similar to production, and with realistic data.
Mean Time To Repair (MTTR), represents the average
time required to repair a failed component or device (i.e.,
the failure of a feature in the pipeline). Assuming failure
is inevitable, it’s important to ensure that mean time to
recover is as fast as possible. In other words, it measures
the time that is needed to “rollback” to a previous build
after a release failure. In this scenario it is clear how
robust monitoring of production is essential. In order to
make this approach more effective, teams should learn
about failures through monitoring and alerts, not through
customer complaints.
Features Per Month (FPM), identifies the number of new
features that have crossed the entire pipeline during a
month. The metric is based on Day-by-the-Hour (DbtH),
which measures quantity produced over hours worked
[
The ever-growing complexity of modern software, which
also involves physical and/or virtual platforms, requires the
adoption of effective analysis techniques to support the design
and operation. Using M&S techniques is it possible to generate
the real-world scenarios that would be hard to get in the
real world, especially in distributed environments [
The solution introduced in section III (see Figure 2) regards the availability of a Simulation activity that combines the static test units with simulation ones. The benefit of the Simulation activity is that it continuously urges the software services, creating a constant and uniform load on them. This allows to monitor the CD development pipeline and notify developers if any activity failed.
Moreover, through the use of the Simulation activity is it possible to evaluate the source code, which is managed by a CD development pipeline, on specific embedded platforms and then overcome typical issues involved in the hardware-based setups, such as:
Hardware Checking. Managing simulation tests is much easier respect to use hardware, especially when complete and actual tests are too expensive to be performed in terms of cost, time and other resources. Though simulation is also easier to handle multiple hardware configurations, since this means to change configuration parameters in the tests.
Test Software Reflecting the Hardware. Generally, Hardware platforms have limited amounts of computational power and memory space. Through simulation is it possible to evaluate the maximum workload.
Configuration Reuse. With a simulation is very easy to create, save and reuse hardware/software configurations. Error Handling. The part of the source code that is responsible for handling faults and errors conditions can be difficult to test on hardware. Through a simulator, inject errors/faults is easy since any part of the simulator code can be accessed and modified.
Environmental aspects. By using simulation is it possible to imitate environmental conditions in order to evaluate how the system responds to environmental stresses. Such tests are very hard to perform using the real system.
In the last decades software production activities have become increasingly challenging and software development has become a demanding area of business. In order to reduce the time to market of new product features, following the Agile and DevOps principles, new approaches as continuous delivery and continuous integration are now widely adopted, enabling the software functional items to efficiently flow from development to production.
Such an approach leverages on methods and tools as the deployment pipeline, which represents a series of tasks to be executed in order to deliver a software product or feature to end users, starting from a commit stage.
From this perspective, the need for analyzing and optimizing such activities emerges.
In this paper, we have discussed the adoption of a wellknown standard to represent the deployment pipeline as a process (i.e., BPMN), in order to make it repeatable and clearly specified.
Moreover, we have discussed a set of candidate metrics which can be suitable to analyze the performance of such process, most of them referable to the concepts of throughput and cycle-time.
Finally, we have discussed the benefits of simulation approaches and techniques to carry on performance predictions over a deployment pipeline.
More specifically, the support provided from simulation can be twofold. On the one hand, end-to-end performance of the whole pipeline can be analyzed over different configurations, and, on the other hand, particular hardware or software components can be simulated (e.g., using contracts in case of microservices or using a virtual hardware component).
As a further step, work is ongoing to implement a full stack
framework to enact modeling and simulation of a deployment
pipeline, leveraging on previous experiences in a cloud-based
environment [