=Paper=
{{Paper
|id=Vol-2470/p20
|storemode=property
|title=Comparison of devops maturity models
|pdfUrl=https://ceur-ws.org/Vol-2470/p20.pdf
|volume=Vol-2470
|authors=Monika Gasparaite,Saulius Ragaišis
|dblpUrl=https://dblp.org/rec/conf/ivus/GasparaiteR19
}}
==Comparison of devops maturity models==
https://ceur-ws.org/Vol-2470/p20.pdf
Comparison of devops maturity models
Monika Gasparaitė Saulius Ragaišis
Institute of Computer Science Institute of Computer Science
Vilnius University Vilnius University
Vilnius, Lithuania Vilnius, Lithuania
monika.gasparaite@gmail.com saulius.ragaisis@mif.vu.lt
Abstract—Since software development using DevOps is still II. D EVO PS
a new phenomenon, analysis of DevOps maturity models is
insufficient. The purpose of this work is to investigate whether
The literature review has shown that definition of DevOps is
perception of DevOps is similar in different models. Models ambiguous [3]. Even though DevOps has no formal definition,
that have been investigated are as follows: BTopham, Samer I. there are a few prevailing opinions about it. Some state that it
Mohamed, and Focus Area models. In order to compare the can be defined as a job title which requires additional skills,
models, assessments of maturity and capability assured by the while others strongly believe that DevOps is more like a
models have been performed. The results have shown that the
examined models are different in terms of their interpretation of
movement with specific concepts covered. However, in simple
DevOps. terms, it can be defined as a combination of development and
Keywords—DevOps, Maturity Model, Process Assessment, Pro- operations. Development is represented by software develop-
cess Capability ers, while operations involve experts who maintain software
in production environment such as database administrators and
network specialists [4].
I. I NTRODUCTION Today DevOps covers culture, collaboration, automation,
lean practices, continuous improvement and delivery, user
Software process is a set of interrelated or interacting satisfaction. Culture can be interpreted as shared goals and
activities that are performed while creating a software product. values, responsibility and effortless communication. Collab-
Although software industry has improved significantly over the oration is about adopting cross-functional teams. Automation
last few decades, many software companies face such prob- involves building, testing, and deployment. Lean practices aim
lems as projects being behind schedule, exceeding the budget, to eliminate waste. DevOps also encompasses practices related
customer dissatisfaction with product quality [1]. Eventually, to monitoring and measurement, which result in continuous
it was acknowledged that most of the problems arise due improvement. Another purpose of DevOps concerns releasing
to immature software process of the company [2]. Software software and reacting on feedback faster [3].
process models were created to improve and assess software
process. Software process model defines essential elements of III. D EVO PS MATURITY MODELS
the process, which can be used to assess the maturity of an This section provides an overview of three considered
organization or the capability of individual processes. models as follows: BTopham, Samer I. Mohamed, and Focus
New software development methods are emerging over time. Area maturity models. From all DevOps models that can be
One of the rapidly growing phenomena is DevOps, the primary found, only three DevOps models were selected for further
goal of which is to bridge the gap between development and investigation. The reason being is that they are more compre-
operations. That can be achieved by combining the objectives hensive in comparison with other models.
of different disciplines and tools, forcing interdisciplinary
A. BTopham maturity model
professionals to communicate frequently.
Since software development using DevOps practices is a Shani Inbar, Sayers Yaniv, Pearl Gil, Schitzer Eran, Shufer
quite new concern, there is a lack of analysis related to DevOps Ilan, Kogan Olga, Srinivasan Ravi present DevOps maturity
maturity models. The purpose of this paper is to examine model which consists of five maturity levels [5]. For the pur-
whether DevOps concept can be interpreted the same way in pose of simplicity, this maturity model is hereinafter referred
various DevOps maturity models. to as BTopham maturity model.
This paper is organized as follows. Section II describes Authors define three dimensions - process, automation, and
DevOps. Section III presents existing DevOps maturity mod- collaboration. It is essential to mention that term "dimension"
els. Section IV defines comparison method which is used when is not used correctly in this context as it has a different
assessing models in sections V-VI. meaning in traditional maturity models. In this case, dimension
is an equivalent term to process area in CMMI [6]. In fact,
©2019 for this paper by its authors. Use permitted under Creative other models give a different name for it. For example,
Commons License Attribution 4.0 International (CC BY 4.0) ISO/IEC 15504 [7] uses term "process" while AgilityMOD [8]
65
model names it aspect. In order to avoid misunderstandings, Model assessment is composed of several steps. Firstly,
in this paper dimensions will be called aspects. maturity assessment of selected model, for example, model
Even though BTopham maturity model is designed to cover X, is done. This assessment is performed as follows. First of
the entire life-cycle of an application or a service for large all, it is assumed that the company meets the requirements of a
enterprises, this model lacks clarity and specificity. This model certain maturity level of the selected model X. Based on those
can be rather perceived as guidelines which state general ideas requirements, the assessment is carried out in accordance with
for software process improvement. Focus Area assessment method. It gives information about
maturity level that can be achieved in accordance with Focus
B. Samer I. Mohamed maturity model Area maturity model. The assessment is performed until all
Samer I. Mohamed maturity model presents an improved maturity levels of the selected model X are assessed.
version of the previously examined BTopham maturity model After maturity assessment, the capability assessment is
[9]. This model has identical five maturity levels, but it performed. This assessment is based on focus areas instead of
defines slightly different aspects as follows: communication, maturity levels as in maturity assessment. It gives information
automation, quality, and governance. about capability level for each focus area that can be assured
Communication refers to effective communication between by the highest maturity level of model X. If a certain level of
teams. Automation specifies improvement of delivery speed, capability is not achieved, the higher capability levels are no
throughput, and repeatability. Quality is about delivering in longer assessed. This assessment provides more information
more lean and faster manner while governance is responsible about the model being assessed.
for controlling how those aspects work seamlessly together. Finally, reverse assessments are performed. Reverse assess-
Although this model is an improved version of BTopham ments are analogous to previously mentioned assessments. The
maturity model, disadvantages remain the same. Both models only difference is that it is based not on Focus Area maturity
are quite abstract in order to assess maturity of real-life model but on the selected model X.
organization. All assessments rely not only on model requirements that
were presented but also on requirements that are established
C. Focus Area maturity model
and well known in IT industry nowadays. For instance, version
Rico de Feijter, Rob van Vliet, Erik Jagroep, Sietse Over- control system usage. If assessments were based only on the
beek, Sjaak Brinkkemper present another DevOps maturity explicitly stated requirements, the results would be slightly
model which is the largest and the most comprehensive model worse. In fact, the purpose of this assessment is to get results
of all models that have been found [3]. For the purpose of that are as realistic as possible.
simplicity hereinafter it will be referred to as Focus Area
model.
V. C OMPARISON OF BT OPHAM AND F OCUS A REA
Focus Area model is a non-traditional model because of its
MATURITY MODELS
focus area architecture. In this context, staged and continu-
ous representations are traditional. Focus area architecture is A. BTopham model maturity assessment
different for two reasons. First of all, there can be unlimited
quantity of maturity and capability levels. Secondly, each focus The assessment of BTopham model maturity allows to
area (which is a synonym for process area in CMMI matu- determine which level of maturity in accordance with Focus
rity model) has different evaluation intervals. For example, Area model each maturity level can assure.
possible capability levels for communication focus area starts Assessment begins with the first maturity level. BTopham
from A to E, while configuration management can be assessed model does not have any requirements for the first maturity
from A to C. For this reason, meaning of C capability level is level. For this reason, each organization is at this level by
different in both contexts. Even though focus areas are named default. Focus Area maturity model defines maturity level 0,
differently in traditional maturity models, original term in this which is equivalent to maturity level 1 in BTopham model as
paper will be used instead. This is because of focus area there is no focus area assigned.
architecture which is quite specific. One of the Focus Area model requirements for maturity
This model describes sixteen focus areas which must be level 1 is “functional and non-functional requirements and
taken into consideration while trying to adopt or improve incidents are gathered from and prioritized with internal
DevOps practices in company. All focus areas are logically stakeholders and customers”. However, BTopham maturity
grouped into three groups: 1) culture and communication, 2) model does not provide any information about requirements
product, process, and quality, 3) foundation. and incidents. For this reason, maturity level 2 in BTopham
maturity model cannot assure maturity level 1 in Focus Area
IV. C OMPARISON METHOD model. Oddly, all other requirements for maturity level 1 are
In order to determine how different DevOps maturity mod- met.
els perceive DevOps, comparison of maturity models has been The assessment of maturity levels 3 - 5 corresponds to
performed. That comparison is accomplished by assessing assessment of maturity level 2 because previously mentioned
maturity models in accordance with Focus Area model. requirement still cannot be satisfied.
66
In short, the results of BTopham model maturity assessment The assessment of maturity levels 2-10 corresponds to
show that all maturity levels in BTopham maturity model can previous assessment because Focus Area model does not have
ensure only maturity level 0 in the Focus Area model. requirements related to documentation.
B. BTopham model capability assessment D. Focus Area model capability assessment
As mentioned before, capability assessment allows to gain The result of Focus Area capability assessment is provided
additional knowledge about models. The assessment result is in Table II. The highest capability level that can be achieved
provided in Table I. A dash indicates that the lowest capability is 5 as model provides exact five levels. It is worth to mention
level cannot be assured. Also, maximum capability levels are that in traditional models the lowest capability level is 0
provided as they are different for each focus area. which means that aspect is either not performed or partially
performed. In this case, the lowest capability is 1.
TABLE I
T HE RESULT OF BT OPHAM MODEL CAPABILITY ASSESSMENT
TABLE II
Achieved Maximum T HE RESULT OF F OCUS A REA MODEL CAPABILITY ASSESSMENT
Focus area
capability level capability level
Communication E E Achieved
Aspect
capability level
Knowledge sharing C D
Process 2
Trust and respect A C
Automation 1
Team organization B D
Collaboration 1
Release alignment C C
Release heartbeat - F
Branch and merge A D E. Disadvantages of BTopham and Focus Area maturity
Build automation B C models
Development quality
- E
improvement The obtained results have shown that BTopham and Focus
Test automation - E Area models are contrastive. First of all, a few requirements
Deployment automation C D in BTopham maturity model are abstract, making it difficult to
Release for production - D check whether the requirement is met, while the Focus Area
Incident handling - D model provides very specific requirements. For instance, Focus
Configuration management A C Area model requires manual code quality monitoring and
Architecture alignment A B examples of how it can be done are provided, while BTopham
Infrastructure A D
maturity model requires that the quality of the overall process
should be measured but it is not clear which areas and how
As shown in Table I, communication and release alignment to measure.
focus areas assure the highest capability level. It means that What is more, requirements that can be checked, such
perception of communication is similar in both models. It is as standardization or documentation of the process, are not
worth to mention that requirement for capability level C is required in Focus Area model. Assessments have shown that
incorrect because it is composed of requirements for lower BTopham maturity model has some drawbacks. For instance,
capability levels. Therefore, if lower capabilities (A and B) requirements for higher maturity levels can be satisfied even
are achieved, the highest capability level C will be always though requirements for lower maturity levels are not met.
achieved as well. Meanwhile, the absolute majority of the requirements in Focus
Area model are formulated so that higher level requirements
C. Focus Area model maturity assessment cannot be met unless lower level requirements are satisfied.
The assessment of Focus Area model maturity allows to de- Another disadvantage of BTopham maturity model is that
termine which level of maturity in accordance with BTopham this model does not define the necessary level implementation
maturity model each maturity level can assure. of the requirements for maturity level. Traditional models do
As mentioned before, Focus Area model does not have any not require complete implementation of the requirements. For
requirements for maturity level 0 because there are no focus example, ISO/IEC 15504 states that the process attribute is
areas assigned to this maturity level. For this reason, maturity “fully achieved” if at least 86% requirements are met, while
level 0 in Focus Area maturity model is analogous to maturity CMMI requires no fundamental shortcomings.
level 1 in BTopham maturity model. Unfortunately, Focus Area model has downsides as not
Maturity level 1 in Focus Area model cannot ensure matu- proper requirements. For example, requirement "a software
rity level 1 in BTopham maturity model because requirements build is created manually" are always fulfilled because it
"automation process is documented and partially automated", cannot be done in a worse way. Furthermore, requirements
"regular sync meetings are held", "there is frequent commu- for release alignment focus area are not correct because they
nication between the teams" are not fulfilled. overlap each other.
67
VI. C OMPARISON OF S AMER I. M OHAMED AND F OCUS Even though requirements related to defect track-
A REA MATURITY MODELS ing/management are met in maturity level 3, requirement
A. Samer I. Mohamed model maturity assessment for documentation still cannot be fulfilled. For this reason,
maturity level 3 in Focus Area model cannot assure maturity
The results of BTopham and Samer I. Mohamed maturity
level 1 in Samer I. Mohamed model.
assessments are identical. The reason being is that both models
The assessment of maturity levels 4-10 is identical to
do not define any requirements related to requirements and
assessment of maturity level 3. In short, maturity level 1
incidents gathering.
in Focus Area model assures maturity level 1 in Samer I.
In short, Samer I. Mohamed maturity model can assure only
Mohamed maturity model.
maturity level 0 in accordance with Focus Area model.
D. Focus Area model capability assessment
B. Samer I. Mohamed model capability assessment
The result of Focus Area capability assessment is provided
The result of Focus Area capability assessment is provided
in Table IV.
in Table III.
TABLE IV
TABLE III T HE RESULT OF F OCUS A REA MODEL CAPABILITY ASSESSMENT
T HE RESULT OF S AMER I. M OHAMED MODEL CAPABILITY ASSESSMENT
Achieved
Achieved Maximum Aspect
Focus area capability level
capability level capability level
Communication 3
Communication E E
Automation 1
Knowledge sharing A D
Governance 2
Trust and respect A C
Quality 2
Team organization A D
Release alignment C C
Release heartbeat - F E. Disadvantages of Samer I. Mohamed and Focus Area
Branch and merge A D maturity models
Build automation B C Assessments have shown that Samer I. Mohamed maturity
Development quality model has the same drawbacks as BTopham maturity model.
A E
improvement
Test automation - E C ONCLUSIONS
Deployment automation C D The assessments have shown that considered models per-
Release for production - D
ceive DevOps in a different manner. This conclusion can be
Incident handling - D
justified by the fact that the highest maturity level in BTopham
Configuration management A C
and Samer I. Mohamed maturity models corresponds to the
Architecture alignment A B
Infrastructure A D
lowest maturity level in Focus Area maturity model and
vice versa. Nevertheless, communication perception is similar.
Assessments have exposed that all considered maturity models
As shown in Table III, the assessment result is similar
have weaknesses.
comparing with BTopham model. Almost all achieved capa-
bility levels are the same except for knowledge sharing, team R EFERENCES
organization, and development quality improvement focus [1] Stasys Peldžius, “Software process assessment using multiple process
areas. assessment models,” Ph.D. dissertation, Vilnius University, 2014, (in
Lithuanian).
C. Focus Area model maturity assessment [2] W. S. Humphrey, W. Sweet, R. Edwards, G. LaCroix, M. Owens, and
H. Schulz, “A method for assessing the software engineering capability
The assessment of Focus Area model maturity allows to of contractors,” 1987.
determine which level of maturity in accordance with Samer [3] R. de Feijter, R. van Vliet, E. Jagroep, S. Overbeek, and S. Brinkkemper,
I. Mohamed maturity model each level can assure. “Towards the adoption of devops in software product organizations: A
maturity model approach,” Department of Information and Computing
Maturity level 0 in Focus Area model is analogous to Sciences Utrecht University, Utrecht, The Netherlands, Tech. Rep. UU-
maturity level 1 in Samer I. Mohamed maturity model because CS-2017-009, 2017.
both maturity levels do not have any requirements. [4] J. Roche, “Adopting DevOps Practices in Quality Assurance,” Commun.
ACM, vol. 56, no. 11, pp. 38–43, Nov. 2013.
Maturity level 1 in Focus area cannot ensure maturity [5] S. Inbar, S. Yaniv, P. Gil, S. Eran, S. Ilan, K. Olga,
level 2 in BTopham maturity model because requirements and S. Ravi, “DevOps and OpsDev: How Maturity Model
"automation process is documented but not yet executed as a Works,” https://community.softwaregrp.com/t5/All-About-the-Apps/
DevOps-and-OpsDev-How-Maturity-Model-Works/ba-p/306787, 2013,
standard", "defect tracking/management is done using proper [Accessed: 2018-06-15].
tools" are not satisfied. [6] C. P. Team, “CMMI for Development, version 1.2,” 2006.
The assessment of maturity level 2 corresponds to previous [7] ISO/IEC 15504 – 1 Information technology – Process Assessment – Part
1: Concepts and vocabulary, “ISO/IEC,” 2004.
assessment because Focus Area model does not have require- [8] Ozcan Top, Ozden, “Software Agility Assessment Reference Model v3.0
ments related to documentation and defect tracking. (AgilityMOD),” 2014.
68
[9] S. I. Mohamed, “DevOps shifting software engineering strategy-value
based perspective,” International Journal of Computer Engineering,
vol. 17, no. 2, pp. 51–57, 2015.
69