-Since software development using DevOps is still a new phenomenon, analysis of DevOps maturity models is insufficient. The purpose of this work is to investigate whether perception of DevOps is similar in different models. Models that have been investigated are as follows: BTopham, Samer I. Mohamed, and Focus Area models. In order to compare the models, assessments of maturity and capability assured by the models have been performed. The results have shown that the examined models are different in terms of their interpretation of DevOps.
Software process is a set of interrelated or interacting
activities that are performed while creating a software product.
Although software industry has improved significantly over the
last few decades, many software companies face such
problems as projects being behind schedule, exceeding the budget,
customer dissatisfaction with product quality [
New software development methods are emerging over time. One of the rapidly growing phenomena is DevOps, the primary goal of which is to bridge the gap between development and operations. That can be achieved by combining the objectives of different disciplines and tools, forcing interdisciplinary professionals to communicate frequently.
Since software development using DevOps practices is a quite new concern, there is a lack of analysis related to DevOps maturity models. The purpose of this paper is to examine whether DevOps concept can be interpreted the same way in various DevOps maturity models.
This paper is organized as follows. Section II describes DevOps. Section III presents existing DevOps maturity models. Section IV defines comparison method which is used when assessing models in sections V-VI. ©2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0)
The literature review has shown that definition of DevOps is
ambiguous [
Today DevOps covers culture, collaboration, automation,
lean practices, continuous improvement and delivery, user
satisfaction. Culture can be interpreted as shared goals and
values, responsibility and effortless communication.
Collaboration is about adopting cross-functional teams. Automation
involves building, testing, and deployment. Lean practices aim
to eliminate waste. DevOps also encompasses practices related
to monitoring and measurement, which result in continuous
improvement. Another purpose of DevOps concerns releasing
software and reacting on feedback faster [
This section provides an overview of three considered models as follows: BTopham, Samer I. Mohamed, and Focus Area maturity models. From all DevOps models that can be found, only three DevOps models were selected for further investigation. The reason being is that they are more comprehensive in comparison with other models.
Shani Inbar, Sayers Yaniv, Pearl Gil, Schitzer Eran, Shufer
Ilan, Kogan Olga, Srinivasan Ravi present DevOps maturity
model which consists of five maturity levels [
Authors define three dimensions - process, automation, and
collaboration. It is essential to mention that term "dimension"
is not used correctly in this context as it has a different
meaning in traditional maturity models. In this case, dimension
is an equivalent term to process area in CMMI [
Even though BTopham maturity model is designed to cover the entire life-cycle of an application or a service for large enterprises, this model lacks clarity and specificity. This model can be rather perceived as guidelines which state general ideas for software process improvement.
Samer I. Mohamed maturity model presents an improved version of the previously examined BTopham maturity model [9]. This model has identical five maturity levels, but it defines slightly different aspects as follows: communication, automation, quality, and governance.
Communication refers to effective communication between teams. Automation specifies improvement of delivery speed, throughput, and repeatability. Quality is about delivering in more lean and faster manner while governance is responsible for controlling how those aspects work seamlessly together.
Although this model is an improved version of BTopham maturity model, disadvantages remain the same. Both models are quite abstract in order to assess maturity of real-life organization.
Rico de Feijter, Rob van Vliet, Erik Jagroep, Sietse
Overbeek, Sjaak Brinkkemper present another DevOps maturity
model which is the largest and the most comprehensive model
of all models that have been found [
Focus Area model is a non-traditional model because of its focus area architecture. In this context, staged and continuous representations are traditional. Focus area architecture is different for two reasons. First of all, there can be unlimited quantity of maturity and capability levels. Secondly, each focus area (which is a synonym for process area in CMMI maturity model) has different evaluation intervals. For example, possible capability levels for communication focus area starts from A to E, while configuration management can be assessed from A to C. For this reason, meaning of C capability level is different in both contexts. Even though focus areas are named differently in traditional maturity models, original term in this paper will be used instead. This is because of focus area architecture which is quite specific.
This model describes sixteen focus areas which must be taken into consideration while trying to adopt or improve DevOps practices in company. All focus areas are logically grouped into three groups: 1) culture and communication, 2) product, process, and quality, 3) foundation.
In order to determine how different DevOps maturity models perceive DevOps, comparison of maturity models has been performed. That comparison is accomplished by assessing maturity models in accordance with Focus Area model.
Model assessment is composed of several steps. Firstly, maturity assessment of selected model, for example, model X, is done. This assessment is performed as follows. First of all, it is assumed that the company meets the requirements of a certain maturity level of the selected model X. Based on those requirements, the assessment is carried out in accordance with Focus Area assessment method. It gives information about maturity level that can be achieved in accordance with Focus Area maturity model. The assessment is performed until all maturity levels of the selected model X are assessed.
After maturity assessment, the capability assessment is performed. This assessment is based on focus areas instead of maturity levels as in maturity assessment. It gives information about capability level for each focus area that can be assured by the highest maturity level of model X. If a certain level of capability is not achieved, the higher capability levels are no longer assessed. This assessment provides more information about the model being assessed.
Finally, reverse assessments are performed. Reverse assessments are analogous to previously mentioned assessments. The only difference is that it is based not on Focus Area maturity model but on the selected model X.
All assessments rely not only on model requirements that were presented but also on requirements that are established and well known in IT industry nowadays. For instance, version control system usage. If assessments were based only on the explicitly stated requirements, the results would be slightly worse. In fact, the purpose of this assessment is to get results that are as realistic as possible.
V. COMPARISON OF BTOPHAM AND FOCUS AREA
MATURITY MODELS
The assessment of BTopham model maturity allows to determine which level of maturity in accordance with Focus Area model each maturity level can assure.
Assessment begins with the first maturity level. BTopham model does not have any requirements for the first maturity level. For this reason, each organization is at this level by default. Focus Area maturity model defines maturity level 0, which is equivalent to maturity level 1 in BTopham model as there is no focus area assigned.
One of the Focus Area model requirements for maturity level 1 is “functional and non-functional requirements and incidents are gathered from and prioritized with internal stakeholders and customers”. However, BTopham maturity model does not provide any information about requirements and incidents. For this reason, maturity level 2 in BTopham maturity model cannot assure maturity level 1 in Focus Area model. Oddly, all other requirements for maturity level 1 are met.
The assessment of maturity levels 3 - 5 corresponds to assessment of maturity level 2 because previously mentioned requirement still cannot be satisfied.
In short, the results of BTopham model maturity assessment show that all maturity levels in BTopham maturity model can ensure only maturity level 0 in the Focus Area model.
The assessment of maturity levels 2-10 corresponds to previous assessment because Focus Area model does not have requirements related to documentation.
As mentioned before, capability assessment allows to gain additional knowledge about models. The assessment result is provided in Table I. A dash indicates that the lowest capability level cannot be assured. Also, maximum capability levels are provided as they are different for each focus area.
As shown in Table I, communication and release alignment focus areas assure the highest capability level. It means that perception of communication is similar in both models. It is worth to mention that requirement for capability level C is incorrect because it is composed of requirements for lower capability levels. Therefore, if lower capabilities (A and B) are achieved, the highest capability level C will be always achieved as well.
The assessment of Focus Area model maturity allows to determine which level of maturity in accordance with BTopham maturity model each maturity level can assure.
As mentioned before, Focus Area model does not have any requirements for maturity level 0 because there are no focus areas assigned to this maturity level. For this reason, maturity level 0 in Focus Area maturity model is analogous to maturity level 1 in BTopham maturity model.
Maturity level 1 in Focus Area model cannot ensure maturity level 1 in BTopham maturity model because requirements "automation process is documented and partially automated", "regular sync meetings are held", "there is frequent communication between the teams" are not fulfilled.
The result of Focus Area capability assessment is provided in Table II. The highest capability level that can be achieved is 5 as model provides exact five levels. It is worth to mention that in traditional models the lowest capability level is 0 which means that aspect is either not performed or partially performed. In this case, the lowest capability is 1.
The obtained results have shown that BTopham and Focus Area models are contrastive. First of all, a few requirements in BTopham maturity model are abstract, making it difficult to check whether the requirement is met, while the Focus Area model provides very specific requirements. For instance, Focus Area model requires manual code quality monitoring and examples of how it can be done are provided, while BTopham maturity model requires that the quality of the overall process should be measured but it is not clear which areas and how to measure.
What is more, requirements that can be checked, such as standardization or documentation of the process, are not required in Focus Area model. Assessments have shown that BTopham maturity model has some drawbacks. For instance, requirements for higher maturity levels can be satisfied even though requirements for lower maturity levels are not met. Meanwhile, the absolute majority of the requirements in Focus Area model are formulated so that higher level requirements cannot be met unless lower level requirements are satisfied.
Another disadvantage of BTopham maturity model is that this model does not define the necessary level implementation of the requirements for maturity level. Traditional models do not require complete implementation of the requirements. For example, ISO/IEC 15504 states that the process attribute is “fully achieved” if at least 86% requirements are met, while CMMI requires no fundamental shortcomings.
Unfortunately, Focus Area model has downsides as not proper requirements. For example, requirement "a software build is created manually" are always fulfilled because it cannot be done in a worse way. Furthermore, requirements for release alignment focus area are not correct because they overlap each other.
VI. COMPARISON OF SAMER I. MOHAMED AND FOCUS
AREA MATURITY MODELS
The results of BTopham and Samer I. Mohamed maturity assessments are identical. The reason being is that both models do not define any requirements related to requirements and incidents gathering.
In short, Samer I. Mohamed maturity model can assure only maturity level 0 in accordance with Focus Area model.
The result of Focus Area capability assessment is provided in Table III. Even though requirements related to defect tracking/management are met in maturity level 3, requirement for documentation still cannot be fulfilled. For this reason, maturity level 3 in Focus Area model cannot assure maturity level 1 in Samer I. Mohamed model.
The assessment of maturity levels 4-10 is identical to assessment of maturity level 3. In short, maturity level 1 in Focus Area model assures maturity level 1 in Samer I. Mohamed maturity model.
The result of Focus Area capability assessment is provided in Table IV.
As shown in Table III, the assessment result is similar comparing with BTopham model. Almost all achieved capability levels are the same except for knowledge sharing, team organization, and development quality improvement focus areas.
The assessment of Focus Area model maturity allows to determine which level of maturity in accordance with Samer I. Mohamed maturity model each level can assure.
Maturity level 0 in Focus Area model is analogous to maturity level 1 in Samer I. Mohamed maturity model because both maturity levels do not have any requirements.
Maturity level 1 in Focus area cannot ensure maturity level 2 in BTopham maturity model because requirements "automation process is documented but not yet executed as a standard", "defect tracking/management is done using proper tools" are not satisfied.
The assessment of maturity level 2 corresponds to previous assessment because Focus Area model does not have requirements related to documentation and defect tracking.
Aspect Communication Automation Governance Quality
Achieved capability level 3 1 2 2
Assessments have shown that Samer I. Mohamed maturity model has the same drawbacks as BTopham maturity model.
The assessments have shown that considered models perceive DevOps in a different manner. This conclusion can be justified by the fact that the highest maturity level in BTopham and Samer I. Mohamed maturity models corresponds to the lowest maturity level in Focus Area maturity model and vice versa. Nevertheless, communication perception is similar. Assessments have exposed that all considered maturity models have weaknesses. [9] S. I. Mohamed, “DevOps shifting software engineering strategy-value based perspective,” International Journal of Computer Engineering, vol. 17, no. 2, pp. 51–57, 2015.