Context. DevOps is about improving the software delivery performance. Lord Kelvin stated: “ If you can not measure it, you can not improve it.” For this reason, well-defined measurement is particularly important when using DevOps. Method. This paper critically examines the measurement process outlined in the ISO/IEC/IEEE-32675 DevOps standard and proposes several improvements based on measurement-related standards (ISO-15939, ISO-25021, VIM, etc.). Findings. The measures (base or derived) stated in the DevOps standard are neither characterized nor defined, as described in several standards. Some measures given as examples are ambiguous. Also, the DevOps standard claims to follow ISO/IEC/IEEE-12207, but some expected measures are missing (e.g. efort and size). This article proposes a way to define the diferent measures and explains why having efort and size measures is important in the context of DevOps. Conclusion. The actual measurement process, as described in ISO/IEC/IEEE-32675, could be improved by better defining the base measures, including efort and size, using a measurement method, defining the derived measures using a measurement function, and avoiding ambiguous measures.
CEUR ceur-ws.org (J. DESHARNAIS) CEUR Workshop Proceedings
The DevOps standard cites ISO/IEC/IEEE 12207:2017 [
These measures provide insights into the DevOps process. However, our analysis reveals a lack of explicit definitions and the use of standardized measurement methods. The absence of such details hinders the comparability and reproducibility of results across diferent organizations (see section 1.4). Furthermore, several implicit measures can be found under a single term. This ambiguity can make it dificult to determine the exact object being measured.
The maturity of the software process relies on key relationships in the measurement
information model as part of the ISO-15939 standards [
The information model indicates the construction steps to obtain a data analysis model.
This approach can be found in the following standards: ISO-15939 [
It is not uncommon that, in the software field, we tend to start by suggesting a certain number
of derived measures without taking the time to define the base measures. For example, the
software quality measure ISO-9126 [
To ensure clarity and establish a common ground for discussion, we provide some concept definitions. A more extensive list of definitions regarding the measurement elements of these three perspectives can be found in Annex A of the ISO-15939 standard.
Entity: “It is an object that is to be characterized by measuring its attributes” [
Attribute: The attributes of an entity may or may not be measurable. It can be measured by human or automated means. First, we choose the attributes most relevant to the information needs of the measurement user.
Base measure: A base measure “captures information about a single attribute” [
Measurement method: It “is a logical sequence of operations, described generically, used in
quantifying an attribute with respect to a specified scale ” [
Derived measure: The combination of two or more base measures is a derived measure. We must distinguish algorithms (for example, SQuaRe) from a derived measure which captures information on several attributes (or entities).
Measurement function: It is the formula that joins the base measures. This formula must be clearly defined. This formula must consider the measurement scales. For example, ordinal values cannot be added.
Analysis model: This model combines a number of derived measures linked to decision criteria. For example, improve productivity by X% or even make an estimate with a confidence factor of plus or minus Y%.
Indicators: They are the basis of analysis and decision-making. This is what is presented
to decision-makers. Since “measurement is always based on imperfect information, therefore
quantifying the uncertainty, precision or importance of indicators is an essential part of presenting
the true value of the indicator ” [
According to ISO-15939, the maturity of the software process relies on key relationships in the measurement information model. To interpret the data of an information product, one perspective is to have base measures, derived measures, and indicators closely related (data preparation). The second perspective is about data analysis. The tendency is to look at the data analysis perspective and forget the data preparation perspective. Our aim in this article is data preparation.
According to Figure 1, the measures outlined by DevOps standard show that: A. The measures are not characterized (e.g. base measure, derived measure, or indicator). B. The base measures are not defined.
C. The measurement functions of derived measures are not defined.
D. Some measures cannot be qualified due to insuficient precision.
E. Some measures are ambiguous.
A. The list does not provide any distinction between the diferent measures (e.g. base measures, derived measures, or indicators). Without characterization, the measures do not allow consistent and reliable data collection.
B. The DevOps standard does not define the base measures. However, it is impossible to use them coherently since organizations could apply diferent definitions of these base measures, resulting in diferent or incomparable results from one organization to another or from one project to another. This article proposes several base measures for DevOps (see Table 3) and suggests improving the measurement process. Finally, we cannot find any measure of size and efort that could be useful for measuring the velocity as defined in Section 4.
C. To create a derived measure, a well-defined measurement function is essential. This function mathematically transforms one or more base measures into the desired derived measure. For instance, the change failure rate is a derived measure calculated using the following measurement function: = represents the change failure rate = represents the number of failures = represents the number of changes
Of the 15 provided measures, the majority are derived measures (for example, MTTR, MTTD) as defined by the ISO-15039 standard, some contain base measures (for example, attack vector details), and some are too imprecise for qualification (example, resource utilization). Some contain two diferent measures (for example, change lead time and volume).
D. Without a precise identification of the measures, it is dificult to reproduce diferent
measurements and interpret them as base measures or in combination with derived measures.
The lack of consistency when using base measures in data collection can afect data preparation
and analysis. Several standards have been proposed for this [
E. Ambiguous measures can lead to misrepresenting, hindering benchmarking, and ofering unclear improvement targets. The change in lead time and volume is an example of an ambiguous measure. First, it combines time and volume, which are two completely diferent concepts. Secondly, there is no clear definition of when the start time begins and where the end time stops to adequately measure the lead time. Lead time is usually defined by the time it takes to go from a customer making a request to the request being satisfied. However, Forsgren et al. who promoted the lead time as one of the four DevOps key measures for measuring performance, separates the lead time into two parts: the time it takes to design and validate a product or feature and the time it takes to deliver the feature to customers. The latter is defined as the product delivery lead time and is the one measured as “the lead time”. This can be misleading for an unadvised reader.
Another example is the “defect rate”. Fenton and Bieman [14, p .450] mention that “The defect density measure is sometimes incorrectly called a defect rate” and it could lead to defining defect rate as the number of defects found during a time period or per software component (e.g. module, API, micro-service, etc.) or as a number of defects found per software size (e.g. per one thousand source lines of code - KSLOC or per one hundred function points). The latter being the derived measure of “defect density”. Separating the measures and defining the base measures and the measurement function would allow a better comprehension of the DevOps performance and benchmarking.
It is not the first time that a standard has proposed several measures that are mainly derived measures (constructed by a measurement function) but with no definition of their related base measures.
The ISO-9126 series was analyzed in 2011 by Desharnais et al. [
Those measures did not cover the size of the product and indirectly covered the efort using the word “time”, which often means “duration”. Size (e.g. functional size) is helpful in the requirements part of the life cycle. Size is also beneficial for establishing the velocity of the development and implementation. According to this definition, DevOps covers all aspects of the life cycle as defined in ISO-12207. The ISO 32765 standard’s sub-section 5.1, Values of DevOps [1, 5.1], mentioned that “DevOps seeks to achieve a balance between velocity and system reliability and stability”.
ISO-32675 defines DevOps as a “ set of principles and practices which enable better communication and collaboration between relevant stakeholders for the purpose of specifying, developing, and operating software and systems products and services, and continuous improvements in all aspects of the life cycle”.
DevOps has diferent concepts. The value of DevOps is the first one: According to ISO-32765,
“DevOps is appropriate for building, packaging, and deploying reliable and secure systems”[
There are also some principles: • Business first means that “DevOps focuses on business and organizational goals ahead of procedural and technical considerations”; • Customer focus: “DevOps takes a customer-centric view, prioritizing and designing work to deliver value to the customer, as well as identifying and managing risk”; • Left-shift means that continuous delivery, testing, security, and quality assurance (QA) practices are integrated earlier in the workflow. • Continuous everything means “using the same practices in development as in operations and sustainment”; • System thinking: DevOps discourages promoting specialists (networking, database specialist, etc.). This means that “in DevOps, taking a comprehensive view encourages technology professionals to fully understand the system from end to end”.
According to the DevOps standard, the life cycle is part of the organizational culture [
Blanco strongly advocates open standards, much like the SI (Système International) units. He
believes that “the adoption of SI units [
Section 6 of ISO-32675 [1, sect. 6] discusses the relationship between software life cycle processes and DevOps. The section covers the agreement processes, the organizational project-enabling processes, the technical management processes, and the technical processes. The measurement processes are part of the technical management process. The definition of the technical management processes is: “to establish and evolve plans, to execute the plans, to assess actual achievement and progress against the plans, and to control execution through to fulfillment ”. Following and controlling the plan to improve it is an important part of the technical management processes. This is why measurement is also important. As Lord Kelvin said, “You can’t improve what you can’t measure”.
We address the following research questions: • RQ1. How can DevOps measures be defined in conformance with software measurement standards? • RQ2. Which base measures can be identified in the ISO-32765 DevOps standards?
For each of the 15 proposed measures in ISO-32675, we should answer the questions in Figure 2.
After applying model in Figure 2, we obtained the results shown in Table 2 and Table 3. Table 2 qualifies each measure outlined in the DevOps standard. It first outlines the type of measure, i.e. if it is a base measure, a derived measure, or if it is undefined and it illustrates the base measures associated if applicable. Table 3 will be explained in sub-section 3.1. Change lead time and volume Change failure rate Defect rate Mean time to recovery (MTTR) Mean time to detection (MTTD) Issue volume (number of issues) and resolution time Time to approval Time to patch vulnerabilities
Type of measure Derived Derived Derived Derived Derived Derived Base & Derived Derived
Derived # 10. 11. 12. 13. 14. 15.
Base measure(s) Number of deployment Time period (hours, days, weeks) Number of changes (volume) Time duration between x and the change being deployed to customers Number of changes causing a failure Number of changes Number of known defects Time period Number of software components Recovery of service duration Number of recoveries Time duration between the detection and the start time of a failure Number of failures Number of issues Duration of resolution Duration of approval Number of items to approve Duration of patching Number of vulnerabilities NA NA NA Logging availability Undefined Retention control compliance Undefined Software assurance requirements Undefined findings count Application trafic Undefined NA Attack vector details (IP, stack trace, Undefined NA time, rate of attack)
Resource utilization Undefined NA In measure 2, change lead time and volume, x can refer to the time when a customer make a request or the time it takes for work to be implemented, tested and delivered into production [13].
The results show that the base measures are not described and therefore not defined and that 6 measures out of 15 can not be qualified: logging availability, retention control appliance, software assurance requirements finding counts, application trafic, attack vector details, and resource utilization.
We also note that there is no mention of the velocity as a measure. The velocity is a derived measure composed of the size over a duration. The measurement function could be, for example, the number of SLOC or the number of function points delivered in a 2-week sprint.
Figure 1 indicates that to obtain a base measure, one must use a measurement method. According to ISO-15939, a measurement method is “a logical sequence of operations, described generically, used in quantifying an attribute with respect to a specified scale” [3, p. 4].
In this section, we analyze each measure and provide, if possible, the name(s) of the base(s)
measure(s) and a definition of each base measure ( 3). The following information will be collected
according to the measurement method as described in [
To apply the measurement function correctly, it is necessary to know the measurement
scale. The scale type is defined in diferent standards [
Table 3 summarizes the collected information according to the list of measures identified in the DevOps standard. Written notification by an authorized representative that an information item appears to satisfy requirements and is complete [18].
The modification of an existing application comprising additions, changes and deletions [18].
Imperfection or deficiency in a work product or a project component. In generic terms, it can refer to a fault or a failure [18].
Phase of a project in which a system is put into operation and
cutover issues are resolved [18, 3.1113]
Termination of the ability of a product to perform a required .
function or its inability to perform within previously
specified limits [
The return to normal operation after a hardware or software failure [18].
Refers to the process of addressing, fixing, or resolving issues, bugs, or problems that have been identified within a software system.
The time in calendar units between the start and finish of a scheduled activity [19].
It signifies the real-world time that elapses, independent of the software development process. A time period could represent a number of hours, days, weeks (or sprints), months, etc.
Weakness in an information system, system security procedures, internal controls, or implementation that could be exploited or triggered by a threat source. [20] Type of scale Ratio Ratio Ratio Ratio Ratio Ratio Ratio Ratio Interval Interval Ratio
Table 3 illustrates that most of the base measures list in table 2 are already defined in diferent known standards.
The measurement process within the DevOps standard does not propose measurement functions. A measurement function “is an algorithm or calculation that combines two or more base measures”. For example, the deployment frequency is the number of deployments for a specific time period (i.e. a specific duration). = represents the deployment frequency; = represents the number of deployments; = represents the time period during which the deployments happened.
In practice, the following example could serve as an expected result: one deployment per week (or per hour, per day, per sprint, per month, per year).
In Table 4, we did propose measurement functions for each of the 9 defined measures.
Mean time to recovery (MTTR) Mean time to detection (MTTD) 7. Issue volume and resolution
time 8. Time to approval 9. Time to patch vulnerabilities
A “software development project” usually has a start and end date consistent with its budget. Applying DevOps, many organizations are adopting the “software development product mode”, where a budget is established for a time period (e.g. the current year, semester, or month). In both modes (project or product), the size of the related software product is required to measure velocity and productivity, whether this size is physical (SLOC) or functional (function points). However, in modern projects and DevOps, it is quite usual that software system components are written in diferent programming languages [ 21, 22]. This makes the number of SLOCs an inappropriate measure of performance unless the performance needs to be measured separately by technology. In most cases, adopting a technology-independent size measurement method, such as COSMIC [23], would be more appropriate. Nonetheless, the size measure is not mentioned directly in the ISO-32675 DevOps standard’s list of measures.
The DevOps standard mentioned “work units” in the “velocity” definition [ 1, p. 14], giving examples such as “story points”, which represent “a largely approximate level of perceived effort” [24]. Other given “work units” examples in this “velocity” definition are also subjective or incomparable values, such as delivered features, functions, user stories, use cases, or requirements (from one of these work units to another, the size may vary greatly). In [13, chap. 2], the velocity based on user story points is declined as a weak subjective measure linked with various unwanted behaviours from team members.
The product’s size and related efort for a given time period must be included to measure productivity. Therefore, the list of proposed measures needs to mention these measures and be updated accordingly. The standard proposes a certain number of measures that are not necessarily linked to each other. A performance model with its characteristics is missing (for example, deployment, security, validation, etc.). This is because, at a high level, we cannot find a model of relationships like the one in quality measures [16]. For this reason, it is dificult to determine whether all measures are relevant or whether some measures are missing. For example, comparing performance between diferent projects and organizations is dificult if there is no common definition of the performance measures. Some performance-related measures are not available to measure velocity and productivity, which are also part of the performance, namely the software development process performance.
Several quality-related measures are proposed in the DevOps standard. It is expected that quality measures be used to improve quality aspects of the software development process and of the software product. In that case, the “Defect density” measure could be added to the DevOps standard, as the number of known defects per software size, such as per 100 COSMIC Function Points (CFP). Also, the evolution of the value of the set of quality-related measures should be looked at over time to observe their variation and confirm if the software process improvements are bringing the expected results.
One can argue that measuring the delivered software’s functional size is a manual task that would take extra efort. Forsgren et al. [ 13, chap. 4] define a list of five (5) DevOps key principles, where the third one relates to the automation of any repetitive manual task. Functional size measurement should also be automated under this key principle. There are available tools that automate software functional size in COSMIC function points from several sources: requirements written in natural language (French and English) [25], Java code developed with the Spring MVC framework [26], and Java code of business applications [27, 28]. Also, current research projects aim to automate functional size measurement from other DevOps artifacts (for example, automated tests and API requirements) [21].
This article aims to answer those two questions:
• RQ1. How can DevOps measures be defined in conformance with software measurement
standards?
• RQ2. Which base measures can be identified in the ISO-32765 DevOps standards?
To answer the question of compliance of measurement standards in software engineering,
we consulted three recognized standards, including two in software engineering [
To answer the second question, we analyzed the proposed measures to find the base measures that could be extracted from the measures listed (Table 2, “Name of the measure” column) and, when possible, define them (Table 3).
We went further and applied, where possible, the measurement function method (Table 4). We found that we could apply the functional measurement method to only sixty percent (60%) of the measures (9 out of 15) listed in the DevOps standard, meaning that 6 measures out of 15 (40%) were too ambiguous to be defined.
Many other measures are proposed for DevOps in diferent publications [ 29, 30, 13, 31]. They propose diferent measures adapted to the immediate situation and do not refer necessarily to their related measurement method and measurement function.
The actual measurement process in the DevOps standard shows that the measurement process
needs to be more mature. To improve this measurement process, it is necessary to:
• Better define the base measures using a measurement method [
DevOps measures can be defined in conformance with software measurement standards. Table 2 shows that DevOps measures can first be categorized as base measures, derived measures, or indicators. Table 3 shows that base measures identified according to Figure 2 can be defined. Finally, Table 4 summarizes the list of base measures identified in the DevOps standard and the measurement functions associated with the ISO-32765 DevOps measure.
When DevOps becomes more mature, it will be necessary to have a global view of the measurement processes like those in SQuaRE [15, 16]. This could take the form of a DevOps measurement model suggesting a number of measures to be calculated automatically. A suggestion of tools could be provided, but not without considering the rapid evolution of tools. In future works, researchers could be interested in applying Blanco’s telemetry idea, using the measurement method and measurement function approaches. [13] N. Forsgren, J. Humble, G. Kim, Accelerate: the science of lean software and DevOps: building and scaling high performing technology organizations, IT Revolution, 2018. [14] N. Fenton, J. Bieman, Software Metrics: A Rigorous and Practical Approach, 3rd ed., Project
Management Institute, 2017. [15] I. O. for Standardization, Iso 25000:2014 systems and software engineering – systems and software quality requirements and evaluation (square), 2014. [16] I. O. for Standardization, Iso 25010:2023 systems and software engineering – systems and software quality requirements and evaluation (square), 2023. [17] D. G. Blanco, Practical OpenTelemetry: Adopting Open Observability Standards Across
Your Organization, Springer, 2023. [18] I. O. for Standardization, Iso 24765:2017 systems and software engineering – vocabulary, 2017. [19] P. M. Institute, A Guide to the Project Management Body of Knowledge (PMBOK® Guide), 6th ed., Project Management Institute, 2017. [20] N. I. of Standards, T. (NIST), vulnerability - glossary | csrc, 2023. URL: https://csrc.nist.gov/ glossary/term/vulnerability. [21] A. Lapointe-Boisvert, S. Mosser, S. Trudel, Towards modelling acceptance tests as a support for software measurement, in: 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), IEEE, 2021, pp. 827–832. doi:10.1109/MODELS- C53483.2021.00129. [22] P. Rani, S. Panichella, M. Leuenberger, A. Di Sorbo, O. Nierstrasz, How to identify class comment types? a multi-language approach for class comment classification, Journal of systems and software 181 (2021) 111047. [23] I. ISO, Iso/iec 19761:2011 software engineering — cosmic: a functional size measurement method, 2011. [24] S. Trudel, Agile & cosmic: A good integration!, 2019. [25] B. Gérançon, S. Trudel, R. Kkambou, S. Robert, Software functional sizing automation from requirements written as triplets, ICSEA 2021 (2021) 33. [26] A. Sahab, S. Trudel, Cosmic functional size automation of java web applications using the spring mvc framework., in: IWSM-Mensura, 2020, p. 7. [27] M. A. Sag, A. Tarhan, Measuring cosmic software size from functional execution traces of java business applications, in: 2014 Joint Conference of the International Workshop on Software Measurement and the International Conference on Software Process and Product Measurement, 2014, pp. 272–281. doi:10.1109/IWSM.Mensura.2014.29. [28] N. Chamkha, A. Sellami, A. Abran, Automated cosmic measurement of java swing applications throughout their development life cycle., in: IWSM-Mensura, 2018, pp. 20–33. [29] G. Kim, J. Humble, P. Debois, J. Willis, N. Forsgren, The DevOps Handbook, Second Edition:
How to Create World-Class DevOps Teams and Culture, 2 ed., IT Revolution Press, 2016. [30] D. Research, A. D. Project, Dora | devops research and assessment, https://dora.dev/, 2024. [31] PuppetLabs, State of devops report. technical report, puppetlabs, 2021.