The increasing adoption of Agile and DevOps has triggered the following question: “How can the performance of Agile/DevOps teams be continuously improved?” In this paper we analyze how combined Agile/DevOps team performance can be appropriately measured and quantified, and identify factors that have an influence on team performance. A literature review was conducted to identify metrics for measuring team performance, combining both Agile and DevOps. Semi-structured interviews were conducted with nine experts in Capgemini, who manage multiple Agile/DevOps teams. They were asked to propose and substantiate factors that are deemed important to team success. These factors are aggregated and validated, and structured in a conceptual model. The resulting performance metrics and factors are compiled into a DMAIC cycle adaptation model, which focuses on providing an actionable process to continuously improve team performance. The method is currently being tested in practice Agil-ISE23: 2nd Intl. Workshop on Agile Methods for Information Systems Engineering, June 13, 2023, Zaragoza, Spain $ thoby.visser@capgemini.com (T. Visser); joris.hulstijn@uni.lu (J. Hulstijn) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CPWrEooUrckResehdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g CEUR Workshop Proceedings (CEUR-WS.org)
Adoption of Agile methodologies has experienced dramatic growth in the last decade [
Teams are central to software development. There are clear diferences in the performance
of teams [
RQ. How can the performance of Agile/DevOps teams be continuously improved?
1. What metrics represent the performance of the Agile/DevOps teams?
2. What factors impact the performance of the Agile/DevOps teams?
This paper only provides a brief summary of [
(1) The problem context is as follows: at GapGemini, like at other IT providers, combined
Agile/DevOps teams are deployed at clients. Teams are managed remotely. Because of the
autonomy of teams in Agile methods, it is hard to monitor performance. Management is
seeking methods to help teams improve their performance. (2) An artefact is developed: an
improvement method. After deliberation, we chose to develop the so called DMAIC framework:
define - measure - analyse - improve - control [
The research proceeds in four steps. First, based on a literature review, the notion of team performance was studied, for both Agile and DevOps, identifying potential factors that influence team performance. Second, based on a review of the literature on metrics in general, and metrics of software development and deployment in particular, a set of potential metrics was proposed. Third, semi-structured interviews were conducted with nine experts in CapGemini, who manage multiple Agile/DevOps teams. These people can be seen as experts, both on the factors that contribute to team performance, and on measurements and monitoring. The interviewees were asked to validate the factors, and also the metrics. Fourth, the resulting set of factors and metrics, were combined in a DMAIC framework, to provide guidance to stakeholders.
The remainder of the paper is structured as follows. Section 2 details factors that have an influence on team performance. Section 3 discusses the set of possible metrics. Finally, Section 4 presents a brief overview of the DMAIC framework. The paper ends with discussion of future research and conclusions. Factor (Test) Automation Expertise (Collaborative) Tools T-Shaped Technical Debt Facilities Business Orientation Technical Overview Retention Collaboration Team Climate Common Goal Seniority Team Diversity Distributed Team Independence Trust (Inside) Soft Skills Respect Workload Focus Transparency Business-IT Alignment Autonomy Planning Stability Technological Innovation Acceptance Project Planning Business Vision Readiness Trustworthiness Facilitation Knowledge Sharing 8
Definition
Technical Factors 38 Automating tasks and processes 26 Relevant technical skills and knowledge 13 Applications used in the Agile/DevOps lifecycle and to collaborate 10 Ability to apply knowledge across situations, and having specific expertise too. 10 Complexity of a software system that makes development more dificult, often
accumulated by choosing simple fixes over suitable solutions 8 A suitable workspace with equipment 8 To take business perspective into account 5 Ability to oversee complex technical landscapes (systems and infrastructure).
Non-technical Factors 41 The time the team members have been together and continue to stay in a team 23 Interpersonal cooperation of team members 20 Culture and environment among the members of a team 16 An objective worked towards by all team members 13 Balance of employer expertise, defined by skill and length of service. 15 Variety in character, expertise and roles. 10 Teams that are geographically spread out. 10 Ability to solve solutions without assistance from outside the team. 8 Belief of reliability, truth, or ability within the team and between its members.
Ability to interact efectively with others. 5 Respect for the other team members. 5 Team members are focused on a single project and not multiple projects. 5 Honesty and openness between team members.
Environment Factors 36 Efective communication and a shared understanding of business and IT. 20 Ability and freedom for teams to manage, govern and organise themselves. 16 No change in planned business requirements during an iteration. 16 The implementation of new technologies. 15 Ability of the organisation to accept Agile/DevOps methodology. 15 Prioritisation, selection and control of the business requirements for projects. 10 Strategic goals, values and aspirations. 10 Prepared to accept change in culture/methodology. 8 Perceived to be reliable, sincere and competent. 5 Be enabled to operate by other departments.
5 Sharing of knowledge and information.
A systematic literature review of the Agile way of working, DevOps, and team performance [11, Ch 2] produced a list of factors, which were discussed in semi-structured interviews. A total of 9 interviews were held. Interviewees were selected among employees of GapGemini because of their role (service coordinator, service delivery manager, project manager) and expertise. Each interviewee manages between 2 and 6 teams. The application domains of the clients are public sector (5 times), retail (1), utilities (1) and energy (1). Client size ranges from 500-2500 (4), 2500-5000 (2) to 5000-77500 (2) employees. The experts were asked to rank the relative importance of factors, on a scale of ⟨10, 5, 3, 2, 1⟩. The outcome is shown in Table 1.
Change failure rate: Time to restore service: Mean time to recovery (MTTR): Defect density:
Deployment frequency: Change lead time: User stories planned versus delivered: Mean time to resolve:
Number of failed deployments in production in a given period.
Average time taken to restore a running application or service since the issue was detected.
Total time taken to recover the applications or services or systems for all failures, divided by number of failures Number of confirmed defects detected in software in a given period, divided by the size of the software or component.
Number of deployments/releases in production in a given period.
The length of time between when a code change is committed/delivered to when
it is deployed to production [
Completed number of story points divided by planned number of story points.
Average time taken to develop a fix and roll out in production.
The most important technical factors are automation, the extent to which tasks and processes are automated in a CI/CD pipeline, and technical skills and knowledge. Consider the following quote: “Automation is important. The automation of processes and the [CI/CD] pipeline. Not only can it save a lot of useful time, but it also makes the team members think of continuous improvement.” (interviewee ). Non-technical factors are seen as relatively more important. Consider factors like retention, collaboration, team climate, and team diversity. Regarding retention, one of the interviewees remarked: “The longer a team is together, the better they will be tuned in to each other. They will be able to better understand each other and know what the others have to ofer. ” (interviewee ). With regard to the environment in which the team must operate, the most important factors are business-IT alignment, and autonomy of teams, according to the interviewees: “There should be a shared understanding of both business and IT. Working with DevOps is not very traditional and undoubtedly requires the business to adapt. Both parties need to be on the same page for it to work.” (interviewee )
Agile methods focus on change [
We conducted a systematic literature review about metrics in general, and metrics for Agile
and DevOps team performance [Ch 2.3][
Given these metrics, it is possible to formulate an equation summarizing team performance, over a given period: () = ( · ()) · ( · ()), where () is performance for team , () and () are throughput and stability metrics respectively, and and are the relative weights for throughput and stability in a given project, such that + = 1.0.
Metrics have to meet certain requirements: they must be relevant to the business and to Agile
and DevOps team performance, they must be measurable, i.e. have standardized values that are
consistent over time [
To make sure that metrics are measured consistently over time and their values have real meaning, the measurement must be well-integrated in existing software development and project management processes, and be taken directly from information systems.
For these reasons, the metric of velocity was not selected in Table 2. Velocity is how much
technical scope a team can develop over a period of time. Velocity is usually expressed in story
points per period, an estimate of the expected efort required to implement a piece of work.
Story points are subjective by nature and therefore do not adhere to the criteria of comparability
and reliability [
The adoption of such metrics was discussed with the experts. Although various interviewees stated that they were already tracking metrics (7 said so), most of them said their use was limited. The metrics that are collected are velocity (mentioned 7 times), change lead time (4), number of defects (1), and employee turnover (1). Several interviewees mentioned that their client has shown interest in establishing metrics to measure performance (5). Not all interviewees support independently measuring team performance: “Metrics do not exhibit trust in the teams, which are supposed to be autonomous and self-organising.” (Interviewee ).
For this research, the DMAIC cycle is used as a framework for facilitating continuous
performance improvement within the context of Agile/DevOps teams. DMAIC is also the methodology
underlying the well known Lean six Sigma quality improvement method. The DMAIC cycle is
chosen over other improvement methodologies – like plan-do-check-act, RADAR or DFSS –
because of the data-driven orientation [
Example: the ‘defects over time’ metric has increased compared to earlier iterations. 3. Analyse: identify problem areas. Select which factors impacting performance can be adjusted and possibly improved (Table 1).
Example (cont.): Analysis shows that the increase in ‘defects over time’ metric is caused by the complexity of the software. The complexity is accumulated by many simple fixes. In response, the ‘technical debt’ factor is identified for improvement. 4. Improve: start projects or interventions to adjust and manage the identified factors.
Example: start a project to reduce complexity, by refactoring the software in crucial modules, replacing quick fixes with a principled solution. 5. Control: verify whether the projects lead to performance improvements, by using the metrics, and whether adjusted factors are controlled and sustained.
Example: ‘defects over time’ and ‘technical debt’ are monitored. If technical debt remains a problem, additional modules can be refactored.
The aim of this research is to find a method to continuously improve team performance in Agile/DevOps teams. A possible method is the DMAIC framework: define, measure, analyze, improve and control. In such continuous improvement methods, team performance must be made measurable and quantifiable. Additionally, factors that may impact the team performance must be identified. Projects can then be started to try and adjust these factors.
Based on a literature study a list of metrics is generated, to measure team performance. The
metrics are obtained from the ‘State of Agile’ report [
Semi-structured interviews are conducted with 9 experts in Capgemini who each manage multiple Agile/DevOps teams. The experts are interviewed on topics like experiences, adoption, team performance and metrics. The experts are asked to propose and substantiate ‘technical’, ‘non-technical’ and ‘environmental’ factors that afect team performance, according to them. These factors are aggregated, ranked and linked to literature. The factors are shown in Table 1.
The resulting performance metrics and factors afecting team performance are combined in the DMAIC cycle, adjusted to the context of Agile/DevOps team performance. A full scale evaluation of the usefulness of the factors and metrics, and the DMAIC framework as a whole, to improve Agile/DevOps team performance, is currently ongoing.
The set-up of this research has important limitations. It is quite possible that a bias results from selecting interviewees from one organization, namely GapGemini. This risk is reduced, because we selected interviewees who manage teams in diferent client organizations. Furthermore, there are big diferences between the situations of the interviewees. They had taken diferent approaches to adopt Agile/DevOps, and each struggled with diferent issues. It seems that no guiding structure was provided to establish Agile/DevOps at the diferent clients. These limitations mean that the framework should be seen as a series of hypotheses.
Measuring the performance of teams goes against the spirit of autonomy that characterizes Agile methods. This may hinder adoption. On the other hand, there is a clear need for guidance and alignment, especially when teams are managed remotely. Further research must point out which metrics will be most popular to be adopted by the teams themselves, which metrics must be obliged, and which metrics must be made part of the CI/CD pipeline tooling.