Large, embedded software development companies increasingly often transform from V-model development to Agile practices, as they want to increase their customer responsiveness. Their teams transform and they are faced with the new challenges on how to measure progress, quality and scope over time. Their management evolve and need new kinds of dashboards to address their information need and ease decision formulation processes. The goal of this paper is to identify the set of measures and indicators important for Agile embedded software development based on SAFe. We studied a large automotive company and identi ed the information needs of their teams and SAFe release train management. The results show that the three main areas to monitor are: scope creep, defects carried over to integration and integration status. Based on the results of our work, some of the elicited information needs and measures were implemented in forms of dashboards (which we present in the paper). By comparing to the existing literature, we concluded that the set of measures prescribed by SAFe is not su cient in practice and needs measures relating to scope creep, defects carried over to integration and integration status.
Agile software development gained high popularity in di erent software
development domains, starting from web development and now becoming increasingly
popular even in the embedded systems domain. One of the modern
achievements is continuous software integration and deployment. They aim to improve
the quality of software products and their availability to the market [
Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
data analytics and optimization of software development towards faster
deliveries. Agile metrics are an important part of that work as they provide insight into
the progress and quality of software product development; they also provide a
means of communication within Agile organizations [
Automotive software companies are no exception, although their context is
more challenging than, for example, the Web 2.0 companies, since:
{ their software needs to ful ll safety critical requirements (e.g. ISO/IEC 26262
standard [
These challenges resulted in the development of Agile-based software
development methods for these companies. SAFe [
Therefore, we set o to study the practice of applying SAFe, in particular we explored what the information needs of Agile software organizations in the embedded software development are. In this paper, we report on the results of studying one large Swedish automotive OEM. We address the research question:
What are the information needs of SAFe teams and train management in the automotive domain?
Our work follows the design science research methodology [
The results show that the main information needs are: { scope creep { new functionality added to the team's backlog after the start of a sprint or program increment, { defects carried over into integration { defects which are not discovered in earlier phases and result in decreasing the speed of integration, and { integration status { availability of test equipment, stability of builds and defect turnaround time.
Based on the results, we concluded that the standard, prescribed, measures of SAFe do not address these information needs su ciently. They do not provide the necessary insight for the teams and the management. Therefore, in practice, new measures,indicators and visualization dashboards need to be added.
This paper is structured as follows. Section 2 presents an overview of the work in SAFe process management, dashboards for Agile teams in general and automotive software measurement. Section 3 presents the measures that are prescribed by SAFe. Section 4 presents the design of our study and Section 5 describes the results. Finally, Section 6 and Section 7 discuss the validity of our study and our conclusions. 2
Measures for Agile software development have been studied in several cases, and
the main consensus is that the standard measures must be complemented with
the domain speci c ones. For example, Meding [
In order to nd this kind of combination of measures, we can explore the
measures for DevOps, as they combine the development and operation measures.
An example of a study on DevOps, which has been done this in the domain of
development of nance systems, has been done by Huijgens et al. [
On the other hand, there are studies that show that standard dashboards
are not su cient, e.g. Liechti [
SAFe development methodology recommends a number of measures,
categorized in four areas: lean portfolio, program, large solution, team. The full set of
measures is available in the process documentation at [
The most important area for our work is from all areas of the category of team measures and from the solution train performance (STP) area of the large solution category. They are presented in Table 2
The measures which we assessed as less relevant were measures related to innovation accounting (e.g. number of customer visits on site), self-assessments (e.g. stakeholder engagement), and pipiline e ciency (e.g. validation on staging).
During the problem awareness identi cation phase, we understood that these
measures may not be su cient, as in our previous work, agile teams in other
companies were asking for more product-oriented measures (e.g. architecture
stability [
We use the design science research method to understand the information needs, to design the dashboards (mock-ups) and to evaluate these mock-ups. In this study, we set o to explore the research question of What are the information needs of SAFe teams and train management in the automotive domain? 4.1
Problem awareness, case and subject selection
In order to address the question, we selected a case company which is an
automotive OEM from Sweden. The OEM is developing software both in-house
and through suppliers, which is a standard way for this market. The OEM has
transformed its operations from the classical, automotive V-model based
development [
The studied company was large, which meant that there were several roles involved in software development, integration, testing, deployment and management. Therefore our study was based on a number of workshops with di erent stakeholders. We selected all stakeholders based on their experience and a speci c information need that they could have (which we elicited beforehand through discussions conducted at the company by one of the authors).
To build awareness of the problem with the existing measures and to elicit information needs from the stakeholders, we conducted a series of workshops. Workshop 1: All stakeholders First we conducted a workshop with all stakeholders, in total 22 persons, with the roles representing: software product managers, software designers, software teams, quality management and release management. The goal of the workshop was to understand the diversity of the information needs. The main method for data collection was the so-called brainwriting, where each participant prepared a set of post-it notes about their information needs and then presented them to the entire group. In this way, we could avoid the problems of dominating the discussion by the most active participants.
During the workshop, we also grouped the information needs and evaluated the groups with the workshop participants. We documented the results by taking notes of the discussions (one of the authors) and by noting the groups of information needs elicited by the workshop.
Towards the end of the workshop, we also agreed on which group of information needs should be discussed next and which group of stakeholders was the most relevant for that discussion. We identi ed which stakeholders should be invited to the subsequent workshops.
Thematic workshops We conducted six thematic workshops with groups of stakeholders, focused on one speci c area identi ed in Workshop 1. Each thematic workshop was conducted with a smaller number of stakeholders, in order to capture the information needs of the right stakholders and not to discuss issues that are out-of-scope for that particular reason. The criteria for selecting the stakeholders were that the stakeholder should: { have the mandate to react upon the measures (e.g. be line manager responsible for integration), { have the ability to make change (e.g. assign extra resources, reduce scope), and { be recognized as the responsible for that particular area of interest (e.g. release train).
SAFe train management (also referred to as Program Manager) was represented by one stakeholder, who had a number of years of experience with working within the organization. He has been working both before and after the transformation to SAFe. The development team was represented by two stakeholders, one of them was an experience engineer working for a number of years at the company, and the second was a junior engineer with experience with software measurement.
The integration and test team was represented by three persons who had similar background to the Program Manager, in particular they worked at the company before and after the transformation from the V-model to SAFe. 4.2
In order to organize the work with prototype development, we used the process
of developing measurement systems from [
The model provided a framework to organize the collected data and to organize the results into dashboards and measurement systems, which were used at the company.
Need porbojebcletimvess, goals, risksand
characterized by measuring itsatributes IPnrfoodrmucattion itTmnhhfaeoetarsmsouauarttitecsiomofinmeesnneetthoepefdrtoshceess Interpretation iitEnnhxffeooprrmlammneaaaatttiisooiounnnreinnrmeeteelhadnetsitniunginsdteitochrsaeqtuolaarnntogtiuttaahtgeiveeof
Variable assigned avaluebyapplying
derived measures (aMnaoldyseils) caAonlgmdobdriietnhcimnisgifoomnrecaristeurrieas Derived Measure Measurement Function Base Measure Measurement
Method Attribute
Variable assigned avalueby applying the measurement function totwo ormore values ofbase measures
Algorithmforcombining two or morebase measures Variable assigned avalueby applying the method toone atribute Operations mapping an atribute toanumber
Property relevant to information needs The analysis methods for all of the evaluations was that one person was conducting the evaluation and two researchers were taking notes. Then one of the researchers compared the notes and summarized the results.
In the workshop with all stakeholders, we used post-it notes and thematic groupings as the analysis methods. For the thematic workshops, in the analysis of notes we used thematic analysis, where we used the results of the rst workshop as the themes.
Each dashboard was evaluated by the stakeholder. We presented the dashboards and the stakeholder was asked to assess whether it ful lled his/her information needs. If the needs were not ful lled, a change request was made. An example of such a case was wrong ltering criteria in one of the dashboards { instead of showing test progress since the last test run, the dashboard showed the cumulative one. The dashboards were also presented for other companies during a workshop (10 companies present ranging from medium to large size, all developing embedded systems and all using variations of Agile software development). 5
Based on the rst workshop, we identi ed three categories of information needs: { Scope creep: in large organizations there is a tendency to understand Agile as a exible way of working and lack of planning, instead of the exibility to plan the sprints and do not change the scope within the sprints. Therefore, in the workshop we identi ed the need to quantify the work that is added to the team's backlog during an ongoing sprint. { Defects into integration: for safety critical systems there is a need for stringent integration and testing phases, in order to secure the safety of the software product. Therefore a high quality software which is integrated and tested is important for the continuous delivery and deployment; if the software does not meet quality requirements, defects are reported and removed, which slows down software development. { Integration status: in distributed organizations, the teams can deliver software many times during the day and it is important that they can do that, in particular it is important that the continuous integration toolchain is working fast and without problems. Therefore, we identi ed the need to monitor the availability and speed of tools used for integration.
In this section we group the results from all workshops per category above and present them per category. 5.1
We identi ed two stakeholders who have the interest, mandate and ability to react upon the indicators of scope creep: product manager (ProdMan) and development team (DevTeam). Their information needs are partially complementary.
The results are presented in Table 3. Stakeholder Information Need Measure(s) ProdMan, DevTeam What is the status of our eld Number of Problem Reports (PR) tests? from the eld tests ProdMan, DevTeam What is our status of problems Number of PRs from customer from the previous scope (from customer) ProdMan, DevTeam What is our status of legacy Number of internal PRs (previous defects? release defects) ProdMan, DevTeam What is our planning accu- Di erence between estimated deracy? velopment time and actual development time ProdMan, DevTeam How much do we support Number of resolved issues (internal other teams? defects) + burn-up + uctuations in velocity ProdMan, DevTeam How much do we support Number of new issues in internal other teams? defect reporting tool DevTeam How many changes in the de- Number of changes introduced to velopment environment do we tooling experience? DevTeam What is the status of our qual- Number of open problem reports ity journal? DevTeam How dependent are we on Expert assessment other teams?
We found that the company provided the teams with the support for di erent tools for categorizing di erent types of defects, which are called problem reports at the company: one for team's defects that need to be resolved within the development organization, another for defects reported by other organizations (e.g. testers), and the third for the defects reported by the customers and from eld testing. The di erent phases, where the defects were found, were also used to provide them with the weight { from 1 point for the internal defects to 100 points for the ones reported from eld or from the customers.
We have also found that dependency on other teams cannot be quanti ed given the current set-up and therefore the information need can be satis ed only through the expert assessment at this time.
The dashboard for the rst information need is presented in Figure 2.
The gure shows how the stakeholders monitor the problem reports { both as individual numbers and as trends. Clicking on each of the widgets leads to the details of which problem reports are counted and their details. 5.2
Defects carried over into integration Three roles were identi ed as stakeholders for the category of defects into integration: integrators (engineers who are responsible for integrating software components into subsystems and software with hardware), testers (responsible for the development, execution and monitoring of integration, system, regression and function tests), and the development team (DevTeam, responsible for the design and implementation of software requirements).
The information needs and the measures are provided in Table 4.
We have found that this category was very important for both the entire company, and for the particular stakeholders. The measures in this category monitor the quality of the software and the potential risk that the software requires re-work and thus increased costs. The number of elicited information needs was higher than in the previous category, becuase of the higher number of stakeholders.
We can also observe that, compared to the standard SAFe measures, these information needs and measures are related to product and organization/infrastructure status. The audience is more towards the technical side of the development organization, not the project or product management.
The dashboard which realizes these information needs is presented in Figure 3. The numbers with orange background correspond to the information needs realized: { What is our test status? { widgets 1, 2 and 3, { What is the status of our test rigs? { widgets 4, 5, and 6, and { What is the availability of our tools? { widget 7.
Clicking on an area in the dashboard leads to details related to the measure, e.g. rig availability over time. rsou sah e i g h t : p o i n t s 1 0 0
W Y e e k X
W e e k X n u m b e r e i g h t : p o i n t s 1 W e e k b e r n X u m 1 2 3 4 5 se se se se se tsca tsca tsca tsca tsca
Te Te Te Te Te 2 g i R 3 g i
R 4 6 5 n o i t a r g e t n i o t n i s t c e f e d r o f d r a o b h s a D . 3 . g i F Uptime / average time between fails What is our test status? Response time for server What is our test status? Test results over time (per sw. re
vision) What is our test status? Requirements coverage over time
(per sw. revision) What is our backlog? Average number of sw. defects over
time What is the extra workload in Burn-up over time (work items not our sprints? planned) What is our planning accu- Schedule slippage racy? What is our release speed?
Time between the designer's readiness of model and model's release What is our test status? Number of smoke tests executed What is our test status? Number of scope tests executed What is the status of our test Test rig availability rigs? What is the quality of our in- Code commits/broken builds tegration? What is the status of our re- Status of release steps per sprint lease? What is our backlog?
Number of changed Electronic
Control Units (to be integrated) What is our integration speed? Total lead time from model to in
tegrated code What is our test status? Number of passed regression test
cases What is our test speed? Execution time per test case What is our defect resolution Defect resolution time speed?
The dashboard provides an overview over the status of the defects carried over to the integration. It shows the status of the testing equipment, the test progress and the test status for the latest few tests.
The version of the tools used, which ful lls the second information need, is presented in Figure 4.
ECU 1 ECU 3
One of the control units has a software that has not been updated for a long time, and therefore the status for this control unit is red. The stakeholder for the integration status is the program manager. The results are presented in 5.
Stakeholder ProgMan ProgMan ProgMan ProgMan ProgMan
We have found that the integration status is important from two perspectives { the availability of the integration toolset (which was already discussed in the previous category { defects into integration), and the speed. The information need of speed related to the management's need to understand whether the organization is performing with the optimal scope and optimal speed compared to the resources. 6
To discuss the validity of our study, we use the framework advocated by Wohlin [19].
The main threat to the construct validity of our study was the fact that we used workshops as the main data collection procedure. To minimize this threat, we developed prototypes which were used at the company (examples presented in the paper), to check whether the results are relevant.
One of the main threats to the internal validity is the fact that we did not use recordings or transcripts for the workshops. We chose this as the better option as it provided us with the more open environment. Our countermeasure to avoid the bias in notes was that two authors took notes independently.
In the conclusion validity category, we minimized the threats by conducting analyses independently by two authors. We collected the notes independently and one of the authors checked the consistency between these notes.
Our study has been conducted at one company only, which creates the threat to the external validity about the generalizability of the results. In order to minimize the threat to validity, we presented the results to other companies as part of our research project. 7
In this paper we explored the information needs of an organization developing, integrating, testing and deploying embedded, safety-critical software. We started by analyzing the existing measures prescribed by the process adopted by the organization { SAFe. We have found that the prescribed measures are not su cient, they lack the focus on product and focus on the process.
Our design research resulted in the identi cation of 28 new information needs.
These information needs were ful lled by a similar number of measures (some
information needs required expert assessment instead of measurement). These
results are aligned with the previous work of our team [
In the further work, we envision the study of more organizations adopting similar processes and designing a portfolio of Quality Measure Elements (according to the templates of the ISO/IEC 25000 standards). These measures could be used as prescribed in the SAFe processes.