=Paper=
{{Paper
|id=Vol-1796/cre-paper-2
|storemode=property
|title=Adding Value Every Sprint: A Case Study on Large-Scale Continuous Requirements Engineering
|pdfUrl=https://ceur-ws.org/Vol-1796/cre-paper-2.pdf
|volume=Vol-1796
|authors=Rashidah Kasauli,Eric Knauss,Agneta Nilsson,Sara Klug
|dblpUrl=https://dblp.org/rec/conf/refsq/KasauliKNK17
}}
==Adding Value Every Sprint: A Case Study on Large-Scale Continuous Requirements Engineering==
https://ceur-ws.org/Vol-1796/cre-paper-2.pdf
Adding Value Every Sprint: A Case Study on
Large-Scale Continuous Requirements
Engineering?
Rashidah Kasauli1 , Eric Knauss1 , Agneta Nilsson1 , Sara Klug2
1
Dept. of Computer Science and Engineering
Chalmers | University of Gothenburg, Sweden
rashida@chalmers.se, {eric.knauss,agneta.nilsson}@cse.gu.se,
2
Ericsson AB
sara.klug@ericsson.com
Abstract. Agile development practices, such as continuous integration
and continuous delivery, promise value through shorter time to market
and increased flexibility. While these practices have been widely adopted
in small-scale, they have shown to be challenging to adopt in large-scale,
system development. This is often due to a distance between customer
and developer in large scale systems, and the need to break down value
from the whole system into manageable parts. The notion of value is fun-
damental for agile methods, especially for practices such as continuous
delivery to the customer. However, how value should be handled in devel-
opment practices is not clearly understood. In this paper, we investigate
how the notion of adding value in every sprint has been perceived in a
large-scale system development. Based on an exploratory qualitative case
study, the outcome shows that it is perceived beneficial by practitioners
although it comes at a price and challenges exist.
Keywords: value, continuous requirements engineering, continuous in-
tegration, continuous delivery, large-scale agile
1 Introduction
Agile software development focuses on customer collaboration and the ability
to deliver customer value quickly and incrementally [8]. For this, popular agile
methods such as Scrum [26] and eXtreme Programming (XP) [4] have powerful
planning mechanisms in place. These methods align well with continuous inte-
gration (CI) and continuous delivery (CD) and can lead to substantially higher
productivity [27] and shorter time to market. While most research on agile prac-
tices (such as continuous integration) focuses on the team scope and software
only (e.g. [12]), they are increasingly applied to more and more complex devel-
opment endeavors, such as large software-intensive systems [17, 23].
?
In: Proc. of 3rd WS on Continuous Requirements Engineering, 2017, Essen, Germany.
Copyright 2017 for this paper by its authors. Copying permitted for private and
academic purposes.
To our knowledge, there is a lack of empirical studies investigating large-scale
systems development, its challenges with respect to requirements engineering,
and suitable advice. Particulary, the notion of value and how it is used in sprints
has not been investigated in literature. Yet, it is an important aspect of continu-
ous requirements engineering, especially for organizations transitioning towards
continuous delivery and deployment. In previous work, we found that particu-
larly the distance between developers and customers can introduce challenges to
the notion of value in agile developments [11]. In this paper, we present results
from an explorative, qualitative case study in a large-scale agile development or-
ganization to address this research gap and to investigate the following research
questions:
RQ1 What is the interpretation of value from the perspective of different roles
in large-scale agile software development?
RQ2 What are the effects of the notion of adding value every sprint of
individual teams? What benefits and challenges exist?
RQ3 How do you check if value has been added in each sprint?
RQ4 What improvements could support adding value every sprint in
large-scale continuous software engineering?
The study shows that all interviewees see value in “adding value every sprint”
as well as widely share the notion of value and how it relates to their daily
work. They however reveal a diverse picture on how to check and control if
value is added in the sprint. Despite being positive towards adding value every
sprint, all interviewees see challenges and present constructive suggestions for
improvement.
2 Background
Agile methods have drastically changed software development, relying on people
skills and close customer collaboration to meet ever changing market needs and
requirements rather than formalized processes and contracts as in traditional
methods [5]. Agile development is characterized by short development cycles
[5], face-to-face collaborations [11], continuous integration of code changes into
the product baseline [12, 20], and continuous delivery of working software to
meet customer demands [10]. While there is relatively rich literature on how to
implement and setup agile practices (such as continuous integration) at small-
scale for a project (e.g. [12, 20]), there is very limited scientific support for how
to transfer this to large-scale environments (e.g. [21]).There is however research
that reports on challenges with scaling of continuous integration [9, 24, 25].
With the wide adoption of agile methods, requirements engineering is no
longer confined to the initial phases of software development [13]. Instead it
has become a continuous process in the software development life-cycle [15]. To
successfully evolve products that bring value to customers, continuous require-
ments engineering needs to take into account the different concerns of all the
stakeholders involved in the process or project [15].
Software today has a major influence on most systems’ cost, schedule, and
value [7], therefore a lack of focus on the value can seriously degrade the project
outcome. Agile methods promote the notion of customer value, e.g. by valuing
working software over comprehensive documentation [5] and there is a rich re-
lated research that discusses the creation of customer value as key element for a
company’s success [3, 16, 21].
Related to this concept of customer value, value based requirements engi-
neering (VBRE) has been proposed as an approach [16]. The VBRE approach
uses selection of requirements to enhance the value of a release [3, 28]. VBRE
promotes software developers to align customers’ requirements, business require-
ments, and technological opportunities, to have a sound understanding of both
technical and business implications of decisions made, and to understand the
business dynamics that drive software development [2].
The notion of delivering value at the end of every sprint is the aim of agile
methods [6]. However, different interpretations of the concept of customer value
exist [14], frequently relating customer value to the trade-off between what the
customer receives and what they invest to acquire and use a product [29]. This
definition is based on the customer’s perspective, and to our knowledge there
exists little research on how software development teams can relate to customer
value, especially in large-scale systems development.
While customer value and its role for prioritization in agile projects has
been discussed critically [22], Alahyari et al. have started to investigate this
gap, i.e. how value is interpreted, prioritized, assured, and measured in agile
software development [1]. Based on a qualitative study with 23 participants from
14 organisations, they identified and prioritized value aspects such as delivery
process with respect to time, quality, and knowledge of feature value for customer.
In this paper, we specifically investigate the latter: What value for a customer
is added during a sprint and how do developers relate to it. In accordance with
Alahyari et al., we also investigate the question on how to measure or evaluate
the value added in each sprint. Based on our smaller scope and stronger focus
on one value aspect, we can shed more light on this aspect, e.g. how acceptance
testing and sprint demos can help measuring which value has been added.
3 Research Method
In this paper, we investigate the notion of value and its use in large-scale agile
system development. We employ collaborative practice research [18], which is a
way to organize and conduct research in close collaboration between researchers
and practitioners, drawing on the combined strengths of the practitioners’ way of
thinking and the reflective researchers. We designed the exploratory, qualitative
case study together with a practitioner at the company and identified key areas
and relevant roles for semi-structured interviews. In total we interviewed five
persons, which we selected based on a convenience sampling strategy in order to
cover relevant roles: two system testers, one product owner, one designer, and
one function tester.
In the interview guide we covered the areas: notion of value, effects of adding
value every sprint, how to check if value has been added, and suggestions of
improvements. Each interview was conducted face-to-face by two researchers
and lasted 45-60 minutes. Interviews were recorded and notes were taken.
Case Company: The study is conducted at one unit of a Swedish-based multi-
national organization offering services, software, and infrastructure in informa-
tion and communication technology for telecom operators and other industries.
All interviewees were involved with the development of one specific product.
The unit has worked with continuous integration since 3-4 years, and employs a
scaled agile approach of a feature development model, with prioritized features
and 30-40 cross-functional teams working in parallel with features.
In the data analysis we focused on synthesizing the views from the different
roles regarding 1) notion of value to understand and distinguish the meaning of
this concept, 2) effects mentioned in terms of benefits or challenges with these
practices in their daily work, 3) how they check if value has been added in the
sprint, and 4) suggestions of improvements.
Discussion of Validity: As an exploratory study, the aim of this paper is to
better understand the notion of value and its use in large-scale agile system
development. Based on the limited size of this study, we cannot generalize our
results beyond the scope of our study. Instead our aim is to identify relevant
aspects for future research on a larger scale and with more companies involved.
We validated our findings during a workshop, where we discussed the synthesis
of results from transcripts and notes.
4 Findings
4.1 Interpretations of Value (RQ1)
We asked our interviewees what they viewed as customer value and product
value to investigate if there was a perceived difference between these notions of
value. While we got rather clear answers on customer value, the interpretation
of product value and its relation to customer was more diverse, leading us to
include market value as a third concept of value.
Customer Value: From the perspective of our interviewees, customer value
relates to a change in the product.
“If we add something that the customer wants, but it shall be a change
in the product.” — System Tester
More specifically, this change should relate to something a customer can sell
or that makes their product or service cheaper. Customer value also relates to
the relationship to the customer and become visible in development sprints as
promised features, functionality, quality, configuration, or documentation.
“[. . . ] also building a relationship and getting them involved. When we
start we give them a demo. Then we break down things into small user
stories. Then we discuss the release plan, priorities and the user stories.
So they can influence and participate in the discussion.” — Product
Owner
One interviewee thought that customer value is also about providing them with
the ability to influence and participate in discussions around new features.
“Different customers, they value different things.” — Function Tester
Our interviewees widely agreed that customer value can differ for each customer.
Product Value: In contrast to customer value, most of our interviewees
referred to product value as “something the customer does not see”.
“If you have certification or peer-review. Such a certification could be
product value. Not sellable by itself. But now that I think about it, re-
design and refactoring could be counted as product value. That, we have
a lot.” — Designer
Above activities add value to the product that does not directly relate to a
customer need. However, as many interviewees indicated, an increase of product
value will indirectly lead to customer value.
“Product value can improve development environment and indirectly im-
prove customer value. ” — System Tester
Market Value: Since customer value differs between customers, there is a
risk that following each customer separately will lead to an unnecessarily com-
plicated product.
“We have a lot of discussions on having customer specific solutions.
For the product, it is not always adding value, but instead introduces
complexity. So we spend a lot of time to abstract and prioritize so we do
not blindly do what one customer says.” — Function Tester
To mitigate this risk one needs to abstract from individual customer wishes to
a combined market value that adds value to more than one customer.
4.2 Effects of Adding Value Every Sprint (RQ2)
We were also interested in the effects of adding value every sprint has for our
interviewees. Specifically, we asked about benefits and challenges.
Benefits: All our interviewees saw benefits of focusing on value.
“If you think about sprint goals, and tie to continuous integration - yes it
is beneficial. It helps with these small changes. You are not in your head
thinking about things that you will do in the next year, but only about
the next three weeks.” — Designer
Table 1. Internal and external benefits of adding value every sprint
Type Benefit Characteristic quote
Internal – Improve quality of tests and “Today, we can always set up an integration
feedback test. We are not blocked. We can go back in the
version history of one of the features.”
– Allows to focus on what is – System Tester
important now
External – Reduce risk to build the wrong “When trying to add customer value in such
system small increments, you gain a lot of flexibility
and can steer away from bad directions.”
– Add flexibility to development – Designer
“Distance to customer is largely improved by
– Bridge distance to customer feature development. Of course the customer is
not sitting next to us, but they visit sometimes.
– Get a feature out early and start And it is always clear what we are working for
learning from customer that is customer visible.” – System Tester
Generally, the benefits we saw can be divided into internal and external benefits,
as shown in Table 1.
Challenges: In addition to the clear benefits, our interviewees also men-
tioned some challenges.
High costs of benefits: Our interviewees recognize that firstly, a high invest-
ment was necessary to get the organization to become efficient with adding value
in every sprint. Secondly, it does not necessarily feel like a speedup on team level.
“As a team we spend a lot of time with trouble reports. In the past we
squeezed the trouble reports in after the feature development. Then there
was a lot to fix. Now we do the trouble reports continuously. So we are
slower with the features. But the overall quality should improve, which is
more important.” — Function Tester
Risk of technical debt: As shown in Table 1, a feature can be pushed out to
a customer early to facilitate learning. However, the learning from the customer
often comes at the expense of technical debt:
“This is software intended for one or two customers. It is usually not
commercially supported, so you can cut corners on robustness. Send it
to the customer, let them use it, get feedback on it, and make sure it is
what the customer wants. So you reduce the risk of not delivering value
after long development. [. . . ] We can half the time and deliver something
they can use [. . . ]. Of course, we have some debt, but we can solve that
in the coming month.” — Product Owner
Trade-off between agile and long term perspective: Our interviewees recognize
the challenge to balance agility with the necessity of long term planning.
“How do you stay agile and maintain a 5-10 year strategy? How can you
see that you go in the right direction with hundreds or thousands of user
stories?” — Designer
This is however not only due to the large scale, but also due to the difference
between customer and market value mentioned earlier:
“You have to be careful about the differences between customer value and
product value [we refer to this as market value]. Because you have more
dialog there is also a risk that you just agree and miss out on the big
picture of where the product is going.” — Function Tester
Maintain high quality on main branch: It is also recognized that the quality of
the main branch becomes paramount. Keeping feedback cycles short and identify
which delivery to the main branch is causing problems quickly becomes critical.
“It is a new way of working. You need a culture where you are following
your deliverable. The goal is to have high quality on [main branch] and
most often we have. If you deliver, you need to check the portal if you
broke something. But that is sometimes hard to see, e.g. when many are
delivering at the same time.” — System Tester
4.3 How to Check if Value is Added in Each Sprint (RQ3):
Our interviewees indicated that there is very little formal measurement:
“[Added value] is nothing that we measure. It is just captured in natural
language.” — Designer
“What I am using daily is ’find a good enough level’ - if it costs too much
to measure, than you need to scrap it.” — Product Owner
However, they talked about how their agile ways of working capture the checking
and control of value to a significant extent, through reviews, demonstrations,
combining user stories with criteria of definition of done, and rigorous testing.
Reviews: Teams are conducting a large number of reviews, both during
sprint planning and sprint review. In addition, there are special meetings across
cross-functional teams that cover all features.
Sprint Demo: Sprint demos are recognized as very efficient way of demon-
strating the value that has been added to the system.
“We demo the product at the end of the Sprint (not so much to the
customer, but to the PO and Scrum Master). There we prove that the
promised value has been added.” — System Tester
Such demonstrations are conducted both for internal and external stakeholders:
“We can demo internally, and we can request external customers to come
here and we can demonstrate. And there are also market and customer
units that we can demo to.” — Product Owner
Testing and Definition of Done: Our interviewees rely a lot on definitions
of done for user stories, hence, testing becomes a powerful measurement tool for
which value has been added.
“[. . . ] you try to define the value when you define the user story. It
might be that we are not there yet to measure the value, but we have user
stories definition of done. When we define the user stories we also define
acceptance criteria. So it is a binary decision on level of user story: pass
or fail” — Function Tester
Table 2. Suggested Improvements with respect to Challenges
Sugg. Improvement Aspects to improve Challenge to be addressed
Be more Know value for customer before they do so Trade-off between agile and
pro-active that we can demonstrate better. long term.
Component guardians Introduce/strengthen role with Trade-off between agile and
responsibility to assess how a change affects long term; Risk of technical
the long term value of a component. debt.
Increase focus on Include requirements as part of delivery to Maintain high quality on
process quality better characterize capabilities of deliveries, main branch.
Include team earlier Allow team to develop an understanding of High Cost, investment for
customer value. general notion of value
More detailed check Based on partial deliveries, improved High Cost, investment for
breakdown of user stories. general notion of value.
Improve tests as Improve partial delivery of value based on High Cost, investment for
measurement explicit notion about how tests are used to general notion of value.
measure added value.
Improve demos as Further spread sprint demos in organization, Trade-off between agile and
check invite more stakeholders, demonstrate main long term.
flow long before feature is implemented.
Transparency Better transparency about (testing) High Cost, ability to speed
resources and dependencies to other teams. up.
4.4 Suggested Improvements (RQ4)
During our interviews, we also asked the interviewees about which improvements
they would suggest with respect to the scope of our investigation. Table 2 gives
an overview of the improvements we collected and how they relate to challenges.
5 Discussion and Conclusion
Implications for Future Research: Value reflects the owner’s or buyer’s de-
sire to retain or obtain a product [19]. Neap [19] defines product value as a
measure expressed in units of currency equal to the cost of the product as a sub-
jective value. Product value is influenced by the quality attributes of the software
product and is related to the product price [3]. Woodruff defines the concept of
customer value: “a customer’s perceived preference for, and evaluation of, those
product attributes, attribute performances, and consequences arising from use
that facilitates (or blocks) achieving the customer’s goals and purposes in use
situations” [29]. The term customer value has many meanings but two dominate
- value for the customer (customer perceived value) and value for the firm (cus-
tomer lifetime value) [29]. While Neap’s concept of product value differs from
the interpretation of our interviewees’, their interpretation of customer value
relates directly to Woodruff’s customer perceived value, while market and prod-
uct value in our study are two aspects of customer lifetime value. In contrast
to the definitions in literature, our interviewees’ did not discuss the concept of
cost. Evobota et al., argue that agile planning is particularly difficult to scale,
because it is hard to bring together the perspectives of planning and cost [11],
and we believe that this is also due to the difficulties to manage value.
Future work should identify suitable definitions of value as well as investigate
how cost and value can be related to each other in a more transparent and
beneficial manner. We believe that this is crucial for understanding what value
that is added in each sprint as well as defining more fine-grained measurements.
In addition, we see a need for more conceptual work on requirements and testing
in order to measure the value added in each sprint.
Implications for Large-Scale Agile Development Practice: Our re-
sults are particularly interesting for companies that want to embrace continuous
delivery of large-scale systems, i.e. that aim at delivering new functionality to
customers continuously. Based on our results we suggest that lack of shared
understanding of customer value is an impediment for continuous delivery and
deployment, highlighting the need for continuous requirements engineering prac-
tices. Specifically, we suggest that distance to customer, lack of focus on sprint
goal, and lack of quality on test infrastructure will lead to inefficient continuous
delivery. The company studied has addressed these critical areas to a large extent
and our interviews suggest that this investment was crucial. Finally, we recom-
mend the continuous requirements engineering practices of adding value every
sprint, establishing a definition of done for each user story, and linking user
stories to requirements and tests as these were deemed beneficial for continuous
delivery in our interviews.
Acknowledgments: We are grateful for the support and insightful discussions
with our industry partners and we thank all interviewees. This work has been
partly supported by the Software Center, Proj. 1 “Implications of Continuous
Deployment” and the SIDA BRIGHT project.
References
1. Alahyari, H., Svensson, R.B., Gorschek, T.: A study of value in agile software
development organizations. Journal of Systems and Software (2016)
2. Aurum, A., Wohlin, C.: A value-based approach in requirements engineering: ex-
plaining some of the fundamental concepts. In: Int. Working Conf. on Reqts. Eng.:
Foundation for Softw. Qual. (REFSQ). pp. 109–115. Springer (2007)
3. Barney, S., Aurum, A., Wohlin, C.: A product management challenge: Creating
software product value through requirements selection. Journal of Systems Archi-
tecture 54(6), 576–593 (2008)
4. Beck, K.: Extreme programming explained: embrace change. Addison-Wesley
(1999)
5. Beck, K., Beedle, M., Van Bennekum, A., Cockburn, A., Cunningham, W., Fowler,
M., Grenning, J., Highsmith, J., Hunt, A., Jeffries, R., et al.: Manifesto for agile
software development (2001)
6. Berczuk, S.: Back to basics: The role of agile principles in success with an dis-
tributed scrum team. In: Agile Conference (AGILE). pp. 382–388 (2007)
7. Boehm, B.: Value-based software engineering. SIGSOFT Softw. Eng. Notes 28(2),
4– (Mar 2003), http://doi.acm.org/10.1145/638750.638776
8. Cohen, D., Lindvall, M., Costa, P.: Agile software development. Tech. rep., DACS
SOAR Report (2003)
9. Debbiche, A., Diener, M., Svensson, R.B.: Challenges when adopting continuous
integration: A case study. In: Proc. of 15th Int. Conf. of Product Focused SW Dev.
and Process Impr. (Profes). pp. 17–32. Helsinki, Finland (2014)
10. Dingsøyr, T., Moe, N.B.: Towards principles of large-scale agile development. In:
International Conference on Agile Software Development. pp. 1–8. Springer (2014)
11. Evbota, F., Knauss, E., Sandberg, A.: Scaling up the planning game: Collaboration
challenges in large-scale agile product development. In: Proc. of 17th Int’l Conf.
on Agile Softw. Dev. (XP). Edinburgh, UK (2016)
12. Fowler, M.: Continuous integration. Tech. rep. (2006), http://martinfowler
.com/articles/continuousIntegration.html last visit: 01 2017
13. Golra, F.R., Beugnard, A., Dagnat, F., Guerin, S., Guychard, C.: Continuous re-
quirements engineering using model federation. In: Proc. of 24th Int. Reqts. Eng.
Conf. (RE). pp. 347–352. IEEE (2016)
14. Kauppinen, M., Savolainen, J., Lehtola, L., Komssi, M., Tohonen, H., Davis, A.:
From feature development to customer value creation. In: Proc. of 17th IEEE Int.
Reqts. Eng. Conf. (RE). pp. 275–280. IEEE (2009)
15. Kirikova, M.: Continuous requirements engineering in freedom framework: A posi-
tion paper. In: Proc. of 2nd WS on Cont. Reqts. Eng. (CRE). Gothenburg, Sweden
(2016)
16. Komssi, M., Kauppinen, M., Töhönen, H., Lehtola, L., Davis, A.M.: Roadmap-
ping problems in practice: value creation from the perspective of the customers.
Requirements Engineering 20(1), 45–69 (2015)
17. Leffingwell, D.: Scaling Software Agility: Best Practices for Large Enterprises.
Addison-Wesley Professional (2011)
18. Mathiassen, L.: Collaborative practice research. Information Technology and Peo-
ple 15(4), 321–345 (2002)
19. Neap, H.S., Celik, T.: Value of a product: A definition. Int. Journal of Value-Based
Management 12(2), 181–191 (1999)
20. Neely, S., Stolt, S.: Continuous Delivery? Easy! Just Change Everything (well,
maybe it is not that easy). In: Proc. of Agile Conference. pp. 121–128. IEEE,
Nashville TN, USA (2013)
21. Olsson, H.H., Bosch, J., Alahyari, H.: Customer-specific teams for agile evolution
of large-scale embedded systems. In: 2013 39th Euromicro Conference on Software
Engineering and Advanced Applications. pp. 82–89. IEEE (2013)
22. Racheva, Z., Daneva, M., Sikkel, K., Herrmann, A., Wieringa, R.: Do we know
enough about requirements prioritization in agile projects: Insights from a case
study. In: Proc. of 18th Int. Requts Eng. Conf. (RE 10). Sydney, Australia (2010)
23. Reifer, D., Maurer, F., Erdogmus: Scaling agile methods. IEEE Software 20(4),
12–14 (2001)
24. Roberts, M.: Enterprise continuous integration using binary dependencies. In: Ex-
treme Progr. and Agile Proc. in Softw. Eng. (XP ’04). pp. 194–201. Springer (2004)
25. Rogers, R.: Scaling continuous integration. In: Extreme Programming and Agile
Processes in Software Engineering. pp. 68–76. Springer (2004)
26. Schwaber, K.: Agile project management with Scrum. Microsoft Press (2004)
27. Stahl, D., Bosch, J.: Modelling continuous integration practice differences in in-
dustry software development. Systems and Software 87, 48–59 (2014)
28. Wohlin, C., Aurum, A.: Criteria for selecting software requirements to create prod-
uct value: An industrial empirical study. In: Value-based software engineering, pp.
179–200. Springer (2006)
29. Woodruff, R.B.: Customer value: the next source for competitive advantage. Jour-
nal of the academy of marketing science 25(2), 139–153 (1997)