Today's software-intensive organizations are experiencing a paradigm-shift with regards to how to develop software systems. With the increasing availability and access to data and with artificial intelligence (AI) and technologies such as machine learning and deep learning emerging, the traditional requirement driven approach to software development is becoming complemented with other approaches. In addition to having development teams executing on requirements specified by product management, the development of software systems is progressing towards a data driven practice where teams receive an outcome to realize and where design decisions are taken based on continuous collection and analysis of data. On top of this, and due to artificial intelligence components being introduced to more and more software systems, learning algorithms, automatically generated models and data is replacing code and the development process is no longer only a manual effort but instead a combination of human and automated processes. In this paper, and based on multi-case study research in embedded systems and online companies, we see that companies use different approaches to software development but that they often take a requirement driven approach even if they would benefit from one of the other two. Also, we see that picking the wrong approach results in a number of problems such as e.g. inefficiency and waste of development efforts. To help address these problems, we develop a holistic development framework and we provide guidelines on how to improve effectiveness in development. The contribution of this paper is two-fold. First, we identify that there are three distinct approaches to software development; (1) Requirement driven development, (2) Outcome/data driven development and (3) AI driven development and we outline the typical problems that companies experience when using the wrong approach for the wrong purpose. Second, we provide a holistic framework with guidelines for when to use what approach to software development.
driven
Today’s software-intensive business is in the midst of profound changes in relation to
development of software systems. With rapid pace, and across industry domains,
sophisticated technologies for data collection and analysis are implemented to provide
developers with real-time input on how the systems they develop perform in the field.
Also, this data helps developers understand what functionality is used by customers and
it allows product managers to confirm whether feature prioritizations were accurate [
Fueled by the increasing availability and access to data, artificial intelligence (AI)
and technologies such as e.g. machine learning and deep learning are rapidly adopted
in a variety of domains [
However, despite the rapid growth of data and the emergence of complementary
approaches to software development, most companies have a strong tradition in
requirement driven development. In this approach, system requirements are specified
in the early stages of development, and although more agile requirements engineering
practices are increasingly applied [
In our research, we see that companies use different approaches to software development but that there are a number of problems associated with selecting the most suitable approach. First, companies with a strong tradition in requirement driven development often take this approach even if they would benefit from an alternative approach. Second, proponents of outcome/data and AI driven development approaches tend to neglect other approaches and instead argue for their approach being the only right one. Third, picking the wrong approach for the wrong purpose results in a number of problems such as e.g. inefficiency and waste of development efforts. In this paper, and based on multi-case study research in companies in the embedded systems and in the online domain, we develop a holistic development framework including three distinct development approaches and we provide guidelines for how to improve effectiveness in development by selecting the optimal one.
The contribution of this paper is two-fold. First, we identify that there are three distinct approaches to software development; (1) Requirement driven development, (2) Outcome/data driven development and (3) AI driven development and we outline the typical problems that companies experience when using the wrong approach for the wrong purpose. Second, we provide a framework with guidelines for when to use what approach to software development.
The paper is organized as follows. In section 2, we detail the background of our research. In section 3, we describe the research method and the case companies involved in our research. In section 4, we report on the software development approaches in the case companies and we summarize our empirical findings. In section 5, we first identify three distinct development approaches and outline the key problems companies experience when picking the wrong approach for the wrong purpose. Second, we provide guidelines for when to use what approach. In section 6, we conclude the paper.
Although the saying that things keep getting faster might sound a little worn-out, the
fact is that the software business of today is experiencing bigger, and more rapid,
transformations than ever before. The driving force of this is the increasing
digitalization of industry that is disrupting companies and society in large to an extent
that we have only seen the early beginnings of. As defined by Gartner [
Interestingly, the transformations we see as a result of digitalization have an
enormous impact not only on the products and the services that companies produce but
also on the ways in which these products and services are produced, i.e. the
development approaches themselves. As a result of digitalization and connectivity of
products, the traditional requirements engineering process that has been the primary
approach for software development for decades is being complemented with other
approaches in which continuous use of data, rather than specification of requirements,
informs development teams and software systems. As well-known to most software
businesses, requirements engineering includes the identification of requirements and
the modeling of these in order to develop an agreed upon understanding of what a future
software system will look like in order to provide value to the customer and there exist
a wide range of techniques to help the development team ensure that the requirements
are complete [
However, with systems being connected to the Internet and technologies that
facilitate data collection and analysis, we see that companies are increasingly
complementing their traditional development approaches with other approaches. As
one of the most influential trends in software industry, continuous deployment of
software is challenging traditional ways-of-working in that it by-passes the notion of
early requirements specification. Continuous deployment is a software engineering
practice in which incremental software updates and improvements are developed, tested
and deployed to the production environment on a continuous basis and in an automated
fashion [
The trends described above reflect an interesting shift from a situation where
traditional requirements engineering practices inform development of new features,
towards a situation in which customer and product data is continuously collected and
where companies use this data to inform development during run-time [
The research reported in this paper builds on multi-case study research [
Over the years, our research collaboration with the Software Center companies has
been reported in a large number of publications, e.g. [
The second domain is the online domain and here we studied companies developing online games, online payment services, media streaming services, travel and accommodation services, online search services and tools for developing artificial neural networks and adaptive systems. These companies are pure Software-as-aService companies and with revenue based on license fees, transaction fees and the digital products and services they produce. They continuously add software functionality to their products and they collect and use data as a basis for product development and improvements. In similar to the embedded systems companies, the online companies have access to large amounts of data and they are exploring different development approaches in order to maximize the benefits of this data. In contrast to the embedded systems companies, the online companies have less of a legacy in terms of requirement driven development. Even if this approach exists also here, they typically use data as the basis for development with teams receiving a quantitative target to realize and are asked to experiment with different solutions to improve a certain metric. In addition, some companies [e.g. …] use artificial intelligence and deep learning technologies as part of development in order to automate tasks and improve speed in problem-solving.
Our research collaboration with the online companies has been reported in a number
of publications, e.g. [
In total, our research collaborations with the different companies in these two
domains cover a time period of more than seven years (2011 – 2018). The collaboration
with the embedded systems companies has been an on-going engagement since 2011,
and in relation to a number of different topics such as e.g. agile transformation,
development feedback cycles, data driven development and value modeling of software
features. The specific work on data collection and analysis, and how data can help
improve software development, was initiated in 2015 and is on-going. The
collaboration with the online companies was initiated in 2015 and is on-going. In all
companies, and throughout this period, we have run frequent meetings, interview
sessions and workshops involving project managers, product managers, product
owners, software developers, software and system architects, data scientists, data
analysts and a number of agile team coaches and scrum masters. Meetings are typically
scheduled for one hour, workshop sessions for two – three hours and interviews for one
hour. The empirical data we build on consists of hundreds of pages of interview
transcripts, as many meeting and workshop notes, notes from informal meetings,
thousands of e-mails and frequent telephone conversations. Throughout our research,
we adopted an interpretive approach to data analysis with the intention to identify
recurring elements and concepts in the transcribed interview protocols [
In this paper, we build on our previous findings from the embedded systems and from the online companies when exploring the different development approaches they use. In particular, we are interested in exploring how the development approaches they have traditionally been using are being complemented with other approaches.
In this section, and based on our previous research in embedded systems and online companies, we summarize our empirical findings in relation to existing and emerging approaches to software development. With selected examples from the two domains, we present the current state as well as the transition that the case companies are experiencing with regards to how to develop software products and services. In Table 1, we provide a summary where we generalize the characteristics of the development approaches in the two domains. It should be noted that the summary does not reflect details and deviations but rather it captures the dominant characteristics of each domain.
The embedded systems companies are in the midst of a challenging transition where the products they develop are rapidly becoming digitalized and where connectivity is key for future innovation and revenue. In this fast-changing environment, the hardware dependencies make development complex as the feedback cycles for hardware are slow while the software cycles are rapid. In most of the companies, the traditional and waterfall approach to development is applied in large parts of the organization while agile practices and methods such as e.g. Scrum are well-established in other parts. It should be noted that many of these companies offer a broad product portfolio which implies that the competence and expertise cover the development and delivery of physical products based on hardware components as well as digital services based on software components. To manage such a disparate product portfolio, the embedded systems companies apply a wide range of development methods and they need to constantly adopt new skills and ways-of-working. Still, and as the most common development approach, requirement driven development characterize both the mindset and the organizational set-up in these companies. As a common practice, teams receive a requirements specification from product management, and the task for the team is to deliver according to specification. Even if many companies apply agile practices today, they have a long-standing and strong culture where requirements dictate development and where decisions and prioritizations are made based on previous expertise and experience. Typically, qualitative approaches are used to learn about customers with interviews, prototypes and observations being common techniques for data collection. Also, and in line with this culture, requirements are agreed upon in the early stages of development and with sudden changes being a costly disruptor and viewed as something to avoid.
However, and co-existing with the requirement driven culture, the embedded
systems companies have been collecting data from their products well before they
became connected as many of them are today. For example, the automotive companies
started collecting diagnostics data from vehicles already in the early 90’s to use as the
basis for maintenance whenever a truck or a car was taken to a garage for service. More
recently, and as a result of vehicles becoming connected to the Internet and with
practices such as continuous deployment in place, car manufacturers can push software
updates to the vehicle on a continuous basis without taking the vehicle out of traffic.
This allows for preventive maintenance and has become key to prolong the lifetime of
a vehicle and avoid costly repairs. Also, effective use of data allows car manufacturers
to detect errors while the vehicle is running and before the customer is even aware of
them. In similar, telecom companies collect huge amounts of traffic and configuration
data as the basis for optimizing performance and operation of their systems as well as
for predictive maintenance and monitoring. Based on our research, we see that many of
the embedded systems companies are in the process of instrumenting their products to
increase and further improve data collection and analysis practices. Also, there are
examples of A/B testing initiatives where companies run experiments with customers
to determine whether version A or B of a software feature is the optimal and most
appreciated one [
Based on our most recent interactions with the embedded systems companies, we
see an emerging interest in artificial intelligence and associated technologies. With
connected systems and with large data sets available, new opportunities arise in terms
of how to manage, process and utilize this data. For many of the companies, automated
practices for collection and analysis of data are already in place as continuous
integration and deployment are becoming critical components of their software
development approaches. Still, however, supporting infrastructures for increasingly big
volumes of data that can handle complexity in terms of variety and velocity [
In contrast to the embedded systems companies, the online companies are less frequent
users of requirement driven practices. Although they exist, they don’t serve their
purpose as the products and systems the online companies develop are inherently
different in characteristics and therefore, require other development approaches.
Instead, practices such as continuous integration and deployment are fully in place and
with products being digital there are no hardware dependencies that slow down the
development cycle. This reduces complexity and increases speed and in the majority of
the companies, new software functionality is released on a daily or weekly basis.
Instead of requirements, the online companies use data collected from their products as
the basis for understanding customer needs and preferences. In our experience, most of
the online companies have instrumented their products in order to collect relevant data
and they have software tools that help them analyze this data. As the basis for data
collection, they run A/B tests in which hypotheses on what adds customer value are
validated. A/B tests are experiments where two versions of software functionality are
compared to determine which one performs the better in relation to predefined criteria
such as e.g. conversion rate, click rate or time to perform a certain task [
Despite the many advantages with using data as the basis for development, the online
companies experience challenges with this approach just like the embedded systems
companies experience challenges with their requirements driven practices. As reported
in our previous research [
In our most recent research, we have had the opportunity to learn about some of the
emerging trends in these companies and especially about their rapid adoption of
artificial intelligence and deep learning solutions. These technologies enable radical
improvements in the development cycle by increasing the effectiveness of development
and by reducing development time of novel functionality. In one of the case companies
[
In this paper, we explore the transition that companies in the embedded systems and in the online domain are experiencing due to digitalization of products and services. Based on our previous research in a large number of companies, we see that companies are complementing their traditional development approaches with other approaches and that this requires a careful understanding of when to use what approach. Below, we identify three distinct development approaches that we see exist in the companies we studied. We summarize the approaches in Table 2. Furthermore, and as an inductively derived model from generalizing our case study findings, we provide a holistic development framework (Figure 1) with guidelines for when to use what approach to software development.
Based on our case study research, we identify that companies use three different
approaches to software development. First, they use a requirement driven development
approach where software is built to specification and where product management is
responsible for collecting and specifying requirements as input for the development
teams. As can be seen in the empirical examples, this development approach is
predominantly used when the new features or new functionality are well understood
and defined and where business revenue is not based on frequent releases of new
functionality. Especially in the embedded systems companies, there is a long and
wellestablished practice of developing software systems based on requirements. In all these
companies, requirements are collected, specified and carefully documented as the main
input for development teams and as the mechanism to confirm that a system is
developed and delivered according to customer preferences and needs [
However, the requirement driven development approach is well suited for situations in which features and functionality are well-understood and where there is a long-term agreement between the customer and the development organization. Typically, this approach applies for products and services that are intended to last over time and where there is less frequent change imposed on the system. When applied in fast changing environments where customer requirements fluctuate, the requirement driven development approach should be avoided as it falls short on managing frequent iterations and short development cycles.
The second approach companies use is the outcome/data driven development
approach where development teams receive a quantitative target, i.e. an outcome, to
realize and are asked to experiment with different solutions to improve the metric. This
development approach is predominantly used for development are new features that are
used frequently by customers, and for innovation efforts when there is uncertainty on
how to realize a new feature. As can be seen in the empirical examples, this approach
is the dominant approach in the online companies where development teams are
assigned a certain metric, e.g. conversion rate, and are responsible for improving this
metric. To do so, a team typically runs A/B tests with selected customers to identify
what version of a website that improves the metric and that moves the needle in the
direction set by the business. The decision is based on data that is collected during the
experiment and the approach differs from the requirement driven development
approach in that continuous collection and analysis of customer and product data
informs development rather than requirements specified in the early stages of
development. Also, while the requirement driven approach is characterized by smaller
amounts of qualitative data as the basis for decision-making, the outcome/data driven
approach uses large and quantitative data sets collected at run-time and by
instrumenting the code to monitor specific metrics. In our research, we see example of
this approach also in the embedded domain where experimentation with different
software solutions are becoming increasingly important to determine and validate
customer value [
The third approach companies use, and that is rapidly emerging as a new approach
to software development, is the AI driven development approach where the company
has a large data set available and uses artificial intelligence techniques such as machine
learning and deep learning to create components that act based on input data and that
learn from previous actions. In the case companies we studied, AI is perceived as a very
powerful approach with the potential to take on far more complex assignments humans
by augmenting our skills, talents and abilities. Examples of this type of development
include e.g. object recognition is autonomous cars as well as speech recognition in
modern user interfaces. As another example, one of the case companies refers to the
use of AI for predicting business sales by pulling data from sales tools together and by
using patterns found in this historical and rich data set. Typically, this approach applies
for product and service development in which a company has access to a large data set
with very many data points and where minimizing prediction errors is critical. Also,
and as recognized in the case companies, it is an approach well suited for development
situations in which there are too many potential alternatives that manual processing of
these would be either too difficult, too time consuming or too expensive. As the AI
approach is fundamentally different from traditional software development in that
much of the responsibility for finding a solution to a problem is left to the computer,
problems arise when an organization e.g. lack mechanisms and infrastructures for
running experiments, has limited resources for large and complex data sets and when
the organizational culture, skills and interests do not align with the cross-functional
collaboration that is critical for building a production-ready AI system. As recognized
in [
Definition Software is built to specification. This development approach is predominantly used when new features or functionality are well understood and defined.
Development teams receive a quantitative target to realize and are asked to experiment with different solutions to improve the metric. Examples of this development approach are new features (used frequently by customers) and innovation efforts.
A company has a large data set available and use artificial intelligence techniques such as machine learning and deep learning to create components that act based on input data and that learn from previous actions. Examples of this development approach include e.g. object recognition in autonomous cars and speech recognition in modern user interfaces.
As reported above, and due to the increasing access and availability to data, companies
are starting to complement their requirement driven development approaches with other
approaches. With the adoption of agile development practices [
However, and as recognized in this research, the integration of these development approaches is not well understood and there exist little guidance for when to select one approach over another. In addition, and as future systems will include both traditional software components as well as artificial intelligence components, the combination of approaches will be critical as most software organization will have to manage not only one approach but all three.
To help address this challenge, and to provide companies with a framework on when to select what approach, we present a holistic development framework (Figure 1). Based on a generalization of our case study findings, we outline the three development approaches and we identify the purposes for which each approach is optimal. As can be seen in this framework, the (1) requirement driven development approach is well suited for regulatory features, for competitor parity features and for commodity features, the (2) outcome/data driven development approach is well suited for value hypotheses, development of new “flow” features, i.e. features used frequently by customers and for innovation and the (3) AI driven development approach is well suited when aiming to minimize prediction errors, when there are many data points and when there is a combinatorial explosion of alternatives.
The framework pictures an overall development environment where the system in operation consists of traditional software components as well as AI components, and where continuous deployment practices allow for behavior data to be continuously collected and used as the basis for development. In this environment, and with systems involving different components, the key challenge is to effectively select, combine and deploy different development approaches. Although successful instantiations exist in research and industry, there are no systematic, repeatable methods for creating, evolving and maintaining systems using these techniques.
Holistic DevOps Framework Requirements driven development - Regulatory features - Competitor parity features - Commodity features
Outcome/data driven development - Value hypothesis - New ”flow” features - Innovation
AI driven development - Minimize prediction errors - Many points in data set - Combinatorial explosion of
alternatives continuous deployment behavior data System in operation
AI component SW component continuous deployment behavior data Today’s software industry is in the midst of dramatic transformations with digitalization challenging existing ways-of-working. With increasingly connected and intelligent products, and with availability and access to massive amounts of data, the traditional requirement driven approach to software development is being complemented with other approaches that reflect these new opportunities and technologies. However, in our research we see that companies often take a requirement driven approach even if they would benefit from one of the other two. Also, we see that picking the wrong approach results in a number of problems such as e.g. inefficiency and waste of development efforts.
In this paper, and based on multi-case study research in embedded systems and online companies, we identify three distinct approaches to software development: (1) Requirement driven development, (2) Outcome/data driven development and (3) AI driven development and we provide a framework with guidelines for when to use what approach to help minimize the problems associated with using the wrong approach for the wrong purpose. With this framework, we aim to help companies effectively select, combine and deploy different development approaches in order to manage the digital transformation they are in.
In our future work, we intend to further validate this framework and explore how software-intensive companies in different domains can benefit from complementary development approaches and how successful selection of approaches can become key for competitive advantage.