Organizations seek to streamline their data and analytics structures in order to meet increasingly demanding business requirements. This can be difficult due to complex and fast-moving data landscapes. DataOps promises a remedy by combining an integrated and process-oriented perspective on data with automation and methods from agile software engineering, like DevOps, to improve quality, speed, and collaboration and promote a culture of continuous improvement. The goal of this on-going research is to elaborate DataOps as a new discipline. For this, it explores the body of knowledge and presents a working definition of DataOps as well as an initial research framework based on an explorative literature review and eight interviews with industry experts.
Data is a key asset to compete in today’s business. Data-driven decision making significantly increases business success [1] and data is essential for many business processes or even entire business models [1, 2]. Consequently, companies seek to streamline their data and analytics processes in order to make them more efficient, provide data faster and in superior quality, and ensure a stable and reliable operation in general. This, however, can be hard due to fractured data landscapes with heterogeneous tools and technologies, a broad scope with various stakeholders, rapidly changing business requirements, and a general lack of standards [3, 4]. To revise these insufficient and inefficient structures, there is a need to shape enterprise data and analytics in a way that enables a stable operation and increases speed, quality, and overall productivity. Discussions on how to achieve this, often includes topics like agile methods, data governance concepts, or the use of automation. Here, many see similarities to challenges in software engineering, where DevOps and continuous integration were introduced to provide highquality software at an every-increasing pace [5]. However, data analytics is different to software engineering, and consequently the new term DataOps emerged [6]. The goal of this ongoing research is to academically elaborate DataOps as a new discipline. This paper explores the body of knowledge and presents a working definition of DataOps and an initial research framework. We firstly depict related topics and our methodical approach. Then, we discuss first results from a qualitative exploration and derive a working definition and an initial research framework. The paper finally concludes with an outlook for future work.
The field of DataOps goes hand in hand with a continuous professionalization of data and analytics processes in companies. It combines ideas from information systems research with different areas like agile and lean thinking and modern software engineering [7]. For one thing, DataOps is taking its cue from DevOps which is “an organizational approach that stresses empathy and cross-functional collaboration within and between teams” [5] in order to accelerate delivery of changes and increase quality of software [5, 8]. For this, DevOps promotes integrated and highly-automated engineering pipelines to bridge the gap between development and IT operations. This can be challenging, since development and operations are historically separate tasks with conflicting goals. Software engineers need to quickly respond to changing requirements and seek to rapidly deploy new features. The operations team, in contrast, is interested in providing stable and reliable services and infrastructure and therefore avoids risks and works as predictably as possible [8]. This is why DevOps requires a mind-shift and strives for change in company culture [5, 8]. As DevOps is a rather novel concept, related research is mainly focusing on exploring ways of adaption [9, 10] and its business value [10, 11]. Next to DevOps, other organizational and technological approaches from software engineering like, behavior-driven development [12] or scrum [13], as well as the general philosophy of the agile manifesto [14] appear in DataOps discussions.
From an information systems research perspective, DataOps fits into the continuous stream of work about agility in business intelligence (BI) and data science. Here, agility is seen as “the ability to react to unforeseen or volatile requirements regarding the functionality or the content of a BI solution” [15]. This encompasses the transformation of processes with agile methods like Scrum or Kanban, as well as the adaption of new technologies and architectural concepts to provide flexibility and increase value [3, 16, 17]. There is also an overlap with research about BI and analytics maturity models that tries to assess the level of development of organizational capabilities and resources in organizations [18]. Here, the associated goals are usually to reveal necessary steps to advance BI and analytics in order to increase productivity and business value [19]. Findings in this area shows that mature analytics solutions involve a high-level of collaboration, strategic alignment, and an enterprise-wide integration [20, 21], what matches the goals of DataOps. 3
In order to examine the field of DataOps, we followed an exploratory research design outlined in Fig. 1. First, we conducted an exploratory literature review for an initial exploration of the body of knowledge and to provide a better general understanding of terms and concepts [22].
Exploratory literature review
Interviews with
industry experts
For this, we followed Levy and Ellis [23] and queried four scientific databases, namely
SpringerLink, ACM, AISeL and IEEE, as they cover numerous important journals and
conferences in the information systems domain. In the search we used the keyword
DataOps as well as a combination of the terms DevOps and Data. We manually
screened the results and filtered out articles that seemed to have no relevancy, e.g. focus
on another domain or too specific topics. This process resulted in 783 documents of
which 243 were selected relevant and were further investigated by screening the
abstract and full text. It turned out that only 6 documents contained DataOps in its title or
abstract. The temporal distribution of the publications confirmed the novelty and
relevancy of the topic, as t
300 200 100 0 18 2012 14 2013 60 2014 210
241 76
157 2015 2016 2017 2018 For a qualitative exploration, we then interviewed selected industry experts to gain deeper insights about the scope of DataOps. To select relevant experts, we searched for companies that offer DataOps services (vendors) or use DataOps in practice (users). We conducted eight interviews with international experts from different companies (c.f. Table 1). Most companies were vendors of which five can be denoted as startups (< 10 years or < 50 employees) and three can be assigned to the enterprise level. The interviews were conducted as semi-structured discussions of 30 – 60 minutes by phone. The experts were asked about their understanding of DataOps, how their services fit into this area and about goals and principles related to DataOps. The interviews were transcribed and coded to identify more general components of DataOps. For this, we first labeled statements (e.g. “we allow unit-tests in databases” as technical tests) and then grouped similar labels (e.g. technical tests and acceptance tests as testing).
Company Company C1 Company C2 Company C3 Company C4 Company C5 Company C6 Company C7 Company C8 The results of the literature review and the insights of the interviews were then triangulated to (i) formulate an initial working definition for the term DataOps and (ii) to derive a research framework for further work. Moreover, the derived working definition was discussed with two experts of the preceding interviews to test its relevance for practice. 4
The exploration of the domain shows, that there is no general accepted definition of DataOps yet. The term itself was introduced by Andy Palmer who described DataOps as a discipline that “addresses the needs of data professionals on the modern internet and inside the modern enterprise” [6]. Since then, there are different understandings of the term within various scopes, like DataOps is “a hub for collecting and distributing data” [25], “the function within an organization that controls the data journey from source to value” [26], or “a new way of managing data that promotes communication between, and integration of, formerly siloed data, teams, and systems” [7]. Similarly, the conducted interviews also included various understandings like “all activities between the data and operation teams” or “an integrated perspective over the entire data lifecycle”.
When describing these vague understandings, most authors emphasize continuous improvement and a culture of collaboration and trust as the underlying goal of DataOps [7, 27, 28] and elaborate their definitions with a set of goals, principles, and components. Moreover, many mentioned the empowerment of citizen-users and an end-to-end thinking as core objectives next to enhanced speed and quality [6, 28, 29]. When it comes to implementation, some use the term data pipeline to describe a process-oriented structure where data is transferred through multiple stages (e.g. extracted, transformed, and visualized) [26, 28]. This concept is often used to support orchestration and automation in complex scenarios. In this context, some advocate the idea that every artifact (e.g. data models or visualizations) can be represented as code (analytics as code) [27]. From an agile perspective, DataOps adopts the strive for short-cycles and incremental change. One interview summarized the DataOps with the goal to “become as fast as a startup, while being as robust as a manufactory”. In addition to that, the examined literature and the interviews revealed data-driven improvements, reuse of artifacts, and testing and monitoring as key principles of DataOps [7, 27, 30]. The results indicate, that DataOps is rather a collection of various practices and technologies, than a particular method or tool. Based on our initial findings, we suggest the following working definition of the term DataOps: DataOps is a set of practices, processes and technologies that combines an integrated and process-oriented perspective on data with automation and methods from agile software engineering to improve quality, speed, and collaboration and promote a culture of continuous improvement.
This definition is intended to be a starting point for a common understanding of the term DataOps and an early limitation of its scope. It does not claim to be final and should be subject of discussion in future work.
Next to the working definition, we propose the research framework in Fig. 3 to further explore the DataOps space. This framework differentiates between the exploration of DataOps as a discipline, which includes methods, technologies and concrete implementations, and the investigation of the business value of DataOps.
Environment Technology
Organization DataOps as a Discipline
Dynamic
Capabilities
Business Value DataOps as a discipline can be further broken down by the means of the TechnologyOrganization-Environment (TOE) Framework [31] that helps to explain the relation between technology, organization and external factors in the diffusion of innovations. This framework brings the necessary flexibility to cover the wide scope of DataOps that spans from technological advancements, like automated data testing and continuous deployments, to organizational initiatives, like end-to-end collaboration throughout various business functions. Moreover, the TOE-framework has already been successfully used in similar areas, e.g. cloud computing [32] or big data [33]. Regarding the examination of the business value of DataOps, the concept of dynamic capabilities [34, 35] seems to be a valid candidate for a theoretical foundation, as it focuses “the ability to integrate, build, and reconfigure internal and external competencies to address rapidly-changing environments” [34] and it was already used in prior work to explain the business value of data and analytics [1, 36, 37] or DevOps [38]. This research framework is intended as an initial structure for future research. The prosed methods and theoretical foundations need to be validated and refined in upcoming work. 6
This research illustrated the broad character of DataOps and showed that it is not a particular method or tool, but rather a collection of principles and a way of doing things on a cultural, organizational, and technological level. With its exploratory character, the research contributes to the body of knowledge by academically defining DataOps and arranging it in the field of information system research. Moreover, the research can help organizations to understand and adopt DataOps in practice.
DataOps is a rather novel term and there is only little experience and many research challenges. The proposed framework provides a starting point for future work that should, first, elaborate DataOps as a discipline, e.g. by developing blueprints for roles, processes and governance, and further explore related technologies, and, second, investigate the business value proposition of DataOps. As the diffusion of DataOps is still low, we suggest to conduct in-depth case studies to compare traditional approaches with DataOps-like implementations in order to gain further insights and refine this work. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21.