Previous studies on software product management (SPM) domain have provided an extensive picture of the work of a software product manager. Still, little evidence exists about what principles should guide their decisions. A product manager's decision-making has a certain level of subjectivity based on managerial intuition. However, sustainable software product development requires effective long-term decision-making practices. Requirements engineering, as well as release planning and roadmapping, are SPM areas with the highest level of evidence-based decision-making. Still, the clear understanding of evidencebased decision-making practices is missing. The paper provides an analysis of decision-making related to SPM, reveals a spectrum of attitudes and approaches and reports assumptions on whether SPM is based on intuition or if it is evidencebased.
Software product management is a growing area of research and practice that bridges the gap between business and engineering aspects of the software business. Many frameworks for SPM in diverse areas have been introduced by both researchers and practitioners [1–5]. A systematic analysis of the frameworks has produced core areas of SPM responsibility and activities [6]. Although product management practices may vary significantly across companies and be determined by many internal and external factors, existing research reveals a range of possible SPM practices and provide quite clear picture of the software product manager role. Still, little evidence exists about how the work of software product managers should be organized and what principles should guide their decisions.
Software developing organizations should establish business processes and practices that enable managers to make decisions using evidence rather than intuition. However, the introduction of evidence-based techniques that lead to informed decision-making can be challenging. Especially start-ups in their early maturity stages make decisions with an ad hoc “gut feeling” approach rather than using evidence-based methods [7]. However, a long-term sustainable software product development requires a transition towards more systematic evidence-driven managerial processes and practices [8].
Requirements engineering along with roadmapping and release planning are the areas from which evidence-driven managerial processes and practices begin. However, it is quite common practice to limit only to these areas when companies transform their decision-making practices and process in SPM. Research on evidence-based SPM has also noted this. While recently some studies have raised the question of making SPM more data and model-driven [9–14], we still lack a comprehensive analysis of evidencedriven decision-making and its potential in software product management.
The objective of this paper is to evaluate the feasibility of evidence-based decisionmaking in software product management as well as to draft a proposal for implementing evidence-based SPM processes, practices and required IT infrastructure. This paper is the very first step towards our research goal to obtain a better understanding of approaches, evidence, and techniques used in SPM decision-making. We also examine product managers’ perceptions on using evidence-based decision-making and identify obstacles of broader implementation of this approach. 2 2.1
Recent studies indicate the progress of SPM as a discipline at the intersection of software engineering and business domains as well as growing attention from practitioners. Still, multiple challenges for companies can be identified that require proper support from the research community [3, 15]. Overwhelming processes, unclear responsibilities of software product managers along with premature decision-making practices and moving targets are often named as prime challenges [15, 16]. Existing studies provide a solid foundation for roles and scope of duties in SPM [6]. However, SPM efficiency and how SPM decision-making processes and practices should be organized have not received much attention in academic literature.
The range of SPM decisions varies across several dimensions. The decisions can be done in three levels: strategic, tactical and operational levels. At the strategic level, decisions are mostly related to designing a product strategy and defining the overall business model. Decisions at the tactical level intend to guide most product management processes including release planning, lifecycle management, and roadmapping. Finally, decisions at the operational level determine the use of technical solutions for product architecture, required infrastructure and deployment as well as schedules operations. However, like other types of managerial decision, problems of SPM decisions are not limited to these three levels. Factors such as whether the decision is made by a group or individual [17] as well as whether it can be programmed or not [18] may affect decision-making processes and their designs.
Effective decision-making in SPM requires considering a large number of factors. These factors have both engineering and business origins and include market characteristics (e.g., concentration, B2B vs. B2C market type), product specifications (e.g., mass-market vs. bespoke, product vs. digital service), technological (e.g., technology obsolescence, infrastructure, project complexity), organizational (e.g., lack of staff with the required skills, uncooperative internal parties) as well as the customers’ needs and expectations (e.g., costs, benefits) [19]. Considering these factors requires collecting vast amount of data and analyzing it with the help of sophisticated techniques and models.
Another attribute of effective decision-making is formalized processes [20]. Using systematic decision-making approaches in SPM creates valuable transparency that in the long run allow all stakeholders to have both clear vision of the situation and to identify a possible room for improvement in vision execution. However, quite often software product managers prefer making decisions in an ad-hoc way driven by their “gut feeling” and to use tacit knowledge, fearing that otherwise their flexibility in decision-making will be reduced [21].
The transition towards SaaS business and development model along with the active use of information management tools that support digitalizing business processes increased the scope and scale of data available for analysis and decision-making in all areas of software product management [22, 23]. However, making sense of this data considering its overwhelming amount and complexity is not trivial. It requires established processes and models for data collection, storage, and analysis with further visualization and integration into existing business landscape and decision-making practices. Without it, many companies still have suboptimal reporting and poor market insight. The inability of obtaining sufficient evidence in the form of data, knowledge or models for informed decisions can push product managers to rely only on the intuitive vision of the product and its life cycle.
For start-up companies, the question of designing decision-making processes and practices with both managerial and technological aspects of new product development has already been raised by Eric Ries [24] and Steve Blank [25]. They proposed “Customer Development Model” and “Build-Measure-Learn” concept that are widely adopted by start-up companies and form the basis for other decision-support solutions for software companies (e.g. The Early Stage Software Startup Development Model [26]). However, the question of how decision-making should evolve through companies’ growth and development and what are the prime challenges and trade-offs of decision-making have not received too much attention in academic literature. 2.2
Evidence-based decision-making has its origins in evidence-based management that can be defined as practices of “making decisions through the conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgment, evidence from the local context, a critical evaluation of the best available research evidence, and the perspectives of those people who might be affected by the decision” [27]. Being more an umbrella term rather than a rigorous decision-making approach, evidence-based management does not neglect intuition as a valuable source of expertise. Instead, it assumes that for effective managerial decision-making this intuition should be formalized in the form of verifiable knowledge and supplemented by datadriven and model-driven business analytics as well as consideration of prior experience and conducted researches [28].
Evidence-based decision-making gains a significant boost with the new opportunities to collect, store and analyze data. A new frontier in data management appeared as the “Big Data” concept [29] twisted the overall focus of evidence-based management. The key challenges in the early days of evidence-based management were lack of data that could be turned into pieces of evidence and tweaking the techniques that could help to get at least some proxy data. Nowadays, companies usually have a lot of data, but turning this available “Big Data” into “Smart Data” that could serve as pieces of evidence is not trivial [30]. While recent business and economic studies showed overall rapid adoption of evidence-based management with significant positive impact on company performance [31, 32], defining these practices within the context of a particular company or even an industry could be quite a challenging task.
Evidence-based management has a strong connection with knowledge management as a discipline that intends to manage the processes of creating, organizing, and using the information and knowledge within an organization [33]. Robust knowledge management processes are crucial for effective evidence-based decision-making as it allows to formalize and integrate managers’ experience and expertise in decision-making. Moreover, quite often the product management activities are distributed among a group of managers, and each of them has her area of expertise with “tacit” knowledge regarding aspects of product management they are responsible for. In this case, effective knowledge management means storing and sharing this knowledge adequately to ensure informed and coordinated decision-making [34].
The evolution of big data analytics and knowledge management have given a new way of exploring new frontiers in decision making in high volume, front-line decisions. These frontiers are associated with new types and sources of data available, as well as new approaches and methods of analysis to identify evidence required for decisionmaking [35]. Software companies are the drivers of this process, providing its customers on the B2B market with the possibility to integrate various processes and gain insight into day-to-day business operations in real-time. They have developed modern Business Intelligence systems to analyze current data and historical facts to improve decision-making. The question remains, to what extent have software companies adopted the data-driven SPM approach by themselves and used rich data for decisionmaking in business, product and project management. 3
The following research questions drive the first step in this study: ─ RQ1: How is the decision-making process in software product management organized by software companies? To what extent are the SPM related decision-making processes and practices are formalized and evidence-driven? ─ RQ2: What are the prime sources and types of pieces of evidence as well as models and tools used for decision-making in SPM? ─ RQ3: What are the obstacles towards broader implementation of evidence-based software product management? To address these questions, we made a study that can be classified as a positivist, exploratory multiple case study. The case sampling strategy was guided by the diverse case approach with its primary objective to achieve maximum variance along relevant dimensions [36]. Referring to the research questions, the goal is to identify decisionmaking practices and processes as well as to understand the logic behind them. To achieve that purpose, a within-case analysis was conducted with the analytical strategy of explanation-building based on the description of the cases, i.e., our study can be classified as exploratory case research.
We present all analysis in this paper in the form of propositions for further research. These propositions are grounded on qualitative data received through the series of semistructured interviews with product managers and executives from five software companies. Companies selected for the study have their operation on EU and/or Russian markets. All companies have developed a SaaS solution for their customers. The SaaS solutions can be considered as mass-market services, where minor possibilities for customization are also available.
The data collection consisted of interviews that we consider as the first step of our longitudinal research project. The length of interviews varies from 2 to 3 hours. Their goal was to identify pressure points of decision-making in SPM, motivate companies to participate in the longitudinal study and assess both current status quo and product managers’ perceptions of existing processes and practices. The data obtained covered the following topics: General information about the company and products: name, industry, market, number of employees, number of customers, maturity level, business model, number of products, products type and critical characteristics, product maturity level, etc. SPM practices and processes: SPM frameworks used, product activities allocation across business units, collaboration principles between business units, development approach, product manager’s roles, and responsibilities, SPM tools used, SPM performance assessment principle, etc. SPM decision-making principles: formal regulation and written policies on SPM activities, factors of risks and uncertainty to consider by the product manager, types of data collected for SPM decision-making, models, and tools used to process provided data, information system support for SPM processes, etc. 4 4.1
A brief overview of the case companies is presented below and summarized in Table 1.
Company A provides a SaaS solution that includes a web service interface and mobile applications for different platforms. However, the company considers it as a single product. The product is highly dependent on the government's regulations. SPM duties are divided between two co-founders. One of them is responsible for business and product strategy, the vision of the firm and targeted market, while another co-founder is responsible for product functionality, product development and project management.
Company B offers highly specialized software fitting both the public and private market needs. The management believes that any possible market ceiling is far away, and the company will continue to grow at a rapid pace, driven by specific legislative initiatives taken in the EU that oblige other companies to use this type of software. The CEO is deeply involved in product management and responsible for defining product strategy, pricing policy, and other business-related issues. Product Owner is responsible for the rest of product management activities, mostly on tactical and operational levels.
Company C develops a fully integrated, software package for automating business processes such as accounting, ERP, CRM, etc. To date, the company has focused mainly on the local market, seeing more opportunities for organic growth within the country. Despite a large number of customers and the company's size, the company remains privately owned, which has a high influence on the culture of informal communication inside the company and decision-making through negotiations, including direct ones with company shareholders and top-level managers. The company has five product managers, each of whom is responsible for different modules in a single system. A product director working closely with the owners of the company is responsible for major strategic planning issues and defining the vision of the product.
Company D is a global player that offers accounting, CRM and industry-specific solutions for SMEs in Accountancy, Manufacturing, Professional Services, and Wholesale. Well-defined corporate governance practices and procedures are already established, and currently, the company is revising its software product management practices to make them more evidence-based. The company expects that it will improve decision-making and allow senior management to have better control over processes in the company and a better capability to assess product management performance.
Company E is a multinational company specializing in Internet-based services. The company provides clients with a variety of services, some of the provided services are technological platforms for taxi booking, work-at-home jobs search, food delivery, etc. Products form the unique ecosystem of services aimed to cover as many aspects of human life as possible and share the brand name, infrastructure and usually have are mutually integration. Still, the products vary widely regarding product management practices and development processes. The company holds oneself out as the analyticaldriven; moreover, the analytical department is inter-product and, on request, provides analytical support to management processes.
Ownership Number of Employees Number of Products (Modules) Market Type Product Type Number of Clients Consolidation of within-case analysis findings with a cross-case analysis facilitates a deeper understanding of the cases and accentuates the differences between them [37].
All companies are aware of evidence-based and data-driven management; still, all interviewees share the vision that this approach can be implemented fully only in large public companies with well-established organizational structure and available resources to establish the data-analytics business unit. Only Companies D and E were ready to implement a full-fledged evidence-based product management approach that includes formalization of evidence-based decision-making principles. Still, even for them it is a resource-intensive and non-trivial task: “It took us more than a year to form a metrics system that we think is suitable to track the product development and measure it market performance... this work we did together with our analytical department and much has been done by analogy with existing metrics for other, more mature products” (Company E).
At the same time, all interviewees specified that decisions in their companies are adequately grounded and to a certain extent data-driven. In Company C product managers try to support all sufficient decisions with analytics, all processes for obtaining pieces of evidence are not formalized. The company, despite the size, is trying to remain the spirit of the startup and afraid that formalization will reduce its ability to react to various market challenges and “keep an ear to the ground." The main reason for providing managers with broad responsibilities and opportunities in decision-making is part of the corporate culture and can be considered even as a competitive advantage that ensures solid growth: “…the product manager can blow up the company. Definitely. Moreover, everyone here has full awareness of that… but we are growing perfectly. Until the situation remains, we do not need formalized and evidence-based processes” (Company C).
The growth issue is even more important for SMEs. Both Companies A and B claimed that the introduction of evidence-based decision-making approach would possibly not only lead to a slowdown in their growth. Additionally, these companies specified that they have a lack of competences if dealing with the data available: "We are collecting a lot of data, but simply not using it… everything falls on the shoulders of the members of our small team… we cannot afford hiring someone else, for now, to do this” (Company A) and “…numbers do not say anything, numbers just say that there is an issue, but they do not supply solutions or the way to go. You have to investigate and… If the information is not enough, emotions or intuition may help…” (Company B).
Evidence-based software product management is considered primarily as an instrument of tactical SPM (or product planning according to ISPMA and similar frameworks [3]). Companies B, C, D, and E used various techniques for requirements prioritization, roadmapping and release planning that can be classified as evidence-driven. These processes require use of technical/structural data in decision-making, including feedback and bug reports collected by support unit, surveys with existing and potential clients collected by the sales team and key account managers, available log information regarding product usage.
The prime reason for product managers’ perception on considering evidence-based SPM only on the tactical level seemed to be a lack of clear vision on what information could serve as proper evidence for strategic purposes. The majority of product managers considered evidence as a synonym to metric. However, more sophisticated pieces of evidence and data processing techniques are required for strategic decision-making. For instance, Company C confirmed that data related to sales and pricing is available, but not used for the decision-making: “I have access to their CRM system...We have signals. If they are losing too many clients in the particular branch but... no, we do not work with this data...”. Additionally, Company B regrets that they are overworked and don’t have competences to deal with such important source of evidence as knowledge collected through the analysis of the decisions made in the past: “It could be interesting if we could take all the iterations for the last five years and you estimate and ask managers whether it was a mistake or not... I think that they are not able to do it right now... they are busier with the product...”. Besides lack of vision towards sources of evidence and competences to work with them, even large companies feel that they can follow the leaders in strategic product management having better product and service quality as a strategic advantage: “The product is very successful, we have exponential annual growth… many decisions related to pricing and other economic-design issues were borrowed from the similar platforms” (Company E). 4.3
We can formulate several propositions from our analysis in the form of answers to the identified research questions. Further field research should test these propositions in practice. ─ RQ1: How is the decision-making process in software product management organized by software companies? To what extent are the SPM related decision-making processes and practices formalized? The practice of decision making varies widely in software companies depending on their size and maturity level. Moreover, inside the same company, practices may vary from product to product, depending on the maturity level of the product and product managers competencies. The transition towards formalized evidence-based decision-making starts with tactical and operational decision-making but rarely comes to strategic decisionmaking level. Tactical evidence-based SPM allow gradual development of the product and getting new consumers without violating the value of the product to the existing ones. Such thoroughness is usually not required in strategic SPM. Formalized evidence-based decision-making processes are very resource consuming, regarding time, money, and people. Therefore, a company starts implementing them only after a particular stage of product (and company) maturity when the product is visible on the market and accepted by customers. ─ RQ 2: What are the prime sources and types of pieces of evidence as well as models and tools used for decision-making in SPM? Quantitative technical data is usually used as evidence for tactical and operational decision-making. However, strategic decision-making requires dealing with various sources of evidence that may also be of non-quantitative nature and challenging to quantify. An important source of evidence is accumulated past experience in the form of knowledge. In companies with well-established product management practices and processes, a lot of decisions on all three levels for new products are made taking into account prior experience. ─ RQ 3: What are the obstacles towards broader implementation of evidence-based software product management? There is a pervasive need for easy-to-use approaches and frameworks to support evidence-based SPM migration. Lack of clear vision regarding typology of evidences that could be served for informed decision-making could be names as prime obstacles towards broader implementation of evidence-based software product management. These issues are supplemented by immature communication and knowledge sharing practices, poor integration between various SPM tools and systems used and scarcity of competency in data analytics, simulation modeling, and knowledge management.
The cross-case analysis revealed that it is possible to identify a clear trend towards the formalization of practices and processes for software product management, along with the company's development and growth. However, to the best of our knowledge, no attempt has been made in the academic literature to describe the logic of transforming product management practices towards evidence-based ones, including the critical milestones on this path. Development of the Product Management Maturity model that specify various aspects of transition towards evidence-based SPM could be used as a valuable supporting instrument. This model could complement other existing ones for development and operations [38] and project management [39, 40].
The in-depth interviews in five companies cannot produce a generalizable nomothetic theory [41]. Instead, we consider this qualitative study as idiographic, as it throws a glance on decision-making related to software product strategy in specific cases. To enhance the validity of this case study, further research is needed to shed light on current decision-making practices in the industry.
Besides a more substantial systematic study on current practices, SPM will benefit from a comprehensive review that will allow providing a typology of evidences for decision-making as well as methods for the analysis. This could consist of a rigorous theoretical and practical analysis of the power and limitations of available evidencebased SPM practices, methods, and techniques as well as development of a software product management maturity model with the focus on decision-making practices. This model could guide software companies in their transition towards evidence-based software product management that in term can reduce the likelihood of poor decision-making that leads to poor business success [42]. 5
This study observed the current state of SPM decision making, managers’ perceptions towards them as well as and the needs of the case companies. The presented perspective on decision-making practices complements and extends the existing literature on status quo and challenges in software product management [14, 15]. Software Product Management is a relatively young practice, and despite the presence of some significant research explaining its aims and objectives, the question of practical significance is still debatable. Software product managers have a strategic, cross-functional role that requires visibility into every phase of the product life-cycle. Immaculate product data, combined with the transactions surrounding each product, should, in theory, provide the product managers the insights they need to ensure product profitability and identify areas for improvement. Although there has been much discussion in the software business community on roles and area of responsibilities of software product managers, relatively little attention has been paid to the decision-making processes, practices, and principles.
This case study reveals that companies tend to try to formalize the existing decisionmaking practices to make them more transparent and evidence-driven. The more difficult question is that having the intention to move from intuition-based decision-making to a data-driven one, managers are often faced with the lack of a clear vision or understanding on what could serve as evidence in SPM and what techniques are required to make informed decisions. This becomes especially evident when dealing with strategic aspects of SPM associated with the product strategy and analysis of the product in relation to its market.
The first author has been partially supported by the Academy of Finland Grant no 310827 (MobiSPSULUT) 13. 14. 15.