Defining and observing the dimensions of success in software engineering and related fields is challenging due to the elusive and changing nature of success [1]. An additional layer of challenge is created by the special characteristics of public procurement of information systems (IS), where a complex process meets a complex information system. No current success model takes into account these characteristics, or aims to measure success on a practical level. This research problem is addressed in the doctoral dissertation described in this research plan. The goal of this research is to create a tool or framework to quantify, measure, and observe success in the context of public procurement of IS. The motif and the research question guiding the dissertation as a whole is: How to measure success in public procurement of information systems?
Public procurement of IS refers to purchases where public money is spent on ICT services and works for government organisations or for the benefit of taxpayers [2,3,4]. The demand for research and a culture change is recognised both in academic publications [5,6,7] as well as on national strategy level [8]. The impact of acquisitions made with public funds is noteworthy both for the economic and social structures of society [5,9,10]. In many high-income countries public agencies produce the majority of requests for IS procurement, which in turn affects the private market [5,6]. Constant evolution and continuous development of procurement practices are essential to meet the growing need for public services, as digitalisation and technological advances, changes in demography, and the decreasing of public resources keep the trend for public procurement of IS ever-growing [9,10,11,12]. As a factor in economic and social sustainability, a successful procurement is in the interest of all stakeholders involved in the process [13,14].
Figure 1 presents the existing knowledge gap this research addresses. While measuring success is a topic of interest in many related research fields, there are no standardised models or methods for measuring success or benefits realisation that would consider the views of various stakeholders and be able to adapt to the changing societal and technological environment with sufficient efficiency. The large number of theories and frameworks from many research fields shed light on defining and measuring success, but no model or tool exists to quantify success in a context or environment similar to public procurement of IS [15,16,17].
Research on public procurement of IS has focused on its special nature and characteristics and the challenges they create. A public procurement is essentially a heavily regulated project with multiple stakeholders who all have their own agenda [7,18]. Being public, an acquisition process is an enabler of current policies and legislation [19]. The procurement process is undergone to acquire a complex IS that is designed and developed to fit a specific purpose [3,7,12]. These characteristics create a myriad of challenges that are well recognised both in academic literature and practice. Acknowledging these challenges is essential in order to identify elements, steps, and factors leading to success with which the procurement has an opportunity to achieve its greatest possible value [13]. Past research on challenges forms a solid foundation for new ideas and enables building of solutions.
The public procurement process follows the rigid guidelines and distinct objectives presented by legislation [2,20]. In public procurement predefined phases follow one another, and the needs and requirements of the end user are identified pre-tender [4,21]. Understanding the procurement process as a whole [6,22], collaboration and effective interaction [20], and careful preparation and assessment [3,23] carry a significant weight throughout the whole procurement process. Off-the-shelf information systems rarely suit the needs of the purchasing unit, and as the acquired products and services create complex, unique systems, comparing software is challenging or impossible in the absence of a comparison base [3,7,12]. The impact of differing stakeholder views on success is recognised in literature [1,18]. Successful communication between procuring parties includes defining common goals, finding consensus, preparing for compromises, and engaging all stakeholders throughout the process [3,24,25]. Long-term relationships between procurement unit and supplier have been found to have a positive effect on the efficiency, quantity, and quality of innovations, which is more evident in the less regulated private sector [5,22].
Despite its growing impact on both government organisations and taxpayers, public procurement of IS has long been a dormant area of research [6,26]. In recent years, academic interest has been on the rise especially in the Nordics, which is evidenced by the growing amount of dissertations and publications covering the topic from various points of view (see e.g. [22,27,28]). Public procurement of IS has similar qualities to innovation and services procurement and a lot can be learnt from research done on these topics. However, while these fields provide valuable insights to the software engineering and information systems research communities, the scope is slightly misaligned [29]. For this reason public procurement of IS needs to be studied as a separate phenomenon.
In project management, plenty of models and methods for observing and measuring success have been developed over the years [30,31,32], and empirical research on project success has concentrated on the views of project management [33,34]. The Iron Triangle of project management, i.e. cost, time, and quality, are the staples for measuring project success [1,30]. These values are measurable e.g. in the forms of budget, project schedule, and technical properties of the IS, while subjective values such as job satisfaction, development of interorganisational communication and decision-making processes, and attention to detail are subjective and thus more challenging to quantify [1,7,32,35]. The impact of these so called softer values may emerge or change over time, and the perceived views on success may need to be observed long-term to see a trend [18,33]. IS success has been approached through many models, with The DeLone and McLean Model of Information System Success [36] being one of the most used [15]. Perceived usefulness and technology acceptance have been at the centre of other popular IS success models [37,38]. Success metrics and dimensions are a topic of interest in software engineering [1,39].
To address the knowledge gap of measuring success in the context of public procurement of IS, a tool or framework is developed. To ensure the tool is built on scientific findings and research, the doctoral dissertation follows the design science methodology [40,41,42]. The artefact is created to observe, measure and ensure continued success throughout the life cycle of a public IS procurement process. To create appropriate metrics for measuring success, a definition for success within the context must be explored. First, the concept of perceived success in the various user groups of public procurement will be observed, existing models examined, and the overall state of the research discussed. Later an artefact is designed, developed, and evaluated based on the findings of the earlier phases. The work is divided into phases similar to Peffers et al. [41]. Figure 2 provides a schedule for the ongoing three years of doctoral studies. The studies and main activities of each phase are described below in more detail.
The doctoral research is guided by the following research questions: How to measure success in public procurement of IS? What are the metrics for assessing success in public procurement of IS?
The initiative for this doctoral research rises originally from practice, which is one of the four identified motives for research following the design science principles [41]. A client-initiated case study on success factors using the Delphi technique has been carried out to identify the research problem, to get tentative results on success definitions, and to draft objectives for a solution. Generalising results are to follow in the second phase.
RQ's: What factors define a successful public IS procurement in a long-standing multi-buyer relationship?
Methods include a case study using the Delphi technique. Main tasks are identifying the research problem, defining motivations to justify the value of a solution, and to establish the current state of research.
Deliverables include one journal article of a client-initiated case study.
The objective of the first two phases is to learn how success is perceived by the various stakeholders participating in procurement (e.g. ICT vendors, public agencies, diverse identifiable user groups of the system). Defining success is crucial in order to create an understanding on the success factors and points of failure surrounding the practices of public IS procurement. To create drafts of the artefact and to define its objectives, a mixed method study is conducted. Quantitative data is gathered from HILMA1 and semi-structured interviews with stakeholders of multiple procurement processes are arranged. All major stakeholders of public procurement (vendor, procurement unit, end users) are expected to participate. The metrics identified in this phase are a compromise or consensus between all stakeholders. RQ's: How is success defined in public procurement of IS by stakeholders? What elements or factors define the success of public procurement of IS? What amount of public IS acquisitions succeed?
Methods include quantitative and qualitative data analysis, and semi-structured interviews.
Main tasks are defining the purpose of the solution, observing the perception of success in past and ongoing IS procurement by various user groups and stakeholders, quantifying softer variables, and identifying and establishing metrics for measuring success.
Deliverables include one conference publication. A large data pool allows for multiple studies and publications also outside the scope of this dissertation.
The solution created in this study aims at ensuring and measuring the success of public procurement of IS as perceived by various stakeholders. The data gathered and analysed in previous phases will provide the foundation and a preliminary requirement specification for the development of the artefact. Existing models for success specification, such as the DeLone-McLean model (see [36]) are invaluable in the creation of a more practical tool. The exact nature of the artefact is yet to be decided. Potential drafts include e.g. a practical tool, a conceptual framework, or a model or method. The solution is designed for the cultural procurement context in Finland or a similar environment.
RQ's: How can success perceived by various stakeholders and user groups be measured? What are the metrics for assessing success in public procurement of IS?
Main tasks include the design and development of an artefact for observing success throughout the life cycle of a public IS procurement process.
Deliverables include the artefact, the requirement specification, and the design of the artefact (e.g. concept design, architecture, user interface etc.). Publications include a technical report and a conference publication.
After a proof of concept level solution has been created, it will be implemented to production in one or more ongoing procurement processes. Criteria for evaluating the solution will be decided on to observe its functionality. A longitudinal study would be needed to observe the long-term impact of the artefact, but in the scope of the dissertation a short-term evaluation is sufficient. This phase allows fine-tuning or more extensive restructuring of the artefact if needed to meet its goals. Accurate research questions and methods to conduct the study will be decided on closer to the phase, after the nature of the solution is uncovered.
Research problem: Is the created solution suitable for evaluating, measuring, and observing success in public procurement of IS?
Main tasks include applying the solution within a suitable context, creating metrics for evaluation and analysis, iterating back to design as needed, and addressing further development and future research.
Deliverables include a minimum of one conference publication.
Academic publications will be compiled into a doctoral dissertation. Findings of each study will be communicated systematically to the academic and practitioner communities throughout the study when appropriate. Findings will be published in peer-reviewed journals or conferences, and all publications will be made open access through self-archiving to ensure open availability, visibility, and long-term preservation.
I have joined the three-year Finnish Acceleration of Scientific Talent (FAST) programme starting in August 2024 and look to complete my PhD within three years in accordance with the programme. My compilation dissertation will consist of up to four scientific articles that seek definitions, answers, and solutions to success in public procurement of IS.
As seen in figure 2, there are five semesters for collecting data, analysing and concluding results, and developing the tool. Articles and conference papers will be composed and submitted for publication when any relevant findings need to be communicated to the academic or practitioner audiences. Assuming at least three articles get successfully published, the final semester will be dedicated to compiling the articles into a coherent dissertation and submitting the work for preliminary inspection. The public examination is to be held before or soon after the scheduled end of the educational FAST-programme pilot in July 2027.
The most practical result of the study is the proposed solution for measuring success. It is planned to be used as a tool by different stakeholders during the lifecycle of a public procurement to quantify, measure, and ensure the success of the procurement process and the acquired IS. On a larger scale the tool aims to advance practices of public procurement. In order to create an artefact for measuring success, a definition and metrics must be uncovered. Defining success in the context of public procurement of IS has not been done before.
The expected contributions are academically significant, as they provide new and updated information on a topic that has not been in the centre of attention for researchers in the recent years. A thorough literature review connects past research and developments in the field and shows the state of the art of public IS procurement research while presenting a new synthesis. The dataset gathered in the mixed method study allows for multiple studies from different angles also outside the scope of the dissertation, and e.g. comparative studies in international collaboration are possible.
Heartfelt thanks to professors Samuli Pekkola and Tommi Mikkonen for their supervision and advice so far, and for guiding me on my journey to becoming a researcher in the first place. Thanks to the rest of the Software Engineering FAST team at University of Jyväskylä for their support.
(S. Marjanen) https://www.softwareengineering.fi/researchers/sanni-marjanen/ (S. Marjanen)