Public procurement of information systems (IS) research is gaining traction as digitalisation of services and the continuous, rapid development of technology keep the number of acquired IS software growing. Its impact on economy and societal structures is significant, yet there is no tool or method to assess and measure the success of the procurement process nor of the acquired IS. A successful procurement is in the interest of all stakeholders involved in the process, and recognising challenges is essential in order to identify elements, steps and factors leading to success with which the procurement has an opportunity to achieve its greatest possible value. This research plan presents the roadmap of a PhD dissertation on the topic. The outlined studies aim at advancing public procurement practices and to develop tools and methods to observe, measure, and ensure success in the context of public procurement of IS. The research problem is identified, motivation and knowledge gap presented, research methodology and timeline for completing the research within three years described, and expected results summarised.
Defining and observing the dimensions of success in software engineering and related fields is
challenging due to the elusive and changing nature of success [
Public procurement of IS refers to purchases where public money is spent on ICT services and works
for government organisations or for the benefit of taxpayers [
Figure 1 presents the existing knowledge gap this research addresses. While measuring success is a topic of interest in many related research fields, there are no standardised models or methods for measuring success or benefits realisation that would consider the views of various stakeholders and be able to adapt to the changing societal and technological environment with suficient eficiency. The large number of theories and frameworks from many research fields shed light on defining and measuring success, but no model or tool exists to quantify success in a context or environment similar to public procurement of IS [15, 16, 17].
Research on public procurement of IS has focused on its special nature and characteristics and the
challenges they create. A public procurement is essentially a heavily regulated project with multiple
stakeholders who all have their own agenda [
The public procurement process follows the rigid guidelines and distinct objectives presented by
legislation [
Despite its growing impact on both government organisations and taxpayers, public procurement
of IS has long been a dormant area of research [
In project management, plenty of models and methods for observing and measuring success have
been developed over the years [30, 31, 32], and empirical research on project success has concentrated
on the views of project management [33, 34]. The Iron Triangle of project management, i.e. cost, time,
and quality, are the staples for measuring project success [
To address the knowledge gap of measuring success in the context of public procurement of IS, a tool or framework is developed. To ensure the tool is built on scientific findings and research, the doctoral dissertation follows the design science methodology [40, 41, 42]. The artefact is created to observe, measure and ensure continued success throughout the life cycle of a public IS procurement process. To create appropriate metrics for measuring success, a definition for success within the context must be explored. First, the concept of perceived success in the various user groups of public procurement will be observed, existing models examined, and the overall state of the research discussed. Later an artefact is designed, developed, and evaluated based on the findings of the earlier phases. The work is divided into phases similar to Pefers et al. [41]. Figure 2 provides a schedule for the ongoing three years of doctoral studies. The studies and main activities of each phase are described below in more detail.
The doctoral research is guided by the following research questions: How to measure success in public procurement of IS? What are the metrics for assessing success in public procurement of IS?
The initiative for this doctoral research rises originally from practice, which is one of the four identified motives for research following the design science principles [41]. A client-initiated case study on success factors using the Delphi technique has been carried out to identify the research problem, to get tentative results on success definitions, and to draft objectives for a solution. Generalising results are to follow in the second phase.
RQ’s: What factors define a successful public IS procurement in a long-standing multi-buyer relationship?
Methods include a case study using the Delphi technique.
Main tasks are identifying the research problem, defining motivations to justify the value of a solution, and to establish the current state of research.
Deliverables include one journal article of a client-initiated case study.
The objective of the first two phases is to learn how success is perceived by the various stakeholders participating in procurement (e.g. ICT vendors, public agencies, diverse identifiable user groups of the system). Defining success is crucial in order to create an understanding on the success factors and points of failure surrounding the practices of public IS procurement. To create drafts of the artefact and to define its objectives, a mixed method study is conducted. Quantitative data is gathered from HILMA 1 and semi-structured interviews with stakeholders of multiple procurement processes are arranged. All major stakeholders of public procurement (vendor, procurement unit, end users) are expected to participate. The metrics identified in this phase are a compromise or consensus between all stakeholders.
RQ’s: How is success defined in public procurement of IS by stakeholders? What elements or factors define the success of public procurement of IS? What amount of public IS acquisitions succeed? Methods include quantitative and qualitative data analysis, and semi-structured interviews.
Main tasks are defining the purpose of the solution, observing the perception of success in past and ongoing IS procurement by various user groups and stakeholders, quantifying softer variables, and identifying and establishing metrics for measuring success.
Deliverables include one conference publication. A large data pool allows for multiple studies and publications also outside the scope of this dissertation. 1Service for notices on public procurement in Finland, https://www.hankintailmoitukset.fi/en/
The solution created in this study aims at ensuring and measuring the success of public procurement of IS as perceived by various stakeholders. The data gathered and analysed in previous phases will provide the foundation and a preliminary requirement specification for the development of the artefact. Existing models for success specification, such as the DeLone-McLean model (see [36]) are invaluable in the creation of a more practical tool. The exact nature of the artefact is yet to be decided. Potential drafts include e.g. a practical tool, a conceptual framework, or a model or method. The solution is designed for the cultural procurement context in Finland or a similar environment.
RQ’s: How can success perceived by various stakeholders and user groups be measured? What are the metrics for assessing success in public procurement of IS?
Main tasks include the design and development of an artefact for observing success throughout the life cycle of a public IS procurement process.
Deliverables include the artefact, the requirement specification, and the design of the artefact (e.g. concept design, architecture, user interface etc.). Publications include a technical report and a conference publication.
After a proof of concept level solution has been created, it will be implemented to production in one or more ongoing procurement processes. Criteria for evaluating the solution will be decided on to observe its functionality. A longitudinal study would be needed to observe the long-term impact of the artefact, but in the scope of the dissertation a short-term evaluation is suficient. This phase allows fine-tuning or more extensive restructuring of the artefact if needed to meet its goals. Accurate research questions and methods to conduct the study will be decided on closer to the phase, after the nature of the solution is uncovered.
Research problem: Is the created solution suitable for evaluating, measuring, and observing success in public procurement of IS?
Main tasks include applying the solution within a suitable context, creating metrics for evaluation and analysis, iterating back to design as needed, and addressing further development and future research.
Deliverables include a minimum of one conference publication.
Academic publications will be compiled into a doctoral dissertation. Findings of each study will be communicated systematically to the academic and practitioner communities throughout the study when appropriate. Findings will be published in peer-reviewed journals or conferences, and all publications will be made open access through self-archiving to ensure open availability, visibility, and long-term preservation.
I have joined the three-year Finnish Acceleration of Scientific Talent (FAST) programme starting in August 2024 and look to complete my PhD within three years in accordance with the programme. My compilation dissertation will consist of up to four scientific articles that seek definitions, answers, and solutions to success in public procurement of IS.
As seen in figure 2, there are five semesters for collecting data, analysing and concluding results, and developing the tool. Articles and conference papers will be composed and submitted for publication when any relevant findings need to be communicated to the academic or practitioner audiences. Assuming at least three articles get successfully published, the final semester will be dedicated to compiling the articles into a coherent dissertation and submitting the work for preliminary inspection. The public examination is to be held before or soon after the scheduled end of the educational FAST-programme pilot in July 2027.
The most practical result of the study is the proposed solution for measuring success. It is planned to be used as a tool by diferent stakeholders during the lifecycle of a public procurement to quantify, measure, and ensure the success of the procurement process and the acquired IS. On a larger scale the tool aims to advance practices of public procurement. In order to create an artefact for measuring success, a definition and metrics must be uncovered. Defining success in the context of public procurement of IS has not been done before.
The expected contributions are academically significant, as they provide new and updated information on a topic that has not been in the centre of attention for researchers in the recent years. A thorough literature review connects past research and developments in the field and shows the state of the art of public IS procurement research while presenting a new synthesis. The dataset gathered in the mixed method study allows for multiple studies from diferent angles also outside the scope of the dissertation, and e.g. comparative studies in international collaboration are possible.
Heartfelt thanks to professors Samuli Pekkola and Tommi Mikkonen for their supervision and advice so far, and for guiding me on my journey to becoming a researcher in the first place. Thanks to the rest of the Software Engineering FAST team at University of Jyväskylä for their support. [12] C. E. Moe, T. Päivärinta, View of Challenges In Information Systems Procurement in the Public Sector, Electronic Journal of e-Government 11 (2013) 308–323. URL: https://academic-publishing. org/index.php/ejeg/article/view/589/552. [13] M. Januska, A. Palacka, Critical success factors for public procurement of innovative solutions in Central Europe: Empirical study, E&M Economics and Management 23 (2023) 24—-41. doi:10. 15240/tul/001/2023-2-002. [14] C. E. Moe, A. C. Risvand, M. K. Sein, Limits of Public Procurement: Information Systems Acquisition, volume 4084, Springer, Berlin, Heidelberg, 2006, pp. 281–292. URL: https://link.springer.com/ chapter/10.1007/11823100_25. [15] G. G. Gable, D. Sedera, T. Chan, Re-conceptualizing Information System Success: The IS-Impact Measurement Model, Journal of the Association for Information Systems 9 (2008) 377–408. doi:10.17705/1jais.00164. [16] C. Herzog, A. Richter, M. Steinhüser, U. Hoppe, M. Koch, Methods And Metrics For Measuring The Success Of Enterprise Social Software – What We Can Learn From Practice And Vice Versa, ECIS 2013 Completed Research 132 (2013). URL: https://aisel.aisnet.org/ecis2013_cr/132/. [17] T. D. Nguyen, T. M. Nguyen, T. H. Cao, The relationship between IT adoption, IS success and project success, in: International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2016. doi:10.1109/ICACCI.2016.7732207. [18] M. Ylinen, S. Pekkola, Searching for Success in a Successful IS Acquisition, in: Proceedings of the 51st Hawaii International Conference on System Sciences, 2018, pp. 4793–4802. URL: http://hdl.handle.net/10125/50492. [19] T. Williams, H. Vo, M. Bourne, P. Bourne, R. Kirkham, G. Masterton, P. Quattrone, C. Toczycka, Benefits realisation: case studies in public major project delivery with recommendations for practice, Production Planning & Control (2023) 1–21. doi:10.1080/09537287.2023.2256287. [20] Y. Caloghirou, A. Protogerou, P. Panagiotopoulos, Public procurement for innovation: A novel eGovernment services scheme in Greek local authorities, Technological Forecasting and Social Change 103 (2016) 1–10. doi:10.1016/j.techfore.2015.10.016. [21] M. Rolfstam, Public Procurement of Innovation, Ph.D. thesis, Lund University, 2008. URL: https: //portal.research.lu.se/en/publications/public-procurement-of-innovation. [22] R. Ghezzi, T. Mikkonen, On Public Procurement of ICT Systems: Stakeholder Views and Emerging
Tension, in: 14th International Conference on Software Business, ICSOB 2023, 2023, pp. 61–76. [23] V. Lember, R. Kattel, T. Kalvet, Quo vadis public procurement of innovation, Innovation: The European Journal of Social Science Research 28 (2015) 403–421. doi:10.1080/13511610.2015. 1043245. [24] H. Kautonen, M. Nieminen, Julkisten ICT-hankintojen käyttäjäkeskeisyyden edellytysten kriittinen tarkastelu [Critical examination of the prerequisites for user-centricity in public ICT procurement], Hallinnon Tutkimus 38 (2019) 155–173. doi:10.37450/ht.98037. [25] C. Edquist, J. M. Zabala-Iturriagagoitia, Public Procurement for Innovation as mission-oriented innovation policy, Research Policy 41 (2012) 1757–1769. doi:10.1016/j.respol.2012.04.022. [26] K. V. Thai, Public Procurement Re-examined, Journal of Public Procurement 1 (2001) 9–50.
doi:10.1108/JOPP-01-01-2001-B001. [27] H. Vilpponen, You Get What You Order: Required Expertise in the Procurement of Public Services,
Ph.D. thesis, University of Jyväskylä, 2021. URL: https://jyx.jyu.fi/handle/123456789/78852. [28] A. Koski, On the Provisioning of Mission Critical Information Systems based on Public Tenders,
Ph.D. thesis, University of Helsinki, 2019. URL: http://hdl.handle.net/10138/303511. [29] B. Lenderink, J. I. Halman, H. Voordijk, Innovation and public procurement: from fragmentation to synthesis on concepts, rationales and approaches, Innovation: The European Journal of Social Science Research 35 (2019) 650–674. doi:10.1080/13511610.2019.1700101. [30] R. Atkinson, Project management: cost, time and quality, two best guesses and a phenomenon, its [sic] time to accept other success criteria, International Journal of Project Management 17 (1999) 337–342. doi:10.1016/S0263-7863(98)00069-6. [31] K. R. T. Larsen, A Taxonomy of Antecedents of Information Systems Success: Variable Analysis Studies, Journal of Management Information Systems 20 (2003) 169–246. doi:10.1080/07421222. 2003.11045768. [32] D. Baccarini, The Logical Framework Method for Defining Project Success, Project Management
Journal 30 (1999) 25–32. doi:10.1177/875697289903000405. [33] K. Davis, A method to measure success dimensions relating to individual stakeholder groups, International Journal of Project Management 34 (2016) 480–493. doi:10.1016/j.ijproman. 2015.12.009. [34] T. Cooke-Davies, The ’real’ success factors on projects, International Journal of Project Management 20 (2002) 185–190. doi:10.1016/S0263-7863(01)00067-9. [35] S. Petter, W. DeLone, E. R. McLean, The Past, Present, and Future of ’IS Success’, Journal of the
Association for Information Systems 13 (2012) 341–462. doi:10.17705/1jais.00296. [36] W. H. Delone, E. R. McLean, The DeLone and McLean Model of Information Systems Success: A Ten-Year Update, Journal of Management Information Systems 19 (2003) 9–20. doi:10.1080/ 07421222.2003.11045748. [37] F. D. Davis, Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information
Technology, MIS Quarterly 13 (1989) 319–340. doi:10.2307/249008. [38] V. Venkatesh, M. G. Morris, G. B. Davis, F. D. Davis, User Acceptance of Information Technology:
Toward a Unified View, MIS Quarterly 27 (2003) 425–478. doi: 10.2307/30036540. [39] M. V. Mäntylä, M. Jørgensen, P. Ralph, H. Erdogmus, Guest editorial for special section on success and failure in software engineering, Empirical Software Engineering 22 (2017) 2281–2297. doi:10.1007/s10664-017-9505-5. [40] A. R. Hevner, S. T. March, J. Park, S. Ram, Design Science in Information Systems Research, MIS
Quarterly 28 (2004) 75–105. doi:10.2307/25148625. [41] K. Pefers, T. Tuunanen, M. A. Rothenberger, S. Chatterjee, A Design Science Research Methodology for Information Systems Research, Journal of Management Information Systems 24 (2007) 45–77. doi:10.2753/MIS0742-1222240302. [42] P. Johannesson, E. Perjons, An Introduction to Design Science, Springer, 2014.