Seeking Success in Public Procurement of Information Systems Sanni Marjanen University of Jyväskylä, PO BOX 35, 40014 University of Jyväskylä, Finland Abstract Public procurement of information systems (IS) research is gaining traction as digitalisation of services and the continuous, rapid development of technology keep the number of acquired IS software growing. Its impact on economy and societal structures is significant, yet there is no tool or method to assess and measure the success of the procurement process nor of the acquired IS. A successful procurement is in the interest of all stakeholders involved in the process, and recognising challenges is essential in order to identify elements, steps and factors leading to success with which the procurement has an opportunity to achieve its greatest possible value. This research plan presents the roadmap of a PhD dissertation on the topic. The outlined studies aim at advancing public procurement practices and to develop tools and methods to observe, measure, and ensure success in the context of public procurement of IS. The research problem is identified, motivation and knowledge gap presented, research methodology and timeline for completing the research within three years described, and expected results summarised. Keywords public procurement of IS, information systems, software engineering, IS success, doctoral dissertation, research plan, objectives and methods, timetable 1. Problem Definition Defining and observing the dimensions of success in software engineering and related fields is challeng- ing due to the elusive and changing nature of success [1]. An additional layer of challenge is created by the special characteristics of public procurement of information systems (IS), where a complex process meets a complex information system. No current success model takes into account these characteristics, or aims to measure success on a practical level. This research problem is addressed in the doctoral dissertation described in this research plan. The goal of this research is to create a tool or framework to quantify, measure, and observe success in the context of public procurement of IS. The motif and the research question guiding the dissertation as a whole is: How to measure success in public procurement of information systems? Public procurement of IS refers to purchases where public money is spent on ICT services and works for government organisations or for the benefit of taxpayers [2, 3, 4]. The demand for research and a culture change is recognised both in academic publications [5, 6, 7] as well as on national strategy level [8]. The impact of acquisitions made with public funds is noteworthy both for the economic and social structures of society [5, 9, 10]. In many high-income countries public agencies produce the majority of requests for IS procurement, which in turn affects the private market [5, 6]. Constant evolution and continuous development of procurement practices are essential to meet the growing need for public services, as digitalisation and technological advances, changes in demography, and the decreasing of public resources keep the trend for public procurement of IS ever-growing [9, 10, 11, 12]. As a factor in economic and social sustainability, a successful procurement is in the interest of all stakeholders involved in the process [13, 14]. The 15th International Conference on Software Business (ICSOB 2024), November 18–20, 2024, Utrecht, the Netherlands $ sanni.e.marjanen@jyu.fi (S. Marjanen) € https://www.softwareengineering.fi/researchers/sanni-marjanen/ (S. Marjanen)  0009-0009-2008-4970 (S. Marjanen) © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR ceur-ws.org Workshop ISSN 1613-0073 Proceedings Figure 1: Knowledge gap: The missing of metrics, methods, models, and solutions to define, measure, and observe success as perceived by the stakeholders in the context of public procurement of IS. 2. Knowledge Gap Figure 1 presents the existing knowledge gap this research addresses. While measuring success is a topic of interest in many related research fields, there are no standardised models or methods for measuring success or benefits realisation that would consider the views of various stakeholders and be able to adapt to the changing societal and technological environment with sufficient efficiency. The large number of theories and frameworks from many research fields shed light on defining and measuring success, but no model or tool exists to quantify success in a context or environment similar to public procurement of IS [15, 16, 17]. Research on public procurement of IS has focused on its special nature and characteristics and the challenges they create. A public procurement is essentially a heavily regulated project with multiple stakeholders who all have their own agenda [7, 18]. Being public, an acquisition process is an enabler of current policies and legislation [19]. The procurement process is undergone to acquire a complex IS that is designed and developed to fit a specific purpose [3, 7, 12]. These characteristics create a myriad of challenges that are well recognised both in academic literature and practice. Acknowledging these challenges is essential in order to identify elements, steps, and factors leading to success with which the procurement has an opportunity to achieve its greatest possible value [13]. Past research on challenges forms a solid foundation for new ideas and enables building of solutions. The public procurement process follows the rigid guidelines and distinct objectives presented by legislation [2, 20]. In public procurement predefined phases follow one another, and the needs and requirements of the end user are identified pre-tender [4, 21]. Understanding the procurement process as a whole [6, 22], collaboration and effective interaction [20], and careful preparation and assess- ment [3, 23] carry a significant weight throughout the whole procurement process. Off-the-shelf information systems rarely suit the needs of the purchasing unit, and as the acquired products and services create complex, unique systems, comparing software is challenging or impossible in the absence of a comparison base [3, 7, 12]. The impact of differing stakeholder views on success is recognised in literature [1, 18]. Successful communication between procuring parties includes defining common goals, finding consensus, preparing for compromises, and engaging all stakeholders throughout the process [3, 24, 25]. Long-term relationships between procurement unit and supplier have been found to have a positive effect on the efficiency, quantity, and quality of innovations, which is more evident in the less regulated private sector [5, 22]. Despite its growing impact on both government organisations and taxpayers, public procurement of IS has long been a dormant area of research [6, 26]. In recent years, academic interest has been on the rise especially in the Nordics, which is evidenced by the growing amount of dissertations and publications covering the topic from various points of view (see e.g. [22, 27, 28]). Public procurement of IS has similar qualities to innovation and services procurement and a lot can be learnt from research done on these topics. However, while these fields provide valuable insights to the software engineering and information systems research communities, the scope is slightly misaligned [29]. For this reason public procurement of IS needs to be studied as a separate phenomenon. In project management, plenty of models and methods for observing and measuring success have been developed over the years [30, 31, 32], and empirical research on project success has concentrated on the views of project management [33, 34]. The Iron Triangle of project management, i.e. cost, time, and quality, are the staples for measuring project success [1, 30]. These values are measurable e.g. in the forms of budget, project schedule, and technical properties of the IS, while subjective values such as job satisfaction, development of interorganisational communication and decision-making processes, and attention to detail are subjective and thus more challenging to quantify [1, 7, 32, 35]. The impact of these so called softer values may emerge or change over time, and the perceived views on success may need to be observed long-term to see a trend [18, 33]. IS success has been approached through many models, with The DeLone and McLean Model of Information System Success [36] being one of the most used [15]. Perceived usefulness and technology acceptance have been at the centre of other popular IS success models [37, 38]. Success metrics and dimensions are a topic of interest in software engineering [1, 39]. 3. Research Method To address the knowledge gap of measuring success in the context of public procurement of IS, a tool or framework is developed. To ensure the tool is built on scientific findings and research, the doctoral dissertation follows the design science methodology [40, 41, 42]. The artefact is created to observe, measure and ensure continued success throughout the life cycle of a public IS procurement process. To create appropriate metrics for measuring success, a definition for success within the context must be explored. First, the concept of perceived success in the various user groups of public procurement will be observed, existing models examined, and the overall state of the research discussed. Later an artefact is designed, developed, and evaluated based on the findings of the earlier phases. The work is divided into phases similar to Peffers et al. [41]. Figure 2 provides a schedule for the ongoing three years of doctoral studies. The studies and main activities of each phase are described below in more detail. The doctoral research is guided by the following research questions: How to measure success in public procurement of IS? What are the metrics for assessing success in public procurement of IS? 3.1. Phase 1: Identify and motivate The initiative for this doctoral research rises originally from practice, which is one of the four identified motives for research following the design science principles [41]. A client-initiated case study on success factors using the Delphi technique has been carried out to identify the research problem, to get tentative results on success definitions, and to draft objectives for a solution. Generalising results are to follow in the second phase. RQ’s: What factors define a successful public IS procurement in a long-standing multi-buyer rela- tionship? Methods include a case study using the Delphi technique. Main tasks are identifying the research problem, defining motivations to justify the value of a solution, and to establish the current state of research. Deliverables include one journal article of a client-initiated case study. Figure 2: Main tasks and timetable for completing doctoral studies within three years. 3.2. Phase 2: Define objectives of a solution The objective of the first two phases is to learn how success is perceived by the various stakeholders participating in procurement (e.g. ICT vendors, public agencies, diverse identifiable user groups of the system). Defining success is crucial in order to create an understanding on the success factors and points of failure surrounding the practices of public IS procurement. To create drafts of the artefact and to define its objectives, a mixed method study is conducted. Quantitative data is gathered from HILMA1 and semi-structured interviews with stakeholders of multiple procurement processes are arranged. All major stakeholders of public procurement (vendor, procurement unit, end users) are expected to participate. The metrics identified in this phase are a compromise or consensus between all stakeholders. RQ’s: How is success defined in public procurement of IS by stakeholders? What elements or factors define the success of public procurement of IS? What amount of public IS acquisitions succeed? Methods include quantitative and qualitative data analysis, and semi-structured interviews. Main tasks are defining the purpose of the solution, observing the perception of success in past and ongoing IS procurement by various user groups and stakeholders, quantifying softer variables, and identifying and establishing metrics for measuring success. Deliverables include one conference publication. A large data pool allows for multiple studies and publications also outside the scope of this dissertation. 1 Service for notices on public procurement in Finland, https://www.hankintailmoitukset.fi/en/ 3.3. Phase 3: Development of the solution The solution created in this study aims at ensuring and measuring the success of public procurement of IS as perceived by various stakeholders. The data gathered and analysed in previous phases will provide the foundation and a preliminary requirement specification for the development of the artefact. Existing models for success specification, such as the DeLone-McLean model (see [36]) are invaluable in the creation of a more practical tool. The exact nature of the artefact is yet to be decided. Potential drafts include e.g. a practical tool, a conceptual framework, or a model or method. The solution is designed for the cultural procurement context in Finland or a similar environment. RQ’s: How can success perceived by various stakeholders and user groups be measured? What are the metrics for assessing success in public procurement of IS? Main tasks include the design and development of an artefact for observing success throughout the life cycle of a public IS procurement process. Deliverables include the artefact, the requirement specification, and the design of the artefact (e.g. concept design, architecture, user interface etc.). Publications include a technical report and a conference publication. 3.4. Phase 4: Demonstration and evaluation After a proof of concept level solution has been created, it will be implemented to production in one or more ongoing procurement processes. Criteria for evaluating the solution will be decided on to observe its functionality. A longitudinal study would be needed to observe the long-term impact of the artefact, but in the scope of the dissertation a short-term evaluation is sufficient. This phase allows fine-tuning or more extensive restructuring of the artefact if needed to meet its goals. Accurate research questions and methods to conduct the study will be decided on closer to the phase, after the nature of the solution is uncovered. Research problem: Is the created solution suitable for evaluating, measuring, and observing success in public procurement of IS? Main tasks include applying the solution within a suitable context, creating metrics for evaluation and analysis, iterating back to design as needed, and addressing further development and future research. Deliverables include a minimum of one conference publication. 3.5. Phase 5: Dissertation Academic publications will be compiled into a doctoral dissertation. Findings of each study will be communicated systematically to the academic and practitioner communities throughout the study when appropriate. Findings will be published in peer-reviewed journals or conferences, and all publications will be made open access through self-archiving to ensure open availability, visibility, and long-term preservation. 4. Timeline I have joined the three-year Finnish Acceleration of Scientific Talent (FAST) programme starting in August 2024 and look to complete my PhD within three years in accordance with the programme. My compilation dissertation will consist of up to four scientific articles that seek definitions, answers, and solutions to success in public procurement of IS. As seen in figure 2, there are five semesters for collecting data, analysing and concluding results, and developing the tool. Articles and conference papers will be composed and submitted for publication when any relevant findings need to be communicated to the academic or practitioner audiences. Assuming at least three articles get successfully published, the final semester will be dedicated to compiling the articles into a coherent dissertation and submitting the work for preliminary inspection. The public examination is to be held before or soon after the scheduled end of the educational FAST-programme pilot in July 2027. 5. Expected Contributions The most practical result of the study is the proposed solution for measuring success. It is planned to be used as a tool by different stakeholders during the lifecycle of a public procurement to quantify, measure, and ensure the success of the procurement process and the acquired IS. On a larger scale the tool aims to advance practices of public procurement. In order to create an artefact for measuring success, a definition and metrics must be uncovered. Defining success in the context of public procurement of IS has not been done before. The expected contributions are academically significant, as they provide new and updated information on a topic that has not been in the centre of attention for researchers in the recent years. A thorough literature review connects past research and developments in the field and shows the state of the art of public IS procurement research while presenting a new synthesis. The dataset gathered in the mixed method study allows for multiple studies from different angles also outside the scope of the dissertation, and e.g. comparative studies in international collaboration are possible. Acknowledgments Heartfelt thanks to professors Samuli Pekkola and Tommi Mikkonen for their supervision and advice so far, and for guiding me on my journey to becoming a researcher in the first place. Thanks to the rest of the Software Engineering FAST team at University of Jyväskylä for their support. References [1] P. Ralph, P. Kelly, The Dimensions of Software Engineering Success, in: ICSE 2014: Proceedings of the 36th International Conference on Software Engineering, 2014, pp. 24–35. doi:10.1145/ 2568225.2568261. [2] G. A. Boyne, Public and Private Management: What’s the Difference?, Journal of Management Studies 39 (2002) 97––122. doi:10.1111/1467-6486.00284. [3] C. E. Moe, M. Newman, M. K. Sein, The public procurement of information systems: dialectics in requirements specification, European Journal of Information Systems 26 (2017) 143––163. doi:10.1057/s41303-017-0035-4. [4] D. Jamieson, K. Vinsen, G. Callender, Agile Procurement to Support Agile Software Development, in: 2005 3rd IEEE Intemational Conference on Industrial Informatics, INDIN ’05, 2005. doi:10. 1109/INDIN.2005.1560413. [5] T. Torvatn, L. de Boer, Public procurement reform in the EU: start of a new era?, IMP Journal 11 (2017) 431–451. doi:10.1108/IMP-09-2015-0056. [6] C. E. Moe, Research on Public Procurement of Information Systems: The Need for a Process Approach, Communications of the Association for Information Systems 34 (2014) 1320–1335. doi:10.17705/1CAIS.03478. [7] A. Alanne, P. Hellsten, S. Pekkola, I. Saarenpää, Three Positives Make One Negative: Public Sector IS Procurement, Tambouris, E., et al. Electronic Government. EGOV 2015. Lecture Notes in Computer Science (LNISA) 9248 (2015) 321–333. doi:10.1007/978-3-319-22479-4_24. [8] Valtiovarainministeriö, Kansallinen julkisten hankintojen strategia 2020 [National strategy for public procurement 2020]], institution = Ministry of Finance (Finland), 2020. URL: http://urn.fi/URN: NBN:fi-fe2020090768680. [9] OECD, Government at a Glance 2021, 2021. doi:10.1787/1c258f55-en, fig 8.1. [10] H. Torvinen, P. Ulkuniemi, End-user engagement within innovative public procurement practices: A case study on public–private partnership procurement, Industrial Marketing Management 58 (2016) 58–68. doi:10.1016/j.indmarman.2016.05.015. [11] OpenProcurement.fi, 2024. URL: https://tutkihankintoja.fi/procurement-categories?lang=en. [12] C. E. Moe, T. Päivärinta, View of Challenges In Information Systems Procurement in the Public Sector, Electronic Journal of e-Government 11 (2013) 308–323. URL: https://academic-publishing. org/index.php/ejeg/article/view/589/552. [13] M. Januska, A. Palacka, Critical success factors for public procurement of innovative solutions in Central Europe: Empirical study, E&M Economics and Management 23 (2023) 24—-41. doi:10. 15240/tul/001/2023-2-002. [14] C. E. Moe, A. C. Risvand, M. K. Sein, Limits of Public Procurement: Information Systems Acquisition, volume 4084, Springer, Berlin, Heidelberg, 2006, pp. 281–292. URL: https://link.springer.com/ chapter/10.1007/11823100_25. [15] G. G. Gable, D. Sedera, T. Chan, Re-conceptualizing Information System Success: The IS-Impact Measurement Model, Journal of the Association for Information Systems 9 (2008) 377–408. doi:10.17705/1jais.00164. [16] C. Herzog, A. Richter, M. Steinhüser, U. Hoppe, M. Koch, Methods And Metrics For Measuring The Success Of Enterprise Social Software – What We Can Learn From Practice And Vice Versa, ECIS 2013 Completed Research 132 (2013). URL: https://aisel.aisnet.org/ecis2013_cr/132/. [17] T. D. Nguyen, T. M. Nguyen, T. H. Cao, The relationship between IT adoption, IS success and project success, in: International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2016. doi:10.1109/ICACCI.2016.7732207. [18] M. Ylinen, S. Pekkola, Searching for Success in a Successful IS Acquisition, in: Proceedings of the 51st Hawaii International Conference on System Sciences, 2018, pp. 4793–4802. URL: http://hdl.handle.net/10125/50492. [19] T. Williams, H. Vo, M. Bourne, P. Bourne, R. Kirkham, G. Masterton, P. Quattrone, C. Toczycka, Benefits realisation: case studies in public major project delivery with recommendations for practice, Production Planning & Control (2023) 1–21. doi:10.1080/09537287.2023.2256287. [20] Y. Caloghirou, A. Protogerou, P. Panagiotopoulos, Public procurement for innovation: A novel eGovernment services scheme in Greek local authorities, Technological Forecasting and Social Change 103 (2016) 1–10. doi:10.1016/j.techfore.2015.10.016. [21] M. Rolfstam, Public Procurement of Innovation, Ph.D. thesis, Lund University, 2008. URL: https: //portal.research.lu.se/en/publications/public-procurement-of-innovation. [22] R. Ghezzi, T. Mikkonen, On Public Procurement of ICT Systems: Stakeholder Views and Emerging Tension, in: 14th International Conference on Software Business, ICSOB 2023, 2023, pp. 61–76. [23] V. Lember, R. Kattel, T. Kalvet, Quo vadis public procurement of innovation, Innovation: The European Journal of Social Science Research 28 (2015) 403–421. doi:10.1080/13511610.2015. 1043245. [24] H. Kautonen, M. Nieminen, Julkisten ICT-hankintojen käyttäjäkeskeisyyden edellytysten kriittinen tarkastelu [Critical examination of the prerequisites for user-centricity in public ICT procurement], Hallinnon Tutkimus 38 (2019) 155–173. doi:10.37450/ht.98037. [25] C. Edquist, J. M. Zabala-Iturriagagoitia, Public Procurement for Innovation as mission-oriented innovation policy, Research Policy 41 (2012) 1757–1769. doi:10.1016/j.respol.2012.04.022. [26] K. V. Thai, Public Procurement Re-examined, Journal of Public Procurement 1 (2001) 9–50. doi:10.1108/JOPP-01-01-2001-B001. [27] H. Vilpponen, You Get What You Order: Required Expertise in the Procurement of Public Services, Ph.D. thesis, University of Jyväskylä, 2021. URL: https://jyx.jyu.fi/handle/123456789/78852. [28] A. Koski, On the Provisioning of Mission Critical Information Systems based on Public Tenders, Ph.D. thesis, University of Helsinki, 2019. URL: http://hdl.handle.net/10138/303511. [29] B. Lenderink, J. I. Halman, H. Voordijk, Innovation and public procurement: from fragmentation to synthesis on concepts, rationales and approaches, Innovation: The European Journal of Social Science Research 35 (2019) 650–674. doi:10.1080/13511610.2019.1700101. [30] R. Atkinson, Project management: cost, time and quality, two best guesses and a phenomenon, its [sic] time to accept other success criteria, International Journal of Project Management 17 (1999) 337–342. doi:10.1016/S0263-7863(98)00069-6. [31] K. R. T. Larsen, A Taxonomy of Antecedents of Information Systems Success: Variable Analysis Studies, Journal of Management Information Systems 20 (2003) 169–246. doi:10.1080/07421222. 2003.11045768. [32] D. Baccarini, The Logical Framework Method for Defining Project Success, Project Management Journal 30 (1999) 25–32. doi:10.1177/875697289903000405. [33] K. Davis, A method to measure success dimensions relating to individual stakeholder groups, International Journal of Project Management 34 (2016) 480–493. doi:10.1016/j.ijproman. 2015.12.009. [34] T. Cooke-Davies, The ’real’ success factors on projects, International Journal of Project Manage- ment 20 (2002) 185–190. doi:10.1016/S0263-7863(01)00067-9. [35] S. Petter, W. DeLone, E. R. McLean, The Past, Present, and Future of ’IS Success’, Journal of the Association for Information Systems 13 (2012) 341–462. doi:10.17705/1jais.00296. [36] W. H. Delone, E. R. McLean, The DeLone and McLean Model of Information Systems Success: A Ten-Year Update, Journal of Management Information Systems 19 (2003) 9–20. doi:10.1080/ 07421222.2003.11045748. [37] F. D. Davis, Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly 13 (1989) 319–340. doi:10.2307/249008. [38] V. Venkatesh, M. G. Morris, G. B. Davis, F. D. Davis, User Acceptance of Information Technology: Toward a Unified View, MIS Quarterly 27 (2003) 425–478. doi:10.2307/30036540. [39] M. V. Mäntylä, M. Jørgensen, P. Ralph, H. Erdogmus, Guest editorial for special section on success and failure in software engineering, Empirical Software Engineering 22 (2017) 2281–2297. doi:10.1007/s10664-017-9505-5. [40] A. R. Hevner, S. T. March, J. Park, S. Ram, Design Science in Information Systems Research, MIS Quarterly 28 (2004) 75–105. doi:10.2307/25148625. [41] K. Peffers, T. Tuunanen, M. A. Rothenberger, S. Chatterjee, A Design Science Research Methodology for Information Systems Research, Journal of Management Information Systems 24 (2007) 45–77. doi:10.2753/MIS0742-1222240302. [42] P. Johannesson, E. Perjons, An Introduction to Design Science, Springer, 2014.