=Paper=
{{Paper
|id=Vol-3921/phd-paper2
|storemode=property
|title=Seeking Success in Public Procurement of Information Systems
|pdfUrl=https://ceur-ws.org/Vol-3921/phd-paper2.pdf
|volume=Vol-3921
|authors=Sanni Marjanen
|dblpUrl=https://dblp.org/rec/conf/icsob/Marjanen24
}}
==Seeking Success in Public Procurement of Information Systems==
https://ceur-ws.org/Vol-3921/phd-paper2.pdf
Seeking Success in Public Procurement of Information
Systems
Sanni Marjanen
University of Jyväskylä, PO BOX 35, 40014 University of Jyväskylä, Finland
Abstract
Public procurement of information systems (IS) research is gaining traction as digitalisation of services and the
continuous, rapid development of technology keep the number of acquired IS software growing. Its impact on
economy and societal structures is significant, yet there is no tool or method to assess and measure the success of
the procurement process nor of the acquired IS. A successful procurement is in the interest of all stakeholders
involved in the process, and recognising challenges is essential in order to identify elements, steps and factors
leading to success with which the procurement has an opportunity to achieve its greatest possible value. This
research plan presents the roadmap of a PhD dissertation on the topic. The outlined studies aim at advancing
public procurement practices and to develop tools and methods to observe, measure, and ensure success in the
context of public procurement of IS. The research problem is identified, motivation and knowledge gap presented,
research methodology and timeline for completing the research within three years described, and expected results
summarised.
Keywords
public procurement of IS, information systems, software engineering, IS success, doctoral dissertation, research
plan, objectives and methods, timetable
1. Problem Definition
Defining and observing the dimensions of success in software engineering and related fields is challeng-
ing due to the elusive and changing nature of success [1]. An additional layer of challenge is created by
the special characteristics of public procurement of information systems (IS), where a complex process
meets a complex information system. No current success model takes into account these characteristics,
or aims to measure success on a practical level. This research problem is addressed in the doctoral
dissertation described in this research plan. The goal of this research is to create a tool or framework to
quantify, measure, and observe success in the context of public procurement of IS. The motif and the
research question guiding the dissertation as a whole is: How to measure success in public procurement
of information systems?
Public procurement of IS refers to purchases where public money is spent on ICT services and works
for government organisations or for the benefit of taxpayers [2, 3, 4]. The demand for research and
a culture change is recognised both in academic publications [5, 6, 7] as well as on national strategy
level [8]. The impact of acquisitions made with public funds is noteworthy both for the economic
and social structures of society [5, 9, 10]. In many high-income countries public agencies produce
the majority of requests for IS procurement, which in turn affects the private market [5, 6]. Constant
evolution and continuous development of procurement practices are essential to meet the growing
need for public services, as digitalisation and technological advances, changes in demography, and the
decreasing of public resources keep the trend for public procurement of IS ever-growing [9, 10, 11, 12].
As a factor in economic and social sustainability, a successful procurement is in the interest of all
stakeholders involved in the process [13, 14].
The 15th International Conference on Software Business (ICSOB 2024), November 18–20, 2024, Utrecht, the Netherlands
$ sanni.e.marjanen@jyu.fi (S. Marjanen)
https://www.softwareengineering.fi/researchers/sanni-marjanen/ (S. Marjanen)
0009-0009-2008-4970 (S. Marjanen)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
Figure 1: Knowledge gap: The missing of metrics, methods, models, and solutions to define, measure, and
observe success as perceived by the stakeholders in the context of public procurement of IS.
2. Knowledge Gap
Figure 1 presents the existing knowledge gap this research addresses. While measuring success is a topic
of interest in many related research fields, there are no standardised models or methods for measuring
success or benefits realisation that would consider the views of various stakeholders and be able to
adapt to the changing societal and technological environment with sufficient efficiency. The large
number of theories and frameworks from many research fields shed light on defining and measuring
success, but no model or tool exists to quantify success in a context or environment similar to public
procurement of IS [15, 16, 17].
Research on public procurement of IS has focused on its special nature and characteristics and the
challenges they create. A public procurement is essentially a heavily regulated project with multiple
stakeholders who all have their own agenda [7, 18]. Being public, an acquisition process is an enabler
of current policies and legislation [19]. The procurement process is undergone to acquire a complex IS
that is designed and developed to fit a specific purpose [3, 7, 12]. These characteristics create a myriad
of challenges that are well recognised both in academic literature and practice. Acknowledging these
challenges is essential in order to identify elements, steps, and factors leading to success with which the
procurement has an opportunity to achieve its greatest possible value [13]. Past research on challenges
forms a solid foundation for new ideas and enables building of solutions.
The public procurement process follows the rigid guidelines and distinct objectives presented by
legislation [2, 20]. In public procurement predefined phases follow one another, and the needs and
requirements of the end user are identified pre-tender [4, 21]. Understanding the procurement process
as a whole [6, 22], collaboration and effective interaction [20], and careful preparation and assess-
ment [3, 23] carry a significant weight throughout the whole procurement process. Off-the-shelf
information systems rarely suit the needs of the purchasing unit, and as the acquired products and
services create complex, unique systems, comparing software is challenging or impossible in the absence
of a comparison base [3, 7, 12]. The impact of differing stakeholder views on success is recognised
in literature [1, 18]. Successful communication between procuring parties includes defining common
goals, finding consensus, preparing for compromises, and engaging all stakeholders throughout the
process [3, 24, 25]. Long-term relationships between procurement unit and supplier have been found to
have a positive effect on the efficiency, quantity, and quality of innovations, which is more evident in
the less regulated private sector [5, 22].
Despite its growing impact on both government organisations and taxpayers, public procurement
of IS has long been a dormant area of research [6, 26]. In recent years, academic interest has been
on the rise especially in the Nordics, which is evidenced by the growing amount of dissertations and
publications covering the topic from various points of view (see e.g. [22, 27, 28]). Public procurement of
IS has similar qualities to innovation and services procurement and a lot can be learnt from research
done on these topics. However, while these fields provide valuable insights to the software engineering
and information systems research communities, the scope is slightly misaligned [29]. For this reason
public procurement of IS needs to be studied as a separate phenomenon.
In project management, plenty of models and methods for observing and measuring success have
been developed over the years [30, 31, 32], and empirical research on project success has concentrated
on the views of project management [33, 34]. The Iron Triangle of project management, i.e. cost, time,
and quality, are the staples for measuring project success [1, 30]. These values are measurable e.g. in
the forms of budget, project schedule, and technical properties of the IS, while subjective values such as
job satisfaction, development of interorganisational communication and decision-making processes,
and attention to detail are subjective and thus more challenging to quantify [1, 7, 32, 35]. The impact
of these so called softer values may emerge or change over time, and the perceived views on success
may need to be observed long-term to see a trend [18, 33]. IS success has been approached through
many models, with The DeLone and McLean Model of Information System Success [36] being one of
the most used [15]. Perceived usefulness and technology acceptance have been at the centre of other
popular IS success models [37, 38]. Success metrics and dimensions are a topic of interest in software
engineering [1, 39].
3. Research Method
To address the knowledge gap of measuring success in the context of public procurement of IS, a tool
or framework is developed. To ensure the tool is built on scientific findings and research, the doctoral
dissertation follows the design science methodology [40, 41, 42]. The artefact is created to observe,
measure and ensure continued success throughout the life cycle of a public IS procurement process. To
create appropriate metrics for measuring success, a definition for success within the context must be
explored. First, the concept of perceived success in the various user groups of public procurement will
be observed, existing models examined, and the overall state of the research discussed. Later an artefact
is designed, developed, and evaluated based on the findings of the earlier phases. The work is divided
into phases similar to Peffers et al. [41]. Figure 2 provides a schedule for the ongoing three years of
doctoral studies. The studies and main activities of each phase are described below in more detail.
The doctoral research is guided by the following research questions: How to measure success in
public procurement of IS? What are the metrics for assessing success in public procurement of IS?
3.1. Phase 1: Identify and motivate
The initiative for this doctoral research rises originally from practice, which is one of the four identified
motives for research following the design science principles [41]. A client-initiated case study on
success factors using the Delphi technique has been carried out to identify the research problem, to get
tentative results on success definitions, and to draft objectives for a solution. Generalising results are to
follow in the second phase.
RQ’s: What factors define a successful public IS procurement in a long-standing multi-buyer rela-
tionship?
Methods include a case study using the Delphi technique.
Main tasks are identifying the research problem, defining motivations to justify the value of a
solution, and to establish the current state of research.
Deliverables include one journal article of a client-initiated case study.
Figure 2: Main tasks and timetable for completing doctoral studies within three years.
3.2. Phase 2: Define objectives of a solution
The objective of the first two phases is to learn how success is perceived by the various stakeholders
participating in procurement (e.g. ICT vendors, public agencies, diverse identifiable user groups of the
system). Defining success is crucial in order to create an understanding on the success factors and
points of failure surrounding the practices of public IS procurement. To create drafts of the artefact and
to define its objectives, a mixed method study is conducted. Quantitative data is gathered from HILMA1
and semi-structured interviews with stakeholders of multiple procurement processes are arranged.
All major stakeholders of public procurement (vendor, procurement unit, end users) are expected to
participate. The metrics identified in this phase are a compromise or consensus between all stakeholders.
RQ’s: How is success defined in public procurement of IS by stakeholders? What elements or factors
define the success of public procurement of IS? What amount of public IS acquisitions succeed?
Methods include quantitative and qualitative data analysis, and semi-structured interviews.
Main tasks are defining the purpose of the solution, observing the perception of success in past and
ongoing IS procurement by various user groups and stakeholders, quantifying softer variables, and
identifying and establishing metrics for measuring success.
Deliverables include one conference publication. A large data pool allows for multiple studies and
publications also outside the scope of this dissertation.
1
Service for notices on public procurement in Finland, https://www.hankintailmoitukset.fi/en/
3.3. Phase 3: Development of the solution
The solution created in this study aims at ensuring and measuring the success of public procurement
of IS as perceived by various stakeholders. The data gathered and analysed in previous phases will
provide the foundation and a preliminary requirement specification for the development of the artefact.
Existing models for success specification, such as the DeLone-McLean model (see [36]) are invaluable
in the creation of a more practical tool. The exact nature of the artefact is yet to be decided. Potential
drafts include e.g. a practical tool, a conceptual framework, or a model or method. The solution is
designed for the cultural procurement context in Finland or a similar environment.
RQ’s: How can success perceived by various stakeholders and user groups be measured? What are
the metrics for assessing success in public procurement of IS?
Main tasks include the design and development of an artefact for observing success throughout the
life cycle of a public IS procurement process.
Deliverables include the artefact, the requirement specification, and the design of the artefact (e.g.
concept design, architecture, user interface etc.). Publications include a technical report and a conference
publication.
3.4. Phase 4: Demonstration and evaluation
After a proof of concept level solution has been created, it will be implemented to production in one or
more ongoing procurement processes. Criteria for evaluating the solution will be decided on to observe
its functionality. A longitudinal study would be needed to observe the long-term impact of the artefact,
but in the scope of the dissertation a short-term evaluation is sufficient. This phase allows fine-tuning
or more extensive restructuring of the artefact if needed to meet its goals. Accurate research questions
and methods to conduct the study will be decided on closer to the phase, after the nature of the solution
is uncovered.
Research problem: Is the created solution suitable for evaluating, measuring, and observing success
in public procurement of IS?
Main tasks include applying the solution within a suitable context, creating metrics for evaluation
and analysis, iterating back to design as needed, and addressing further development and future research.
Deliverables include a minimum of one conference publication.
3.5. Phase 5: Dissertation
Academic publications will be compiled into a doctoral dissertation. Findings of each study will be
communicated systematically to the academic and practitioner communities throughout the study when
appropriate. Findings will be published in peer-reviewed journals or conferences, and all publications
will be made open access through self-archiving to ensure open availability, visibility, and long-term
preservation.
4. Timeline
I have joined the three-year Finnish Acceleration of Scientific Talent (FAST) programme starting in
August 2024 and look to complete my PhD within three years in accordance with the programme. My
compilation dissertation will consist of up to four scientific articles that seek definitions, answers, and
solutions to success in public procurement of IS.
As seen in figure 2, there are five semesters for collecting data, analysing and concluding results, and
developing the tool. Articles and conference papers will be composed and submitted for publication when
any relevant findings need to be communicated to the academic or practitioner audiences. Assuming
at least three articles get successfully published, the final semester will be dedicated to compiling the
articles into a coherent dissertation and submitting the work for preliminary inspection. The public
examination is to be held before or soon after the scheduled end of the educational FAST-programme
pilot in July 2027.
5. Expected Contributions
The most practical result of the study is the proposed solution for measuring success. It is planned to be
used as a tool by different stakeholders during the lifecycle of a public procurement to quantify, measure,
and ensure the success of the procurement process and the acquired IS. On a larger scale the tool aims
to advance practices of public procurement. In order to create an artefact for measuring success, a
definition and metrics must be uncovered. Defining success in the context of public procurement of IS
has not been done before.
The expected contributions are academically significant, as they provide new and updated information
on a topic that has not been in the centre of attention for researchers in the recent years. A thorough
literature review connects past research and developments in the field and shows the state of the art of
public IS procurement research while presenting a new synthesis. The dataset gathered in the mixed
method study allows for multiple studies from different angles also outside the scope of the dissertation,
and e.g. comparative studies in international collaboration are possible.
Acknowledgments
Heartfelt thanks to professors Samuli Pekkola and Tommi Mikkonen for their supervision and advice
so far, and for guiding me on my journey to becoming a researcher in the first place. Thanks to the rest
of the Software Engineering FAST team at University of Jyväskylä for their support.
References
[1] P. Ralph, P. Kelly, The Dimensions of Software Engineering Success, in: ICSE 2014: Proceedings
of the 36th International Conference on Software Engineering, 2014, pp. 24–35. doi:10.1145/
2568225.2568261.
[2] G. A. Boyne, Public and Private Management: What’s the Difference?, Journal of Management
Studies 39 (2002) 97––122. doi:10.1111/1467-6486.00284.
[3] C. E. Moe, M. Newman, M. K. Sein, The public procurement of information systems: dialectics
in requirements specification, European Journal of Information Systems 26 (2017) 143––163.
doi:10.1057/s41303-017-0035-4.
[4] D. Jamieson, K. Vinsen, G. Callender, Agile Procurement to Support Agile Software Development,
in: 2005 3rd IEEE Intemational Conference on Industrial Informatics, INDIN ’05, 2005. doi:10.
1109/INDIN.2005.1560413.
[5] T. Torvatn, L. de Boer, Public procurement reform in the EU: start of a new era?, IMP Journal 11
(2017) 431–451. doi:10.1108/IMP-09-2015-0056.
[6] C. E. Moe, Research on Public Procurement of Information Systems: The Need for a Process
Approach, Communications of the Association for Information Systems 34 (2014) 1320–1335.
doi:10.17705/1CAIS.03478.
[7] A. Alanne, P. Hellsten, S. Pekkola, I. Saarenpää, Three Positives Make One Negative: Public
Sector IS Procurement, Tambouris, E., et al. Electronic Government. EGOV 2015. Lecture Notes in
Computer Science (LNISA) 9248 (2015) 321–333. doi:10.1007/978-3-319-22479-4_24.
[8] Valtiovarainministeriö, Kansallinen julkisten hankintojen strategia 2020 [National strategy for
public procurement 2020]], institution = Ministry of Finance (Finland), 2020. URL: http://urn.fi/URN:
NBN:fi-fe2020090768680.
[9] OECD, Government at a Glance 2021, 2021. doi:10.1787/1c258f55-en, fig 8.1.
[10] H. Torvinen, P. Ulkuniemi, End-user engagement within innovative public procurement practices:
A case study on public–private partnership procurement, Industrial Marketing Management 58
(2016) 58–68. doi:10.1016/j.indmarman.2016.05.015.
[11] OpenProcurement.fi, 2024. URL: https://tutkihankintoja.fi/procurement-categories?lang=en.
[12] C. E. Moe, T. Päivärinta, View of Challenges In Information Systems Procurement in the Public
Sector, Electronic Journal of e-Government 11 (2013) 308–323. URL: https://academic-publishing.
org/index.php/ejeg/article/view/589/552.
[13] M. Januska, A. Palacka, Critical success factors for public procurement of innovative solutions
in Central Europe: Empirical study, E&M Economics and Management 23 (2023) 24—-41. doi:10.
15240/tul/001/2023-2-002.
[14] C. E. Moe, A. C. Risvand, M. K. Sein, Limits of Public Procurement: Information Systems Acquisition,
volume 4084, Springer, Berlin, Heidelberg, 2006, pp. 281–292. URL: https://link.springer.com/
chapter/10.1007/11823100_25.
[15] G. G. Gable, D. Sedera, T. Chan, Re-conceptualizing Information System Success: The IS-Impact
Measurement Model, Journal of the Association for Information Systems 9 (2008) 377–408.
doi:10.17705/1jais.00164.
[16] C. Herzog, A. Richter, M. Steinhüser, U. Hoppe, M. Koch, Methods And Metrics For Measuring
The Success Of Enterprise Social Software – What We Can Learn From Practice And Vice Versa,
ECIS 2013 Completed Research 132 (2013). URL: https://aisel.aisnet.org/ecis2013_cr/132/.
[17] T. D. Nguyen, T. M. Nguyen, T. H. Cao, The relationship between IT adoption, IS success and
project success, in: International Conference on Advances in Computing, Communications and
Informatics (ICACCI), 2016. doi:10.1109/ICACCI.2016.7732207.
[18] M. Ylinen, S. Pekkola, Searching for Success in a Successful IS Acquisition, in: Proceedings
of the 51st Hawaii International Conference on System Sciences, 2018, pp. 4793–4802. URL:
http://hdl.handle.net/10125/50492.
[19] T. Williams, H. Vo, M. Bourne, P. Bourne, R. Kirkham, G. Masterton, P. Quattrone, C. Toczycka,
Benefits realisation: case studies in public major project delivery with recommendations for
practice, Production Planning & Control (2023) 1–21. doi:10.1080/09537287.2023.2256287.
[20] Y. Caloghirou, A. Protogerou, P. Panagiotopoulos, Public procurement for innovation: A novel
eGovernment services scheme in Greek local authorities, Technological Forecasting and Social
Change 103 (2016) 1–10. doi:10.1016/j.techfore.2015.10.016.
[21] M. Rolfstam, Public Procurement of Innovation, Ph.D. thesis, Lund University, 2008. URL: https:
//portal.research.lu.se/en/publications/public-procurement-of-innovation.
[22] R. Ghezzi, T. Mikkonen, On Public Procurement of ICT Systems: Stakeholder Views and Emerging
Tension, in: 14th International Conference on Software Business, ICSOB 2023, 2023, pp. 61–76.
[23] V. Lember, R. Kattel, T. Kalvet, Quo vadis public procurement of innovation, Innovation: The
European Journal of Social Science Research 28 (2015) 403–421. doi:10.1080/13511610.2015.
1043245.
[24] H. Kautonen, M. Nieminen, Julkisten ICT-hankintojen käyttäjäkeskeisyyden edellytysten kriittinen
tarkastelu [Critical examination of the prerequisites for user-centricity in public ICT procurement],
Hallinnon Tutkimus 38 (2019) 155–173. doi:10.37450/ht.98037.
[25] C. Edquist, J. M. Zabala-Iturriagagoitia, Public Procurement for Innovation as mission-oriented
innovation policy, Research Policy 41 (2012) 1757–1769. doi:10.1016/j.respol.2012.04.022.
[26] K. V. Thai, Public Procurement Re-examined, Journal of Public Procurement 1 (2001) 9–50.
doi:10.1108/JOPP-01-01-2001-B001.
[27] H. Vilpponen, You Get What You Order: Required Expertise in the Procurement of Public Services,
Ph.D. thesis, University of Jyväskylä, 2021. URL: https://jyx.jyu.fi/handle/123456789/78852.
[28] A. Koski, On the Provisioning of Mission Critical Information Systems based on Public Tenders,
Ph.D. thesis, University of Helsinki, 2019. URL: http://hdl.handle.net/10138/303511.
[29] B. Lenderink, J. I. Halman, H. Voordijk, Innovation and public procurement: from fragmentation
to synthesis on concepts, rationales and approaches, Innovation: The European Journal of Social
Science Research 35 (2019) 650–674. doi:10.1080/13511610.2019.1700101.
[30] R. Atkinson, Project management: cost, time and quality, two best guesses and a phenomenon, its
[sic] time to accept other success criteria, International Journal of Project Management 17 (1999)
337–342. doi:10.1016/S0263-7863(98)00069-6.
[31] K. R. T. Larsen, A Taxonomy of Antecedents of Information Systems Success: Variable Analysis
Studies, Journal of Management Information Systems 20 (2003) 169–246. doi:10.1080/07421222.
2003.11045768.
[32] D. Baccarini, The Logical Framework Method for Defining Project Success, Project Management
Journal 30 (1999) 25–32. doi:10.1177/875697289903000405.
[33] K. Davis, A method to measure success dimensions relating to individual stakeholder groups,
International Journal of Project Management 34 (2016) 480–493. doi:10.1016/j.ijproman.
2015.12.009.
[34] T. Cooke-Davies, The ’real’ success factors on projects, International Journal of Project Manage-
ment 20 (2002) 185–190. doi:10.1016/S0263-7863(01)00067-9.
[35] S. Petter, W. DeLone, E. R. McLean, The Past, Present, and Future of ’IS Success’, Journal of the
Association for Information Systems 13 (2012) 341–462. doi:10.17705/1jais.00296.
[36] W. H. Delone, E. R. McLean, The DeLone and McLean Model of Information Systems Success:
A Ten-Year Update, Journal of Management Information Systems 19 (2003) 9–20. doi:10.1080/
07421222.2003.11045748.
[37] F. D. Davis, Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information
Technology, MIS Quarterly 13 (1989) 319–340. doi:10.2307/249008.
[38] V. Venkatesh, M. G. Morris, G. B. Davis, F. D. Davis, User Acceptance of Information Technology:
Toward a Unified View, MIS Quarterly 27 (2003) 425–478. doi:10.2307/30036540.
[39] M. V. Mäntylä, M. Jørgensen, P. Ralph, H. Erdogmus, Guest editorial for special section on
success and failure in software engineering, Empirical Software Engineering 22 (2017) 2281–2297.
doi:10.1007/s10664-017-9505-5.
[40] A. R. Hevner, S. T. March, J. Park, S. Ram, Design Science in Information Systems Research, MIS
Quarterly 28 (2004) 75–105. doi:10.2307/25148625.
[41] K. Peffers, T. Tuunanen, M. A. Rothenberger, S. Chatterjee, A Design Science Research Methodology
for Information Systems Research, Journal of Management Information Systems 24 (2007) 45–77.
doi:10.2753/MIS0742-1222240302.
[42] P. Johannesson, E. Perjons, An Introduction to Design Science, Springer, 2014.