Deliverology Implementation at the Local Government Level of Ukraine Andrii Buriachenko 1[0000-0002-7354-7491], Tetiana Zhyber 1[0000-0002-4557-023X], and Tetiana Paientko 1[0000-0002-2962-308X] 1 Kyiv National Economic University named after Vadym Hetman, Peremohy avenu 54/1, Kyiv 02000, Ukraine andrii.buriachenko@kneu.ua, tzhiber@gmail.com, tpayentko109@gmail.com Abstract. The article is devoted to the application of deliverology at the local government level in countries comparable to the stage of fiscal decentralization implementation as in Ukraine. The aim of the article is to show how the applica- tion of informational technology can contribute to the implementation of deliv- erology at the local government level as in the example of Ukraine. The method- ology is based on the application of Difference in Difference method and the development of basic program codes for the implementation of deliverology at the local government level. It is proposed that multi-criteria methods of decision making be used to analyze information which is most often formalized in the format of groups of performance indicators of local budget programs. Further research may be necessary to develop recommendations for the use of a single standard of electronic databases for regional development indicators and local budgets. This will allow the full automation of the process of managing data on local government performance. Keywords: Deliverology, Multi-criteria methods, Difference in Difference, Program code. 1 Introduction Deliverology ideas were first introduced by former UK Prime Minister Tony Blair's office to ensure that election promises were kept. The main components of deliverology are setting goals that are based on previously stated priorities, evaluating target indica- tors, and using tough feedback channels to monitor the achievement of goals. This ap- proach has proven effective in public administration not only at the national level, but also at the sub-national level [1, 2]. Small group of professionals monitors the ways to improve the quality of public services focused exclusively on achieving impact and improving outcomes. To project a high-level public management vision into reality they build optimal models for the services provision based on widely collected data, sets goals for their implementation, and disseminate the good practices [3]. The approach is centralized and requires external transparency for public service data. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). Deliverology is, in fact, a citizen-centered approach. In addition, it is a performance- based approach that requires transparency and accountability. The implementation of this approach requires the organization of work with large amounts of information, which cannot be organized without the use of information technology (IT). Information technologies are those that allow a reduction in the cost of database development, up- dating, analysis and monitoring of data for stakeholders, and so on. This approach is partly used in post-communist countries, although the need for it is much higher there due to higher risks of corruption, including political corruption. It should be noted that there is no single approach to the development of deliverology, as the process of deliverology implementation is influenced by the institutional environ- ment, the legal and regulatory framework, and the economic condition of the country. The prerequisites for local development in Ukraine emerged after the consolidation of mid-term budget planning and amendments to the Budget Code in 2019 [4]. The amendments to the Budget Law consolidated typical forms of local budgets and orga- nized the consolidation of tasks of local authorities. Deliverology ensures that planning for budget expenditures is not only decentralized but also focused on achieving national strategic objectives. It should be noted that implementation of deliverology at the local level in Ukraine is slow as there are no appropriate software tools to deal with hetero- geneous information on the implementation of budget programs. The aim of the article is to show how the application of information technology can contribute to the introduction of deliverology at the local level using the example of Ukraine. The article is organized as follows: the literary review of the investigated problem is conducted in the second section. The third section describes the research methodology. The fourth section displays the main results of the study. The fifth section presents findings and prospects for further research. 2 Theoretical Background According to Watkins (2013) deliverology is defined by many researchers as a control system that ensures maximum success [5]. Success is achieved through an approach that closely links expected results to current activities [5-7]. At the same time, current results are continuously monitored to adjust the actions of the implementers. It should be noted that this approach requires working with large volumes of data, their continu- ous processing and analysis. Rapid response to changes in performance indicators al- lows timely adjustment of tactics to achieve the set objectives. Deliverology is seen by many researchers as an approach to improving public ad- ministration [8-9]. Gash, T., Hallsworth, M., Ismail, S., and Paun, A. believe that im- proving the quality of public administration is achieved through clear goal-setting and rigorous monitoring [10]. The achievement of objectives is often linked to the provision of adequate quality public services. Therefore, timely information on the current state of affairs is needed to achieve the stated outcomes. Bouchal, P., Kidson, M., Norris, E., and Rutter, J. argue that the implementation of deliverology contributes to social justice [11]. This effect is achieved by increasing government accountability, both at the central and local levels. Publishing timely infor- mation on the funds received and the results of their use helps reduce the risks of the inefficient use of taxpayers' funds. Some researchers compare deliverology in the public sector with performance-based management in the business sector. They believe that it is the focus on results that con- tributes to the effectiveness of government [12, 13]. Many scholars have also studied the impact of deliverology on improving the quality of education, health care [14], and government cost effectiveness [15]. The most recent publications, for example Birch, L., Jacob, S., & Baby-Bouchard, A., Haws, E., Jakobsen, M. L., Baekgaard, M., Moyni- han, D.P., & van Loon, N., discuss the role of deliverology in delivering on election promises and increasing government accountability [16-18]. In other words, the role of deliverology is increasing, and in the future the need to implement this approach in the public sector will increase. Implementation of deliverology ideas in practice requires working with large amounts of information, so the use of the deliverology approach in practice requires the application of information technology. Information technology has been most success- ful in introducing deliverology into education and medicine [19]. Ideas on the use of computer technology for evaluation of results are presented in Barua A, Kriebel C, Mukhopadhyay T [20]. Critical evaluation of the information technology approaches used to measure results is found in Dedrick J, Gurbaxani V, Kraemer KL (2003) and Jarvis, C. B.; MacKenzie, S. B.; and Podsakoff, P. M [21, 22]. The main idea of the application of information technology in the implementation of deliverology is to create simple and quickly adaptable software products that allow the quick collection of in- formation, processing of it, and publishing of the results in the required format. 3 Methodology The research methodology is based on qualitative and quantitative methods. Qualitative methods were used to describe the implementation of deliverology principles in the work of local authorities, as well as to determine performance indicators of budget pro- grams. As a main qualitative approach case study methodology was used. The main purpose of the case study is to provide an in-depth analysis of the problem in terms of understanding the qualitative characteristics of the object of a study [23]. In our re- search the object is the local government level of Ukraine. The data for the research were selected based on an analysis of local government press releases on key ap- proaches to improving public service delivery. The performance indicators in the selected budget programs are designed for public services, delegated by the central to the local level within the framework of public func- tions, such as provision of public goods in the sphere of education, health care and social protection. Since Ukraine consumes a lot of energy resources, energy savings indicators are envisaged as targets in the budget programs of local authorities for recent years. Quantitative methods are used to develop an algorithm for collecting data on indica- tors, as well as to calculate performance indicators. This algorithm has been developed for implementation in local government practices. For this purpose, the software envi- ronment R was used. With the help of R, two goals were achieved: 1. An algorithm for collecting statistical data for calculations has been developed. 2. An algorithm for calculating performance indicators using the difference in dif- ference method has been developed. The data is available on the websites of local authorities. The technic of “data scrap- ing” was used in this research. The difficulty of collecting statistical data is that most reports on the implementation of budget programs are presented in PDF format. Such libraries were used to transform this data into a table form and form a database: library(rJava) Needed for tabulizer library(tabulizer) Tool for PDF Scraping library(tidyverse) Core data manipulation and visualization libraries A fragment of the program code to obtain a table on the performance of budget pro- grams is presented below: # PDF Scrape Tables Budget_indicators_scrape <- extract_tables( file = "pro- gram_oblast.pdf", method = "decide", output = "data.frame") The package DID was used to calculate the indicators by the Difference in Differ- ence (DID) method. DID is usually implemented as an interaction term between time and treatment group dummy variables in a regression model (1) [24]. 𝑌 = 𝛽0 + 𝛽1 × [𝑇𝑖𝑚𝑒] + 𝛽2 × [𝐼𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛] + 𝛽3 × [𝑇𝑖𝑚𝑒 × 𝐼𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛] + 𝛽4[𝐶𝑜𝑣𝑎𝑟𝑖𝑎𝑡𝑒𝑠] + 𝜀 (1) DID is typically used to estimate the effect of a specific intervention or treatment (such as a passage of law, enactment of policy, or large-scale program implementation) by comparing the changes in outcomes over time between a population that is enrolled in a program (the intervention group) and a population that is not (the control group) [24]. In this research the intervention group is the group of regions which have imple- mented deliverology approach. As dependent variables the indicators of energy effi- ciency and energy effectiveness were used. The covariates are normative indicators for the country. The calculations were done for two periods: 2016-2017, 2017-2018. The DID package contains tools for computing average treatment effect parameters in Difference in Differences models with more than two periods, with variation in treat- ment timing across individuals, and where the DID assumption possibly holds condi- tional on covariates. 4 Results 4.1 Deliverology Organization at the Local Government Level A key aspect of local government effectiveness is to provide public services of appro- priate quality to taxpayers. One of the important factors in ensuring local government performance is to raise awareness among all actors in the supply chain of such services about their results and costs. The founder of the deliverology Barber et al (2011) identified three key components of the approach: 1. Centralizing decisions in the hands of a small group highly skilled professionals who provide a systematic approach. 2. Accumulation of reliable and relevant performance data to determine goals and trajectories for achieving them. 3. Establishing daily implementation of the chosen concept [25]. To implement deliverology at the local government level, standard budget programs should be established first. Further development of a constantly updated information system which will accumulate the information on planned and actual indicators of per- formance of budgetary programs is necessary. This information system should be uni- form for all local governments so that information on the implementation of budget programs is collected, processed and published according to a single standard. The sys- tem should make possible the accumulation of data over several years and analysis of their dynamics. The initial information should be presented in fields such as the following: 1. By groups of indicators of implementation of local budgets (expenses in absolute value, product/service, indicators of efficiency and effectiveness). 2. By functions of budget programs. Currently in Ukraine the first step in the creation of a complex information system has been made, namely, the data portal of the "Open Budget" system. Information on local budgets is now stored in a data repository, where it should be timely updated as new data becomes available. The original information is displayed on a special website (link openbudget.gov.ua), but efforts should be made to separate results in a graphical form and sample data analysis be available upon request of a particular user. Coordination of information exchange in deliverology can be achieved by providing information to different user groups. For example, Corrado lo Storto (2014) suggests using a "cognitive user system" [26]. The idea is that public officials need information that immediately generates ways to implement country and local government policies. First, it means compliance with targets, implementation of strategic criteria, selection of indicators by form in dynamics with grouping by administrative units or budgets, etc. For analysts and researchers, information can be provided from primary sources grouped according to common criteria: period, type of budget, cost objectives, planned and actual indicators. For ordinary citizens the information should be summarized in graphical form with necessary details and maximum transparency. With the availability of such a service, citizens can easily see the quality of public services in their region and beyond. According to Barber and others. (2011), the evaluation of the beneficiaries of public services and goods is an integral and important part before the end of the service cycle [9]. Based on users' information needs, deliverology at the local level should be orga- nized as shown in Figure 1. Source: Developed by authors based on [8, 9, 25] Fig 1. Deliverology stages at the local level Possible difficulties for establishing credible performance indicators for the public goods and services provision in Ukraine are: 1. Qualitative indicators of budget programs measure the result of the activity of the performer, but not the ultimate objective achievement. 2. According to the volumes of information, in the accumulation of results it is rea- sonable to consider a future system by big data characteristics in the public sector [27, 28]. The general scheme of the Big Data system construction at the local government level is shown in Fig. 2. Fig. 2. Local deliverology Big Data system Big Data system for local deliverology should be implemented in at least two as- pects. 1. The first part is a framework with indicators for the general monitoring of policy implementation, development indicators, and forecast trends for available resources in the region. The system's use of these indicators is aimed at maximum objectivity and automation to determine deviations. This part includes, for example, such indicators as employment, average official income, number of employees in the public sector with the projected amount of taxes paid by them and other similar indicators. This part should also include indicators of the real estate tax base, the amount and cost of energy consumed, the level of infrastructure development, its depreciation, cost and profitabil- ity, and other indicators. The problem of Ukrainian governance, in particular, is to as- sign the role of a forecasting tool that simulates a desired result that is far from reality. This leads to the fact that plans based on such forecasts cannot be implemented in prac- tice. After implementation of deliverology, the dynamics of key indicators built into forecasts and displayed in the system of indicators would publicly outline the frame- work of forecast realism using IT tools. Thus, the central government and local author- ities will strengthen the consensus in the provision of services, stipulated in the budget legislation from 2020. Taxpayers will be able to understand the real business environ- ment. 2. The second part consists of consolidated information in the form of standard dash- boards and detailed individual data sets for analytics by type of services. This part has the primary task not to standardize but report the completeness and consistency of col- lected data on services. Further requests from the population and authorities, performed by IT specialists, will form the environment for the most convenient use of information about the activities of the authorities in the regions. 4.2 Simulation of Results As noted above, the development of an information system for the implementation of deliverology approaches requires software that allows timely collection of data, their processing and publication in the required format. To simulate this process, software codes have been developed to collect and analyze data related to energy efficiency as- sessment. Energy efficiency was chosen as a pilot indicator because one of the strategic objectives at the national level is to reduce energy consumption. That is why energy efficiency indicators are subject to special monitoring in all budget programs. Some performance indicators from typical passports of the local budget program, which relate to the task of energy saving by the executing agency, are presented in Table 1. These indicators are used to assess outcomes for individual budget entities. They allow a comparison of the results of local government energy efficiency policies with and without declared energy saving measures by regions. Most of the reports on budget execution are published in PDF format. To collect the indicators, a software code in R has been developed that automatically reads data from the reports and creates worktables. A fragment of the program code is shown in the Methodology section. Table 1. Performance indicators of the local budget programs for the energy saving task The effectiveness - The norm indicator Indicator accom- Indicator accom- utility consumption (for the whole coun- plished in the budget plished in the budget level try or in a particular program for the pe- program for the pe- region) riod 1 riod 2 Heat supply (H), Nh H1 H2 Gcal per 1 m3 of heated space Electricity (E), kWh Ne E1 E2 / m2 Water supply (W), Ng W1 W2 cubic meters per square meter Efficiency (Quality) Cost savings for a Accomplished in Accomplished in – the level of savings specific region or the the budget program the budget program country for the period 1 for the period 2 The level of heat NE EH1 EH2 supply saving (H), % The level of electric- NE EE1 EE2 ity saving (E), % The level of water NE EW1 EW2 supply saving (G), % The "Difference in Difference" method described in Wooldridge's (2009) was used to estimate the difference in energy saving indicators across regions [24]. The DID method allows the budget program implementer to compare homogeneous metrics in the service group over a number of years and measure the results of the energy effi- ciency policy. The difference comparison method shows how the energy consumption of each individual producer changes over time in different regions. It differentiates re- gions with different approaches over time to account for their impact on typical changes in indicators. Differences in groups of indicators of typical tasks of budget programs under this method aggravate exceptions in indicators of efficiency of consumption or saving of consumed public utilities between regions. Estimates of consumption by re- gions may indicate the presence or absence of changes in the dynamics of energy effi- ciency. Interpretation of the results will depend on the goals and conditions of the utility company. A country specific norm or efficiency indicator for any region, where an as- sessment of efficiency and quality of utility consumption has shown optimal results, can be considered as a benchmark for using this method. Therefore, it is necessary to compare the changes in dynamics between the first and second periods in the activities of local public companies in Regions A and B with the optimal energy-saving results in Administrative Territory C, which in this case are adopted as "normative". For example, Regions A and B declare that they implement energy efficiency policies, while Region C uses energy in the usual way. It is necessary to determine whether Region C shows the best projected energy saving results or over- estimates the efficiency targets in order to get the best cost savings in its budget pro- grams? The adapted sequence of formulas for the calculation is as follows: ∆uA= (I1 –I2 ) – (N1 – N2 ). ∆uB= (I1 –I2 ) – (N1 – N2 ). P(AB) = [∆uA] – [∆uB] were 1 and 2 – periods when the performance indicators done, A and B – finally comparable regions, I – the performance indicator within a certain group, N – the indicator of a comparable certain group from the standard (neutral) region C. Instead of manually calculating the four means and their difference-in-differ- ences, it is possible to estimate the difference-in-differences estimator and its statis- tical properties by running a regression that includes indicator variables for treat- ment and after and their interaction term [29]. As stated in the Methodology section, for the automatization of calculations the package DID was used for R. A fragment of the program code is presented below: library (did) data (locdta) summary (out$aggte, type="dynamic") Also, the following options were used:  "selective" - results when there is selective treatment timing  "calendar" - results when there are calendar time effects A sample of indicators on nine regions of Ukraine was created for analysis. Eight of them were compared with the ninth "normal" region in terms of consumption of se- lected utilities after the implementation of the budget programs for 2016-2018. For ear- lier periods, the reports are available only for the city of Kyiv. Therefore, calculations for earlier periods have not been performed. The results of the calculations are presented in Table 2. Table 2. DID method result in the energy saving calculation Group number DID in the ef- DID in the ef- DID in the effec- DID in the effi- fectiveness ficiency 2016- tiveness 2017- ciency 2017-2018 2016-2017 2017 2018 1. Heat 0,6 5,5 0,3 5,6 2. Heat 2,2 2,5 2,6 2,5 3. Heat 0,9 7,0 1,4 7,2 4. Heat 1,0 3,9 1,1 3,8 1. Electricity 6,1 8,7 5,9 10,7 2. Electricity 3,4 7,5 2,2 7,6 3. Electricity 6,8 9,1 6,7 15,1 4. Electricity 8,8 11,6 9,7 14,5 1. Water 0,05 2,8 0,06 3,4 2. Water 0,07 5,2 0,13 8,2 3. Water 0,02 4,4 0,02 3,7 4. Water 0,08 2,9 0,09 3,8 The results of the analysis showed that electricity consumption is characterized by significant deviations. Therefore, electricity consumption by local authorities should be a priority in further evaluation. The performance indicators that are assessed in local public services are consistent with the level of social orientation of the territory (average salary, emigration, etc.). Local governments need such an assessment in order to discuss a centralized approach to service delivery, with the implementation of the most reasonable and cost-effective solutions. Local government leaders should not, at present, simultaneously assess the various options for making such decisions on many variables, because their task is to centralize service delivery. However, some of the 'top-down' directions are conflicting (e.g., the impact of small school and kindergarten decisions on community well-being), sometimes factors that interfere with the final decision increase or disappear altogether. Therefore, the correctness and effectiveness of the decision depends heavily on the es- tablished utility metrics and assessment methods. Multi-criteria decision-making methods, in our opinion, will ensure the best further assessment in these conditions. They allow the evaluation and analysis of multi-purpose tasks using combinations of different criteria. 5 Conclusions The proposed information system for implementing the deliverology approach at the local level offers a number of advantages in the use of official information. First, the independence of local government is not compromised, but its patronage and account- ability are ensured. Secondly, the collection and analysis of performance indicators over a period of time allows their quantitative and qualitative composition to be as- sessed and local government actions to be adjusted where necessary. Third, multi-cri- teria decision-making methods used to analyze performance indicators, combined with benchmarks, will create new opportunities for researchers in local governance. The concept of development of deliverology at the local level proposed in this pub- lication, based on the performance indicators of local budget programs in regions, is necessary to obtain unified data. In the long term, with the use of "Big Data" at the local level, data storage systems can be expanded through performance indicators of budget- ary institutions, staff evaluation indicators, indicators of environmental and economic progress, etc. One of the key obstacles to the introduction of deliverology at the local level is the lack of requirements for reporting information in a single format. Therefore, the sample for the calculations was limited to nine regions of Ukraine. Therefore, for further de- velopment of deliverology, it is necessary to develop a single format for publishing reports on the implementation of budget programmes. The prospect of further research is to develop algorithms and program codes for the analysis of key performance indicators of budget programs and create on their basis a unified system of data storage and analysis for management decision-making at the local level. References 1. Cleary, S., The Prime Minister’s Delivery Unit (PMDU), 2005-2007: What impact did the machinery of government and leadership changes have on the PMDU? 2. Freeguard, G., and Gold, J., Data-driven Delivery: Lessons from the O’Malley administra- tion of Maryland, Institute for Government, January 2015, retrieved 16 January 2020. (2015). 3. Stratton, T. S. Deliverology in Community Economic Development: Enhancing Results Frameworks and Performance Measurement. Papers in Canadian Economic Development, 18, 1-16 (2019). 4. Budget Code of Ukraine. Retrieved February 10 2020, from https://za- kon.rada.gov.ua/laws/show/2456-17 (2010). 5. Watkins, K. Jim Kim’s ‘science of delivery’: what role for politics? | Overseas Development Institute (ODI). Retrieved December 20, 2019, from http://www.odi.org/comment/7703- jim-kims-science-delivery-role-politics (2013). 6. Bald, J. (n.d.). Deliverology - a science? Retrieved December 20, 2019, from http://johnbald.typepad.com/language/2013/07/deliverology-a-science.html (2016) 7. Cummings, D. Dominic Cummings (who ought to know) is not impressed by Michael Bar- ber, Tony Blair’s former adviser and self-styled ‘delivery man’. Retrieved December 20, 2019, from http://www.spectator.co.uk/2015/03/dominic-cummings-who-ought-to-know- is-not-impressed-by-michael-barber-tony-blairs-former-adviser-and-self-styled-delivery- man/(2015). 8. Barber, M., Instruction to Deliver: Fighting to transform Britain’s public services, Methuen, (2008). 9. Barber, M., et al., Deliverology 101: A Field Guide for Educational Leaders, Corwin, (2011). 10. Gash, T., Hallsworth, M., Ismail, S., and Paun, A., Performance Art: Enabling better man- agement of public services, Institute for Government, December 2008, retrieved 18 January 2020 (2008). 11. Bouchal, P., Kidson, M., Norris, E., and Rutter, J., Doing Them Justice: Lessons from four cases of policy implementation, Institute for Government, July 2014, retrieved 16 January 2020 (2014). 12. Box, R. C. Running government like a business: Implications for public administration the- ory and practice. The American Review of Public Administration, 29(1), 19-43 (1999). 13. Boyne, G. A. Public and private management: what’s the difference? Journal of management studies, 39(1), 97-122 (2002). 14. Barber, M. How to deliver improved outcomes for schools. Retrieved 26 January 2020 from https:// www.wise-qatar.org/2017-wise-research-improved-outcomes-school-systems/ (2017). 15. Shepherd, R. P. Expenditure reviews and the federal experience: Program evaluation and its contribution to assurance provision. Canadian Journal of Program Evaluation, 32 (3), 347– 370. https://doi.org/10.3138/cjpe.43180 (2018). 16. Birch, L., Jacob, S., & Baby-Bouchard, A. The Trudeau government’s legislative agenda: Election promises and a dual mandate. In L. Birch & F. Pétry (Eds.), Assessing Justin Tru- deau’s Liberal government: 353 promises and a mandate for change (pp. 27–42). Quebec, QC: Presses de l’Université Laval (2019). 17. Haws, E. Oft critiqued Liberal mandate tracker to become “more useful,” says government “deliverology” chief. The Hill Times. Retrieved from https:// www.hilltimes.com/2018/01/24/liberals-mandate-letter-tracker-great-conversationgetting- refresh-momentarily-mendelsohn/131770 (2018). 18. Jakobsen, M. L., Baekgaard, M., Moynihan, D. P., & van Loon, N. Making sense of perfor- mance regimes: Rebalancing external accountability and internal learning. Perspectives on Public Management and Governance, 1(2), 127–141. https://doi. org/10.1093/ppmgov/ (2018). 19. Arellano, N. Can deliverology help solve education, health problems in remote areas? Ca- nadian Government Executive. Retrieved 15 January 2020 from http://canadiangovernment executive.ca/can-deliverology-help-solve-education-health-problems/(2016). 20. Barua A, Kriebel C, Mukhopadhyay T Information technology and business value: an ana- lytic and empirical investigation. Inform Syst Res 6(1):3–23 (1995). 21. Dedrick J, Gurbaxani V, Kraemer KL Information technology and economic performance: a critical review of the empirical evidence. ACM Comput Surv 35(1):1–28 (2003) 22. Jarvis, C. B.; MacKenzie, S. B.; and Podsakoff, P. M. A critical review of construct indica- tors and measurement model misspecification in marketing and consumer research. Journal of Consumer Research, 30, 2, 199-218 (2003). 23. Stewart, A. Case study. In Jane Mills & Melanie Birks (Eds.), Qualitative methodology: A practical guide (pp.145-159). Thousand Oaks, CA: Sage (2014). 24. Wooldridge, Jeffrey M. “Difference-in-differences estimation,” Quantile, No.6, pp.25–47 (2009). 25. Barber, M., Kihn, P., & Moffit, A. Deliverology: From idea to implementation. McKinsey on Government, 6, 32-39 (2011). 26. lo Storto, C. Benchmarking website performance in the public sector: a non parametric ap- proach. Journal of Computers, 9(3), 636-643 (2014) 27. Bouckaert, G., Halligan, J. Comparing performance across public sectors. In Performance Information in the Public Sector. Palgrave Macmillan, London, 72-93 (2008). 28. Lægreid, P., Roness, P. G., Rubecksen, K. Performance information and performance steer- ing: integrated system or loose coupling? In Performance information in the public sector. Palgrave Macmillan, London, 42-57 (2008). 29. Callaway, Brantly and Sant’Anna, Pedro. "Difference-in-Differences with Multiple Time Periods and an Application on the Minimum Wage and Employment." Working Paper (2018).