Regional E-Participation Portals Evaluation: Preliminary Results from Russia Andrei Chugunov*, Yury Kabanov**, Georgy Panfilov*** *ITMO University, St. Petersburg, Russia, chugunov@itmo.ru **(1) National Research University Higher School of Economics; (2) ITMO University, St. Petersburg, Russia, ykabanov@hse.ru ***ITMO University, St. Petersburg, Russia, panfilovgeorg@mail.ru Abstract: This ongoing research presents the framework and preliminary results of the regional e-participation portals evaluation in the regions of Russia. Based on the system approach in Political Science and taking into account the Russian context, we have developed the framework that allows the evaluation of various e-participation tools, as well as cross-regional and cross- platform comparisons alongside the key stages of the e-participation process: the input, "black box", output and feedback. The framework was applied to evaluate 205 e-participation portals in 85 Russian regions, representing six different types of e-participation options. Findings suggest substantial discrepancies in the development of e-participation in Russia. The advantages and implications of this framework for further analysis are discussed. Keywords: E-Participation, Russian regions, Evaluation, Ranking, System approach Acknowledgement: The research has been supported by the Russian Scientific Fund (RSF) as part -18-00360 «E-participation as Politics and Public Policy Dynamic Factor». 1. Introduction Evaluation is an important, though challenging aspect in e-participation research. A lot of methodologies have been developed to provide the assessment of e-participation (e.g. see Garcia et al. 2005; Tambouris et al. 2007; Fedotova et al. 2012). Yet, the techniques vary significantly due to different theoretical underpinnings (Panapolou et al. 2008) and usually lack validity because of the inability to consider contextual factors (Sundberg 2018), and, as argued by Kubicek and Aichholzer of multichannel participation processes" (p. 23). This "evaluation gap" is further widened by a variety of e-tools, as the taxonomy of e-participation is continuously expanding (Bohman 2014). This ongoing paper attempts to address these challenges, providing an evaluation framework that deals with the variety of e-participation tools. Based on the system approach in Political Science, founded by Easton (1957), as well as previous research on e-participation evaluation in Russia Copyright ©2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 72 Ongoing Research (Chugunov & Kabanov 2018; Vidiasova et al. 2016) we present a new methodology that stresses the process of e-participation. The evaluation methodology is designed in a way so it can be adapted to other national and regional contexts, hence can be of interest for both scholars and practitioners. At the same time, it considers the Russian context to get a more nuanced view on the development of e-participation in the regions of Russia. In this ongoing paper, we first overview the methodology and evaluation framework, and then present some of the findings, followed by the discussion of results, shortcomings, and future steps. 2. Background: Challenges of E-Participation Evaluation There are plenty of evaluation techniques of e-government and e-participation applications, yet there are sufficient discrepancies between them (Berntzen & Olsen 2009). The first challenge here is related to the methodology used. For instance, Machintosh and Whyte (2008) proposed a coherent model including democratic, project and socio-technical layers. Yet, this set of layers is not exclusive, and there is a variety of other criteria used (Panopolou et. al 2008), further expanded by the diversity of methods, ranging from quantitative analysis (Fedotova et al. 2012; Vidiasova et al. 2016) to expert surveys and qualitative assessment (Sá et al. 2016). The second challenge comes from the multiple channels citizens may use in public policy (Kubicek and Aichholzer 2016), with all these tools requiring their own criteria for assessment. Usually scholars pay attention to certain e-participation technologies, like government websites (Xenaxis & Loukis 2010; Freeney & Brown 2017) or social media accounts (Elsherif & Azab 2019). As the variety of e-tools is rising (Bohman 2014), it is crucial to provide unified criteria in which to evaluate them. The third challenge is related to the fact that the evaluation of e-participation initiatives should not concentrate on the readiness of the websites on which they are hosted as a goal on its own right: rather this assessment should focus on the substantial outcomes of e-participation for the public and policy-making. For example, Vidiasova et al. (2018) in their evaluation methodology, stress the social efficiency of e-participation, while Chugunov & Kabanov (2018) highlight the importance of the e- participation institutionalization and institutional design. Finally, there is a challenge of inclusion country-specific contextual factors: on the one hand, the evaluation framework should be rather general and universal to allow cross-country comparisons (Berntzen & Olsen 2009), yet, more emphasis on a particular social and political context can help to provide a more nuanced view on a particular e-participation practice and thus be more relevant for decision - makers. Scholars emphasize the impact of various institutional and policy variables on the development of e-government and e-participation (Gulati & Yates 2011), including the level of decentralization in e-tools implementation (Kassen 2015). The Russian case is very peculiar in this regard. Russia is a federal state with a high level of power centralization (Busygina 2018). Thus, the development of regional e-participation in Russia is closely connected to the federal initiatives. From 2012 onwards, the federal government has paid special attention to the implementation of regional online services, from complaint mechanisms to Ongoing Research 73 participatory budgeting. Yet the centralization of the policy has not led to the elimination of various divides among the regions, and the latter still perform differently in e-participation development (Kabanov & Sungurov 2016; Chugunov & Kabanov 2018). As a result, currently there are basically six major e-participation tools used in the Russian regions, namely: 1) initiative / participatory budgeting; (2) open budget; (3) e-petitions; (4) crowdsourcing; (5) complaint mechanisms and (6) e- voting. But they neither can be found in all 85 regions, nor they have the same quality of implementation and performance. It is therefore important to estimate these discrepancies of outcomes amidst a relatively centralized policy. In brief, while there are numerous attempts to evaluate e-participation, the development of a unified technique is far from complete. The new methodology should not only be based on a clear theoretical and methodological framework, but also tackle the e-participation tools diversity, provide a clear assessment of the difference that e-participation makes to political and policy processes, as well as to balance between a general approach allowing cross-contexts comparisons and a more nuanced context-related view. 3. Evaluation Methodology: Framework and Procedure The methodology proposed here is based on the previous technique developed by Chugunov & Kabanov (2018). Though it allowed to range the regions according to the level of e-participation institutionalization, its scope is quite limited, especially in terms of the tools analyzed. Yet, regional e-participation in Russia is much more diverse, hence the new methodology encompasses the six types of e-participation: 1) initiative / participatory budgeting; (2) open budget; (3) e-petitions; (4) crowdsourcing; (5) complaint mechanisms and (6) e-voting. Our methodology stresses the importance of e-participation as a facilitator of the normal public policy process on its various stages (Coelho et al. 2017; Scherer & Wimmer 2011). In line with this process orientation, we take a broader vision of the political process, derived from the system approach in politics. As was argued by Easton (1957), there are some crucial elements of the political system: (1) the system itself (the "black box"), (2) the inputs (demands and support from citizens); (3) the outputs (decisions, policies); and (4) the feedback (inputs - outputs correspondence) (Easton 1957). However simplified this vision might be, we argue it still portrays the basic elements of the political process accurately, and in fact, reflects the essentials of the e-participation workflow. Citizens formulate a demand (complain, petition, etc.) and submit it to the "black box", where the system processes the request and provides an output - a certain policy or action. This stimulates the "feedback", and, if necessary, another input. Each of these four concepts corresponds to a criterion. These criteria include: (1) openness for the "black box", i.e. how open, transparent and comprehensive the available information about the e- participation process is; (2) availability for the input, i.e. how e-participation allows the access for various groups of citizens; (3) decision-making capacity for the output, i.e. the availability of the information related to the outcomes of e-participation; (4) feedback quality, denoting the spectrum of opportunities for the citizens to give feedback on e-participation results. These criteria are 74 Ongoing Research universal for all six types of e-participation tools under analysis. Additionally, a fifth criterion - Specific requirements - was developed to evaluate unique features of each type. Every criterion was then decomposed to 3 indicators. The selection of this indicators has been based both on the previous studies (Chugunov & Kabanov 2018) and after a series of consultations with the experts on e-government and e-participation. So, the methodology consists of 15 indicators in general (Table 1). Each indicator is evaluated from 0 to 2 points: "0" - the indicator is not present, "1" - the indicator is partially present and "2" - the indicator is fully present. Hence for each e- participation tool a region could get a maximum of 30 points (22 points for open budget portals). The evaluation of the Russian e-participation portals was carried out in December 2019. We found and evaluate overall 205 Internet resources, attributed to one of the six types of e-participation. Seven online resources were then excluded from the analysis, as they had not been updated for one year and hence were considered irrelevant. Table 1: E-Participation Evaluation Framework. Source: Authors' Elaboration Concept Criterion Indicator & Explanation "Black box" Openness 1.1. Topicality: Has the website been updated within the last month? 1.2. Information about Responsibility: Is there information about the goals, objectives and operators of the portal? 1.3. Comprehensiveness: Are there infographics / opportunities to get the basic information within 2 clicks? Input Availability 2.1. Special conditions: Is there a version for people with disabilities? 2.2. Mobility: Is there a mobile version / app? 2.3. Alternative: Is there an offline alternative to e-participation? Output Decision- 3.1. Legislation: Is there a regional legislation regulating this portal? making 3.2. Reports: Are there reports on activities available? capacity 3.3. Routing: Are there any markers on this portal allowing to trace the stage at which the application is? Feedback Feedback 4.1. Contact: Is there contact information? Quality 4.2. Evaluation: Are citizens able to evaluate their satisfaction with this portal or leave a feedback? 4.3. Loop: Can citizens re-apply if they disagree with a decision? - Special 5.1. - 5.3. Narrow technological functionality questions pertinent to Requirements types of e-participation portals (e.g. for e-complaints we assess the availability of the GIS to position complaints, the classifier of complaints and the "public control"). Full list is in the online annex (https://clck.ru/MkdWF). Notes: indicators 2.3, 3.3., 4.2, 4.3. are not applicable to the Open Budget portals, as they do not allow direct citizens' involvement. Ongoing Research 75 4. Preliminary Findings All the results of the evaluation and visualizations are presented in the online annex to this paper (https://clck.ru/MkdWF). The first general finding about e-participation in Russian regions corresponds to previous studies (Chugunov & Kabanov 2018), stressing moderate and significantly disproportionate regional development in terms of e-participation (Table 2): the general score for the regions ranges from 0 to 83 points. Table 2: E-Participation Evaluation in the Russian Regions Summary. Source: Authors' Calculation E-Participation Tool Number of Regions The Average Score Initiative - participatory budgeting 51 8 / 30 Open Budget 83 12 / 22 E-Petition 12 12 / 30 Crowdsourcing 9 11 / 30 Complaint Mechanisms 30 14 /30 E-Voting 13 12 / 30 The regions can be divided into three types: those with the high (41 points and more), moderate (21 - 40) and low (20 and fewer) levels of e-participation development. Currently, 17 regions can be referred to as highly developed in these terms. They have, on average, from 4 to 6 online resources, which have the following features: (1) a separate open budget portal; (2) a developed initiative / participatory budgeting portal; (3) a cross-platform solution for e-complaints, e-voting and e- petitions; (4) more rarely, a separate crowdsourcing portal. Yet, the quantity of facilities does not necessarily correlate with higher scores, as these facilities can have problems with performance. Among the moderately developed are 29 regions with approximately 3 online resources, or two highly developed resources (e.g. open and participatory budget, or open budget and e-complaint). In less developed regions there is usually one resource (open budget), or two poorly performing resources. Currently, this group contains 39 regions, which is nearly half of the country (46 per cent). The ranking also reflects the macro-regional dynamics: while the North-Western regions are among the leaders, Southern and Caucasian regions are usually among the outsiders. Figure 1: The Average Score Across E-Participation Tools and Components. Source: Authors' Calculation 76 Ongoing Research The second finding is the related to the availability of tools (Table 2). The most frequent online tool is the open budget, that can be found in 83 out of 85 regions. On the contrary, crowdsourcing platforms are the rarest: only 9 regions had this facility in 2019. Hence it seems that the regional governments are still keen on developing information portals (like the open budget), prioritizing passive information acquisition rather than active citizens engagement. Thirdly, there is also an interesting dynamic across the stages of the process reflected by the four concepts (fig. 1). While the "black box" (openness) and the "output" (decision-making capacity), on average, perform well, the "input" (availability), and especially the "feedback", are visibly less developed: in many cases the features that we relate to them were simply non-existent. Many regional governments seem to underestimate the importance the problem of digital divide that hinder the availability of e-participation for all citizens, as well as the feedback quality that ensures the effectiveness of e-participation. 5. Discussion & Conclusion In general, this pilot analysis has several methodological and practical outcomes. First, it has proven the applicability of the framework to the Russian case, and its usefulness to make interregional comparisons. The preliminary results of the evaluation are in line with the previous findings (Chugunov & Kabanov 2018; Kabanov & Sungurov 2016) that e-participation in Russia is highly disproportionate across the regions. At the same time, we have explored such disproportions further, across the e-participation tools and stages, unveiling important contexts of e-participation diffusion and development. More effort should be put to estimate all the stages of the policy cycle (Coehlo 2017). Though the framework was designed to get a nuanced view on Russia, it is based on the widely accepted system approach and is universal enough. The results obtained can be further used in both large-N studies and for the deep analysis of cases. Secondly, the framework allows cross-platform evaluation, which is especially important when one needs to estimate various types of online tools. Thirdly, the framework has shown its capacity to evaluate e-participation throughout the whole process, from citizens' demands to the government's response, which is one of the key values of the proposed methodology. Of course, there are still some methodological problems to be solved. First, the genuine effectiveness and impact of e-participation are yet to be further explored: what we can evaluate are rather online manifestations of the real process, and it is hard to ensure that e-participation does make a difference. Secondly, it is difficult to evaluate the level of citizens' engagement with the portals, as these data are usually unavailable. Thirdly, there may usually be problems in access to the online resources due to the necessity for users to register on them. Finally, the next step should be the estimation of the internal and external validity of the scores obtained. We are planning to solve these problems in the forthcoming research. As for the practical results, the survey has allowed us to explore the regional e-participation in Russia, both in terms facility diversity and performance discrepancies. Despite the general federal trends towards digital government, we may however see a variety of outcomes, the reasons for which are yet to be found. Ongoing Research 77 References Berntzen, L., & Olsen, M. G. (2009). Benchmarking e-government-a comparative review of three international benchmarking studies. In 2009 Third International Conference on Digital Society (pp. 77-82). New York: IEEE. DOI:10.1109/ICDS.2009.55 Bohman, S. (2014). Information technology in eParticipation research: a word frequency analysis. In Tambouris E., Macintosh A., Bannister F. (Eds.), Electronic Participation. ePart 2014. Lecture Notes in Computer Science, vol 8654 (pp. 78-89). Berlin, Heidelberg: Springer. DOI: 10.1007/978-3-662-44914-1_7 Busygina, I. (2018). Russian Federalism. In Studin E. (Ed.), Russia (pp. 57-64). London: Palgrave Macmillan. Chugunov, A. V., & Kabanov, Y. (2018). Evaluating e-participation institutional design. A pilot study of regional platforms in Russia. In Edelmann N., Parycek P., Misuraca G., Panagiotopoulos P., Charalabidis Y., Virkar S. (Eds.) Electronic Participation. ePart 2018. Lecture Notes in Computer Science, vol 11021. (pp. 13-25). Cham: Springer. DOI: 10.1007/978-3-319-98578-7_2 Coelho, T. R., Cunha, M. A., & Pozzebon, M. (2017). eParticipation and the policy cycle: Designing a research agenda. In Proceedings of the 18th Annual International Conference on Digital Government Research (pp. 368-376). DOI: 10.1145/3085228.3085277 Easton, D. (1957). An approach to the analysis of political systems. World politics, 9(3), 383-400. Elsherif, M., & Azab, N. (2019). A Framework to Measure E-Participation Level of Government Social Media Accounts. In Proceedings of the 9th International Conference on Information Systems and Technologies (pp. 1-6). DOI: 10.1145/3361570.3361572 Fedotova, O., Teixeira, L., & Alvelos, H. (2012). E-participation in Portugal: evaluation of government electronic platforms. Procedia Technology, 5, 152-161. DOI: 10.1016/j.protcy.2012.09.017 Feeney, M. K., & Brown, A. (2017). Are small cities online? Content, ranking, and variation of US municipal websites. Government Information Quarterly, 34(1), 62-74. DOI: 10.1016/j.giq.2016.10.005 Garcia, A. C. B., Maciel, C., & Pinto, F. B. (2005, August). A quality inspection method to evaluate e- government sites. In Proceedings of the 4th international conference on Electronic Government, -209). Berlin, Heidelberg: Springer. DOI: 10.1007/11545156_19 Gulati, J., & Yates, D. J. (2011). Strategy, competition and investment: explaining the global divide in e- government implementation with policy variables. Electronic Government, An International Journal, 8(2/3), 124-143. Kabanov, Y., & Sungurov, A. (2016). E-Government development factors: evidence from the Russian regions. In Chugunov A., Bolgov R., Kabanov Y., Kampis G., Wimmer M. (Eds.), Digital Transformation and Global Society, CCIS, vol. 674. (pp. 85-95). Cham: Springer. DOI: 10.1007/978-3-319-49700-6_10 Kassen, M. (2015). Understanding Systems of e-Government: e-Federalism and e-Centralism in the United States and Kazakhstan. Rowman & Littlefield: Lexington Books, Lanham, MD. Kubicek, H., & Aichholzer, G. (2016). Closing the evaluation gap in e-participation research and practice. In Aichholzer G., Kubicek H., Torres L. (Eds.), Evaluating e-Participation. Public Administration and Information Technology, vol 19 (pp. 11-45). Cham: Springer. DOI: 10.1007/978-3-319-25403-6_2 78 Ongoing Research Macintosh, A. and Whyte, A. (2008) Towards an evaluation framework for eParticipation. Transforming Government: People, Process & Policy, 2 (1), pp. 16-30. DOI: 10.1108/17506160810862928/ Panopoulou, E., Tambouris, E., & Tarabanis, K. (2008). A framework for evaluating web sites of public authorities. In Aslib Proceedings. Emerald Group Publishing Limited. DOI: 10.1108/00012530810908229 Sá, F., Rocha, Á., Gonçalves, J., & Cota, M. P. (2017). Model for the quality of local government online services. Telematics and Informatics, 34(5), 413-421. DOI: 10.1016/j.tele.2016.09.002 Scherer, S., & Wimmer, M. A. (2011). Reference framework for E-participation projects. In Tambouris E., Macintosh A., de Bruijn H. (eds) Electronic Participation. ePart 2011. Lecture Notes in Computer Science, vol 6847 (pp. 145-156). Berlin, Heidelberg: Springer. DOI: 10.1007/978-3-642-23333-3_13 Sundberg, L. (2018). Shaping up e-Participation Evaluation: A Multi-criteria Analysis. In Edelmann N., Parycek P., Misuraca G., Panagiotopoulos P., Charalabidis Y., Virkar S. (Eds.), Electronic Participation. ePart 2018. Lecture Notes in Computer Science, vol 11021 (pp. 3-12). Cham: Springer. DOI: 10.1007/978- 3-319-98578-7_1 Tambouris, E., Liotas, N., & Tarabanis, K. (2007). A framework for assessing eParticipation projects and tools. In 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07) (pp. 90-90). New York: IEEE. DOI: 10.1109/HICSS.2007.13 Vidiasova, L. (2016). The applicability of international techniques for E-participation assessment in the Russian context. In International Conference on Digital Transformation and Global Society (pp. 145- 154). Cham: Springer. Vidiasova, L., Tensina, I., & Bershadskaya, E. (2018). Social efficiency of E-participation portals in Russia: assessment methodology. In Alexandrov D., Boukhanovsky A., Chugunov A., Kabanov Y., Koltsova O. (eds) Digital Transformation and Global Society. DTGS 2018. Communications in Computer and Information Science, vol 858 (pp. 51-62). Cham: Springer. DOI: 10.1007/978-3-030-02843-5_5 Xenakis, A., & Loukis, E. (2010). An investigation of the use of structured e-forum for enhancing e- participation in parliaments. International Journal of Electronic Governance, 3(2), 134-147. About the Authors Andrei Chugunov Andrei Chugunov is the director of e-Government Center at ITMO University. He received PhD in political science in 2000 and has published more than 100 papers on information society development, e-government, and e-participation technologies implementation as well as the interdisciplinary research of digitalization. Yury Kabanov Yury Kabanov is a Senior Lecturer and a Research Fellow at the Center of Comparative Governance Studies of the Higher School of Economics (St. Petersburg). Also, he is a Researcher at the e-Government Center at ITMO University. He has more than 20 academic publications on e-government and e-participation research. Georgy Panfilov Georgy Panfilov is a MA student, engineer at the Institute of Design and Urban Studies at the ITMO University Russia with a bachelor's degree in Political Science.