<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Archiving and Interchange DTD v1.0 20120330//EN" "JATS-archivearticle1.dtd">
<article xmlns:xlink="http://www.w3.org/1999/xlink">
  <front>
    <journal-meta />
    <article-meta>
      <title-group>
        <article-title>A Cloud-Based Platform to Support the Policymaking: The Case of the DECIDO Project.</article-title>
      </title-group>
      <contrib-group>
        <contrib contrib-type="author">
          <string-name>Antonio Filograna</string-name>
          <email>antonio.filograna@eng.it</email>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Philipp Martin</string-name>
          <email>philipp.martin@fokus.fraunhofer.de</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Raphael Sturm</string-name>
          <email>raphael.sturm@fokus.fraunhofer.de</email>
          <xref ref-type="aff" rid="aff1">1</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Francisco Sanz</string-name>
          <email>frasanz@ibercivis.es</email>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Lucía Moreno</string-name>
          <email>lucia.moreno@ibercivis.es</email>
          <xref ref-type="aff" rid="aff2">2</xref>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Engineering Ingegneria Informatica S.p.A.</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Piazzale dell'Agricoltura</string-name>
        </contrib>
        <contrib contrib-type="author">
          <string-name>Italy</string-name>
        </contrib>
        <aff id="aff0">
          <label>0</label>
          <institution>D C, C. de Mariano Esquillor Gómez</institution>
          ,
          <addr-line>Zaragoza</addr-line>
          ,
          <country country="ES">Spain</country>
        </aff>
        <aff id="aff1">
          <label>1</label>
          <institution>Fraunhofer FOKUS</institution>
          ,
          <addr-line>Kaiserin-Augusta-Allee 31, Berlin</addr-line>
          ,
          <country country="DE">Germany</country>
        </aff>
        <aff id="aff2">
          <label>2</label>
          <institution>Ibercivis Foundation</institution>
          ,
          <addr-line>Campus Río Ebro Edificio I</addr-line>
        </aff>
      </contrib-group>
      <abstract>
        <p>Policy making is the process of creating and monitoring policies to solve societal challenges. In this respect, it is often conceptualized as a policy cycle [15], consisting of several different phases, such as agenda setting, policy formulation, policy implementation &amp; monitor and policy evaluation. Citizens often have the perception that the policy decided by the public authorities doesn't meet their needs. They would like to be engaged to provide their contribution to the policy decision, supporting the decision takers in a bottom-up approach. Citizens are called to take an active role in public services definition and implementation through co-production and co-creation. In this paper we illustrate how DECIDO project can support the policy makers along with citizens, organisations, businesses, public authorities to create/improve better policies, exploiting the power of data through the use of the disruptive technologies. Evidence-based and data-drive approaches are strongly recommended to take informed design decisions affecting the policies. The solution proposed by DECIDO is a combination of a co-creation methodology to involve all the actors in the Policy Life Cycle, and an easy-to-use web portal to support the creation of the policy exploiting big data analysis and cloud services in each phase of the Policy Life Cycle.</p>
      </abstract>
      <kwd-group>
        <kwd>Cloud services</kwd>
        <kwd>policymaking</kwd>
        <kwd>co-creation</kwd>
        <kwd>open data</kwd>
      </kwd-group>
    </article-meta>
  </front>
  <body>
    <sec id="sec-1">
      <title>1. Introduction</title>
      <p>DECIDO (eviDEnce and Cloud for more InformeD and effective pOlicies) is an European project
funded under the H2020 Programme (under the topic DT GOVERNANCE 12-2020). The project
started the 1st March of 2021 and will end the 29th of February 2024.</p>
      <p>The mission of DECIDO project is to demonstrate the ground-breaking impact of the adoption of
innovative methodologies, tools and data, enabling the effective development of better evidence-based
policies by public authorities. DECIDO will serve as an intermediary between the public sector, the
citizen science world and the European Cloud Infrastructure (ECI) through the direct collaboration with
European Open Science Cloud (EOSC) and will provide storage capacity and processing power through
EGI infrastructure. The project will test the DECIDO solutions in 4 cities/regions across Europe:
1. Pilot on Forest fires in Kajaani, Finland: prevention and protection against forest fires;</p>
      <p>Procedures to mitigate damage to nature, infrastructure and life.
2. Pilot on Floods and on the Social Crises following a pandemic or the outbreak of a conflict,
Italy: improve design of emergency policies related to floods and weather alerts in two areas of
the City of Turin (Meisino Park and Murazzi), as well as to the social crises following a big
pandemic event or the outbreak of a conflict in a neighbouring country.
EMAIL:</p>
      <p>2023 Copyright for this paper by its authors.
3. Pilot on Power Outage in Greek Municipalities, Greece: power outage management of public
infrastructure and cultural assets of Greek municipalities via emergency response mechanisms.
4. Pilot on Wildfires in the Aragon Region, Spain: improve the design of emergency policies
related to wildfires.</p>
      <p>The project has several objectives:
• Objective 1: Enable public authorities in adopting data and cloud technologies (from the PA
and research sector) to support evidence-based policies
• Objective 2: Support emergent strategic management in order to ensure that data can contribute
to evidence-based policy making processes aligned with political objectives and priorities
• Objective 3: Facilitate the active involvement of local actors in data generation, how it is
analysed and used within the policy making life cycle
• Objective 4: Pursue sustained use of data analytics and cloud infrastructure in policy making
According to the aforementioned objectives, the main results the project is developing are:
• An easy-to-use portal to define, manage and evaluate PA policies in a collaborative manner
leveraging services offered by EOSC cloud services, data made available by EOSC and by
other data providers (e.g. data.europa.eu), including Public Administrations themselves.
• A co-creation methodology on how to improve the collaboration among all stakeholders
involved in the policy life cycle (PLC), using the idea of Hackathons. This enables bottom-up
and externally collaborative ideation of innovative policies.</p>
      <p>The paper is structured as follows: Section 2 presents how the DECIDO project exploits data to
achieve the decision making. Section 3 introduces how the disruptive technologies can facilitate the
policy makers and Section 4 describes the case study in Aragon region on wildfires. Finally, Section 5
concludes and outlines future directions.</p>
    </sec>
    <sec id="sec-2">
      <title>2. DECIDO Solution: From Data and Co-Creation to Decision Making</title>
      <p>
        Democracy, a concept that originated in ancient Greece approximately 2400 years ago, has seen
significant evolution over centuries. Until the 1960s, this evolution was primarily towards
representative democracy, a system where citizens exercise control over the government by voting in
open elections. However, post-1960s, new avenues emerged for citizens to influence their governments
through stakeholder engagement, marking the rise of participatory democracy (1960s-1980s) [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ].
      </p>
      <p>
        From the 1980s to the 2010s, citizens were afforded the choice between free public or private service
providers. This period saw citizens being viewed as customers with individual needs in the paradigm
known as new public management [
        <xref ref-type="bibr" rid="ref2">2</xref>
        ].
      </p>
      <p>
        Starting from 2010, a shift towards the co-creation paradigm was observed, where citizens are seen
as knowledgeable contributors who participate in public governance discussions in (virtual) arenas [
        <xref ref-type="bibr" rid="ref3">3</xref>
        ].
This shift marked a significant paradigm change, wherein citizens transitioned from being regarded as
customers [
        <xref ref-type="bibr" rid="ref6">6</xref>
        ] to being recognized as partners [
        <xref ref-type="bibr" rid="ref7">7</xref>
        ]. Co-creation fosters deliberative democracy [
        <xref ref-type="bibr" rid="ref4">4</xref>
        ] and
is particularly pertinent in local and regional government settings, where decision-makers interact with
citizens and stakeholders on a daily basis [
        <xref ref-type="bibr" rid="ref1">1</xref>
        ]. This interaction can significantly influence the creation
of public value.
      </p>
      <p>DECIDO harnesses co-creation in public policies, involving all stakeholders in the policy life cycle.
The project's main outcome is a co-creation methodology to optimize tasks across the policy life cycle
phases. This methodology is backed by digital tools such as a data catalog, dashboards, survey tools,
and a co-creation tool, all unified in one accessible platform - the DECIDO Portal.</p>
      <p>
        To create a new policy, policy actors (which include decision makers, stakeholders, organizations,
citizens, etc.) should leverage data to develop evidence-based policies. Additionally, these policies
should take into account information derived from co-creation activities. The emphasis on
evidencebased policy-making, a movement that has gained significant traction in recent years, is intended to
make the public policy-making process more efficient and effective [
        <xref ref-type="bibr" rid="ref5">5</xref>
        ].
      </p>
      <p>2.1.</p>
    </sec>
    <sec id="sec-3">
      <title>Policymaking Co-Creation Methodology</title>
      <p>
        Assessments of the utility and effectiveness of co-creation in policy-making have yielded varying
and sometimes contradictory findings. Certain studies have illustrated the added value in policy
construction through e-participation [
        <xref ref-type="bibr" rid="ref8">8</xref>
        ][
        <xref ref-type="bibr" rid="ref9">9</xref>
        ], capitalizing on the rise of digital media platforms in
egovernance. However, there are also studies that highlight instances of failure in such initiatives
[
        <xref ref-type="bibr" rid="ref11">11</xref>
        ][
        <xref ref-type="bibr" rid="ref12">12</xref>
        ]. For instance, Estonia's Osale.ee saw a decline in user participation when efforts to promote
the e-participation system and engage citizens were discontinued, leading to the termination of the
experiment [
        <xref ref-type="bibr" rid="ref10">10</xref>
        ]. Main impediments encountered during these co-creation trials encompass
technological progression and data management, along with inexperience regarding the topics under
discussion, data scarcity, and the absence of a guiding figure. These factors often result in a lack of
impact on the decision-making process [
        <xref ref-type="bibr" rid="ref13">13</xref>
        ].
      </p>
      <p>
        Considering the previous weakness of the co-creation approach, we created a guide for the
implementation of co-creation activities to be followed during the development of DECIDO project.
Co-creation can be defined as “a process through which two or more public and private actors attempt
to solve a shared problem, challenge, or task through a constructive exchange of different kinds of
knowledge, resources, competences, and ideas that enhance the production of public value in terms of
visions, plans, policies, strategies, regulatory frameworks, or services, either through a continuous
improvement of outputs or outcomes or through innovative step-changes that transform the
understanding of the problem or task at hand and lead to new ways of solving it” [
        <xref ref-type="bibr" rid="ref14">14</xref>
        ]. This means that
co-creation takes place when citizens take an active role in public services.
      </p>
      <p>The DECIDO co-creation activities will take place at the four steps of the policy development cycle:
Agenda setting, Policy formulation, Policy implementation and simulation, Policy evaluation.
Considering the previous four steps and the stakeholders involved in each step, the co-creation process
in DECIDO can be structured in the following way:
1. Step 1 - Agenda setting: it is made up of the following sub-steps:
a. Understanding the problem: analysis of available information (data) on the emergency
situation.</p>
      <p>b. Resource planning: planning of personal, material (tools, equipment).
2. Step 2 - Policy formulation: consists of the following sub-steps:
a. Formulation of the challenge: based on the complete problem definition
b. Generation of ideas: gathering ideas from various sources to meet the challenge.
c. Selection and prioritization of ideas based on the criteria.
d. Conceptualisation: means envisaging some solutions to the challenge.
e. Prototyping: means making a decision on which concept will be implemented and
prototyped for validation.
3. Step 3 - Implementation and simulation: it is made up of the following sub-steps:
a. Implementation: means implementing the solution that has been tested.</p>
      <p>b. Launching: the solution is implemented, and now it is delivered to the stakeholders.
4. Step 4 - Evaluation: involves monitoring the functioning of the solution and the interaction of
the different stakeholders with it.</p>
      <p>The DECIDO co-creation methodology provides a comprehensive description of each stage,
detailing the activities required during all sub-stages, the stakeholders involved, and the supporting
methods, techniques, and tools. This concept established the criteria for developing the DECIDO
toolkit, ultimately leading to the construction of the DECIDO Portal solution (refer to Section 3 for
further details).</p>
    </sec>
    <sec id="sec-4">
      <title>2.2. DECIDO</title>
    </sec>
    <sec id="sec-5">
      <title>Making</title>
    </sec>
    <sec id="sec-6">
      <title>Methodology: From</title>
    </sec>
    <sec id="sec-7">
      <title>Data &amp; Co-Creation to Decision</title>
      <p>The DECIDO methodology was designed to transform user needs into valuable information that
facilitates improved decision-making. This process integrates co-creation sessions with the use of
digital services for data collection. The raw data collected is then analyzed, and the derived information
bolsters the decision-making process (as depicted in Figure 1).</p>
      <p>Each pilot identified their needs and challenges, outlining how DECIDO's tools could improve their
policies. Alongside the technical team, they evaluated and selected services from the European Open
Science Cloud (EOSC) based on their needs and DECIDO's requirements, promoting awareness of
cloud services in public administration. Next, they shifted focus to data discovery and collection,
selecting data for policy discussion from open data portals or public administrations' (closed) data.</p>
      <p>The DECIDO Data Catalogue, our first initiative, compiles data for policy creation. Besides this,
co-creation sessions yielded implicit knowledge, converted into data via participant surveys, co-creation
documents, etc. Upon data collection completion, stakeholders determined its usage for data-driven
policy. Algorithms transformed data into information, visualized on interactive dashboards. The
information enabled evidence-based policy creation. Finally, policy participants evaluate the policy,
providing feedback for future improvements..</p>
    </sec>
    <sec id="sec-8">
      <title>3. The Use of Technologies to Support the Policy Making</title>
      <p>There are several technological requirements that can facilitate the DECIDO policy-making
process. The following examples could be directly derived from the DECIDO policy life cycle
methodology and its corresponding set of user roles:
•
•
•
•
•</p>
      <p>Workflow support: A workflow should guide and support all collaborating stakeholders
throughout the different phases of the DECIDO policy-making process, documenting the
main results of each phase.</p>
      <p>Access to information: Policy and decision makers as well as a broader public need access to
information about the creative process, participants, methods, data and evidence, conclusions,
solutions, and finally proposals for new policies or policy alterations.</p>
      <p>Cloud-based approach: A cloud-based solution can provide scalability, accessibility,
costeffectiveness, security, collaboration, and flexibility, enabling other creative "policy labs" to
engage and make use of the solution.</p>
      <p>Collaborative tools: Collaborative tools such as shared workspaces, Wiki-like solutions, and
instant messaging platforms can enhance communication, coordination, and cooperation
among policymakers.</p>
      <p>Data management and visualization: A policy-making process requires robust data
management systems that can collect, store, process, analyse, and share data in a secure and
timely manner. Visualization tools such as graphs, charts, and maps can help policy-makers to
communicate complex information in a clear and understandable manner.</p>
      <p>An integrated and user-friendly platform meeting these technological requirements can help boost
policy making initiatives, engage more stakeholders and consequently help policymakers to make
informed decisions based on accurate data, collaborate effectively with stakeholders, and
communicate their findings to the wider public.
3.1.</p>
    </sec>
    <sec id="sec-9">
      <title>Re-Use of Existing Infrastructure</title>
      <p>A key objective of the DECIDO project is to leverage the pre-existing infrastructure within the
European Open Science Cloud (EOSC). The EOSC has the potential to be a platform that supports
policy-making by offering access to data and scientific resources. Services based on the EOSC can
provide crucial functionality to bolster policy-making in several ways:
• Identity and Access Management: The IAM services within the EOSC, which include user
authentication, authorization, and access control, are designed to be compatible with other IAM
systems employed in research and academic institutions.
• Data Sharing: The EOSC can facilitate the exchange of data between different organizations,
easing access to and analysis of data pertinent to policy-making.
• Collaborative Research: The EOSC can foster collaborative research efforts that tackle policy
questions, forming interdisciplinary research teams capable of addressing complex policy
problems.
• Policy-Relevant Research Services: The EOSC can offer services relevant to policy research,
such as simulation models, data visualization tools, and predictive analytics.
• Standards and Interoperability: The EOSC can encourage the use of open standards and
interoperability between various data sources and tools, making it simpler for policy makers to
access and utilize data from diverse sources.</p>
      <p>In summary, the EOSC can provide foundational functionality for supporting evidence-based
policymaking. However, to expand the range of available tools and services specifically supporting the
DECIDO policy approach, the DECIDO project partially developed and integrated complementary
services within the EOSC ecosystem.</p>
    </sec>
    <sec id="sec-10">
      <title>3.1.1. The DECIDO Portal Solution</title>
      <p>The DECIDO Portal solution is a user-friendly web application designed to enable collaborative
definition, management, and evaluation of public administration policies. Leveraging the services
provided by the European Open Science Cloud (EOSC) and external tools/services, as well as data from
the EOSC and other providers like data.europa.eu, the portal integrates a range of components that can
help public authorities harness the power of big data and cloud technologies while placing the citizen
at the centre of the policy life cycle.</p>
      <p>Figure 2 below provides a comprehensive overview of the DECIDO functional architecture and its
different layers. This suite of tools aims to aid all stakeholders engaged in the policy life cycle in
adopting the DECIDO co-creation methodology. It achieves this by offering a complete set of digital
services designed to simplify and streamline each phase of the methodology.</p>
      <p>Built on the cloud services available through the EOSC, the DECIDO Portal integrates a range of
services and tools, including EGI “Check-in”2 for single sign-on authentication and authorization
activities, “Jupyter Notebook”3 for data analysis algorithm development, EGI “Data Hub”4 for
persistent data storage and sharing, and “Amnesia”5 for anonymizing personal data, all provided in a
Software-as-a-Service (SaaS) model. The data management layer is furthermore supported by robust
2 https://www.egi.eu/service/check-in/
3 https://jupyter.org/
4 https://www.egi.eu/service/datahub/
5 https://amnesia.openaire.eu/
components for data-flow management based on “Apache Nifi”6 and a data catalogue based on the
Fraunhofer “piveau”7 platform.</p>
      <p>Additionally, the functionality has been upgraded integrating a survey tool (“LimeSurvey”8) to
collect feedback, and a wiki (“XWiki”9) that can be used for collaborative documentation. Both tools
can facilitate discussion forums to enhance stakeholder collaboration.</p>
      <p>The EOSC Cloud Infrastructure is also utilized to provide additional processing and storage
capacity. The main function of the portal is to manage policy creation throughout its entire value chain,
from data collection to analysis and visualization, through co-creative processes and stakeholder
collaboration on policy related challenges.</p>
    </sec>
    <sec id="sec-11">
      <title>4. The Real Case of Aragon Wildfire</title>
      <p>Data sourced from the Aragon government reveals that between 2005 and 2021, over half of the fires
(53.3%) were ignited accidentally or through negligence. In light of this, the paper presents a novel
methodology. This approach is designed to contribute to resolving this problem by establishing a more
efficient policy-making system. Specifically, the system would focus on the management and
surveillance of alert scenarios resulting from wildfires.</p>
      <p>This real-world case study and pilot of the DECIDO project utilizes a citizen science approach. This
method is an effective means of collecting and analyzing large volumes of data, leveraging the resources
and knowledge of the public. General citizens can contribute by issuing alerts of potential fire sightings
and providing valuable information about factors that may affect fire progression, such as the presence
of combustible material in the area.</p>
      <p>Our methodology champions co-creation and stakeholder engagement, acknowledging their crucial
role in the design phase. To ensure the output is relevant to the target groups, their input is essential. To
achieve this, we will conduct co-creation workshops where we will gather information and requirements
from all stakeholders. The project has identified the following target groups:
• Affected citizens: Citizen living in risks zones in Aragon
• Fire-fighters, to use the data for a better management of fire extinction.
• Policy Makers and local authorities, to use the information provided by the pilot
• General public, to be aware if they are near a risk zone and/or a fire</p>
      <p>In the Aragon region pilot, three significant events were conducted. These included a technical
hackathon focused on space applications, a datathon where relevant datasets were identified, and a
6 https://nifi.apache.org/
7 https://www.piveau.de/
8 https://www.limesurvey.org/de/
9 https://www.xwiki.org/xwiki/bin/view/Main/WebHome
policy co-creation process based on a "data alliance" concept. Each of these events will be elaborated
further in the subsequent sections.</p>
      <p>First co-creation event: Space apps</p>
      <p>On October 1st, 2022, over 60 individuals convened at Etopia, a center for art and technology in
Zaragoza, to partake in the ninth installment of the NASA SpaceAppsChallenge hackathon, reputedly
the largest such event globally. The premise of this event was to utilize data sources provided by NASA,
as well as any other information deemed necessary by participants, to address a series of problems and
challenges proposed by NASA itself ahead of the competition. This event was sponsored by Ibercivis
and H2020-DECIDO, who supported the local Zaragoza event by mentoring participants tackling the
challenge titled "Earth Data Analysis, Developers Wanted." The event resulted in 14 work teams
developing 14 unique solutions to the challenges presented, all of which were aligned with the
objectives of the DECIDO project.</p>
      <p>Second co-creation event: Datathon identifying datasets</p>
      <p>On November 25th, Aragon Open Data hosted a datathon titled "The Future of Data", coinciding
with the tenth anniversary of this open data portal established by the Government of Aragon. Teams
participating in this event were tasked with creating new services beneficial to society, drawing on the
open data provided via the Aragon Open Data platform. The challenges proposed to the teams spanned
a wide range of topics. These included fostering a connection with nature, promoting vibrant
neighborhoods, generating employment opportunities, enhancing university life, promoting awareness
of Aragon, focusing on renewable energies, and understanding and mitigating the issue of forest fires.</p>
      <p>Third co-creation event: The data alliance</p>
      <p>On February 17th, middle and high school students from Aragon participated in a test of the
datainformed public policy creation and evaluation process. This decision-making cycle includes data
analysis, policy proposal design, implementation, and effectiveness measurement.</p>
      <p>Students simulated the first two stages, focusing on Aragon's forest fires, specifically from the past
year, as part of the DECIDO pilot. The DECIDO platform was used during these co-creation sessions
to visualize forest fire data, enabling rapid understanding of the problem.</p>
    </sec>
    <sec id="sec-12">
      <title>5. Conclusion</title>
      <p>The role of evidence-based policy making is increasingly recognized in contemporary governance.
This approach, bolstered by the use of co-creation, can streamline the policy-making process. The
inclusion of various stakeholders allows the process to draw from a rich diversity of perspectives and
knowledge, enhancing the quality and relevance of policy decisions. Co-creation sessions have proven
to foster inclusivity, transparency, and accountability, leading to more effective and sustainable
policies. Furthermore, these sessions can stimulate collaboration and partnerships between government
entities and civil society organizations, strengthening and making the policy-making process more
responsive.</p>
      <p>The DECIDO project showcases the potential of utilizing co-creation to enhance evidence-based
policy-making, a notion that could be of great interest to the CAiSE community. By involving
stakeholders in policy-making, it ensures a more comprehensive approach to the development and
implementation of information systems. This approach aligns the systems more closely with user needs
and encourages transparency, fostering mutual understanding and cooperation among diverse
stakeholders. These stakeholders include researchers, practitioners, and policymakers, which is
essential when addressing complex issues in Information Systems Engineering.</p>
      <p>
        The project's progress thus far has resulted in numerous contributions to the policy-making
community. This includes participation in the Data for Policy conference, the formation of the Data
Driven Policy Cluster, which encompasses all projects funded under the same H2020 topic, and a paper
detailing the DECIDO experience with a focus on the Italian pilot [
        <xref ref-type="bibr" rid="ref16">16</xref>
        ].
      </p>
      <p>In this framework, DECIDO introduces a new method for policy creation. This method combines
cloud-based technologies with the insights of actors involved in the policy life cycle, offering a fresh
and effective approach to policy-making.</p>
    </sec>
    <sec id="sec-13">
      <title>6. Acknowledgements</title>
      <p>This project has received funding from the European Union’s Horizon 2020 research and innovation
programmed under grant agreement No 101004605.</p>
      <p>This publication, available as an open access document in EU repositories, is produced with the
support of the European Commission (EC). However, it's important to clarify that the EC's assistance
does not signify an endorsement of the content within. The views and perspectives presented in this
publication are solely those of the authors. Therefore, the Commission cannot be held accountable for
any potential application of the information enclosed herein.</p>
    </sec>
    <sec id="sec-14">
      <title>7. References</title>
    </sec>
  </body>
  <back>
    <ref-list>
      <ref id="ref1">
        <mixed-citation>
          [1]
          <string-name>
            <given-names>V.</given-names>
            <surname>Siebers</surname>
          </string-name>
          ,
          <string-name>
            <surname>J. Torfing,</surname>
          </string-name>
          <article-title>Co-creation as a new form of citizen engagement: Comparing Danish and Dutch experiences at the local government level</article-title>
          .
          <source>International Public Management Review</source>
          ,
          <volume>18</volume>
          (
          <issue>1</issue>
          /2) (
          <year>2018</year>
          ),
          <fpage>187</fpage>
          -
          <lpage>208</lpage>
          .
        </mixed-citation>
      </ref>
      <ref id="ref2">
        <mixed-citation>
          [2]
          <string-name>
            <given-names>A.</given-names>
            <surname>Fung</surname>
          </string-name>
          ,
          <article-title>Minipublics: deliberative designs and their consequences</article-title>
          . In: Rosenberg,
          <string-name>
            <surname>S.W</surname>
          </string-name>
          . (eds) Deliberation, Participation and Democracy. Palgrave Macmillan, London. https://doi.org/10.1057/9780230591080_
          <fpage>8</fpage>
        </mixed-citation>
      </ref>
      <ref id="ref3">
        <mixed-citation>
          [3]
          <string-name>
            <given-names>T.</given-names>
            <surname>Nabatchi</surname>
          </string-name>
          , M. Leighninger (Eds.),
          <source>Public Participation for 21st Century Democracy</source>
          ,
          <year>2015</year>
          . doi:
          <volume>10</volume>
          .1002/9781119154815
        </mixed-citation>
      </ref>
      <ref id="ref4">
        <mixed-citation>
          [4]
          <string-name>
            <given-names>C.</given-names>
            <surname>Pateman</surname>
          </string-name>
          , Participatory Democracy Revisited. Perspectives on Politics (
          <year>2012</year>
          ),
          <volume>10</volume>
          (
          <issue>1</issue>
          ):
          <fpage>7</fpage>
          -
          <lpage>19</lpage>
          . doi:
          <volume>10</volume>
          .1017/S1537592711004877
        </mixed-citation>
      </ref>
      <ref id="ref5">
        <mixed-citation>
          [5]
          <string-name>
            <given-names>M.</given-names>
            <surname>Howlett</surname>
          </string-name>
          ,
          <article-title>Policy analytical capacity and evidence-based policy-making: Lessons from Canada</article-title>
          .
          <source>Canadian Public Administration</source>
          (
          <year>2009</year>
          ),
          <volume>52</volume>
          (
          <issue>2</issue>
          ),
          <fpage>153</fpage>
          -
          <lpage>175</lpage>
          . doi:
          <volume>10</volume>
          .1111/j.1754-
          <fpage>7121</fpage>
          .
          <year>2009</year>
          .
          <volume>00070</volume>
          _1.x
        </mixed-citation>
      </ref>
      <ref id="ref6">
        <mixed-citation>
          [6]
          <string-name>
            <given-names>M.</given-names>
            <surname>Long</surname>
          </string-name>
          ,
          <article-title>Beyond traditional boundaries: Government in the information age</article-title>
          .
          <source>Australian Journal of Public Administration</source>
          (
          <year>2002</year>
          ),
          <volume>61</volume>
          (
          <issue>1</issue>
          ),
          <fpage>3</fpage>
          -
          <lpage>12</lpage>
        </mixed-citation>
      </ref>
      <ref id="ref7">
        <mixed-citation>
          [7]
          <string-name>
            <given-names>J.</given-names>
            <surname>Owen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Connor</surname>
          </string-name>
          ,
          <string-name>
            <given-names>H.</given-names>
            <surname>Linger</surname>
          </string-name>
          ,
          <article-title>Project management as a tool of policy implementation</article-title>
          .
          <source>Paper presented at PMI® Research and Education Conference</source>
          , Limerick, Munster, Ireland. Newtown
          <string-name>
            <surname>Square</surname>
          </string-name>
          (
          <year>2012</year>
          ), PA: Project Management Institute
        </mixed-citation>
      </ref>
      <ref id="ref8">
        <mixed-citation>
          [8]
          <string-name>
            <given-names>O.</given-names>
            <surname>Gil</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.E.</given-names>
            <surname>Cortés-Cediel</surname>
          </string-name>
          ,
          <string-name>
            <surname>I. Cantador</surname>
          </string-name>
          ,
          <article-title>Citizen Participation and the Rise of Digital Media Platforms in Smart Governance and Smart Cities</article-title>
          .
          <source>International Journal of E-Planning Research</source>
          (
          <year>2019</year>
          ),
          <volume>8</volume>
          (
          <issue>1</issue>
          ),
          <fpage>19</fpage>
          -
          <lpage>34</lpage>
          . doi:
          <volume>10</volume>
          .4018/ijepr.2019010102
        </mixed-citation>
      </ref>
      <ref id="ref9">
        <mixed-citation>
          [9]
          <string-name>
            <given-names>I.</given-names>
            <surname>Cantador</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Bellogín</surname>
          </string-name>
          ,
          <string-name>
            <given-names>M.E.</given-names>
            <surname>Cortés-Cediel</surname>
          </string-name>
          ,
          <string-name>
            <given-names>O.</given-names>
            <surname>Gil</surname>
          </string-name>
          ,
          <article-title>Personalized recommendations in eparticipation</article-title>
          .
          <source>Proceedings of the International Workshop on Citizens for Recommender Systems</source>
          (
          <year>2017</year>
          )
          <article-title>- CitRec '17</article-title>
          . doi:
          <volume>10</volume>
          .1145/3127325.3127330
        </mixed-citation>
      </ref>
      <ref id="ref10">
        <mixed-citation>
          [10]
          <string-name>
            <given-names>M.</given-names>
            <surname>Toots</surname>
          </string-name>
          ,
          <article-title>Why E-participation systems fail: The case of Estonia's Osale.ee</article-title>
          .
          <source>Government Information Quarterly</source>
          (
          <year>2019</year>
          ),
          <volume>36</volume>
          (
          <issue>3</issue>
          ),
          <fpage>546</fpage>
          -
          <lpage>559</lpage>
          . doi:
          <volume>10</volume>
          .1016/j.giq.
          <year>2019</year>
          .
          <volume>02</volume>
          .002
        </mixed-citation>
      </ref>
      <ref id="ref11">
        <mixed-citation>
          [11]
          <string-name>
            <given-names>J. I.</given-names>
            <surname>Criado</surname>
          </string-name>
          ,
          <string-name>
            <given-names>F.</given-names>
            <surname>Rojas-Martín</surname>
          </string-name>
          ,
          <article-title>Adopting Social Media in the Local Level of Government: Towards a Public Administration 2.0?, Public Administration and Information Technology</article-title>
          , in: Mehmet Zahid Sobaci (ed.) (
          <year>2016</year>
          ),
          <source>Social Media and Local Governments, edition 127, chapter 0</source>
          , pages
          <fpage>135</fpage>
          -
          <lpage>152</lpage>
          , Springer.
        </mixed-citation>
      </ref>
      <ref id="ref12">
        <mixed-citation>
          [12]
          <string-name>
            <given-names>L.</given-names>
            <surname>Brainard</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J. McNutt</given-names>
            ,
            <surname>Virtual</surname>
          </string-name>
          <string-name>
            <given-names>Government-Citizen</given-names>
            <surname>Relations</surname>
          </string-name>
          .
          <source>Administration &amp; Society - ADMIN SOC</source>
          (
          <year>2010</year>
          ).
          <volume>42</volume>
          .
          <fpage>836</fpage>
          -
          <lpage>858</lpage>
          .
          <fpage>10</fpage>
          .1177/0095399710386308.
        </mixed-citation>
      </ref>
      <ref id="ref13">
        <mixed-citation>
          [13]
          <string-name>
            <given-names>R.</given-names>
            <surname>Olszowski</surname>
          </string-name>
          ,
          <article-title>How Collective Intelligent Decisions in Public Policymaking are Made: Case Study of Participatory Budgeting in Kraków</article-title>
          . (
          <year>2021</year>
          ).
          <volume>10</volume>
          .20944/preprints202104.
          <fpage>0422</fpage>
          .
          <year>v1</year>
          .
        </mixed-citation>
      </ref>
      <ref id="ref14">
        <mixed-citation>
          [14]
          <string-name>
            <given-names>E.</given-names>
            <surname>Sorensen</surname>
          </string-name>
          ,
          <string-name>
            <given-names>J.</given-names>
            <surname>Torfing</surname>
          </string-name>
          ,
          <article-title>Collaborative Innovation in the Public Sector</article-title>
          . Working paper series:
          <article-title>Studies in Collaborative Innovation (</article-title>
          <year>2010</year>
          ). Roskilde Universitet
        </mixed-citation>
      </ref>
      <ref id="ref15">
        <mixed-citation>
          [15]
          <string-name>
            <given-names>D.</given-names>
            <surname>Benson</surname>
          </string-name>
          ,
          <string-name>
            <given-names>A.</given-names>
            <surname>Jordan</surname>
          </string-name>
          , Environmental Policy: Protection and Regulation,
          <source>International Encyclopedia of the Social &amp; Behavioral Sciences (Second Edition)</source>
          , Elsevier,
          <year>2015</year>
          , Pages
          <fpage>778</fpage>
          -
          <lpage>783</lpage>
          , ISBN 9780080970875
        </mixed-citation>
      </ref>
      <ref id="ref16">
        <mixed-citation>
          [16]
          <string-name>
            <surname>Filograna</surname>
            <given-names>A</given-names>
          </string-name>
          ,
          <string-name>
            <surname>Perossini</surname>
            <given-names>F</given-names>
          </string-name>
          and
          <string-name>
            <surname>Prette D.</surname>
          </string-name>
          <article-title>Evidence-based policy lifecycle management: The H2020 DECIDO experience [version 1; peer review: 1 approved]</article-title>
          .
          <source>Open Res Europe</source>
          <year>2023</year>
          ,
          <volume>3</volume>
          :70 (https://doi.org/10.12688/openreseurope.15697.1)
        </mixed-citation>
      </ref>
    </ref-list>
  </back>
</article>