Amongst the most contemporary and challenging problems of the policy making process is the generalized requirement of keeping governments and policy makers in general, accountable, via assessing the actual impact of the implemented policies. The need for transparency tends to empower solutions that are end-user oriented and do not simply rely on predefined metrics and Key Performance Indicators. Policy Compass aims to exploit the collective intelligence of citizens and policy makers alike via a user friendly tool that will let the users utilize open data and existing metrics towards analysing and assessing the perceived impact of specific policies of interest.
Prosperity, although being by default a broad and rather vague term, represents a
concept that many different approaches and mathematical equations try to quantify
and measure. Nevertheless, the concept of prosperity can be conceptualized very
differently and, therefore, there are ample instances of the measurement of prosperity
[
What is interesting to notice is that, in spite of the different approaches, many of the aforementioned indexes are used by individuals, organisations and public bodies in order to assess (amongst others) the success or failure of specific policy making measures. Since governments and policy makers look for a way of setting specific Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes. goals for their time in power, (prosperity) indexes and similar indicators appear to be the most comprehensible tool towards this direction. After a specific time frame, the indicators are calculated and the effectiveness of e.g. governmental policies on “prosperity” is estimated.
Another issue of active discussion and dispute is this of the source of information, when trying to evaluate the actual results of endeavours such as legislative acts, wide governmental policies etc. Who provides the necessary data? Is the information accurate? Is it credible? Is it complete? If the information is not available to its entirety to all stakeholders, doubts on the credibility of the assessment will always be present. Opacity in such processes leaves no room for commonly accepted results. This is the main reason behind the continuously growing demand for the release and use of open (governmental) data in as many policy making processes as possible; including the ones relevant to policy impact assessment.
Along the above lines, the purpose of the paper at hand is to showcase how Policy Compass will combine know-how from carefully selected scientific domains in order to allow stakeholders experiment with causal models, open data, argumentation and prosperity indexes in order to evaluate themselves policy impact from the end users point of view.
The paper at hand is structured as follows: Section 1 serves as an introduction; Section 2 presents the main axes of the Policy Compass approach, followed by Section 3 that presents the project’s methodology. Section 4 provides a detailed use case, while Section 5 concludes the document.
As also mentioned earlier in the document at hand, the main purpose of Policy Compass is to heavily empower stakeholders of different backgrounds in the process of analysing and evaluating the actual impact of (governmental) policies. However, one can fairly claim that there is not a single dimensional way to achieve that.
It is self-evident that a single stakeholder (or even a group of stakeholders) can face great obstacles when trying to evaluate policies and/or measure prosperity. First and foremost, credible and complete data are necessary. Secondly, the user should be able to effectively and efficiently model the parameters that he/she considers important for the calculation of the metric. Last but not least, a single stakeholder’s opinion cannot be treated as correct by default; but what if other stakeholders could come in and validate or fine-tune the initial proposition?
Policy Compass tries to cope with the latter, as well as with all aforementioned issues, through a holistic and well-defined approach. This approach is based on five main pillars, namely: • Prosperity Indexes; • Open Public Data; • Fuzzy Cognitive Maps; • Argumentation Technology; • Deliberation Platforms and Social Media.
Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes.
These pillars, although highly interdependent from the methodological point of view, will be realized through independent web components in order to grant the user the ability to separately use each one of them upon desire.
In the following paragraphs, each of the aforementioned pillars is briefly presented, in order to provide the reader with a quick overview of the Policy Compass approach.
Policy Compass, recognising that significant work has been put over the years from various high-calibre organisations in the field of prosperity indexes, will offer its end users a plethora of popular and well-established paradigms of prosperity indexes. What is even more interesting and constitutes a step forward, is the ability that will be given to all users that wish to do so, to propose, formulate and even calculate new, innovative indicators. Diving one step deeper in the process, Policy Compass will facilitate the aforementioned process via a wizard-like process, aiding the user identify relevant existing prosperity indexes and combinations amongst them. In other words, end users will be able to bring to life their own perception of prosperity, based on relevant existing work.
As in almost any innovative initiative relevant to policy making and impact
assessment, open (governmental) data and public sector information play a vital role in
Policy Compass. Policy Compass approach will take advantage of many different
sources of open public data of any geographical scope (e.g. local, national,
international etc.) and will make use of widely accepted metadata specifications; facilitating
the exploitation, enrichment and reuse of existing data. Of course, Policy Compass
will give its users the opportunity to publish themselves metadata for the open data
sets of their choice, offering thereby a wide range of capabilities for creating own
mash-ups and visualizations of metrics and historical events, currently not possible in
other e-participation or open data platforms [
Fuzzy Cognitive Maps [
Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes. 2.4
Argumentation techniques have a rather old background relevant to policy making
and especially eParticipation [
As in every initiative that aims to engage citizens, organisations, NGOs and other stakeholders in the policy making process and the actual impact evaluation of policy making activities, building an active community and exploiting the wisdom of the crowd is critical. Towards this direction, Policy Compass will incorporate in its integrated solution deliberation platforms and social media. Through these facilities, end users will be able to freely discuss their opinions and propositions, feeding also the argumentation functionality. In addition, various social media channels will be also utilised in order to disseminate developments and findings, in order to reach the maximum community possible. 3
In order for the Policy Compass methodological framework to be as concrete and comprehensible as possible, it is built around three pillars, each of which takes advantage of the axes presented in the previous chapter in a different way, namely: • Policy Performance Evaluation, aiming to fulfil stakeholders’ desire to check or verify whether a specific policy action, policy directive law etc. has actually achieved or failed to meet the initially set goals, and thereby whether the relevant or accompanying KPIs have actually reached or not the target values promised • Causal Policy Models’ Construction, aiming to facilitate the user discover relevant data and causal network models, construct (or ameliorate existing) respective Fuzzy Cognitive Maps, simulate the constructed models and visualize and disseminate the accruing results.
• Online Deliberation and Argument Mapping,
Each pillar puts particular emphasis on a different aspect of the suggested
approach. It has to be noted that these three pillars are not isolated from each other, but
are meant to work together for providing end users a unified experience for more
factual, evidence- based, transparent and accountable policy analysis and evaluation,
by combining, as already discussed, the features and capabilities of metrics
construcCopyright © 2015 for this paper by its authors. Copying permitted for private and
academic purposes.
tion and visualization, fuzzy cognitive maps, and structured argumentation, based on
deliberations [
The overall methodological framework, underpinning the Policy Compass approach is illustrated in Figure 1 in the form of three interacting workflows, each one corresponding to a different pillar of the proposed approach.
A highly debated question in society is: “What is a good policy on drugs?” On the one hand, using drugs can be considered a matter of personal freedom. On the other hand, the use of drugs is related to health problems and crime. Thus, an effective policy on drugs should: • reduce drug related health problems, • reduce drug related crime and • limit restrictions to personal freedom to a minimum.
While EU member states share the “acquis communautaire”, they have very distinct policies in the health and justice domains. Drug policies connect these domains. Thus, there is a variation of policies across Europe and one would expect different outcomes. However, the national policies are not static but subject to change and thus one would expect a change in outcome and impact with a change of a policy.
In this scenario, we analyse impacts of a drag policy change in Spain and compare it with the situation in Germany. The analysed countries differ greatly regarding their Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes. policies on drugs. A look on the data and comparison with the policies should reveal the comparative performance of national policies and which policies yield what effects. The analysis will focus on the policies on the use of cannabis. While drugs like alcohol and nicotine are legal in these countries, the legalization of cannabis is a very controversial issue in public debates across Europe. There are significantly less proponents of a legalization of other prohibited drugs.
According to proponents of strict drug regulation, cannabis is considered an entry drug. Therefore, legalization or an easier access to cannabis would increase the number of drug dependent people. At the same time, the numbers for drug related crime should go up1. Data on these issues can be found at the European Monitoring Centre for Drugs and Drug Addiction (EMCDDA) and at Eurostat, where the user can download appropriate data tables. Selected tables can be uploaded to Policy Compass. The upload wizard of Policy Compass assists in structuring the data in the right way. It helps to find the data for the needed rows and columns in the source table – not all tables are structured in the same way and often, data tables2 as found on the Internet contain more fields than necessary, additional annotations, etc.
Once the data is uploaded to Policy Compass, the user defines the indicators “number of drug deaths”3 and “drug related crime” (EMCDDA provides data for “number of drug law offences”4). Then, the user defines a metric on these indicators. Using these two factors only, the metric could for example count the number of drug deaths (weighted twice) and add the number of drug law offences (weighted once). Although not all drug law offences might be caught, it takes several incidents for an individual to grow drug dependence from drug use and even more to die from it. Moreover, one might value the health of people more than the sheer number of drug related crimes, as a “drug crime” is firstly a mere problem of non-compliance. The term “drug crime” insinuates gang crime, violence, or mafia activities. However, this is not what is measured in this indicator and data on “drug related violence” or “drug related organized crime” could not be found. A more lax policy on the use of cannabis could have a negative effect on drug crimes recorded by the police (if it is legal, it is 1 A better test of the underlying hypotheses would monitor drug dependence and drug related crime also for the case of a country that tightens its drug laws. However, there has been no recent case with appropriate data in the EU to examine this. 2 The data has been archived on the Policy Compass GitHub repository: https://github.com/policycompass/policycompass-scenarios/tree/master/Drug%20Scenario 3 Cp. http://appsso.eurostat.ec.europa.eu/nui/show.do?query=BOOKMARK_DS063189_QID_3DAE9910_UID_3F171EB0&layout=TIME,C,X,0;GEO,L,Y,0;UNIT,L,Z,0;SEX,L,Z,1;AGE,L,Z,2;ICD10,L,Z ,3;INDICATORS,C,Z,4;&zSelection=DS-063189INDICATORS,OBS_FLAG;DS063189ICD10,A-R_V-Y;DS-063189SEX,T;DS-063189AGE,TOTAL;DS063189UNIT,NBR;&rankName1=ICD10_1_2_-1_2&rankName2=UNIT_1_2_1_2&rankName3=AGE_1_2_-1_2&rankName4=INDICATORS_1_2_1_2&rankName5=SEX_1_2_1_2&rankName6=TIME_1_0_0_0&rankName7=GEO_1_2_0_1&sortC=ASC_1_FIRST&rStp=&cStp=&rDCh=&cDCh=&rDM=true&cDM=true&footnes=false&empty=f alse&wai=false&time_mode=NONE&time_most_recent=false&lang=EN&cfo=%23%23%2 3%2C%23%23%23.%23%23%23 4 Cp. http://www.emcdda.europa.eu/stats13#display:/stats13/dlotab1a Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes. not criminal). However, one has to consider that it could be only the (personal) use being legalized and not the commercial trade. The data would also have to be normalized. First, the numbers from both sources would have to be set in relation to the population of the respective countries. Second, the per capita numbers would have to be mapped to a qualitative scale indicating the severity of the number of deaths and criminal offences. For example, the numbers could be mapped to real numbers in the range of 0.0 to 1.0, where 1.0 is the worst value and 0.0 is the best. One way to do this would be to map the highest per capita value in the historical data to 1.0 and all other values could be set relative to this.
The above is an example for argumentation processes in the mind of the user when approaching the evaluation of policy performance. Of course, the reader might prefer using other indicators or weighing them differently. Also, the data could be normalized in a different way (here, the user just adds up the numbers). When constructing a metric, the user implicitly takes decisions when selecting indicators and weighing them. The pros and cons of these design decisions would have to be debated. In our case, since the Policy Compass metrics designer is still in development, we will operate with two indicators “number of drug law offences” and “number of deaths from drug dependence”.
After deciding on the metrics and their indicators, the user applies the metric to the datasets. Studying the data available at the EMCDDA website, one notices that data is not available for all countries for the same time period. The user may have to decide what to do about missing data. For example, it could be added from different sources, calculated or estimated or left blank. Applying the metric provides a measure for each country for a selected year (unless the data for this year was left blank).
In our case data for the selected indicators is available for Germany and Spain. Policy Compass visualizes the indicators and the metric in a diagram (for example in a curve chart; Figure 1). This would reveal that the numbers for deaths from drug dependency are falling in Spain from 2004 onwards. At the same time, the number of all drug law offences has exploded since 2004 in Spain. In Germany there is no significant changes in the time period. The values in the chart are presented in percent compare to the first available value in each dataset. This is done to better illustrate the change dynamic.
Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes. Searching for explanations of these effects, the user could search the Policy Compass event database or add events that he or she sees as being linked to the subject. Here, Policy Compass might suggest (if it is in its database) a link to the 2003 Spanish Supreme Court rulings that stated that the possession of even large quantities of cannabis was not a criminal offence if there was no evidence of an intent for trafficking or sale-for-profit. In the chart this event marked as a vertical orange stripe.
The Policy Compass user could now search for a causal policy model that explains the mechanisms behind drug policies, drug use, crime and drug dependencies. This model could explain the connections further and quantify them, explain puzzles in the data and – applying the model on the metric and indicators – forecast future developments. For our scenario the explanation can be the following. Liberalisation of cannabis increases its legal possession by private persons. That however, drives the criminality since more people possessing it are trying sell or distribute cannabis. At the same time, our guess may be, that higher availability of cannabis decreases the consumption of hard drugs, which are the main reason for drug related deaths. The causal model below presents this theory. It has to be mentioned that the presented model is very primitive. For proper analyses it has to be further extended and supported by additional data. However, even a simple model is a good start for a discussion and collaborative analyses of the issue. Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes.
The findings of the application of the metric and the explaining causal policy model could now be shared in the Social Web. The selection of the data, the construction of the indicators and the metric, the selection and construction of a causal policy model – all these decisions could now be further discussed online.
Along the above lines, open (governmental) data and the social web constitute a catalyst towards engaging various stakeholders in an open and collaborative process of calculating, discussing and evaluating the actual impact of policies and governmental initiatives in general, particularly through the exploitation of various (prosperity) indicators.
In the paper at hand, the authors have provided a comprehensive description of the Policy Compass main axes and tools, aiming to constitute an important contribution in the aforementioned fields. The use case described, aimed to provide an easy to understand example, showcasing (part of) the added value of such an approach.
It is expected that the experimentation in the project’s two pilot applications will strengthen the use of indicators (especially in the UK and the Russian Federation where the pilots are located), providing the Policy Compass consortium and the research community in general with valuable insight and feedback on the methodology employed and the platform deployed within Policy Compass.
The research leading to these results has been supported by the EC FP7 under the project “Policy Compass” Grant Agreement 612133. Copyright © 2015 for this paper by its authors. Copying permitted for private and academic purposes.