Whom? Investigating the Swedish Case of Open Comparisons

Abstract

Today, the Swedish public sector is afected by a focus on measurement, control, evaluation, and systematic follow-up to improve work quality. Open comparisons are a series of surveys, initiated by National Board of Health and Welfare (NBHW) and works as a system for performance measurements. It is presented to serve comparisons of quality among municipalities, and to provide information to citizens who are supposed to choose public social services. This paper aims to investigate perceptions of Open Comparisons by professionals working with quality management in social services. The study uses focus group interviews and document analysis. The results reveal that professionals in social services considered Open Comparisons as time consuming, with excessive focus on quantitative data, lack of feedback, and too much interpretive flexibility, leading to mismatched data. These challanges raised questions regarding the validity of comparisons and the point of/meaning of Open Comparisons. This outcome leads to the question, 'Who do open comparisons serve?' from 2014 (25 June 2014), systematic follow-up is claimed to provide transparency for operations, support the fulfilment of requirements and expectations of the organisation in processing, documenting, and ensuring the quality of the work, and create and adapt documentation in databases to compare quality. It is also assumed to generate the grounds for decision-making regarding resources and achievements. Last, it is supposed to generate knowledge about best practices [5]. The value of systematic follow-up is associated with three levels.

Keywords

• Indicators
• measurement
• systematic follow-up
• open comparisons
• practice

1. Introduction

Swedish welfare organisations are under the requirements to evaluate their activities through systematic follow-up [ 1 ]. These requirements can be understood in the context of audit society [ 2, 3 ], and evaluation society [ 4 ], in which local governments performances are measured, evaluated, and exposed to public scrutiny. This can be seen as a result from the increasing rate of digitalisation where the possibilities to measure has also been enabled through information systems and the generating of data. In a National Board of Health and Welfare (NBHW) report rOcid - (A. Hasselblad); - (S. Zimic)

© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). III Central follow-up - the national level (e.g. open comparisons; Swedish: Öppna jämförelser [ 5 ]). 1.1. Aim This study aims to investigate perceptions of the use of systematic follow-up on the third level, the Central follow-up (national level) as it is the level where open comparisons takes place. We take on the perspective of quality managers in social work practices in six Swedish municipalities. The empirical material is gathered through focus group interviews and a document analysis of one document from 2017 describing the results of Open comparisons. The analytical framework in this study builds on previous research on standardisation [ 6 ] and classifications [ 7 ].

2. Open Comparions

The NBHW annually presents a set of indicators called ‘open comparisons’ [ 5 ]. Open comparisons comprise a national follow-up system that aims to gather information for comparison and primarily targets oficials and decision-makers in municipalities and regions and politicians at various levels. The information is gathered through a survey distributed by the NBHW in coordination with the Swedish Association of Local Authorities and Regions. According to the NBHW, open comparisons aim to create openness and improve transparency in publicly funded healthcare and social services, providing a basis for follow-up, analysis, improvement and learning in activities. They aim to initiate local, regional, and national analyses and discussions on the quality of the activities and provide a basis for governance and management. The information is presented either from each municipality, county council, region or county and is compared using rankings. The areas included in open comparisons are financial aid, disability services, municipal health care, crisis preparedness, addiction care, homelessness services, child and youth care, social psychiatry, violence in close relationships, and elderly care.

Open comparisons have previously been looked upon from multiple researchers e.g in elderly care [ 8, 9, 10 ], in health care [ 11 ], and social services [ 12, 13, 14 ] in with we position this paper. Historically, the purpose has been quality improvements for the medical profession, but today they are employed as a national strategy for comparing, controlling, and auditing tools for principals, interest groups and authorities. An argument for using open comparisons is providing information to patients to choose where to obtain care. Carlstedt and Jacobsson [15] discusses how open comparison constructs quality as something definable and measurable which leads to a reduction of the complexity of concept. Quality is therefor in some sense created in in open comparisons as it can be connected to documents, plans and agreements. How the actual quality (the content) is created are mot measured, open comparisons does not give information of the quality is implemented but rather if it exists or not (e.g. number of action plans).

3. Categorisation and Standardisation

National systematic follow-up surveys, such as open comparisons, build on standardisation under the argument of enabling the comparison of municipalities. In many cases, standardisation has provided remarkable success, such as using standardised container freight, where improved eficiency could be reached. Nevertheless, standardisation can also provide misguidance by making the infrastructure invisible [ 6 ]. In more complex processes, such as social services, standardisation hides the individuality present. For example, the structures of social services are diferent in each municipality, making them challenging to compare. One way to address this is to create standardised indicators for targeted municipalities, which each municipality reports. However, although indicators are standardised, the municipal processes still difer, and the problem is passed down to municipalities to interpret what to report. Bowker and Star [ 7 ] described how, as individuals, we sort almost everything around us. We sort our clothes into distinct categories of black or coloured, and our computers show how we sort information into folders from various aspects, such as more or less important. On a societal level, we use standards to create conformity by a common acceptance of diferent rules (e.g. internet standards). These standards work on various levels and may not always be coherent, creating dificulties for the individuals to manage their standards. Some researchers argues that usage of measurements and comparisons involve indirect control through the selections and categorisations that are put into focus [ 16, 7 ].

4. Previous Research

The use of indicators in public-sector organisations has endured much critique, often traced back to the New Public Management and Evidence movements [17, 18, 19, 20, 21]. The use of indicators is also related to the focus on numbers and statistics in social welfare, which has been highlighted in previous research [22], and many scholars have criticised the over-simplification with performance indicators [ 4 ] and its consequences for welfare organisations [ 2 ]. One of the consequences of focusing too much on performance indicators is that only scalar or simple measurable values are visible, reducing the organisation to a few aspects visualisable in numbers, statistical diagrams, and tables. Values such as trust, or safety become invisible and are pushed out to the margins or even disappear from the organisation’s priorities. By only measuring visible numbers, the systematic follow-up procedure is sometimes ‘hitting the target but missing the point’ [ 4, 20, 23 ].

5. Method

The empirical material in this paper comprises one main survey document from open comparison and information on the websites www.socialstyrelsen.se and www.skr.se regarding open comparisons. The study also uses focus group interviews with quality managers within social services as empirical material. The review of the statistical document related to open comparisons is included to reach a deeper understanding regarding how open comparisons afect the quality management profession and how the use of Open Comparisons is perceived. The review of the statistical document related to open comparisons aims to clarify the data used for follow-up, such as the amount, time, monetary data, or quantified qualitative data. The result is later discussed concerning the results from the focus group interviews to compare them.

5.1. Focus Group Interviews

One part of the empirical material in this study is focus group interviews with quality managers on social work practices in six municipalities in Västernorrland County, Sweden. The focus group interviews were performed in a study in 2018 at the Research and Development Unit of the Association of Local Authorities in Västernorrland County. The municipalities are of various sizes. The largest is almost 100,000 inhabitants, and the smallest is 10,000 inhabitants. The municipalities are organised in diferent manners and have several types of professionals supporting the management of quality work. Besides quality managers, professionals such as system managers, medical nurses, and sometimes the heads of a specific unit participated in the study. The number of participants in the focus group interviews varied from three to nine. Altogether, 31 participants participated in the study.

The interviews were semi-structured, posing questions focused on systematic follow-up to gather their overall perceptions of systematic follow-up. Questions included the following: • ‘How is systematic follow-up described by the participants?’ • ‘What is the participants’ mission, and what is expected of them concerning follow-up?’ • ‘What kind of follow-ups are performed, and how are they used?’ • ‘What kind of questions or need for knowledge do the follow-ups aim to answer?’ • ‘If so, how do the results of the follow-ups contribute to the development of the social work practice?’ • ‘What prerequisites do the participants observe for future use of systematic follow-up in social work practices?’

As systematic follow-up works on three levels: individual, operational/business, and central, the interview questions included all levels. The answers focusing on open comparisons and systematic follow-up on a national level were used as the primary empirical material in this paper.

6. Results and Analysis

The focus group interviews demonstrate how several respondents emphasised that quantitative measures were the main information requested and that there was nothing they were supposed to report regarding the quality of work, as stated in the description of the open comparisons. Thus, according to the quality managers, open comparisons aim to produce numbers and statistics and presume some classifying principle to structure the information and later code it to become statistics in the form of variables. For oficial statistics, the classification is scaled up, meaning that the structured information according to a classification system is not applicable in only one context (e.g. one municipality) but many (e.g. all municipalities in Sweden). Oficial statistics require standardisation - a set of agreed-upon rules for producing (textual or material) objects that span more than one community. This standardisation is not established among quality managers using open comparisons.

In one of the focus group interviews, the respondents explained their encounters with some of the key performance indicators they must report. When reporting data to the Council for promotion of municipality-based analysis (in Swedish, Rådet för främjande av kommunala analyser, RKA), the respondents often experienced dificulties regarding the interpretation of the indicators, leaving them with uncertainty regarding exactly which data they should report. Thus, a mismatch often exists regarding what they are supposed to report and what is available in the system. The discrepancy is a sign of the lack of conformity at the national and local levels. The perception corresponds to what Hanberger and Lindgren [24] describe regarding how data form national benchmarking systems can be problematic because of varying data quality, and governing by numbers that are not trustworthy is risky. Indicators are uncertain because of their change over time and that the municipality’s ranking depends on the scoring of others’. Benchmarking is not a solid mechanism for ensuring quality in e.g. eldercare even though it can have value for quality work.

In the focus group interviews, the quality managers also mentioned that some of the indicators, which come from the central agencies are not validated. Sometimes the indicators create a gap between the political goals and the knowledge needs of the profession. Therefore, some of the indicators are not perceived as useful for the profession and are describes by one study participant as ‘just a number no one cares about’. Another study participant articulated that: ‘many times the politicians do not know what they inquire’, and they ‘only measure what is available’ This outcome reveals the lack of an agreed acceptance regarding the standardised method of measuring. Politicians and the professionals seem to have somewhat diferent perceptions of the national systematic follow-up. Johansson and Liljegren [ 13 ] came to similar conclusion - that the politicians had slightly more positive perception of Open Comparisons than the public oficials had, although their study show that OC is considered to be a mature system, accepted by both groups of actors.

According to the study participants, the information gathered through systematic follow-ups is considered as ’reporting to someone else’. It means that the study participants did not always know the use of data they reported. One respondent stated that the reported data ‘goes up in cyberspace’, and another stated, ‘We submit statistics that go to the NBHW. There is nothing that we process ourselves’. This remark indicates that the respondents do not experience the ‘follow-up’ in the systematic follow-up actions, such as open comparisons, as they just provide the requested data.

The respondents also referred to the process of reporting data as time-consuming. They used such descriptions as ‘it is energy-consuming’ and ‘one gets sweaty when it comes’ (referring to the surveys and data reports). The main reason is that most reports must be performed manually. For example, the information required in national surveys, such as open comparisons, cannot always be retrieved from the system for several reasons. One of the reasons is that the information required in the survey (or report) and the information registered in the system do not perfectly match. Municipalities and counties use routines in such a diverse ways that the way the data are compared today is invalid to some extent. The systems used within the compared municipalities must be more similar to each other to enable a correct comparison. One of the respondents wished for a communal system that everybody uses to enable proper comparison between counties and municipalities, meaning that every unit can generate the corresponding data from the system. The quality managers expressed that they need to: ‘be aware and have knowledge about how to extract the numbers representing certain information from the system because if you do not, it will be a dificult task to perform’. Because of the diverse construction of the systems, the study participants said: ’we measure a modified truth’. If the extracted indicator data are diverse and not correlated, the usefulness of systematic follow-up becomes insuficient.

In the interviews with quality managers, all respondents discussed the need for feedback and the importance of making sense of the results from the follow-ups. A social worker or a nurse focuses on the individual they are helping. The professionals engage in follow-up activities to gain feedback and determine whether these ‘follow-up activities’ lead to improvement for the individuals they help. Much too often, large-scale follow-ups cannot deliver the type of information that can make sense to the professionals regarding the knowledge of improvement for individuals.

6.1. Document Analysis

Multiple indicators are presented in the open comparisons survey result document (Socialstyrelsen). And the main part of the these indicators are represented by either a number, per cent or a ratio, for example the number of children per citizen. Under the areas of addiction care, homeless services, and child and youth care, indicators focus on numbers, such as the number of patients re-enrolled in treatment, suicides, mortalities, and those who sought care. The numbers are also used as ratios. For example, in financial aid, the numerator is the number of citizens (children and adults), and the denominator is the total citizen population in that region or municipality, which does not reflect the limits of what makes citizens a recipient for financial aid that can vary among municipalities. Another frequently occurring type of indicator is categorical variables. For example, a municipality may or may not have routines or strategic documents for certain coordinations or agreements within diferent areas. The answers to these indicator types are ‘yes’ or ‘no’. In this sense these indicators do not give information of the quality is implemented but rather if it exists or not.

In this case, one reflection that the use of indicators brings is their bluntness. By only indicating whether the organisation has routines or strategic documents for a certain event, the indicators fail to identify the quality that the documents should provide. For example, when is a ‘routine’ a routine? Is it when the routine is documented or orally agreed upon within the organisation? This situation puts pressure on the interpretive capacity of the person reporting data and may lead to inconsistencies in multiple reporting parts, which can be related to the concept of data representation as ‘hitting the target but missing the point’ [ 4, 20, 23 ]. The current indicators answer the question ‘Does this organisation have a strategic document?’ and do not address its background information, such as whether the routine is used and contributes value to the organisation. The motivation behind using these indicators is organisational development, which is questionable regarding whether it can be achieved with indicators that do not represent or measure the organisation’s actual value. This outcome is also reflected in the focus group interviews, where the respondents pointed out that the indicators are meaningless due to their dificulty to interpret and their use at someone else’s request.

Another aspect of the document is the frequent use of a nominal scale (yes or no) that creates high uncertainty due to the wide intervals it represents. For example, in computing, high uncertainty evaluations often use fuzzy data represented by intervals. Using such blunt measures raises the question of how much the information contributes from the aspect of organisational development. The number of indicators is also high and demands considerable time from the data reporting part. For example, do the 36 indicators in disability services all use a nominal scale? With the high uncertainty from both the nominal scale and interpretative aspects, such a high number of indicators may be unnecessary. As the focus group interviews demonstrated, quality managers need to spend considerable time providing data for the indicators. A possible solution could be to focus on fewer indicators agreed upon by those at the national and local, professional levels in a standardisation. [15] E. Carlstedt, K. Jacobsson, Indications of quality or quality as a matter of fact?, Statsvetenskaplig tidskrift 119 (2017) 47–70. [16] P. Miller, N. Rose, Political rationalities and technologies of government, in: Texts,

Contexts, Concepts. Studies on Politics and Power in Language, 1990, p. 166–183. [17] G. Bevan, C. Hood, What’s measured is what matters: Targets and gaming in the english public health care system, Public Administration 84 (2006) 517–538. [18] P. Jackson, Governance by numbers: What have we learned over the past 30 years?, Public

Money & Management 31 (2011) 13–26. [19] G. Bouckaert, W. Balk, Public productivity measurement: Diseases and cures, Public

Productivity & Management Review (1991) 229–235. [20] Z. Radnor, Hitting the target and missing the point? developing an understanding of organizational gaming, in: Performance Information in the Public Sector, Springer, 2008, p. 94–105. [21] H. de Bruijn, Managing performance in the public sector, Routledge, 2003. [22] T. Hjärpe, Mätning och motstånd: Siferstyrning i socialtjänstens vardag, 2020. [23] J. Green, Education, professionalism, and the quest for accountability: Hitting the target but missing the point, Routledge, 2011. [24] A. Hanberger, L. Lindgren, Evaluation systems in local eldercare governance, Journal of

Social Work 19 (2019) 233–252. [25] U. Eriksson-Zetterquist, M. Hansson, F. Nilsson, Theories and Perspectives in Business

Administration, Studentlitteratur AB, 2020. [26] A. Forssell, A. Westerberg, Administrationssamhället. Studentlitteratur Lund, 2014. [27] A. Hasselblad, L. Sundberg, When Worlds Collide: Comparing the Logic of the Industrial and Welfare Societies, EGOV-CeDEM-EPart-*, 2020.

References

[1] A. Bergmark , T. Lundstrom, Mot en evidensbaserad praktik?-om färdriktningen i socialt arbete , Socialvetenskaplig Tidskrift 13 ( 2006 ).

[2] M. Power , The audit society: Rituals of verification , OUP Oxford , 1997 .

[3] M. Bowerman , H. Raby , C. Humphrey, In search of the audit society: Some evidence from health care, police and schools , International Journal of Auditing 4 ( 2000 ) 71 - 100 .

[4] P. Dahler-Larsen , Constitutive efects of performance indicators: Getting beyond unintended consequences , Public Management Review 16 ( 2014 ) 969 - 986 .

[5] Öppna jämförelser av socialtjänst och kommunal hälso- och sjukvård , n.d. URL: https: //www.socialstyrelsen.se/statistik-och-data/oppna-jamforelser/socialtjanst/.

[6] M. Lampland , S. Star , Standards and their stories: How quantifying, classifying, and formalizing practices shape everyday life , Cornell University Press, 2009 .

[7] G. Bowker , S. Star , Sorting things out: Classification and its consequences , MIT press, 2000 .

[8] S. Johansson , Utvärdering av äldreomsorg: Examplet Öppna jämförelser, 2011 .

[9] P. Kajonius , A. Kazemi , Rankning av sveriges kommuners äldreomsorg i Öppna jämförelser , Socialmedicinsk Tidskrift 91 ( 2014 ) 323 - 331 .

[10] A. Hanberger , L. Nygren , K. Andersson , L. Lindgren , Tillsyn och öppna jämförelser: Exemplet äldreomsorgen. utvärderingars värde-värden i utvärderingar: Svenska utvärderingsföreningens nationella utvärderingskonferens , 2015 .

[11] M. Blomgren , C. Waks , Öppna jämförelser inom hälso-och sjukvården-en utmaning mot den professionella byråkratin? , Arbetsmarknad & Arbetsliv 17 ( 2011 ) 95 - 108 .

[12] E. Carlstedt , En socialtjänst med kvalitet? Öppna jämförelser som skyltfönster och verktyg , 2015 .

[13] S. Johansson , A. Liljegren , Socialtjänsten som granskningsobjekt: Politikers och tjänstemäns användning av Öppna jämförelser , Socialvetenskaplig Tidskrift 28 ( 2021 ) 3 - 28 .

[14] S. Trygged , Öppna jämförelser i socialtjänsten-varför, hur och till vilken nytta? founordväst , 2016 .