The introduction of algorithmic decision-making in the social welfare domain has contributed to the emergence and amplification of harm to citizens. In several cases, the use of algorithmic decision-support systems has led to the formation of so-called “digital cages”; administrative exclusion by digital information architectures. To date we still lack comprehensive theoretical concepts to describe and analyze the systemic hazards introduced by algorithmic systems. Our study illustrates the afordances of system-theoretic concepts and methods, drawing on system safety, to understand and analyze algorithmically induced hazards in public governance. We show, on example of the Dutch Childcare Benefit Scandal , that the system safety discipline ofers powerful concepts and tools to understand, prevent, and address algorithmically induced system hazards in social welfare. By applying system safety concepts, our study contributes to the development of sociotechnical assessment approaches for algorithmic systems in public governance.
[
CEUR Workshop
ISSN1613-0073
“toezichtslijst” (monitoring list). The list consisted of citizens who were suspected of fraud or gross
negligence. Once a person was on this list, any signal associated with them would be flagged by one of
several big data feed risk models and forwarded to case workers for personal control. Furthermore, until
2019, data from the list was used to train and test these risk models. This closed-loop process meant
that registration on the list significantly increased scrutiny and the chances of (incorrect) determination
of fraud. The average time spent on the list was 4,8 years [
3. Mental Model Misalignment; is a function of the lack of information and feedback exchange between
system components [
Final Remarks: This research underscores the critical need for safety engineering methods for algorithmic interventions in social welfare. System Safety provides practitioners with a structured framework to navigate the complexities of public algorithmic systems. By illustrating how wellunderstood safety problems manifested in the Childcare Benefit Scandal, we demonstrate STPA’s efectiveness in identifying and addressing complex system interactions in digital social welfare systems. Declaration on Generative AI During the preparation of this work, the author(s) used Grammarly to: Grammar and spelling check, Paraphrase, and reword. Additionally, a RAG pipeline utilizing OpenAIAPI-GPT4 was developed to query relevant documents related to the case study. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content.