Artificial intelligence (AI) driven technological advances have increased the concerns over the bias problem, especially in high stake applications such as healthcare. This research proposes an AI-based healthcare framework with a feedback loop to ensure a fair real-world practical solution. We argue the need to consider continuous quality improvements of such technologies by including feedback at each development and deployment stage. Furthermore, medical experts, AI experts and policymakers must work together to ensure fairness. We use simple New Zealand-based patient data as a case study. We provide early-stage experimental results using machine learning algorithms where fairness and mitigation are also considered in addition to accuracy measures.
care framework incorporating mechanisms to ensure fairIn a technological era driven by artificial intelligence ness and consider CQI. We emphasise that at each stage (AI), the benefits of incorporating such advances in real- of the process –data collection, algorithm development, world high stake applications, including healthcare, are evaluation and implementation– there is a need to inbecoming the norm [1, 2]. Equally, there is an increase in clude a feedback loop where AI experts, medical profesconcerns and awareness of the issues related to the bias sionals and policymakers are involved. We also provide problems in AI models [3, 4, 5]. In general, fairness is a simple real clinical example in the New Zealand (NZ) seen as being impartial and fair. AI models can be biased setting to demonstrate the stages of the framework. The and can make “unfair” decisions, where such decisions research presented in this paper is at the development are skewed toward a particular group of people [6, 7]. stage, and as such, we acknowledge there are limitations While there is an increase in urgency towards handling and scope for improvements. the bias problem and developing fair AI models, especially in clinical settings, such research is limited by the 2. Framework need to consider practical deployment. Furthermore, pre- Figure 1 provides an overview of the proposed framedominant research is focused on the Black and White work. We have split the process into four parts, (i) data, racial issues in the US population [8, 7]. (ii) problem definition, (iii) algorithm selection and (iv)
Recent studies have emphasised a need for continual feedback, and also list the vital stages of each part. We monitoring and update to ensure the long-term reliability use dash lines to show connections between the parts and efectiveness of AI-based clinical algorithms [ 9, 10]. where feedback is beneficial and may include several cyContinuous quality improvement (CQI) [11] is a common cles, and arrows indicate the flow’s direction. Figure 1 phenomenon in healthcare, where incremental and pro- also provides experts’ level of involvement at each stage, gressive improvements are considered at various stages and decisions that require predominant input from polof operations to ensure patient care and safety. As such, icymakers are provided in ‘red’. In this framework, we using AI in clinical settings to aid clinical decisions and view patients as stakeholders, and as such, their engagerisk predictions requires CQI. ments can improve research appropriateness, acceptabilThis paper proposes developing an AI-based health- ity, feasibility, delivery, and dissemination [12]. In New Zealand, the indigenous population Ma¯ori and other minority groups are important stakeholders, so it is crucial to handle their data with care [13, 14, 15] and incorporate feedback from the minority groups.
The 6th international workshop on Knowledge Discovery in Healthcare Data (KDH), August 20, 2023, MACAO, China $ vithya.yogarajan@auckland.ac.nz (V. Yogarajan) https://profiles.auckland.ac.nz/vithya-yogarajan (V. Yogarajan) (G.0D00o0b-b0i0e0);20-6000504-0-9050413-9(9V3.9Y-8o7g7a3ra(jSa.nL);ei0t0c0h0)-0001-7245-0367 2.1. Fairness measures
© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Standard group fairness measures include demographic CPWrEooUrckReshdoinpgs IhStpN:/c1e6u1r3-w-0s.o7r3g ACttEribUutRion W4.0oInrtekrnsahtioonpal (PCCroBYce4.0e).dings (CEUR-WS.org) parity, disparate impact, equalized odds, and equalized 3.3. Algorithm Selection opportunity, where fairness concerns a group rather than an individual [16]. In this paper, we use disparate impact (DI) [17] as a quantitative measure of fairness. DI measures the ratio of rates at which the outcomes occur for one group of patients over the rest [17, 16, 18]. According to US legislation, the > threshold was set at = 0.8. While in practice, the acceptable range for the DI ratio is generally between 0.8 and 1.25. It is vital to point out that setting an acceptable threshold for NZ society in high stake applications, such as healthcare, is a good example of the need for input from policymakers and a feedback loop among all experts.
This paper uses logistic regression, XGBoost and random forest as the three machine-learning algorithms for predicting medication-related harm in NZ-GP-small datasets. We use specificity and sensitivity to evaluate 3. Case Study the performance of these three algorithms. In addition, 3.1. Data we consider the fairness measures using DI, where in For this simplified case study, we use a small subset of accordance with the US regulations, we expect the score the New Zealand General Practice (GP) Electronic Health to be between 0.8 and 1.25.
Record (EHR) data from [19]. The complete data includes Figure 2 provides all three algorithms’ DI measure, three years’ medical records for over 9,000 patients from specificity and sensitivity scores. Firstly, considering DI 44 diferent GP practices across NZ, collected using a plots, the fairness is measured based on ethnicity and stratified random sampling method to minimise data col- gender, where protected groups are Ma¯ori and female, lection bias. The data includes various categories of free- respectively. In both cases, when the disparate impact text and tabular data, and were manually processed, an- remover repair level is at ‘0’, the DI measures of XGBoost notated and verified by eight GP researchers (For further are within the red dotted lines, indicating the model fairdetails of the NZ-GP Harms data, see [19]). ness is within the acceptable range. However, the logistic
The small section of the above data, referred to as regression sensitivity score is the best, and the specificity NZ-GP-small data, includes patients from only Urban of all three algorithms is similar. locations and only the privileged ethical group ‘NZ Euro- Disparate impact remover [17] is a debias technique peans’ and the protected indigenous group Ma¯ori. This used at the preprocessing stage, where it is designed to results in a total of 3,768 patient data. DI for the NZ- edit feature values to increase group fairness while preGP-small data is 1.23 for Gender (the protected group is serving rank-ordering within groups. The repair level (a) DI measures for ethnicity with protected group Ma¯ori (b) DI measures for gender with protected group Female (c) Sensitivity scores (d) Specificity scores 4. Discussions and Future Work can be set from ‘0’, indicating no change, to 1, indicating maximum repair. In Figure 2a, the increase in repair level This research proposed a development stage AI-based varies the DI score, with values 0.4 to 0.6 being the best healthcare framework with a feedback loop to ensure a for all three algorithms. In Figure 2b, the increase in fair real-world practical solution. We argue the need to repair level marginally improves the DI values of logistic consider CQI by including feedback at each stage and regression and random forest, with repair values between working together as a team of experts. Recognising the 0.8 and 0.9 being the best. In contrast, for XGBoost val- importance of ongoing formal audits and reviews of efect ues, 0 to 0.7 is better for the DI scores (see Figure 2b). and eficacy in the implementation phase is vital. We use Unfortunately, the increase in repair levels decreases sen- simplified NZ patient data as a case study to demonstrate sitivity scores for all three classifiers and the specificity the framework. We provide early- stage experimental score of logistic regression. results using machine learning algorithms where it is evident in addition to the accuracy measures, there is 3.4. Feedback also a need to consider fairness measures and mitigation At each stage, there is a need for a feedback loop, where techniques. the decisions impact the final solution. At the data col- As indicated before, the NZ GP dataset is very complex lection and processing stage, clinicians need to discuss and includes multi-sourced data. Hence, there is a need the details of the approach with AI experts and ensure to extend the process and algorithm selection to enable that ethical and legal policies are considered. Data bias a fair, practical solution. Furthermore, we only consider and mitigation of bias, if any, must also be handled. At ethnicity and gender as binary cases. However, the NZ the algorithm selection stage, as demonstrated in Sec- population and data include other ethnic and minority tion 3.3, the accuracy measures cannot simply determine groups. While we believe the proposed framework allows the choice, but the algorithmic fairness measures must the flexibility to adopt the simple case presented in this be considered. The feedback loop can vary based on the paper to a more complex scenario, it is a much-needed specification of the problem at hand. future direction for this research.
[12] M. Maurer, R. Mangrum, T. Hilliard-Boone, VY is supported by the University of Auckland Faculty A. Amolegbe, K. L. Carman, L. Forsythe, R. Mosof Science Research Fellowship program. bacher, J. K. Lesch, K. Woodward, Understanding the influence and impact of stakeholder engagement in patient-centered outcomes research: a qualReferences itative study, Journal of General Internal Medicine 37 (2022) 6–13. [1] C. S. Webster, S. Taylor, C. Thomas, J. M. Weller, So- [13] M. L. Hudson, K. Russell, The Treaty of Waitangi cial bias, discrimination and inequity in healthcare: and research ethics in Aotearoa, Journal of Bioethimechanisms, implications and recommendations, cal Inquiry 6 (2009) 61–68.
BJA Education 22 (2022) 131–137. [14] L. Esmail, E. Moore, A. Rein, Evaluating patient [2] V. Yogarajan, Domain-specific language models for and stakeholder engagement in research: moving multi-label classification of medical text, Ph.D. the- from theory to practice, Journal of Comparative sis, The University of Waikato, 2022. Efectiveness Research 4 (2015) 133–145. [3] M. A. Ahmad, A. Patel, C. Eckert, V. Kumar, A. Tere- [15] S. Kalkman, J. van Delden, A. Banerjee, B. Tyl, desai, Fairness in machine learning for healthcare, M. Mostert, G. van Thiel, Patients’ and public views in: Proceedings of the 26th ACM SIGKDD Interna- and attitudes towards the sharing of health data for tional Conference on Knowledge Discovery & Data research: a narrative review of the empirical eviMining, 2020, pp. 3529–3530. dence, Journal of Medical Ethics 48 (2022) 3–13. [4] B. Giovanola, S. Tiribelli, Beyond bias and discrimi- [16] D. Pessach, E. Shmueli, A review on fairness in manation: redefining the AI ethics principle of fairness chine learning, ACM Computing Surveys (CSUR) in healthcare machine-learning algorithms, AI & 55 (2022) 1–44.
Society (2022) 1–15. [17] M. Feldman, S. A. Friedler, J. Moeller, C. Scheidegger, [5] S. S. Biswas, Role of Chat GPT in public health, S. Venkatasubramanian, Certifying and removing
Annals of Biomedical Engineering (2023) 1–2. disparate impact, in: proceedings of the 21th ACM [6] N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, SIGKDD International Conference on Knowledge A. Galstyan, A survey on bias and fairness in ma- Discovery and Data mining, 2015, pp. 259–268. chine learning, ACM Computing Surveys (CSUR) [18] P. Besse, E. del Barrio, P. Gordaliza, J.-M. Loubes, 54 (2021) 1–35. L. Risser, A survey of bias in machine learning [7] V. Yogarajan, G. Dobbie, S. Leitch, T. T. Keegan, through the prism of statistical parity, The AmeriJ. Bensemann, M. Witbrock, V. Asrani, D. Re- can Statistician 76 (2022) 188–198. ith, Data and Model Bias in Artificial In- [19] S. Leitch, S. Dovey, W. Cunningham, K. Waltelligence for Healthcare Applications in New lis, K. Eggleton, S. Lillis, A. McMenamin, Zealand, Frontiers in Computer Science 4 (2022). M. Williamson, D. Reith, A. Samaranayaka, et al., URL: https://www.frontiersin.org/articles/10.3389/ Epidemiology of healthcare harm in New Zealand fcomp.2022.1070493. doi:10.3389/fcomp.2022. general practice: a retrospective records review 1070493. study, BMJ open 11 (2021) e048316. [8] M. K. Lee, K. Rich, Who is included in human [20] S. Leitch, S. M. Dovey, W. K. Cunningham, A. J. perceptions of AI?: Trust and perceived fairness Smith, J. Zeng, D. M. Reith, K. A. Wallis, K. S. Egglearound healthcare AI and cultural mistrust, in: Pro- ton, A. W. McMenamin, M. I. Williamson, et al., ceedings of the 2021 CHI conference on human Medication-related harm in New Zealand general factors in computing systems, 2021, pp. 1–14. practice: a retrospective records review, British [9] E. Yoshida, S. Fei, K. Bavuso, C. Lagor, S. Maviglia, Journal of General Practice 71 (2021) e626–e633.
The value of monitoring clinical decision support interventions, Applied clinical informatics 9 (2018) 163–173. [10] J. Feng, R. V. Phillips, I. Malenica, A. Bishara, A. E.
Hubbard, L. A. Celi, R. Pirracchio, Clinical artificial intelligence quality improvement: towards continual monitoring and updating of AI algorithms in healthcare, NJP Digital Medicine 5 (2022) 66. [11] C. P. McLaughlin, A. D. Kaluzny, Continuous quality improvement in health care: theory, implementation, and applications, Jones & Bartlett Learning, 2004.