We formalize a notion of diferential robustness. An algorithm is diferentially robust to an event if the event has disparate impact on the performance of an algorithm for diferent groups in the population. We illustrate it with a case study of the real world deployment of a predictive algorithm in a European public employment service. diferential robustness, causal mechanism, disparate impact, distribution shift, concept drift, algorithmic An algorithm may be diferentially robust to likely distribution shifts as a result of variance in causal mechanisms. This happens when there are distinct causal mechanisms at work for diferent groups. Even if causal inference is used to model the relationship between features and label, the distinct causal mechanisms at work for diferent groups are not invariant [ 1] and may be more or less susceptible to changes in background conditions. Possible or likely changes in the world will then have disparate impact on the algorithm's performance, leading to less accurate interventions for some group(s). Thus, algorithmic fairness is not simply a criterion that obtains under certain distributional assumptions. One should be able to either anticipate possible and likely changes in the real world or evaluate their efects on the model post hoc in a timely manner. In our paper we develop a notion of diferential robustness with a case study of the real world deployment of a predictive algorithm in a European public employment service prior to and during the COVID-19 pandemic.
We formalize the notion of diferential robustness for an algorithm as follows: https://wan-charles.github.io/ (C. Wan); http://www.zejnilovic.com/ (L. Zejnilović) © 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). 1. Given training data set {( , )} ∼ with ∈ ℝ , ∈ {0, 1}, and group membership ∈ {1, 2} and hypothesis class ℋ, train a predictor ℎ ∶ ℝ ↦ {0, 1} that minimizes empirical risk. Individuals are assigned interventions 0 and 1 respectively based on model predictions. 2. Assume that there is an ordering of pairs of outcome and intervention with respect to welfare: ( = 0, 0) = ( = 1, 1) > ( = 0, 1) ≫ ( = 1, 0). That is, assigning intervention 0 to individuals who actually belong to the positive class generates the greatest harm. 3. Suppose ℎ ⟹ ′ for test time with {( ′, ′)} ∼ ′. If ( ′ = 1|ℎ( ′) = 0, = 2) − ( = 1|ℎ( ) = 0, = 2) > ( ′ = 1|ℎ( ′) = 0, = 1) − ( = 1|ℎ( ) = 0, = 1) , then ℎ is more robust to this particular distribution shift for group 1 than for group 2 with respect to welfare. In other words, the false negative rate for group 2 increases relative to group 1. ℎ 4. If the above holds for all likely shifts ∶ ⟹ ′, then ℎ is more robust to distribution shifts for group 1 than for group 2 with respect to welfare.
This can be illustrated with a toy example. Suppose there are two groups: { = 0, = 1} =1 and { = 2, = 3} =2 . Given background conditions the causal mechanism is as follows: = 0 or = 2 → = 0 ; = 1 or = 3 → = 1 . An intervention improves welfare if and only if = 1 . Suppose an algorithm ℎ is able to predict perfectly from and an intervention is ℎ assigned if ℎ() = 1 . Now imagine a change in background conditions ⟹ ′ that induces a ℎ change in the causal mechanism ⟹ ′, with ′ ∶ = 0 → = 0 ; = 1 , = 2 or = 3 → = 1 . The predictor ℎ is diferentially robust to such a change since under ℎ the false negative rate for group 2 increases relative to group 1.
We elucidate the notion of diferential robustness with a case study from the European Union.
The context is a public employment service where an XGBoost-trained model was deployed to
help counselors assess whether unemployed individuals are at risk of long-term unemployment
(LTU), defined as being unemployed for a year or longer. We ran a pilot study from October 2019
to June 2020 and subsequently collected data on the employment outcomes of the unemployed
individuals. During the period of the pilot study the COVID-19 pandemic hit and considerably
changed the causal structure of unemployment with service workers in the tourism industry
being among the most afected [
While it is arguable whether a shock such as the COVID-19 pandemic a priori represents a likely change, the case nonetheless demonstrates how events in the world, expected and unexpected, can change causal mechanisms and shift data distributions in such a way that the model’s performance is diferentially afected for diferent groups. Indeed, our analyses show that as a result of the COVID-19 pandemic the model’s false negative rate increases in particular for service workers in the tourism industry. This means that as a group they were more vulnerable to not being allocated the needed interventions conditional on having received a negative prediction from the model. The model — and the policy designed around its predictions — is diferentially robust to a shock such as the COVID-19 pandemic for diferent groups in the population.
service workers industry workers non-EU/EEA
Figure 1 shows the evolution of the false negative rates for service vs industry workers and non-EU/EEA vs Portuguese nationals before and during the first few months of the COVID-19 pandemic. We observe that the false negative rates for service and industry workers track each other fairly closely before the pandemic but the gap between them expands drastically in favor of industry workers after the onset of the pandemic in Portugal. The gap between the false negative rates for non-EU/EEA and Portuguese nationals is more volatile before the onset of the pandemic but settles into a stable pattern in favor of Portuguese nationals after the onset of the pandemic. (a) False negative rates for service vs industry workers (b) False negative rates for non-EU/EEA vs Portuguese nationals
( |) ( | , ) event the model’s performance for diferent groups
This leaves open the question of whether the observed diferences in the false negative rate
are due to covariate shift, where ( ) changes, or concept drift, where ( | ) changes [
FN = 0 + 1service_workers + 2covid + 3service_workers × covid + 4risk_score + + 4risk_score + FN = 0 + 1industry_workers + 2covid + 3industry_workers × covid
FN = 0 + 1non_EU_EEA + 2covid + 3non_EU_EEA × covid
+ 4risk_score + FN = 0 + 1Portuguese + 2covid + 3Portuguese × covid
+ 4risk_score +
The regression results in Tables 2 and 3 show that the coeficients for service_workers × covid and non_EU_EEA × covid are positive and statistically significant. We can interpret this as evidence that concept drift caused by COVID-19 led to an increase in the false negative rate for service workers (non-EU/EEA nationals) relative to other groups. Conversely, the coeficients for industry_workers × covid and Portuguese × covid are negative and statistically significant. Concept drift caused by COVID-19 led to a decrease in the false negative rate for industry workers (Portuguese nationals) relative to other groups. covid
Our work shows that to ensure algorithmic fairness it is not suficient for criteria of statistical fairness to obtain. If the model is especially or uniquely vulnerable to performance degradation for a particular group in possible scenarios, fairness will be elusive. Possible and likely changes in the world should be considered and their diferential impact on the algorithm’s performance for diferent groups carefully evaluated. This requires the judicious incorporation of domain knowledge into the decision-making process. Alternatively, there should be continual post hoc evaluation of an algorithm’s diferential robustness, for which the analysis in our paper could serve as a model of performance monitoring. covid risk_score Observations Log-likelihood Note: 1.355*** (0.028) 0.080 (0.042) 0.448*** (0.057)
(3)