We study human reliance on (un)fair algorithmic recommendations. We train two separate models to predict the employment status of (anonymized) persons in the US-Census 2018. The two models difer in their fairness, with their overall accuracy being the same for (difering by) binary 1 gender ("overall accuracy equality") [2]. We use the predictions by the two models in a pilot experiment, in which participants performed the same prediction task while receiving assistance from either the fair or unfair model. We find that people rely more on the predictions by the fair algorithm with unknown gender, yet stronger on unfair models once the gender is revealed. The present data remains limited in size, but we aim to identify these efects and potential sources of individual treatment heterogeneity using a much larger experimental sample in the future.
1We fully acknowledge that this dichotomy is far from ideal (see also the discussions in Pinney et al. [
EWAF’24: European Workshop on Algorithmic Fairness, July 01–03, 2024, Mainz, Germany * Corresponding author. insignificant) evidence that prior beliefs, i.e., the general and potentially biased expectation about the employment status of men and women are a potential source of treatment heterogeneity.
The main goal is to deploy two models with difering fairness levels, measured by accuracy, to ofer predictions to participants. Both models are trained on the same 2018 U.S. Census data, predicting employment status using variables like age, education, disability, marital status, and other sociodemographic factors. We exclude the sensitive attribute, gender, from the model’s input. We use BinaryMI [? ] for both models, which aims to make a fair prediction by learning a representation of the data that is invariant to the sensitive attribute, by using a stochastically quantized binary neural layer. All parameters for training are set to the same values for both models, except for , which is the tradeof parameter in the loss function weighting the importance of representation invariance and accuracy on the target variable, i.e., the employment status. For the fair model, we set to 0.05, whereas for the unfair one, we set it to 0. Both models are trained for 50 epochs. Before model training, the dataset is subsampled to retain only 10% of the instances of employed women, reducing their representation in the learning process. While the fair model reaches an accuracy of 79.37% for male and 79.18% for female, the unfair version reaches an accuracy of 81.98% for male, but only 79.24% for female. In the experiment, we select 10 random profiles from the validation dataset, ensuring a balance of gender and employment status, and representing two profiles each from the prediction quintiles of the fair, unfair models, and their diference.
The experiment, centering on a prediction task and supplemented by auxiliary measures, was
conducted on Prolific (www.prolific.com) in December 2023 with 103 participants. It lasted an
average of 27.23 minutes, with 77% of participants receiving a 2-pound bonus. Main Task: For
participants in the online experiment, the main task is to give an accurate probabilistic statement
if a given (anonymized) person is employed or not based on shown personal characteristics.
The statement ranges from 0 to 100 and the closer the statement is to the true employment
status (either 0 or 1 (=100)), the higher the chances that a participant obtains a bonus prize. We
apply the randomized quadratic scoring rule [
Additional tasks and measures: We carry
out several additional tasks and surveys. We gather data on prior beliefs about the employment
status of men and women, a survey with attitudes towards the statistical model by Zhou et al.
[
Main Treatment Diferences: Panel (d) in Figure 1 plots the average WOA for the two groups. On average, the WOA is 0.48 in the unfair group and 0.52 in the fair group. This diference is statistically significant on the 10% level ( = 0.061) in a two-sided t-test. Participants showed a stronger reaction to predictions from the fair model than those from the unfair model. Despite the small diference, it’s significant considering the low "treatment intensity" between the models, making a 4 percentage point diference noteworthy. Contrary, the mean WOA values between phases 3 and 2 (i.e., once the gender of the candidate has been revealed) show a lower reliance on the predictions by the fair model (panel (d)). On average, the WOA on the fair model is 0.41, yet 0.45 on the unfair model ( = 0.052). This suggests that participants react more strongly to fair predictions if the sensitive attribute remains undisclosed but more strongly to unfair predictions once the sensitive attribute is known. Regressions and Heterogeneous Treatment Efects: The Table in panel (e) in Figure 1 regresses the (final) revised belief statement on the initial (revised) statement, the model predictions from the main task between phase 2 and 1 (3 and 2) and a dummy for the treatment group. Columns 1 and 3 include data from phases 2-1, whereas column 2 and 4 from phases 3-2. Focussing on the first two columns, we observe a stronger reliance on the participant’s own initial (revised) estimates compared to the model predictions, while this asymmetry is much more pronounced between phases 3 and 2. In line with the evidence from panel (d), the interaction efect of the fair treatment with the model prediction has a positive (negative) coeficient, implying that participants in the fair treatment rely more (less) strongly on the model predictions during phases 2-1 (3-2). The interaction efects were not statistically significant, likely due to the low power of the pilot experiment. The analysis includes a treatment heterogeneity segment, examining an interaction between fair treatment, model predictions, and "Belief Delta" – a measure of the diference in participants guesses regarding the employment rates of men versus women. The coeficient of this interaction is positive (negative) for phases 2-1 (3-2), implying that the more "biased" a person the stronger (weaker) their reliance on the fair predictions. Again, these interactions are, however statistically insignificant.
The key finding of this study is that participants in our pilot experiment reacted more strongly to the predictions by a fair model compared to an unfair one in the presence of unknown sensitive attributes, yet less once the sensitive attribute was revealed. This novel insight indicates that the efectiveness of fair models depends on the decision-makers using them. We recognize the limitations of our pilot study, notably its low power, leading to provisional conclusions. Methodologically, we didn’t consider the statistically optimal posterior belief (assuming Bayesian participants), potentially explaining some treatment diferences. We plan to overcome these limitations with a larger online experiment and by theoretically modeling task behavior.