Mechanisms used in privacy-preserving machine learning often aim to guarantee diferential privacy (DP) during model training. Practical DP-ensuring training methods use randomization when fitting model parameters to privacy-sensitive data (e.g., adding Gaussian noise to clipped gradients). We demonstrate that such randomization incurs predictive multiplicity: for a given input example, the output predicted by equally-private models depends on the randomness used in training. Thus, for a given input, the predicted output can vary drastically if a model is re-trained, even if the same training dataset is used. The predictive-multiplicity cost of DP training has not been studied, and is currently neither audited for nor communicated to model designers and stakeholders. We derive a bound on the number of re-trainings required to estimate predictive multiplicity reliably. We analyze-both theoretically and through extensive experiments-the predictive-multiplicity cost of three DP-ensuring algorithms: output perturbation, objective perturbation, and DP-SGD. We demonstrate that the degree of predictive multiplicity rises as the level of privacy increases, and is unevenly distributed across individuals and demographic groups in the data. Because randomness used to ensure DP during training explains predictions for some examples, our results highlight a fundamental challenge to the justifiability of decisions supported by diferentially private models in high-stakes settings. We conclude that practitioners should audit the predictive multiplicity of their DP-ensuring algorithms before deploying them in applications of individual-level consequence.
2.5 0.0
2.5
settings [
Predictive multiplicity formalizes the arbitrariness of decisions based on a model’s output.
In practice, predictive multiplicity can lead to questions such as “Why has a model issued
a negative decision on an individual’s loan application if other models with indistinguishable
accuracy would have issued a positive decision? ” or “Why has a model suggested a high dose of a
medicine for an individual if other models with comparable accuracy would have prescribed a lower
dose? ” These examples highlight that acting on predictions of a single model without regard for
predictive multiplicity can result in arbitrary decisions. Models produced by training algorithms
that exhibit high predictive multiplicity face fundamental challenges to their credibility and
justifiability in high-stakes settings [
In this paper, we demonstrate a fundamental connection between privacy and predictive multiplicity: For a fixed training dataset and model class, DP training results in models that ensure the same degree of privacy and achieve comparable accuracy, yet assign conflicting outputs to individual inputs. DP training produces conflicting models even when non-private training results in a single optimal model. Thus, in addition to decreased accuracy, DP-ensuring training methods also incur an arbitrariness cost by exacerbating predictive multiplicity. We show that the degree of predictive multiplicity varies wildly across individual inputs and can disproportionately impact certain population groups. Fig. 1 illustrates the predictive-multiplicity cost of DP training in a simple synthetic example.
In summary, the level of privacy in DP training significantly impacts the level of predictive multiplicity. This, in turn, means that decisions supported by diferentially-private models can have an increased level of arbitrariness: a given decision would have been diferent had we used a diferent random seed in training, even when all other aspects of training are kept fixed and the optimal non-private model is unique. Before deploying DP-ensuring models in high-stakes situations, we suggest that practitioners quantify the predictive multiplicity of these models over salient populations and—if possible to do so without violating privacy—measure predictive multiplicity of individual decisions during model operation. Such audits can help practitioners evaluate whether the increase in privacy threatens the justifiability of decisions, choose whether to enact a decision based on a model’s output, and determine whether to deploy a model in the ifrst place.