Fair Semi-supervised Representation Learning for Tabular Data Classification

Abstract

Semi-supervised learning has shown its potential in many real-world applications where only few labeled examples are available. However, when some fairness constraints need to be satisfied, semisupervised classification models often struggle as they are required to cope with the lack of suficient information for predicting the target variable while forgetting its relationships with any sensitive and potentially discriminatory attribute. To address this issue, we propose a fair semi-supervised representation learning architecture that leads to fair and accurate classification results even in very challenging scenarios with few labeled (but biased) instances. We show experimentally that our model can be easily adopted in very general settings, as the learned representations may be employed to train any supervised classifier. Moreover, when applied to several real-world datasets, our method is competitive with state-of-the-art fair semi-supervised approaches.

Keywords

• eol>semi-supervised autoencoder
• fairness
• deep neural networks

1. Introduction

In an ideal scenario, modern supervised machine learning algorithms are able to get the most from all available training data instances so to accomplish the task at hand, be it classification, regression or ranking. Unfortunately, in real-world applications, this is almost never the case due to several reasons, among the others, the necessity to access huge amounts of labeled instances to train supervised algorithms. Labels often require cost-intensive collection procedures and huge eforts from human experts, especially in challenging domains such as medical and financial ones. Semi-supervised learning precisely addresses this issue by considering, together with a small amount of labeled information, unlabeled instances during the learning process, leveraging the so-called smoothness and cluster assumptions: if two data instances are close to each other or belong to the same cluster in the input distribution, then they are likely to belong to the same class [ 1, 2 ]. If the few available labels are of good quality, and clusters are well separated, unlabeled instances contribute to improve the accuracy significantly. Nonetheless, the labels might contain biases against certain groups. This might be an efect of historical explicit discriminations which may be reflected in a human expert’s beliefs, data scarcity or even biases in the data generation/measuring process itself [ 3 ]. Beyond ethical issues, fairness in machine learning models is becoming an increasingly pressing concern at a practical level as regulators and the general public become more aware of the potential for automatic discrimination.

If the lack of labeled training instances and fairness are complex problems individually, avoiding biases in a semi-supervised learning scenario is even more challenging. In a worst-case scenario, the few available labeled instances could be all or almost all associated to unfair sources, thus leading to very biased results or preventing any debiasing process. On the other hand, unlabeled instances do not carry any explicit bias and could be useful for driving the learning algorithm towards a fairer model. Despite its clear potential, fair semi-supervised learning has not been deeply investigated. The few existing approaches are based on preprocessing strategies that seek to extract fair training datasets by leveraging unlabeled instances [ 4, 5 ]. In short, these strategies train the model on a “fair subset” of the original data [ 4 ], although it is also possible to perform pseudo-labeling over the remaining data [ 5 ]. These techniques bear some resemblance to well-known preprocessing strategies in fully-supervised fair classification [ 6 ]. However, to the best of our knowledge, no representation learning method specifically designed for semi-supervised learning with fairness constraints has been proposed so far.

Representation learning allows one to automatically construct a new feature space that better captures the diferent factors of variation behind the data [ 7 ]. Such new representation can then be used to feed any machine learning algorithms, including supervised and unsupervised ones. Autoencoders are among the most popular representation learning methods and both fair [ 8 ] and semi-supervised [ 9 ] versions of them have been proposed. Louizos et al.’s Variational Fair Autoencoder (VFAE) [ 10 ] could be employed in semi-supervised settings, in principle, however, it has only been tested in the fully supervised ones.

This paper is an extended abstract of [ 11 ], where we propose a fair semi-supervised autoencoder that leads to fair and accurate classification results even in very challenging scenarios with few labeled (but biased) instances. The classic auto-encoding architecture [ 12 ] is enhanced with two components. One is trained to classify instances and employs the available labeled training instances. The second is a debiasing component that removes as much information as possible about the sensitive attribute, in an adversarial fashion. Additionally, our model is inductive and, as such, it can be used to classify unseen examples as well. We name our contribution FairSwiRL, which stands for Fair Semi-supervised classification with Representation Learning.

Through an extensive experimental validation on synthetic and real world datasets, we show that the representations learned by FairSwiRL as the training data for diferent classifiers leads to reasonably accurate models while respecting the fairness constraint. Moreover, our method compares favorably to other state-of-the-art fair semi-supervised classification approaches.

2. Problem Setting

In this section, we describe the problem of semi-supervised fair classification. In this scenario, one seeks to learn a classifier by using both labeled instances and unlabeled ones. Moreover, we would also like to satisfy a fairness constraint with respect to a given sensitive attribute, i.e. a feature representing an individual’s membership in an historically underprivileged group. The rationale here is to avoid potentially discriminatory decisions by the learned classifier [ 3 ].

We denote with (X, s, y) the features, the sensitive attributes, and the target variables of labeled instances, with (X, s) the features and the sensitive attributes of unlabeled instances. In semi-supervised fair classification, we seek to learn a classifier which is able to leverage both (X, s, y) and (X, s) such that the predictions of target variable y computed on an unseen test set (X, s) are accurate and satisfy some fairness constraints. In the following, capital non-bold letters will be used to denote random variables (e.g., , , will denote the stochastic variables associated with examples, labels and sensitive attributes).

As a fairness constraint, we here consider independence, or statistical parity (SP [ 13, 3 ]). Thus, we require that the probability of assigning a positive outcome to an individual is independent of the sensitive information . Formally, we require that (ˆ = 1 | = 0) = (ˆ = 1 | = 1), where ˆ is the stochastic variable associated with the prediction of the model. As a way to quantify how far we are from the statistical parity, we consider the statistical absolute diference (SAD) measure, [ 14 ]:

SAD = ⃒⃒⃒ E[ˆ | = 0] − E[ˆ | = 1]⃒⃒⃒ . (1) The lower the SAD, the better it is, with statistical parity at SAD = 0. We note here that removing the sensitive attributes s and s is usually insuficient to achieve statistical parity as some information about may be present in the remaining variables X and X or the labels y. Thus, FairSwiRL seeks to optimize the SAD metric by learning a debiased representation of the original data - i.e. a new representation of the data in which all information about has been removed. After the debiasing, any classifier trained on the latent representation will be able to achieve low SAD values without being specifically optimized for this metric. Our proposal is a semi-supervised representation learning method which is able to leverage the unlabeled examples and obtain a less biased representation of the data. We describe our contribution in detail in the next section.

3. Fair Semi-supervised Representations for Classification

In our problem setting, label scarcity is paired with fairness constraints. To face these issues, we design an inductive and fair semi-supervised model which leverages representation learning techniques. We employ an auto-encoding architecture [ 12 ] which is able to leverage both labeled X and unlabeled data X. This architecture maps the original data X = {X ∪ X} into a compact representation z via a series of fully-connected layers, a process which is commonly referred to as encoding. In the following we will refer to this section of our model as the encoder (x), where are the learnable parameters for the fully connected layers, and the learned latent representation as z. The dimension of this representation is a hyperparameter for the algorithm and may be set up to be lower than x, therefore compressing information. Another series of fully connected layers, a decoder (z), then maps back the latent representation into an approximation xˆ of the original data. This architecture may be learned via gradient descent over a reconstruction loss ℒrec which is defined as follows:

∑︁ x∈(X∪X) ℒrec( , ) = ‖x − ( (x))‖2.

In the semi-supervised setting there is also the additional opportunity to exploit the limited amount of class information provided by the labeled examples x ∈ X. Exploiting this is paramount to obtain representations that are also useful for classification. Therefore, we employ an auxiliary network (z) and train it on the representations z = (x) for which label data are available. As is commonly done in classification with neural networks, we exploit the cross entropy loss to drive the training of this component of the network: ⎞ ⎞ ℒcla( , ) = , · ( ( (x))) ⎠ , where the notation , assumes the one-hot encoding of the class for the labeled example x ∈ X, and is the set of possible labels (numbered from 1 to ||). Lastly, we employ a component which is able to remove information about the sensitive attribute s from the obtained representations z. This is possible by training another auxiliary classifier which predicts the sensitive attribute from the representation, which we will refer to in the following as . Once again, this may be trained via cross-entropy, albeit over both labeled and unlabeled examples, as we assume that sensitive information is available for all data samples: ∑︁ x∈X ⎛ ⎝− || ∑︁ =1 ∑︁ x∈X∪X ⎛ ⎝− || ∑︁ =1 ℒfair( , ) = , · ( ( (x))) ⎠ , sensible values. Formally, the overall training objective for our method is as follows: where , is the -th component of the one-hot-encoded s vector and is the set of possible ℒtot( , , , ) = claℒcla( , ) + recℒrec( , ) − fairℒfair( , ), where fair, cla, rec are hyperparameters which may be picked to control the fairness/classification/reconstruction trade-of. The networks are pitted against one another in an adversarial fashion. This implies setting up a min-max game where networks , and are employed to respectively minimize the reconstruction and classification losses; the network the other hand, should have maximal loss, i.e., it should be impossible to reconstruct information about the sensitive attribute from the learned representations . This leads to the following , on multi-objective optimization problem: ˆ, ˆ, ˆ, ˆ = arg ︂{

min , , ︂[ claℒcla( , ) + recℒrec( , ) − fair min ℒfair( , ) . ︂]} The equilibrium point in the above problem can be found via gradient reversal [ 15 ], a procedure where the gradient information from a sub-network is multiplied by − 1 when backpropagating into the main architecture. Specifically, we invert the gradient from when updating the parameters in our encoder .

In summary, the proposed network (FairSwiRL) is a fairness focused extension of the semisupervised autoencoder. One core property of FairSwiRL is that it leverages representation learning to obtain feature vectors which are both useful and fair. The obtained representations may then be used for further downstream tasks with no restriction on the employed model, allowing a practitioner to use the model that best fits the domain knowledge on the task or any business requirements. We show the flexibility of our approach in Section 4.1, where we report experimental results for diferent classifiers trained on FairSwiRL’s representations.

4. Experiments

Our experiments are aimed at evaluating the representations learned by FairSwiRL. To this purpose, we experiment on four real-world datasets that have been extensively employed for fair classification and fair representation learning [ 8, 10 ] (see Table 1 for summary statistics).

4.1. FairSwiRL in combination with diferent supervised classifiers

Here, we compare diferent classifiers in combination with FairSwiRL, namely: random forest (FairSwiRL+RF), k-nearest neighbors (FairSwiRL+KNN), logistic regression (FairSwiRL+LR), support vector machines (FairSwiRL+SVC) and neural network (FairSwiRL+NN).

We now define the data splits and the evaluation metric we will employ in this section and in the rest of the paper. Let , , , and be the number of labeled, unlabeled, validation and test examples. We start with the following configuration: = 100, = 10000, = 100, = 10000 (in case of the COMPAS dataset = 100, = 1900, = 100, = 1900). We use the validation examples to find a good configuration of the hyperparameters and then, by using the same hyperparameters, we increase the number of labeled instances from 100 to 2000. For each combination of (, , , ) we repeat the experiments ten times by sampling diferent datasets from the original data, and compute the average performance metrics. We stress that the number of available examples for a given experimental run is computed in absolute terms, not relative. This lets us compare the performance of the methodologies across the same number of test examples, no matter how many labeled examples are available. To measure the fairness level we employ 1-SAD (see Equation 1) while for the predictive performance we compute the Matthews Correlation Coeficient (MCC [ 16, 17 ]).

0.5 Y f oC0.4 C M 0.3 1 D0.95 A S 1

0.9 0 1,000 2,000 0 1,000 2,000

In Figure 1, we report only the results for ADULT , but the behaviors of diferent combinations of FairSwiRL+classifier are similar across the datasets. We note that the trends of the diferent classifiers are the same in predicting the target variable and in being fair. These results show that the latent representations induced by FairSwiRL can be used by diferent classifiers and, as the number of labeled examples increases, the performances on the target variable tend to increase. While the 1-SAD value (higher is better) slightly sufers from the bias introduced by the additional examples, we note that it remains very close to optimal values (> 0.9) nonetheless.

In the next section, we will compare FairSwiRL with competing approaches. In order to enable a fair comparison, we choose the worst combination (FairSwiRL+RF, according to the previous experiment) and keep it fixed in all the experiments presented in this work.

4.2. FairSwiRL +RF compared to competitors

In this experiment, we test the efectiveness of FairSwiRL on diferent datasets and against diferent competitors. The experiment setting is the same as in Section 4.1, but we choose only the worst performing combination (FairSwiRL+RF) as our candidate combination. In addition to FairSwiRL+RF, we include the following competitors: FESF, an implementation of FairnessEnhanced Sampling Framework [ 4 ]; FairSSL, an implementation of the algorithm presented by Chakraborty et al. [ 5 ] with Label Spreading [ 18 ] as the pseudo-labeling algorithm; VFAE an implementation of the Variational Fair Autoencoder [ 10 ] used to get the latent representation on which a random forest is then trained for the classification task, as in FairSwiRL+RF.

The results are reported in Figure 2. The plots report on the -axis the performance metric (MCC) and on the -axis the fairness metric (1-SAD). We vary the number of labeled examples and run the experiments ten times for each configuration. Each point in the plot represents one experiment, shapes vary according to the algorithm used and colors vary according to the number of labeled examples in the dataset. The best possible point in each plot is at coordinates (1,1), but this is usually unattainable. The gray dashed line has slope -1 and, as such, points on that line have the same trade-of between accuracy and fairness. The lines showed in each plot pass through the point closest to (1,1) under the 1 metric. These points are, thus, the 0.95 0.9 best performers under the assumption that fairness and accuracy are equally important. We can see that, with the exception of the plot concerning the COMPAS dataset, the points (⋆) representing FairSwiRL+RF are always in the upper half of the plots. Higher values of 1-SAD mean that the debiasing component of FairSwiRL is working as expected.

The comparisons with FairSSL ( ), FESF ( ) and VFAE (▶) are also favorable. Except for the CARD dataset, FairSwiRL lies on the optimal tradeof line. In CARD, where the best results are attained by FESF, FairSwiRL has a better fairness, but the lower MCC leads the FESF model to prevail in terms of the linear trade-of we are assuming here. This is a typical case of accuracyfairness dilemma: higher 1-SAD implies also lower predictive power when the sensitive attribute and target variable are correlated. On the COMPAS dataset we have a mixed situation, while the best points are attained by FairSwiRL, we can see that for some experiments (specifically, those with fewer labeled examples) it attains worse performances than the competitors. Overall, we would not judge this experiment as a clear win for FairSwiRL, but we still maintain that it is a competitive approach also in this case.

As far as more general trends are concerned, we observe that more labeled instances (warmer colors in Figure 2) lead all methodologies to more accurate, but less fair results. This result, in our view, justifies further future employment of semi-supervised techniques in fair classification: a small amount of labeled data does not impact fairness negatively.

Beyond the linear tradeof discussed above, we also experiment in an hypothetical context in which fairness is paramount and performance may be pursued only when fairness is already guaranteed. To model this situation, we repeated the experiments recording the discounted MCC metric: DisMCC = MCC · − SAD, where MCC is the MCC computed on the target variable. It is worth noting that, in this metric, the fairness performances, as measured by the SAD statistic, are weighted exponentially. Figure 3 plots the average rankings of the competing 0.06 C0.04 C sM0.02 i D0.00

BANK 1,000 BANK 0.04 C C isM0.02 D 0.00

COMPAS 1,000 COMPAS 2 k n a R3 4 0.15 CC0.10 M s iD0.05 0.00 1,000 CARD 1,000

ADULT 0 2,000 0 2,000 0 2,000 0 2,000 approaches for increasing number of labeled examples. Rankings are evaluated according to the value of DisMCC with = 30. We note that lower rankings, which are better, are displayed higher in the picture. The actual values of DisMCC obtained in the corresponding experiment are displayed in the right column (higher values are better). In SYNTHETIC the PD+RF method dominates, as expected, because it represents the theoretical upper-bound, unreachable in a real setting since the data generation process is usually unknown. However, the second best candidate is FairSwiRL+RF. In CARD FairSwiRL+RF reaches the best performance only sometimes but if compared to VFAE and FairSSL it has a more stable trajectory when the number of labeled instances changes. FairSwiRL is overall the strongest performer on both ADULT and BANK. In COMPAS we observe worse performances than the competitors, while the other fair representation learning strategy we tested (VFAE) is the strongest performer. Overall, even in a context where the fairness is exponentially weighted, FairSwiRL+RF performs well on average.

5. Conclusion

We have proposed a neural network for representation learning that addresses two challenging issues simultaneously: the lack of suficient labeled examples in the training data, and the presence of sensitive attributes potentially leading to unfair decisions. We have shown that unlabeled examples help the learning algorithm to cope with both problems, leading to fair and accurate semi-supervised classification of unseen examples. The experiments have shown the efectiveness of our approach, even in comparison with state-of-the-art fair semi-supervised methods which employ preprocessing strategies. We have also performed a full comparison with another fair representation learning strategy (VFAE) [ 10 ] which had so far never been tested in the SSL setting. Our experiments show that fair representation learning approaches are able to outperform feature preprocessing strategies in the semi-supervised setting and such a result transfers across diferent tradeofs for fairness vs. accuracy.

References

[1] O. Chapelle , B. Schölkopf , A. Zien , Introduction to semi-supervised learning , in: SemiSupervised Learning , The MIT Press, 2006 , pp. 1 - 12 .

[2] J. E. van Engelen , H. H. Hoos , A survey on semi-supervised learning , Mach. Learn . 109 ( 2020 ) 373 - 440 .

[3] S. Barocas , M. Hardt , A. Narayanan , Fairness and machine learning ., URL: http://www.fairmlbook.org ( 2019 ).

[4] T. Zhang , T. Zhu, J. Li , M. Han , W. Zhou , P. S. Yu , Fairness in Semi-Supervised Learning: Unlabeled Data Help to Reduce Discrimination , IEEE Trans. Knowl. Data Eng . 34 ( 2022 ) 1763 - 1774 .

[5] J. Chakraborty , H. Tu , S. Majumder , T. Menzies, Can we achieve fairness using semisupervised learning? , CoRR abs/2111 . 02038 ( 2021 ).

[6] F. Kamiran , T. Calders, Classifying without discriminating , in: Proceesings of IEEE-IC4 2009 , 2009 .

[7] Y. Bengio , A. C. Courville , P. Vincent , Representation learning: A review and new perspectives , IEEE Trans. Pattern Anal. Mach. Intell . 35 ( 2013 ) 1798 - 1828 .

[8] D. Madras , E. Creager, T. Pitassi , R. S. Zemel , Learning adversarially fair and transferable representations , in: Proceedings of ICML 2018 , 2018 , pp. 3381 - 3390 .

[9] A. Gogna , A. Majumdar , Semi supervised autoencoder , in: Proceedings of ICONIP 2016 , 2016 , pp. 82 - 89 .

[10] C. Louizos , K. Swersky , Y. Li , M. Welling , R. S. Zemel , The variational fair autoencoder , in: Proceedings of ICLR 2016 , 2016 .

[11] S. Yang , M. Cerrato , D. Ienco , R. G. Pensa , R. Esposito, FairSwiRL : Fair Semi-supervised Classification with Representation Learning, Mach . Learn. ( 2023 ) 1 - 26 . URL: https://doi. org/10.1007/s10994-023-06342-9. doi: 10 .1007/s10994-023-06342-9.

[12] G. E. Hinton , R. S. Zemel, Autoencoders, minimum description length and helmholtz free energy , in: Proceesings of NIPS 1993 , Morgan Kaufmann, 1993 , pp. 3 - 10 .

[13] A. Castelnovo , R. Crupi , G. Greco , D. Regoli , The zoo of fairness metrics in machine learning , CoRR abs/2106 .00467 ( 2021 ).

[14] R. K. E. Bellamy , K. Dey , M. Hind , S. C. Hofman , S. Houde , K. Kannan , P. Lohia , J. Martino , S. Mehta , A. Mojsilovic , S. Nagar , K. N. Ramamurthy , J. T. Richards , D. Saha , P. Sattigeri , M. Singh , K. R. Varshney , Y. Zhang, AI fairness 360 : An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias , CoRR abs/ 1810 . 01943 ( 2018 ).

[15] Y. Ganin , E. Ustinova, H. Ajakan , P. Germain , H. Larochelle , F. Laviolette , M. Marchand , V. S. Lempitsky , Domain-adversarial training of neural networks , J. Mach. Learn. Res . 17 ( 2016 ) 59 : 1 - 59 : 35 .

[16] P. Baldi , S. Brunak , Y. Chauvin , C. A. F. Andersen , H. Nielsen , Assessing the accuracy of prediction algorithms for classification: an overview , Bioinform . 16 ( 2000 ) 412 - 424 .

[17] D. Chicco , G. Jurman, The advantages of the matthews correlation coeficient (mcc) over f1 score and accuracy in binary classification evaluation , BMC genomics 21 ( 2020 ) 1 - 13 .

[18] D. Zhou , O. Bousquet , T. N. Lal , J. Weston , B. Schölkopf , Learning with local and global consistency , in: Proceesings of NIPS 2003 , MIT Press, 2003 , pp. 321 - 328 .