Many-Objective Feature Selection (MOFS) approaches use four or more objectives to determine the relevance of a subset of features in a supervised learning task. As a consequence, MOFS typically returns a large set of non-dominated solutions, which have to be assessed by the data scientist in order to proceed with the final choice. Given the multi-variate nature of the assessment, which may include objectives (e.g., fairness) unrelated to predictive accuracy, this step is often not straightforward and sufers from the lack of existing tools. For instance, it is common to make use of a tabular presentation of the solutions, which provides little information about the trade-ofs and the relationships between objectives over the set of solutions.
CEUR ceur-ws.org
of huge size and dimensionality. Efective Feature Selection (FS) is then more and more important in order to extract valuable information by identifying relevant features and discarding irrelevant or redundant ones.
number of possible subsets is exponential (exactly 2 − 2, niques to find one or more subsets of features (in the following, also called solutions) containing the most relevant features based on certain objectives.
The set of objectives is highly dependent on the task and goal of the data analysis. For example, if we aim at predicting hospital readmission of patients with diabetes [1], the objectives could include generalisation-related measures (e.g., size of the subset, predictive performance, redundancy) as well as fairness. The inclusion of fairness objectives is more and more required in the application of artificial intelligence (AI) to domains involving conifdential information and having a potential impact on citizens’ rights and welfare [2] (e.g., healthcare).
Moving from one to several objectives requires adapting the FS strategy to account for the trade-ofs among the considered objectives. In particular, many-objective FS methods return a set of non-dominated solutions, among which the data scientist is expected to select the most convenient one. In the case of two objectives, the set of solutions can be illustrated with 2D line plots [3], scatter plots [4], bar charts [5], or simply tables [6]. With three tions on a single graph, requiring 3-D scatter plots [7], multiple charts [8] or tables [9]. When we move to four or more objectives (MOFS), it is even harder for the data analyst to extract some useful insights.
based on a large number of descriptive tables [10] or rely on a single measure [11, 10, 12] hiding the multi-variate nature of the objective space to present the solutions.
module (Figure 1) to enable efective exploration of the
considering two classification tasks with six objectives (i.e., the subset size, balanced accuracy, F1-score, Variance
odds). We use a Genetic Algorithm (GA), namely NSGA
DOLAP 2024: 26th International Workshop on Design, Optimization, sical data science pipeline with a MOFS interpretation © 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License tions with Naive Bayes (NB) and Logistic Regression (LR) CEUR htp:/ceur-ws.org ISN1613-073
CEUR
Workshop Proceedings (CEUR-WS.org)
as classifiers.
User
Data sources
Data preprocessing
Result presentation
Modeling
Evaluation Data Flow User Interaction Control Flow
Results
Many-objective optimisations are problems that require We apply the MOFS method to two well-known fairness the simultaneous optimisation of four or more objectives. datasets. The MOFS simultaneously optimises six objecThe goal is to find diverse solutions that lie close to the tives: subset size, balanced accuracy, F1-score, VIF, statistrue Pareto font [15]. On the other hand, FS aims to find tical parity, and equalised odds. We use NSGA-III as the a subset of features from a dataset that is most relevant search method to find a set of non-dominated solutions. to the task at hand; a key part of this is defining their This section presents NSGA-III, the MOFS implementarelevance, which cannot be fully described by a single tion, datasets used in this work, and the execution details objective [16]. Depending on the task, it requires consid- for reproducibility. ering up to four or more objectives, termed MOFS.
MOFS has been used to optimize multiple objectives in 3.1. NSGA-III various applications, such as network anomaly detection [10], motor imagery prediction [17], multi-label classifica- NSGA-III is a pareto and reference-based elitist GA for tion [18], and large-scale feature selection [19]. However, many-objective problems [14]. It uses well-spread-out there is still a need for research on how to choose a final reference points to maintain population (i.e., a set of solution from the large set of non-dominated solutions. candidate solutions) diversity. Also, it is elitist because
The mechanisms used to present and choose a final it is designed to preserve the set of best individuals at solution in previous works include box plots [10], bar each iteration. Thus, NSGA-III begins by initialising a charts [17], scatter plots [18], Parallel Coordinate Plots population and optimises towards better solutions until (PCP) [19], and tables [10, 17, 18]. Additionally, measures a termination criterion is satisfied. such as the hypervolume indicator [17] and inverse generation distance — which is the distance from the ideal 3.2. MOFS implementation solution vector [10], are also used. However, all these ignore one or more viewpoints (perspectives), especially We describe the starting point (i.e., initialisation of the the feature viewpoint, unique to the FS problem. MOFS ifrst population), search strategy, feature subset evaluarequires collective (solutions) and individual (features) tion (i.e., the objectives), and the termination criterion evaluation of the resulting set of solutions to find the for the MOFS used in this work. most appropriate one.
In many-objective optimisation, providing support for 3.2.1. Starting Point the data scientist remains challenging because the com- We begin with a population of one-sized candidate solumonly used mechanisms for presenting the solutions tions, where each feature is selected at least once. The become inadequate with the growing number of dimen- population size is set as an even number greater than sions. To remedy this, the dimensions can be reduced, [o2r0m].uYlteitp,ltehperfeosremntearticoonms ecsanwbitehuaselodsfsoorfcoinmfoprlmetaentieosns, tohded.oOvetrhaelrlwniusme, b=er o+f2feat.ures . So, = + 1 if is and the latter requires clear guidelines.
It is, therefore, imperative to design a methodology 3.2.2. Search Strategy and Subsets Evaluation that facilitates the interpretability of MOFS results holistically, taking into account all three viewpoints that make up the process: objectives, solutions, and features.
strategy for MOFS. In particular, we set the mutation probability to 1/ and the crossover probability to 1.
Knowing that one measure of relevance for a feature subset is insuficient, we evaluate the relevance of a subset of features by up to six objectives: 1. The subset size is a fundamental objective of FS, and the goal is to minimise it (↓). 2. Balanced accuracy measures accurate predictions. It is a commonly used predictive performance metric in classification problems, and we aim to maximise it (↑). 3. F1-Score, which is the harmonic mean of precision and recall [16], is another predictive performance measure which we maximise (↑). 4. Variance Inflation Factor (VIF) measures multicollinearity between features [21]. A high VIF implies redundancy, and we want to minimise it (↓). 5. Statistical parity checks if classifier predictions are void of group membership sentiments [2], and we aim to maximise it (↑). 6. Equalised odds measures if the classifier performs equally well across sensitive groups [2], and we aim to maximise it (↑).
3.2.3. Termination Condition
maximum number of evaluations, which is set up to 2 2. When this condition is satisfied, the non-dominated [ 14] solutions in the current population are returned.
For this dataset, we try to predict hospital readmission of patients with Diabetes using the NB classifier, and the sensitive feature on which we strive for fairness is gender. The MOFS produced 52 non-dominated solutions, and the ranges of values for the objectives are as follows: subset size [7, 19], balanced accuracy [0.5049, 0.5422], F1score [0.0456, 0.1790], VIF [0.0093, ∞], statistical parity [0.8432, 0.9246], and equalised odds [0.7130, 0.8403].
With 52 solutions for the Diabetes dataset, this becomes overwhelming to depict with a table. Also, with greater subset sizes, it becomes impossible to fully display all features that have been selected for each solution. Using a table, in this case, gives little support to the data scientist in choosing a final solution.
Table 1 the Parallel Coordinate Plot (PCP) shown in Figure 2. For Dataset properties. each solution, PCP shows the values for each objective,
Name #Features #Instances #Classes Sensitive Pop. Size Max. Evals. and it is typically interactive (i.e., a range of values can be GerDmiaabnetceresdit 5271 1011,700660 22 GenSdeexr 5282 6796288 chosen for each objective, and the solutions that satisfy the configuration will be filtered). With this, the data scientist can find solutions that satisfy the given criteria.
However, it is unlikely that only one solution satisfies 3.4. Execution this configuration. Whether this returns many or a few solutions, there is still a need to compare alternatives and We implemented the experiments using Python 3 and par- with more alternatives, the complexity increases. Also, ticularly, the jMetalPy3 library implementation of NSGA- the PCP does not show the content of the solutions (i.e., III [22]. The source code for our experiments is available the features that have been chosen). Finally, as shown on GitHub.4 in Figure 2, with more solutions, the PCP becomes overcluttered.
The task for this dataset is to predict the credit risk MOFS produces a potentially large set of solutions with of bank account holders using an LR classifier, having completely diferent values for the considered objectives. famges (Sex) as a sensitive feature. MOFS produced Thus, we have shown the need for a methodology to sys19 non-dominated solutions where the range of val- tematically analyse the results obtained by MOFS through ues for each objective is: subset size [1, 10], balanced two concrete fairness benchmark datasets. The results accuracy [0.5000, 0.6607], F1-score [0.7776, 0.8305], VIF of our research framed under the studies performed in [0, 16.668], statistical parity [0.7488, 1], and equalised the project DEDS (MSCA-ITN G.A. No. 955895) allows odds [0.2122, 0.8000]. Most commonly, MOFS results are us to conclude that MOFS is required in practical use presented in descriptive tables, such as the one shown in cases like those requiring fairness; however, it still lacks Figure 3, which shows all 19 MOFS results for the German methodologies and tools for interpretability. Credit dataset. In most cases, we can see all the features Future work includes developing methodologies for in each solution except for the nineteenth solution with MOFS interpretability and incorporating them into FS 10 features. Looking at the objectives, one after the other tools for accessibility. (since that is all the support the table provides), solutions one to three ofer the smallest subset size ( 1), solution 18 has the best balanced accuracy (0.6607), solution eight 6. Acknowledgements has the highest F1-score (0.8305), solutions one to three ofer the smallest VIF ( 0), and finally, solution two has the The project leading to this publication has received fundbest statistical parity (1) and equalised odd (0.8) scores. ing from the European Union’s Horizon 2020 research Solution two (beszeit) appears to have a better score for and innovation programme (G.A. No. 955895). A. Abelló four out of six objectives. However, building a model and B. Bilalli are funded by the Spanish Ministerio de with one feature is not a good idea. Therefore, we must Ciencia e Innovación under project PID2020-117191RBcontinue comparing alternatives until a satisfactory so- I00, funding scheme AEI/10.13039/501100011033. G. Bonlution is found. tempi is supported by the Service Public de Wallonie
Indeed, a better, more informative and systematic Recherche under G.A. No. 2010235–ARIAC by Digitalmethodology will accelerate this process and make it Wallonia4.ai. Computational resources have been proeasier for the data scientist. vided by the Consortium des Équipements de Calcul Intensif, funded by the Fonds de la Recherche Scientifique de Belgique (G.A. No. 2.5020.11) and Walloon Region. artificial intelligence, Electronic Markets 32 (2022) 2159–2184. [1] J. Clore, K. Cios, J. DeShazo, B. Strack, Diabetes [14] K. Deb, H. Jain, An evolutionary many-objective op130-US hospitals for years 1999-2008, UCI Ma- timization algorithm using reference-point-based chine Learning Repository, 2014. DOI: https://- nondominated sorting approach, part i: solving doi.org/10.24432/C5230J. problems with box constraints, IEEE transactions [2] T. Le Quy, A. Roy, V. Iosifidis, W. Zhang, E. Ntoutsi, on evolutionary computation 18 (2013) 577–601.
A survey on datasets for fairness-aware machine [15] S. Chand, M. Wagner, Evolutionary many-objective learning, Wiley Interdisciplinary Reviews: Data optimization: A quick-start guide, Surveys in OperMining and Knowledge Discovery 12 (2022) e1452. ations Research and Management Science 20 (2015) [3] E. Hancer, B. Xue, M. Zhang, D. Karaboga, B. Akay, 35–42.
Pareto front feature selection based on artificial [16] S. A. Hicks, I. Strümke, V. Thambawita, M. Hambee colony optimization, Information Sciences 422 mou, M. A. Riegler, P. Halvorsen, S. Parasa, On (2018) 462–479. evaluation metrics for medical applications of arti[4] Y. Zhang, S. Cheng, Y. Shi, D.-w. Gong, X. Zhao, ifcial intelligence, Scientific reports 12 (2022) 5979.
Cost-sensitive feature selection using two-archive [17] M. Pal, S. Bandyopadhyay, Many-objective feature multi-objective artificial bee colony algorithm, Ex- selection for motor imagery eeg signals using difpert Systems with Applications 137 (2019) 46–58. ferential evolution and support vector machine, in: [5] J. Grzyb, M. Topolski, M. Woźniak, Application of 2016 International Conference on Microelectronmulti-objective optimization to feature selection for ics, Computing and Communications (MicroCom), a dificult data classification task, in: International IEEE, 2016, pp. 1–6.
Conference on Computational Science, Springer, [18] H. Dong, J. Sun, X. Sun, R. Ding, A many2021, pp. 81–94. objective feature selection for multi-label classifica[6] Z. Liu, B. Chang, F. Cheng, An interactive filter- tion, Knowledge-Based Systems 208 (2020) 106456. wrapper multi-objective evolutionary algorithm for [19] H. Li, F. He, Y. Liang, Q. Quan, A dividing-based feature selection, Swarm and Evolutionary Compu- many-objective evolutionary algorithm for largetation 65 (2021) 100925. scale feature selection, Soft computing 24 (2020) [7] Y. Xue, Y. Tang, X. Xu, J. Liang, F. Neri, Multi- 6851–6870.
objective feature selection with missing data in clas- [20] P. Korhonen, J. Wallenius, Visualization in the sification, IEEE Transactions on Emerging Topics multiple objective decision-making framework, in: in Computational Intelligence 6 (2021) 355–364. Multiobjective optimization: interactive and evolu[8] P. Barbiero, E. Lutton, G. Squillero, A. Tonda, tionary approaches, Springer, 2008, pp. 195–212.
A novel outlook on feature selection as a multi- [21] J. Cheng, J. Sun, K. Yao, M. Xu, Y. Cao, A variable objective problem, in: International conference on selection method based on mutual information and artificial evolution (evolution artificielle), Springer, variance inflation factor, Spectrochimica Acta Part 2019, pp. 68–81. A: Molecular and Biomolecular Spectroscopy 268 [9] Y. Zhou, J. Kang, S. Kwong, X. Wang, Q. Zhang, An (2022) 120652.
evolutionary multi-objective optimization frame- [22] A. Benítez-Hidalgo, A. J. Nebro, J. García-Nieto, work of discretization-based feature selection for I. Oregi, J. Del Ser, jmetalpy: A python frameclassification, Swarm and Evolutionary Computa- work for multi-objective optimization with metation 60 (2021) 100770. heuristics, Swarm and Evolutionary Computation [10] Z. Zhang, J. Wen, J. Zhang, X. Cai, L. Xie, A 51 (2019) 100598.
many objective-based feature selection model for anomaly detection in cloud environment, IEEE
Access 8 (2020) 60218–60231. [11] K. Jha, S. Saha, Incorporation of multimodal multiobjective optimization in designing a filter based feature selection technique, Applied Soft Computing 98 (2021) 106823. [12] D. Rodrigues, V. H. C. de Albuquerque, J. P. Papa,
A multi-objective artificial butterfly optimization approach for feature selection, Applied Soft Computing 94 (2020) 106442. [13] J. Zacharias, M. von Zahn, J. Chen, O. Hinz, Designing a feature selection method based on explainable