In many fields, such as functional genomics or finance, data analysis, and predictive modeling are always challenging for the course of dimensionality and noisy data. In these cases, efective feature selection algorithms, based on Machine and Deep Learning, can perform and improve the identification of important features, leading to more treatable problems in terms of dimensionality. The paper proposes a novel algorithm to perform Feature Selection on highly dimensional data, which exploits the reconstruction capabilities of autoencoders and an ad-hoc defined Explainable Artificial Intelligence-based score to select the most informative feature for predictions. We benchmark such an approach on several state-of-the-art datasets and against the previously proposed algorithm in the literature, showcasing its efectiveness.
In the field functional genomics, starting from the results of the Human Genome Project, the evolution
of sequencing techniques provides big volumes of data for each single patient by taking advantage of
the high-throughput and next-generation sequencing i.e., a set of time and cost-efective techniques for
sequencing DNA and RNA. By means of them, it is possible to measure the expression of thousands of
genes for each individual and hence to collect quantitative gene expression profiles (GEP) to be used for
research and clinical purposes. But despite GEP datasets represent a valuable source of information in
healthcare—they are indeed used for diagnosis, prevention, and precision medicine—their analysis
results challenging for three main reasons. The first one is the course of dimensionality: a genomics
dataset typically consists of a very large number of features (genes) and a small number of samples
(patients); the second problem concerns imbalanced classes: in the analysis of diferent groups of
patients, genomics data are often stratified in classes according to diferent pathologies. In most cases,
there is a significant diference between the number of instances in each class; finally, sequencing data
are typically collected from multiple sources, diferent laboratories, and sequencing tools. This results
in noisy datasets which are dificult to analyze [
In recent years, Machine Learning (ML) and Deep Learning (DL) have been widely adopted in this field,
providing breakthrough results and meaningful insights into the relationship between genomics and
cancer [
We propose a new algorithm, based on DL and Explainable Artificial Intelligence (XAI), for genomics whose aim is threefold: first, select the most meaningful genes for a regression/classification problem; second, provide a more accurate prediction model; third, quantify and evaluate the efect of features on the predictions, through XAI. We used our algorithm for the GEP analysis of acute lymphoblastic leukemia (ALL) patients, identifying a meaningful subset of genes for the disease prognosis. The following sections are organized as follows. First, we review the most relevant related works in Section 2, and we then give a formal definition of the algorithm in Section 3. The application and the results obtained by the algorithm for the CLL study are discussed in Section 4. Finally, directions for further research are proposed in Section 5.
A number of recent studies propose and evaluate new approaches for feature selection (FS) on GEP
datasets for cancer diagnosis and prognosis[
We moreover use an ad-hoc defined XAI-based score in order to iteratively select the features by taking
advantage of the Shapely Additive ex-Planation method (SHAP)[
∑︁
⊆ ∖{}
||!(| | − | | − 1)!
| |!
[∪{}(∪{}) − ()]
(1)
with ∈ R and where ∪{} and ∪{} denote the prediction model and the sample considering
the only subset of features without the -th one. In words, SHAP computes the contribution of a
feature by comparing the model predictions obtained with and without a feature, for all the possible
combinations . Since the computation of the Equation 1 is ineficient in the case of NN as a prediction
model—a NN should be re-trained for each combination of features (2| |)—the authors demonstrate in
[
The proposed algorithm is based on two main ideas: (1) we use a clustered correlation matrix in order to group features that enclose similar patterns and we then filter the redundant information for each group by using AEs. In contrast with previous works, in which AEs are used for dimensionality reduction, we still work at the level of the original features. In particular, we take advantage of the encoding and reconstruction abilities of the AEs assuming that the more accurate is the reconstruction of a feature, the more that feature is representative of the cluster it belongs to. We hence provide a more treatable dataset in terms of dimensionality, without loss of representativeness, by filtering redundant features; (2) we train NNs and we iteratively select the most meaningful features using a new ad-hoc defined SHAP score. We repeat the analysis by removing at each iteration, the previously selected features. We
Clusters k1 . . . kq Number of genes ≥ N Collect selected genes Number of genes < N
Explainability-based Selection Select the most meaningful genes according to an ad-hoc defined
SHAP-based score
Correlation Clustering
Autoencoder (AE) Filter
Patients used a1sfea.tu.re.sforneach k1 . . . kq
AE.1 .
. AEq Remove selected genes from k1 . . . kq
NN Training eventually use the set of selected features (from all the iterations) to train and explain a final model. Figure 1 shows the main algorithm phases.
Let be = {, } a dataset such that ∈ R× is the matrix of inputs, and ∈ R× is the matrix of the corresponding labels. Let us further assume ≫ meaning that the dataset is characterized by a way larger set of features with respect to the number of samples.
As a novelty contribution, we introduce a new impact score, which, by means of the SHAP local explanation, measures the global impact of each feature on model predictions. We hence associate to each feature (column) of , used to train a model N, a couple ( ,N , ,N) were ,N is the correlation between the -th columns of and their shapely values {1, , ..., , }, and ,N is defined as follows: ∑︀=1 |, | * 2, ,N = ∑︀ℎ=1 ∑︀=1 |,ℎ| * 2,ℎ (2) With ,N and ,N we want to emphasize how and how much, respectively, a feature globally afect the predictions of N.
For sake of clarity, we introduce our algorithm by first defining a set of sub-procedures. The first one (Algorithm 1) computes the pairwise correlation matrix ∈ R× between the features (columns) of a generic real-valued matrix . Finally it clusters in order to return a set = {1, ..., } such that for each = 1, .., , is a set of indexes—a partition (cluster) for the columns of . The second sub-procedure, defined in Algorithm 2, trains an AE for each cluster, by using the transpose of the input matrix —meaning that, for the AE model, each feature represents a sample and vice versa. The rationale here is that we assume the best-reconstructed feature (over the samples) to be the most representative of the cluster it belongs to. We denote ∈ R×| | as a matrix including the only columns of which indexes are in . The function provides the column indexes of associated with the best-reconstructed feature. Finally, the sub-procedure returns a set of indexes—one for each cluster.
function CorrClustering() ← () ← () return end function
function AEFiltering(,) ← ∅ for ∈ do
AE ← ( )
← ∪ (AE , )
end for
return
end function
The last sub-procedure, reported in Algorithm 3, takes as input: the data, a matrix of shapely values
Φ and the threshold ∈ R, with ∈ [
Algorithm 3 Selection function select(Φ, , ) ← (Φ, ) ←← |1|∑︀∈(Φ, ) ˜ ← { | | | > ∧ > , ∀ ∈ , ∀ ∈ } return ˜ end function
The main procedure is described by Algorithm 4. After clustering the correlation matrix, it selects a set of meaningful features index to be added to . It then removes the selected indexes from their corresponding clusters in and proceeds by repeating the analysis. Here we denote ∈ R×| | (and accordingly ) as a matrix including the columns of which indexes are in , and N (and accordingly N) as a NN trained on { , }. The iterative analysis stops when ∈ N, ≤ features are selected or on a maximum number of iterations. The algorithm eventually trains and explains a ifnal NN using the set .
To validate the efectiveness of the method, we tested it on a synthetic toy dataset. This allowed us
to verify that the method correctly selected the centroid features for each cluster, ensuring that the
Algorithm 4
Require: ,
← CorrClustering()
← ∅
while || < ∨ not do
← AEFiltering(, )
, ← dataBalancing( , )
N ← findModel( , )
Φ ← Shap(N , )
˜ ← select(Φ, )
← ∪ { ∈ | ∈ ˜}
← ∖
end while
N ← findModel(, )
Φ ← Shap(N, )
* ← select(Φ, )
◁ Model Selection & Training
◁ matrix of shapely values Φ ∈ R×| |
◁ remove from their corresponding cluster
most representative features were identified. We applied our algorithm for analyzing GEP of patients
from Leukemia from the Kent Ridge biomedical data repository [
We finally use our SHAP scores (defined in Section 3.1) to select the most meaningful genes, by setting = 0.85. After selecting a set of = 50 genes through the iterations of the algorithm, we use them to train and explain a final NN.
The algorithm has been implemented using the Python (v3.8.11) programming language. NNs have
been implemented by taking advantage of the Pytorch (v2.4.1) framework. XAI analysis was performed
by means of the SHAP library [
The overall results are reported in Table 1. In particular, for each iteration of the algorithm, we measured the accuracy of all the models obtained during cross-validation, for which we report the confidence interval. As we expected, the classification accuracy decreases with the algorithm iterations: the reason is that the previously chosen features—expected to be the most representative of each cluster—are no more considered for the subsequent analysis. An improvement in accuracy is instead reported for the ifnal step of the algorithm, by which a model is trained using the set of genes selected during each iteration. The accuracy of the best final model is 100%.
Figure 3 reports, on the left side, a summarized representation of the shap values and, on the right side,
the values for correlation and intensity for the most interesting genes found by our algorithm. In this
context, it is important to compare our findings with the work of Al-Azani et al. [
Conversely, Bennet et al. introduced a hybrid gene selection technique that integrates Support Vector Machine-Recursive Feature Elimination (SVM-RFE) with the Based Bayes Error Filter (BBF). Their approach involved ranking attributes with SVM-RFE and subsequently using BBF to eliminate redundant attributes, followed by classification with the SVM algorithm. Their eforts yielded an impressive classification accuracy of 97.2% on the Leukaemia dataset, underscoring the power of hybrid techniques in attribute selection.
The algorithm proposed in this work can be used as a valuable tool in genomics to identify protective (or not) sets of genes for a disease, suggesting potential pathways for further medical investigation. A natural direction for future development is to perform a large-scale assessment of the algorithm performances, by using state-of-the-art benchmark GEP datasets.
The research reported in the paper was partially supported by the PNRR projects “FAIR (PE00000013) - Spoke 9” and “Tech4You (ECS00000009) - Spoke 6”, under the NRRP MUR program funded by the NextGenerationEU. The National Plan for NRRP Complementary Investments (PNC, established with the decree-law 6 May 2021, n. 59, converted by law n. 101 of 2021) in the call for the funding of research initiatives for technologies and innovative trajectories in the health and care sectors (Directorial Decree n. 931 of 06-06-2022) - project n. PNC0000003 - AdvaNced Technologies for Human-centrEd Medicine (project acronym: ANTHEM).