Radiomics, involving analysis of calculated, quantitative features from medical images with machine learning tools, shares the instability challenge with other high-dimensional data analyses due to variations in the training set. This instability afects model interpretation and feature importance assessment. To enhance stability and interpretability, we introduce grouped feature importance, shedding light on tool limitations and advocating for more reliable radiomics-based analysis methods.
Radiomics [
Radiomics analyses typically involve three main steps: image acquisition and segmentation,
feature extraction, and statistical analysis (see Fig. 1). In the first step, medical images are
acquired and segmented to isolate the region of interest. In the second step, quantitative
features are extracted from the segmented region using mathematical algorithms and statistical
methods. These features can include shape, texture, and intensity-based metrics, among others.
In the final step, statistical / machine learning (ML) based analyses are performed to identify
patterns and associations between the extracted features and clinical outcomes, such as disease
diagnosis, prognosis, and treatment response. [
Radiomics relies on measuring feature importance to understand their impact on
predictions. [
Hypothesizing low feature stability (FS) in radiomics-based prediction models, similar to
other high-dimensional data analyses, we propose grouped feature importance [
To investigate the instability of radiomics-based analyses, we constructed a working example utilizing 136 pre-calculated radiomics features [14] from the Lung Image Database Consortium image collection (LIDC-IDRI) [15]. This database contains thoracic computed tomography (CT) scans with annotated lesions, classifing 616 benign and 281 malignant nodules. We established a standard machine learning (ML) classification pipeline using the R package mlr3 [ 16]. The pipeline encompassed a random forest (rf) with feature preprocessing (imputation, factor encoding, and correlation-based feature selection). To refine the pipeline, we utilized nested resampling with 5-fold cross-validation for both outer and inner loops. Hyperparameters, including the fraction filtered in preprocessing, the count of features randomly sampled per decision tree split, and the number of trees within the rf model, were fine-tuned. The correlationbased filter’s optimization involved a correlation cutof range from 0.1 to 0.9. AUC served as the basis for optimization and performance assessment.
Using a methodology akin to test-retest, we trained our ML pipeline over 1000 bootstrap iterations, varying solely the underlying training set (seed) [17]. For model-agnostic feature assessment, we employed minimal depth (MD) and permutation feature importance (PFI). The variable’s ‘importance‘ threshold was determined based on the mean PFI or MD within one iteration, revealing how often the variable was deemed important across 1000 bootstraps.
To test our hypothesis on improved model stability with grouped radiomics features, we’ll form these groups and assess their stability using Group Feature Importance (GFI) methods.
We will consider various approaches to organizing radiomics features into groups: 1. Grouping based on Semantic Meaning / Clinical Relevance: Categories according to the anatomical or physiological aspects (shape, intensity, texture). 2. Feature Type Grouping: Groups based on calculation nature, e.g., original vs. processed (wavelet, log-filter) image features. 3. Statistical Grouping: Use statistical techniques like clustering or intercorrelation analysis to group features based on their statistical properties. 4. Task-Specific Grouping: Adapt feature grouping to the research question; e.g. for predicting treatment response, cluster treatment-related features. 5. Expert Knowledge-based Grouping: Categories guided by physicians or domain experts, based on clinical significance or feature relevance.
To assess grouped feature importance [18] in radiomics analyses, we will use permutationbased [19], refitting[ 20], and Shapley-based [21] methods.
1. Permutation-based method: Randomly permuting grouped features measures their impact on the model’s predictive accuracy. 2. Refitting method: Fit the model multiple times, excluding specific feature groups, to assess the change in performance. 3. Shapley-based method: Assign values to grouped features based on their contribution to predictions using cooperative game theory.
Furthermore, we’ll employ the combined features efect plot (CFEP) [
In our classification demonstration, employing 136 pre-calculated radiomics features from the LIDC-IDRI dataset, we trained a ML pipeline over 1000 bootstrap rounds. The models achieved an average AUC of 0.880 (inter-quartile range: [0.871; 0.891]). In each bootstrap iteration, we calculated MD and PFI values, marking variables exceeding the mean MD/PFI as ‘important‘.
In Figure 2, the relative frequency of each variable’s importance is depicted. Among the 136 radiomics features, 46 were chosen at least once by PFI, and 38 by MD. The top four vital features were selected between 54.7% and 75.6% of the time for PFI, and 61.0% to 78.3% for MD. These four features—promenance0_N, promenance0_P, sphericity, and uniformity_N—appeared as the most important in diferent orders. The fith feature (PFI: diameter_mm, MD: skewness) was chosen less than 40% of the time in both cases.
In conclusion, our study highlights challenges in achieving feature stability and interpretability
in radiomics-based analyses, notably during feature importance interpretation. Using a
testretest approach training one ML model (pipeline) 1000 times on varied training set combinations,
we found that the most crucial feature was selected in only about 75% of cases, clearly revealing
variability and uncertainty. This inconsistency persisted despite incorporating a correlation
iflter into our ML pipeline, signifying that the issue extends beyond feature correlations [
To tackle this challenge and augment the interpretability of radiomics-based ML models,
we advocate for the adoption of grouped feature importance techniques. Among these, the
combined features efect plot (CFEP) [
Instability in radiomics feature calculations arises from various factors like acquisition modes, reconstruction parameters, and segmentation thresholds. [22] [23] This instability extends to radiomics-based ML model performance due to high-dimensional test set variations. Hence, future research should focus on refining grouped feature importance methods, including CFEP, to enhance feature stability and interpretability.
Ultimately, by fortifying the stability and interpretability of radiomics-based ML models, we can revitalize the potential of radiomics in medical imaging, enabling more precise diagnoses, prognoses, and informed treatment choices for various medical conditions.