Central nervous system tumors, particularly gliomas, rank among the top 10 causes of cancer-related deaths worldwide. Thus, precise differentiation of these tumors is crucial for effective treatment, which can reduce patient suffering and lower mortality rates. We propose a deep learning-based technique for glioma differentiation using MRI imaging, which incorporates two novel approaches. First, we represent perfusion sequences as time series, which serve as inputs for a Deep Neural Network (DNN) classifier. This classifier is trained to distinguish the sequences between low-grade glioma (LGG) and high-grade glioma (HGG). Second, we employ Conformal Prediction to calibrate the results, ensuring they include the true category with a 90% probability. This approach has been rigorously tested to evaluate its performance. Our experimental results demonstrate that our deep neural network not only provides accurate but also ensures trustworthy predictions.
Central nervous system (CNS) tumors have traditionally been regarded as a lethal form of cancer, ranking among the top 10 causes of deaths globally. The GLOBOCAN project using nationwide data from 184 countries estimated the annual incidence of malignant-only CNS tumors to be 3.4 per 100,000 individuals [1]. Within the variety of tumor types, gliomas are the most frequent primary intracerebral tumors in adults. Despite their high fatality rates, the prognosis of CNS tumors has considerably improved over the last decades, possibly because of the prompt detection, the optimization of treatment protocols including the introduction of temozolomide and the advances in neurosurgical procedures.
Accurate preoperative differentiation of primary CNS tumors is crucial because the treatment strategies differ substantially (e.g. stereotactic biopsy + chemotherapy vs. gross total resection + chemotherapy in lymphoma vs. glioblastoma). Magnetic Resonance Imaging (MRI) is the most important non-invasive technique in the diagnosis and differentiation of CNS tumors. In this context, MRI-perfusion is an imaging technique frequently used to assess the vascularization of brain lesions in a minimally invasive way. There are different approaches for the study of perfusion and lesions of CNS.
Machine learning has been employed in numerous medical problems as part of expert decision support systems. However, in most instances, these systems do not provide any information regarding the confidence of their predictions. In this paper we propose an innovative approach that treats perfusion curves as time series data and apply Deep Neural Networks (DNN) and Conformal Predictors (CP) to assess the type of glioma a patient suffers, along with measures of confidence.
In essence, this involves framing the differentiation problem as a classification task using the perfusion curve as input. This requires finding a suitable representation for the series and then identifying a technique that delivers high classification performance. On top of the classifier, a conformal predictor is built to quantify the level of uncertainty inherent in the model's predictions using specific metrics. Our research has resulted in an effective automated procedure to assist radiologists in their work, contributing to the quantification of uncertainty at the patient level, with applications in personalized and precision medicine.
The structure of the article is as follows: In Section 2, we present state-of-the-art methods that use deep learning and conformal prediction for glioma differentiation, including medical context for a better understanding of the problem. In Section 3 we detail Conformal Predictors. In Section 4, we present our proposal for glioma differentiation. In Section 5, we include the empirical study and discussion of the results. Finally, Section 6 contains the conclusions of the work.
As noted in the introduction, gliomas are the most frequent primary intracerebral tumors in adults. The age-adjusted annual incidence of histologic verified glioma has been reported to be 7.3 cases per 100,000 person-years [2]. Most patients with gliomas have a fatal prognosis, and the disease has considerable impact on patients and their families' physical, psychological, and social status. A recent study on the epidemiology of gliomas found that high-grade gliomas -HGG, grades 3 and 4 according to World Health Organization (WHO)-were present in 85% and low-grade glioma (LGG, WHO 1 and 3 grades) in 15% of the cases, with 5-year overall survival of 82, 54, 22 and 3% for grade 1, 2, 3, and 4, respectively [3].
Accurate preoperative differentiation of primary CNS tumors is essential because the treatment strategies differ substantially. MRI is the most important non-invasive technique in the diagnosis of CNS tumors. In particular, MRI-perfusion is frequently used to assess the vascularization of brain lesions. There are different approaches for the study of perfusion and lesions of CNS. The most widely used MRI-perfusion approach in clinical practice is T2 gradient echo or dynamic susceptibility contrast (DSC) due to its high sensitivity and specificity to differentiate lesions and its short acquisition time. DSC perfusion allows the study of changes in the signal intensity of the tissues derived from the passage of an intravenous gadolinium bolus administered in a peripheral vein. The passage of the contrast bolus leads to specific imaging patterns in different tissues and tumors, with changes in their signal intensity proportional to the amount of gadolinium that passes through or is deposited in them. Changes in signal intensity over time allow a curve to be obtained, from which hemodynamic parameters are inferred.
At present, pathological anatomy techniques are essential in the diagnosis and genetic and histological differentiation of gliomas. However, in the presence of lesions such as metastases, an adequate diagnostic imaging can avoid a brain biopsy. The procedure of stereotactic brain biopsy entails a considerable risk of complications (e.g., haemorrhage, infection, wound breakdown), which can be as high as 15.3% in tumors located in midline areas, while 28.8% of post-procedural intracranial haemorrhages have been reported [4].
A recent Cochrane review [5] reported limited available evidence on the ability of perfusion MRI to distinguish HGG from LGG, which precludes reliable estimation of the performance of DSC MRI perfusion-derived parameters (e.g., relative cerebral blood volume or rCBV) for determining the tumor grade, specifically in untreated solid and non-enhancing LGG vs. HGG. MRI-perfusion has also been advocated as a useful tool to differentiate between gliomas and other tumors of the CNS, such as lymphomas or metastases, but with variable diagnostic yields depending on the publications. Furthermore, it demonstrated to be useful in differentiating between tumor recurrence of gliomas and post-treatment changes such as pseudo progression or radionecrosis. However, the available evidence is restricted to isolated non-clinically validated studies.
Most of these publications have focused on a classical analysis of MRI perfusion data, with special emphasis on the quantification of rCBV (an estimate of tumor perfusion), which is not quantifiable in absolute terms (thus requiring comparison with areas of normal cerebral parenchyma). A number or publications that have applied Artificial Intelligence algorithms to the analysis of brain perfusion can be found. They are discussed in section 2.2.
In this section, we present the most relevant articles in the state of the art related to the topic. To the best of our knowledge, there is no technique in the literature that utilizes the perfusion curve as input for a time series classification model in low-grade gliomas (LGG) and high-grade gliomas (HGG). Moreover, there are numerous approaches in the literature that apply CP to quantify the uncertainty of deep learning models ( [6], [7]). In fact, the combination of deep learning and CP is especially useful in biomedical problems. For example, in skin lesion classification ( [8]), for the diagnosis and grading of prostate biopsies ( [9]), for rating breast density in mammography ( [10]), for grading the severity of spinal stenosis in lumbar spine MRI ( [11]); among other applications ( [12], [13], [14]), we have not encountered in the literature any work that combines DNN and CP methods in the differentiation of central nervous system tumors.
However, the non-invasive differentiation of gliomas through the application of machine learning, specifically distinguishing between LGG and HGG gliomas, has been extensively investigated in recent years. For instance, in [15], a substantial number of radiological features were extracted from MRI sequences, including T1-weighted, T1-weighted contrast-enhanced, T2-weighted, and FLAIR, across a total of 285 cases (210 HGG, 75 LGG). These features were used to train three classifiers (logistic regression, random forest, and support vector machine) to determine the glioma type, achieving an average AUC of 0.9030 for test cohorts.
In [16], a deep multi-scale 3D Convolutional Neural Network (CNN) architecture was proposed to categorize gliomas into LGG and HGG using volumetric T1 contrast-enhanced MRI sequences, achieving an accuracy of 96.49%.
Recently, some studies advocate mixed approaches incorporating molecular profiling for the differentiation of LGG and HGG ( [17]). For instance, in the work by authors in [18], clinical and laboratory data were integrated to create a tool for predicting the molecular status (ATRX, IDH1/2, MGMT, and 1p19q co-deletion), distinguishing between low-grade and high-grade gliomas. The system achieved an AUC of 0.885 for this specific learning task.
Finally, other studies such as [19], [20], [21], and [22] utilized cases of high-grade or low-grade gliomas to conduct specific studies on various features, but did not present tools for classification between these two grades of gliomas.
In spite of their generally accurate performance, many machine learning models are known to estimate poorly prediction probabilities. To deliver reliable estimates and avoid over or under-confidence predictions, prediction uncertainty must be quantified. This decision aligns with the demand of trustworthy Artificial Intelligence, which, as expressed in recent European Parliament AI regulations [23], includes as a requirement the models ability to quantify and communicate their confidence in their outputs ( [24], [6]) with special focus on high-risk applications as AI-supported medicine [25].
Performance metrics, such as accuracy or F1-score, cannot tell about the confidence or uncertainty of the model's predictions. Even for a model that is right most of the time, not all the cases are equally easy to classify. Some of them might even be doubtful (e.g., they belong to a different distribution or their kind was mostly under represented, overlooked, or misclassified). For example, we can think about patients images from a different hospital, taken from a different MRI scanner, or labeled by another team. The model may provide unnoticed overly confident predictions because machine learning algorithms do not include a built-in warning mechanism to prevent well-informed predictions from looking the same as wild guesses [26] and refers to the alignment between predicted probabilities directly provided by the model and relative frequencies in the actual data. Well-calibrated models are those where the predicted probabilities match the empirical probabilities. Thus, it is about calculating "the probability that the predicted probability is right". For example, given an MRI scan, if the model predicts class A with 80% probability, then class A should occur approximately 80 times out of 100 predictions. More formally, a well-calibrated classifier satisfies the following formula [27]:
To provide confidence to classification results, calibration models should be applied. Two of the most popular calibration methods are Platt's Scaling and Isotonic Regression. Platt's Scaling [28] is a very popular method that maps each prediction to its empirical frequency by passing it through a sigmoid. The method is therefore parametric and presupposes normally distributed and heteroscedastic per-class scores, a notably limiting assumption [29]. On the other hand, Isotonic Regression [30] is more general because the map is performed with an isotonic (monotonically increasing) function. It assumes that the classifier perfectly ranks objects in the test set, essentially implying an ROC AUC of 1 and it is not recommended for small sets [29]. A detailed explanation of how both calibration methods can be found in [31].
One step ahead of calibration is conformalization. To conformalize means that each single-point prediction is not only calibrated but also provided with a prediction interval, with statistical guarantees of including the true value at patient-level ( [27], [32], [29]). Conformal Prediction (CP) methods conformalize the prediction. CP methods, which are distribution-free and lightweight, have shown some successful application in health-related domains, mostly for cancer prediction ( [33], [34], [35], [32], [36], [34], [37], [38], [39]).
Venn-ABERS methods ( [40], [27]) are members of the CP family [41], [42], [43]). They share the skill to evaluate how different is an unseen instance from the training dataset. These methods were created to work on top of binary classifiers (although new implementations work for multi-class problems too [44]). They conformalize the predicted probability to match the actual frequency of its class, while delivering in the process the upper (𝑝 1 ) and lower (𝑝 0 ) bounds of the probability interval that includes the true class (a property known as coverage or validity) with statistical guarantee. This prediction interval quantifies the uncertainty of predictions at instance level: the larger the interval, the lower the confidence the model has in that prediction. Venn-ABERS are an adaptation of Isotonic Regression (where it is applied twice, to fit the probability of each class) and a special case of Venn Predictors ( [45], [46] , [29]), from which they inherit guaranteed validity in the form of well-calibrated probability prediction.
There are two possible implementations of the Venn-ABERS algorithm depending on whether or not an independent calibration set is required separate from the training set. On one hand, an inductive approach, or Inductive Venn-ABERS (I-V-A), where the calibration is trained over a hold-on split of the training set, thus the conformalization is model-agnostic [46]. On the other hand, Cross Venn-ABERS (C-V-A) where the splits are performed via k-fold, thus there is not need to reserve data just for calibration as in the inductive case and we use all available data to train the classifier but the classifier must be retrained each time we predict and conformalize the probabilitiy of a new patient [33].
This work presents a novel methodology for the non-invasive differentiation of a specific family of CNS tumors, specifically distinguishing between high-grade (HGG) and low-grade gliomas (LGG). Patient data are 3D MRI scans focused on the T2 sequence. The T2 sequence is designed to measure the evolution of tissue signal intensity as the contrast bolus passes through. From these changes in signal intensity in the image, once processed, we construct the perfusion curve, which serves as input for our machine learning model. Thus, we transform an image processing problem into a time series classification one, which we address by combining two impactful technologies: on one hand, a deep The process initiates with the reception of a new perfusion sequence. After preprocessing the images, we extract the perfusion curve by calculating the mean luminance of the voxels at different time points. Subsequently, a CP-DNN is trained to extract the final prediction and quantify its uncertainty.
neural network capable of extracting patterns with high precision; and on the other hand, conformal predictors for quantifying the uncertainty of predictions. Consequently, we develop a decision support system with significant generalization capacity, providing valuable and calibrated information to the radiologist. The presented pipeline is divided into three phases: preprocessing of the sequence, generation of the perfusion curve, and finally, glioma differentiation with a machine learning model, specifically, a conformal predictor based on a deep neural network classifier. Below, we provide a detailed description of each phase. Figure 1 show the complete pipeline.
The differentiation process begins with the arrival of a new perfusion sequence. The first phase, related to preprocessing, involves normalizing the image in both size and grayscale levels using the Statistical Parametric Mapping technique [47]. Additionally, it is crucial that the image is segmented. Despite numerous initiatives aiming to develop deep learning models for brain segmentation in MRI (as illustrated in [48] and other reviews), there is, to the best of our knowledge, no automatic technique for segmenting an image in the perfusion sequence. Therefore, in our approach, cases have been segmented by radiologists using the 3D Slicer tool (https://www.slicer.org, [49]).
Once the perfusion sequence is normalized and segmented, we generate the curve. The perfusion sequence consists of a series of snapshots taken over a specified period. This procedure generates 𝑛 3D images of the brain and, specifically, the glioma. By processing each of these 𝑛 images, calculating the mean luminance value of each voxel within the image in each snapshot, we generate a time series with as many points as snapshots. This way, a time series is generated out of the perfusion sequence, the perfusion curve. After that, the time series is normalized by subtracting the mean of the points and dividing by their standard deviation.
The differentiation of gliomas into low grade (LGG) and high grade (HGG) requires a machine learning system capable of extracting relevant patterns to distinguish between the two tumor types. Consequently, we constructed a deep dense neural network that takes a perfusion curve as input and classifies it into LGG and HGG. This is an unbalanced classification problem due to the prevalence of the HGG class in the data set over the LGG class. Thus, to evaluate the DNN performance, we calculate the accuracy and F1-score.
To enhance the trustworthiness of the classifier outputs, its probabilities must be calibrated. We calibrate the predictions of the model applying four independent calibration techniques: Platt's Scaling (PS), Isotonic (IR), Cross Venn-ABERS (C-V-A), and Inductive Venn-ABERS (I-V-A).
The performance of the calibrators is evaluated with classification and calibration metrics. In this way, physicians have access not only to artificial intelligence-based tools with a high level of accuracy but also to the ability to assess the confidence that these models have in each of the predictions associated with the patients, lending credibility to those predictions in which the model has sufficient confidence. To achieve this, we employ a Conformal Predictor with a specified confidence level, 𝜂, (e.g. 90%) on top of the DNN. Thus, the radiologist will know, with each prediction, whether the model has confidence exceeding 𝜂 in determining whether the tumor is a low-grade or high-grade glioma, what would increase his or her trust on the differentiation model performance.
To the best of our knowledge, there is no state-of-the-art method that addresses the problem of glioma differentiation by applying explainability and trustworthiness methods, using perfusion sequences as time series and applying DNN and CP in this binary classification context.
To demonstrate the ability of our method to differentiate between high-grade and low-grade gliomas satisfactorily, we have conducted a rigorous experimentation. It is detailed in this section along with the results and their analysis.
We have compiled a cohort of 58 patients from the radiology department of the Virgen de las Nieves Hospital in Granada, Spain, selected for their exceptional quality and homogeneity in the MRI scans taken for their diagnosis and treatment.
Due to the rather small size of the dataset, we have designed a data augmentation strategy to effectively train our machine learning models. For the training partition, we generate perfusion curves not only at the level of the entire volume but also across the different 2D slices that compose it along the Z-axis. All slices of the same sequence inherit the label, either LGG or HGG, from the original sequence. This extends the training set to exceed 700 curves. Besides the anticipated imbalanced character of the problem arises. The distribution of instances between the two classes is 70% (HGG) and 30% (LGG). To address this issue, the models need to be adjusted to weigh the prediction error, making it more costly to make errors in predicting the LGG class than the HGG class.
Taking into account that the patient cohort is not very extensive, the dense neural network to be used cannot be very large, as we might incur in overfitting issues. After different attempts, an architecture with five layers (input, output, and three hidden layers with 70, 30, and 5 neurons) is considered, which yields competitive results.
For the selection of hyperparameters, a Grid Search Cross Validation is applied to choose the optimization algorithm and the parameters 𝛼 (strength of the L2 regularization term), initial learning rate, 𝛽 1 (exponential decay rate for fist moment vector), 𝛽 2 (exponential decay rate for second moment vector), and 𝜖 (value for numerical stability). The result of this hyperparameter tuning process produced the following value selection:
• Optimization algorithm: Adam • Initial Learning rate: 0.001
• 𝛼 : 0.0001
• 𝛽 1 : 0.9
• 𝛽 2 : 0.9
• 𝜖 : 0.000000001
Our proposal combines deep learning and conformal prediction. We use classic metrics to evaluate the goodness of the prediction and calibration metrics to assess the level of uncertainty in the predictions. For classification metrics, we use accuracy and F1-Score. For calibration metrics, we use Brier Score and Log Loss. The first one is defined as:
where 𝑝 is the prediction probability of occurrence of the event 𝑖, and 𝑜 𝑖 is defined as follows:
Log Loss is defined as
with 𝑦 ∈ {0, 1} and a probability estimate 𝑝 = 𝑃 𝑟(𝑦 = 1).
For both of them, Brier Score and Log Loss, the smaller the value, the better is the calibration.
We calculated the accuracy and F1-score to compare the performance of our novel DNN for time series classification approach with the leading state-of-the-art method for time series classification, which is a 1NN with Dynamic Time Warping ( [50], [51]).
We have designed a data strategy to ensure the rigor of the study. In all cases, we guarantee the separation between patients is maintained. Firstly, to confirm the reliability of the results, a Stratified 5-fold Cross Validation is used. This involves generating five datasets where patients are split into 80% for training and 20% for testing each time, maintaining class proportions in each fold.
The values in Tables 1 and 2 represent the mean values from the five runs for each metric on the test set, demonstrating that our DNN approach outperforms the state-of-the-art 1NN-DTW in both performance and calibration. These results confirm previous analyses where Inductive Venn-ABERS and other CP methods outperformed Platt's Scaling and Isotonic Regression [46].
In order to unleash the true power of the DNN+CP (actually, Inductive Venn-ABERS) combination, we conduct another experiment where the dataset is partitioned into training and test sets, with 80% of the patients used for training and 20% for testing, again in a stratified manner. We plot in Figure 2 the predicted and conformalized probabilities for the 12 instances (MRI scans) in the test set. On the X-axis, we can see the 12 predictions made by the model. On the Y-axis, we find the probability expressed by the model, with 0 representing a prediction of the LGG class and 1 representing a prediction of the HGG class. The black dots indicate the actual labels of the test examples. As can be seen, there are 3 cases of LGG and 9 cases of HGG, maintaining the stratification of the original dataset. The larger light blue dot corresponds to the predictions made by the dense neural network without conformalization. When the black dot and the blue dot coincide, it indicates that the neural network correctly predicted the class. Conversely, if they do not coincide, the neural network made an incorrect prediction.
The smaller blue dot (𝑝 0 ) and the red dot (𝑝 1 ) represent the probability interval containing each of the calibrated predictions (𝑝), depicted by the orange dot. This orange dot represents the predictions of our DNN+CP model expressed in terms of probability, where a probability 𝑝 ≥ 0.5 results in a predicted label of 1 (HGG class). Conversely, if 𝑝 < 0.5, the predicted label is 0 (LGG class). The distance between 𝑝 0 and 𝑝 1 (𝑝 1 − 𝑝 0 ) indicates the uncertainty in the model's decision: the greater the distance between them, the higher the uncertainty in the prediction. This uncertainty is represented by the pink curve.
As observed, the neural network without conformalization makes two errors (the second and the eleventh cases). Thanks to the conformalization of the predictions, the DNN+CP model correctly predicts the second case, as the predicted probability is below 0.5, resulting in an LGG class label. Regarding the eleventh case, the DNN+CP, like the non-conformalized DNN, makes the error of categorizing the case as HGG when it is actually LGG. However, it can be seen that the uncertainty in that prediction is very high, showing confidence of less than 60%. Similarly, predictions with some uncertainty are made in the first (slightly over 60%) and third cases (close to 80%). The rest of predictions are more reliable, showing a confidence ranging from 85% to 99%.
Therefore, the combination of DNN+CP is not only more accurate, as it corrects errors that an uncalibrated neural network would make, but it also provides very valuable information to doctors about the confidence of the predictions, allowing specialists to assess each case with much more trustworthy information.
In this article, we address a global public health issue, ranked among the top 10 causes of cancerrelated mortality worldwide: the detection of central nervous system tumors. Specifically, we focus on differentiating gliomas into their two types: high-grade gliomas (HGG) and low-grade gliomas (LGG). To achieve this, we introduce a novel methodology based on time series classification. By utilizing MRI-perfusion, we transform the image into a time series, called the perfusion curve, which reflects tissue vascularization in a non-invasive manner. To classify these perfusion curves between LGG and HGG, a conformal predictor based on a deep neural network is trained. We have compared our proposal with the state-of-the-art technique in time series classification through rigorous experimentation, based on a dataset of patients from the radiology department of the "Virgen de las Nieves Hospital" in Granada, Spain, obtaining satisfactory results. We demonstrate that our methodology is not only innovative in the way it transforms the image problem into sequences, but also that the combination of deep neural networks and conformal prediction for time series classification generates an ideal tool for radiologists This tool displays exceptional generalization capability and the reduced uncertainty in its predictions, making it useful, reliable and trustworthy. This study contributes to quantifying uncertainty at patient-level with applications in personalized and precision medicine. Besides, by improving the confidence in AI-medical applications, this study also aligns with the requirements of the European Parliament AI regulation to achieve Trustworthy AI.
This research has been partially funded by Spanish Ministry of Economy, Industry and Competitiveness (PID2020-118224RB-100 and PID2023-151336OB-I00), con-financed by the European Union (FEDER).