This work is focused on the intelligent processing of MRI brain images to detect malignant tumors, which in comparison with tumors of other organs have their own specificity, making their correct segmentation and classification difficult. Due to the time-consuming nature of labeling the training sample, a semisupervised learning method based on 4D atlas priors was developed in this paper to solve the segmentation problem for the efficient use of unlabeled images. In the proposed method, a probabilistic 4D atlas is constructed based on the coordinates and voxel intensities of the labeled segments, and a generalization of this atlas was made based on gaussian mixture models and consideration of tumor contrast values. Three uses of this atlas were proposed in the form of two loss functions and pseudomask validation. The performance of the method was tested on a real MRI dataset of brain tumors of T1 modality axial view. The results showed a positive increase in segmentation accuracy compared to existing methods.
According to recent studies [
The main role of conventional/morphologic MRI in making the diagnosis is to determine the size
and anatomic location of the lesion in the brain for treatment or biopsy planning, to assess mass
effect and edema in surrounding healthy brain tissue, to assess the relationship to ventricular failure
of brain systems and vascular structures, and finally, along with other "functional" MRI sequences
to suggest a possible diagnosis [
The primary task of MRI brain image processing is the segmentation task, which is efficiently solved by deep neural networks. The main problem in training such a network, whether it is a convolutional neural network or a vision transformer, is the availability of a sufficiently labeled training sample. Unfortunately, in real-world situations, the samples are limited and with a small number of labeled scans, since high-quality labeling requires the availability and a huge amount of time of a qualified medical radiologist, which is not always possible.
To address the problem of insufficient sample and limited resources for labeling and training the model, there are different approaches, the most popular ones are: transfer learning - transferring knowledge from a more general known dataset to a specific limited dataset, active learning - methods for identifying the most relevant data for manual partitioning by a specialist, semi-supervised learning - model training using unlabeled data among others.
Each of the approaches deserves individual attention, but this paper proposes the use and improvement of semi-supervised learning as one of the most promising approaches using unlabeled sampling.
The use of semi-supervised learning in medical image segmentation has several advantages. First, it significantly reduces the need for large amounts of labeled data, which is time consuming and expensive to create. In the medical domain, where labeling requires the expertise of specialists, this is particularly important.
In addition, SSL can improve model quality by using information from unlabeled data without requiring additional datasets or labeling costs. This helps the model to better generalize and more accurately segment medical images, even with a limited amount of labeled data.
Another aspect is the ability to utilize a variety of data. SSL methods can efficiently handle heterogeneous data, which increases their flexibility and applicability in various medical applications.
Ultimately, such methods help accelerate the development and implementation of more accurate and reliable segmentation models in medical systems, which can lead to improved patient diagnosis and treatment.
One type of SSL is knowledge priors (KP) based learning [
Prior knowledge is information that the learner already has before learning new information, and sometimes it helps to cope with new tasks. Compared with non-medical images, medical images have many anatomical priors such as shape and position of organs, and incorporating anatomical prior knowledge into deep learning can improve the performance of medical image segmentation.
In this paper, a new SSL method has been developed that uses KP as a 4D anomaly atlas and its generalization in the form of gaussian mixture models (GMM) to capture potential anomalous regions more generally. Novel is the use of not only organ position but also organ contrast, as well as a probabilistic generalization to not be limited to only anomalies captured in a labeled sample. Testing of the approach was performed on a dataset of brain tumors.
In recent years, there has been a significant amount of research in the field of medical segmentation
with deep neural networks [
In the paper [
The [
In [
The paper [12] presents a method that integrates dense networks with deep anatomical priors and region adaptation techniques. This approach is particularly effective for fine segmentation tasks, such as renal artery segmentation, where precise anatomical details are critical. The use of dense networks allows for efficient capture of both local and global features, while the deep anatomical priors ensure anatomical plausibility. However, the model's complexity and the need for extensive labeled data for initial training can be significant drawbacks.
The work on [13] introduces a unique approach to incorporating shape priors through topological constraints. By using persistent homology, the method enforces topological consistency in the segmentation results, which is particularly beneficial for capturing complex anatomical structures. The primary advantage of this approach is its ability to ensure topological correctness in the segmentation output, reducing the likelihood of anatomical errors. However, the computational complexity of calculating persistent homology and the reliance on high-quality topological priors can be limiting factors.
A common limitation across the reviewed papers is their focus on specific aspects of the segmentation problem without simultaneously addressing the proposed generalized localization of anomalies and their pixel values. While each method brings innovative solutions to one or two aspects of the segmentation task, none of them comprehensively integrate all of the critical factors. And sometimes overly strict atlas or shape regularization can degrade the generalizability of the model on new data.
A private MRI dataset of brain tumors of the brain was used for experimentation and validation of the proposed method (Fig. 1). T1 modality and axial view were used. The dataset consists of 34 patients and 1144 images. The labeling was performed manually by medical professionals with more than 10 years of experience. An example of binary labeling of tumors by a specialist is shown in Fig. 2.
Figure 1: The brain MRI dataset used Figure 2: Manually segmented tumors on MRI images (represented in orange)
For the labeled sample : ( , ), … , ( , ), where ∈ × × is the MRI scan tensor and ∈ {0, 1} × × is a binary mask of the same size as the original tensor, where element 0 means absence, and 1 the presence of anomaly on a given voxel of the scan, and the unlabeled sample : ( , … ,
) create a classifier ( ) that correctly predicts the binary matrix ∈ {0, 1} × × of the new scan tensor ∈ × × utilizing both samples and . 4.2
Based on the works [
To account for anatomical structures and the natural location of anomalies, as well as their color, we propose to create 4D atlas priors based on labeled data segments. The first three dimensions of the atlas are the coordinates of the anomaly location in some section of the scan in 3D. The fourth dimension is the observed color or contrast of anomaly pixels.
To generalize the atlas and remove the limitation of the input data, we propose modeling the pixel distributions of a given atlas through GMMs that would completely cover the atlas with centers at the most frequent locations (Figs. 3b and 5). The use of modeling through GMM will help the models generalize better to new data without restricting them to the existing tumor atlas, as GMM assigns the probability of tumor presence to a wider range of pixels, unlike a conventional atlas.
Let us consider a method of representing segment voxels as GMMs. For this purpose, let's set the set of quadruples of all segment voxels from the dataset : = ( ), ( ), ( ), (( )) ( ) ( ), | = 1,2, … , = 1,2, … , × × , where ( ) is the row number of the pth segment voxel on 3D scan ; ( ) is the column number of the pth segment voxel on 3D scan ; ( )is the number of the 2D image on 3D scan ; ( ) ( ) ( ) is the ( ) voxel intensity value located in the ( )th row, ( )th column and ( )th 2D image on the th 3D scan; is the number of 3D scans in the labeled sample ; , , are the 3D scan dimensions.
An example of segments is shown in Fig. 2. By aggregating segments from all images, their 2D histogram can be constructed. An example of 2D histogram of the location of segments (tumors in the brain) by image space is presented in Fig. 3a. It is worth noting that the pixel intensities were not used for visualization, but only their 2D locations.
This distribution of location and intensity of all voxels in the dataset is modeled as follows via GMM: ~ ℊ( ) ≜ ( ) =
( | , ), ( | , ) = − ( − )
( − ) 1 (2 ) | | 1 2 = 1 (1) (2) where is the vector-quadruple from the dataset ; is the mean vector of the ith normal distribution in the GMM; Σ is the covariance matrix of the ith normal distribution of the model; is the weight of the ith distribution in the model; is the total number of Gaussian distributions in the model.
The parameters of GMM models can be found using the Expectation-Maximization algorithm [14].
An example of GMM representation of the location of aggregated 2D segments from Fig. 3a is shown in Fig. 3b. In 3D, the tumor atlas may look as in Fig. 4, with its modeling via GMM shown in Fig. 5 (no voxel intensity, only location). An additional atlas for voxel estimation will be an averaged 3D atlas of the intensities of all regions. , (3) where ⨀ is the element-by-element multiplication operation; is the 3D tensor of the MRI image scan of size × × mask of size × ×
(height, width, and depth, respectively); is the 3D tensor of the binary with elements taking values 0 or 1; is the number of 3D scans in the
Three SSL training options based on the proposed 4D atlas are proposed: A new loss function that maximizes the likelihood of the location and voxel intensities of unlabeled image pseudomasks and atlas values in the region of the resulting pseudomask. 3. Loss function based on point 1, but likelihood is maximized by location only, and MSE based on the intensity atlas
is used for intensity.
Validation of pseudomasks for atlas-based sampling expansion. 4.2.2
We propose the negative log-likelihood (NLL) of the data under the GMM model as a loss function based on the 4D GMM atlas (Eq. 2). The NLL for a single quadruple of a voxel is: where ⨀ is the element-by-element multiplication operation; pseudo segment (Eq. 3) from a pseudo mask ∈ {0, 1} × × ; ∈ ℝ × × is a generated ∈ ℝ × × is the tensor of the voxel intensity atlas; ( ) the pseudomask values are 1.
∈ ℝ × × are the voxel intensity atlas values only in voxels where To calculate the second option of the loss function, the following steps are performed: Pseudomask calculation
∈ {0,1} × × .
Creating triplets according to Eq. 1, but without pixel intensity.
Calculation of Summation of for each resulting triplet (Eq. 4).
of all triplets (Eq. 5).
Pseudomask segment selection (Eq. 3).
MSE calculation for intensities (Eq. 7).
For a dataset with samples, the total loss is the sum of the NLL over all data points (voxels): = Then to calculate the first option of the loss function the following steps are performed: Then the full proposed loss function would look like Eq. 6, provided that the combined Dice + Binary cross entropy (BCE) loss is used for labeled data: 4.2.3
As a loss function of the second option, MSE is used for voxel intensity:
Pseudomask calculation Creation of quadruples by Eq. 1.
∈ {0,1} × × .
Calculation of Summation of for each obtained quadruple (Eq. 4).
of all quadruples (Eq. 5). = +
( ( )
= ) = ⨀
, ( − ( ) ) (4) (5) (6) (7) = Then the full proposed loss function will look like this: = + + where , , , are the corresponding weights of each component of the loss function, which are included in the formula, they can be specified in advance.
Eq. 8 to the training sample and iterative training of the model.
Pseudomask validation based on the 4D atlas involves adding all masks, voxels preprocessed by ( ) is its probability density from the GMM atlas (Eq. 2); is the probability of an anomaly in a given voxel, calculated by the model segmenter; is the threshold for selecting voxels into the pseudomask.
The training in this case is only on Dice Loss and BCE loss.
The basizc segmenter neural network DeepLabV3+ was used for training. Training was performed on different configurations of the proposed loss functions. The dataset was divided by patient into training, validation and test samples by percentages of 70/10/20 respectively, in number of patients - 23/4/7. Adam optimizer and proposed loss functions were used.
Training the network on the full dataset by Dice Loss gave a baseline IoU metric of 0.93 for the The results of semi-supervised learning of the network by the proposed methods on the test Resulting IOU of the proposed approach on the test sample
Method/Percentage of data used
4D GMM atlas loss 3D GMM loss + voxel intensity loss
Pseudomasks validation
Method [
The results showed that the proposed SSL methods based on the atlas achieved IoU almost similar to that of training on the full dataset. The best performing method was 3D GMM loss + voxel intensity loss, which achieved an accuracy of 0.9 with only 30% of the used data.
Comparison with existing methods [
In this paper, we have introduced an advanced semi-supervised learning framework for brain tumor segmentation that leverages 4D atlas priors to utilize both labeled and unlabeled data effectively. Our approach constructs a probabilistic 4D atlas based on labeled segments, generalizes this atlas using GMM, and incorporates additional dimensions to account for contrast variations within the anomalies.
The proposed method addresses the common limitations found in existing research, which often focus on either shape priors, atlas priors, or semi-supervised learning in isolation. By integrating generalized localization of anomalies, their contrast, and precise boundary delineation into a single framework, we achieve a more comprehensive and robust segmentation solution.
Our experiments on a real MRI dataset of brain tumors demonstrate that the proposed method significantly improves segmentation accuracy compared to existing methods. The inclusion of 4D atlas priors enhances the model's ability to generalize across different types of anomalies, ensuring both anatomical plausibility and precise boundary detection.
Future work will explore further enhancements to the atlas generation process and the integration of additional modules into the neural network to extend the applicability of the proposed method to other medical imaging scenarios. This research contributes to the field of medical image analysis by providing a more effective and generalized approach to semi-supervised segmentation, paving the way for improved diagnostic and treatment planning tools in clinical practice.
[12] He, Yuting & Yang, Guanyu & Yang, Jian & Chen, Yang & Kong, Youyong & Wu, Jiasong & Tang, Lijun & Zhu, Xiaomei & Dillenseger, Jean-Louis & Shao, Pengfei & Zhang, Shaobo & Shu, Huazhong & Coatrieux, Jean-Louis & Li, Shuo. (2020). Dense Biased Networks with Deep Priori Anatomy and Hard Region Adaptation: Semi-supervised Learning for Fine Renal Artery Segmentation. Medical image analysis. 63. 10.1016/j.media.2020.101722. [13] Clough, J.R., Byrne, N., Oksuz, I., Zimmer, V.A., Schnabel, J.A., & King, A.P. (2019). A Topological Loss Function for Deep-Learning Based Image Segmentation Using Persistent Homology. Ieee Transactions on Pattern Analysis and Machine Intelligence, 44, 8766 - 8778. [14] Reynolds, D. (2009). Gaussian Mixture Models. In: Li, S.Z., Jain, A. (eds) Encyclopedia of Biometrics. Springer, Boston, MA. https://doi.org/10.1007/978-0-387-73003-5_196.