According to recent studies [1,2], brain cancer remains a significant global health concern, accounting for 1.9% of all cancer cases and 2.5% of cancer-related deaths worldwide. In 2019, 347,992 new cases were reported, with higher incidence rates in males (54%) compared to females (46%). The highest age-standardized incidence rates were observed in Europe, while Africa reported the lowest. Notably, Denmark had the highest incidence rate at 17.1 per 100,000 people. The mortality rate also varied significantly, with Palestine reporting the highest at 7.2 per 100,000 people. Trends from 1990 to 2019 indicate a significant increase in incidence globally, highlighting the need for enhanced research and preventive strategies.
Glioblastoma, the most prevalent malignant brain tumor, comprises approximately 49% of all malignant cases. Despite therapeutic advancements, the prognosis for glioblastoma remains poor, with a five-year relative survival rate increasing only slightly from 4% in the mid-1970s to 7% in recent years [2].
In the era of precision medicine, early diagnosis and accurate follow-up are essential for better patient care. In this case, magnetic resonance imaging (MRI) contributes significantly to diagnosis and plays a key role in therapy planning as well as in the assessment of response to treatment and/or relapse.
The main role of conventional/morphologic MRI in making the diagnosis is to determine the size and anatomic location of the lesion in the brain for treatment or biopsy planning, to assess mass effect and edema in surrounding healthy brain tissue, to assess the relationship to ventricular failure of brain systems and vascular structures, and finally, along with other "functional" MRI sequences to suggest a possible diagnosis [3].
The primary task of MRI brain image processing is the segmentation task, which is efficiently solved by deep neural networks. The main problem in training such a network, whether it is a convolutional neural network or a vision transformer, is the availability of a sufficiently labeled training sample. Unfortunately, in real-world situations, the samples are limited and with a small number of labeled scans, since high-quality labeling requires the availability and a huge amount of time of a qualified medical radiologist, which is not always possible.
To address the problem of insufficient sample and limited resources for labeling and training the model, there are different approaches, the most popular ones are:
ο· transfer learning -transferring knowledge from a more general known dataset to a specific limited dataset, ο· active learning -methods for identifying the most relevant data for manual partitioning by a specialist, ο· semi-supervised learning -model training using unlabeled data among others.
Each of the approaches deserves individual attention, but this paper proposes the use and improvement of semi-supervised learning as one of the most promising approaches using unlabeled sampling.
The use of semi-supervised learning in medical image segmentation has several advantages. First, it significantly reduces the need for large amounts of labeled data, which is time consuming and expensive to create. In the medical domain, where labeling requires the expertise of specialists, this is particularly important.
In addition, SSL can improve model quality by using information from unlabeled data without requiring additional datasets or labeling costs. This helps the model to better generalize and more accurately segment medical images, even with a limited amount of labeled data.
Another aspect is the ability to utilize a variety of data. SSL methods can efficiently handle heterogeneous data, which increases their flexibility and applicability in various medical applications.
Ultimately, such methods help accelerate the development and implementation of more accurate and reliable segmentation models in medical systems, which can lead to improved patient diagnosis and treatment.
One type of SSL is knowledge priors (KP) based learning [4].
Prior knowledge is information that the learner already has before learning new information, and sometimes it helps to cope with new tasks. Compared with non-medical images, medical images have many anatomical priors such as shape and position of organs, and incorporating anatomical prior knowledge into deep learning can improve the performance of medical image segmentation.
In this paper, a new SSL method has been developed that uses KP as a 4D anomaly atlas and its generalization in the form of gaussian mixture models (GMM) to capture potential anomalous regions more generally. Novel is the use of not only organ position but also organ contrast, as well as a probabilistic generalization to not be limited to only anomalies captured in a labeled sample. Testing of the approach was performed on a dataset of brain tumors.
In recent years, there has been a significant amount of research in the field of medical segmentation with deep neural networks [5][6][7][8] and semi-supervised medical image segmentation [9][10][11][12][13]. This review focuses on the integration of shape priors, atlas priors, and semi-supervised learning approaches in various methods, highlighting their strengths and weaknesses.
In the paper [9] the authors propose a dual-task framework to improve segmentation performance by leveraging both labeled and unlabeled data. The framework consists of two tasks: a pixel-wise segmentation task and a geometry-aware level set representation task. The dual-task consistency regularization ensures that the predictions from both tasks are consistent, thereby enhancing the model's ability to utilize unlabeled data. This method's strength lies in its ability to incorporate geometric constraints, which helps in achieving more accurate and robust segmentation results. However, the increased model complexity and the need for careful balance between the two tasks can pose challenges in implementation and training.
The [10] paper introduces a novel approach using transformer networks that leverage shape priors through the use of template-based deformable models. This method uses a pre-defined template that captures the general shape of the target organ and deforms it to match the specific instance in the input image. The strength of this approach is its ability to maintain global shape consistency while allowing local variations, which is particularly useful for anatomical structures with high variability. However, the reliance on large amounts of training data and the computational demands of transformer networks can be limiting factors.
In [11] the authors utilize an atlas prior within a GAN framework to enhance liver segmentation. The atlas prior provides a strong anatomical reference, guiding the segmentation process and ensuring anatomical consistency. This approach effectively combines labeled and unlabeled data, improving segmentation accuracy even with limited labeled data. The primary strength of this method is its ability to incorporate detailed anatomical knowledge, which is crucial for accurately segmenting complex organs like the liver. However, the adversarial training process can be unstable and requires careful tuning of hyperparameters.
The paper [12] presents a method that integrates dense networks with deep anatomical priors and region adaptation techniques. This approach is particularly effective for fine segmentation tasks, such as renal artery segmentation, where precise anatomical details are critical. The use of dense networks allows for efficient capture of both local and global features, while the deep anatomical priors ensure anatomical plausibility. However, the model's complexity and the need for extensive labeled data for initial training can be significant drawbacks.
The work on [13] introduces a unique approach to incorporating shape priors through topological constraints. By using persistent homology, the method enforces topological consistency in the segmentation results, which is particularly beneficial for capturing complex anatomical structures. The primary advantage of this approach is its ability to ensure topological correctness in the segmentation output, reducing the likelihood of anatomical errors. However, the computational complexity of calculating persistent homology and the reliance on high-quality topological priors can be limiting factors.
A common limitation across the reviewed papers is their focus on specific aspects of the segmentation problem without simultaneously addressing the proposed generalized localization of anomalies and their pixel values. While each method brings innovative solutions to one or two aspects of the segmentation task, none of them comprehensively integrate all of the critical factors. And sometimes overly strict atlas or shape regularization can degrade the generalizability of the model on new data.
A private MRI dataset of brain tumors of the brain was used for experimentation and validation of the proposed method (Fig. 1). T1 modality and axial view were used. The dataset consists of 34 patients and 1144 images. The labeling was performed manually by medical professionals with more than 10 years of experience. An example of binary labeling of tumors by a specialist is shown in Fig. 2.
For the labeled sample π : (π , π ), β¦ , (π , π ), where π β π Γ Γ is the MRI scan tensor and π β {0, 1} Γ Γ is a binary mask of the same size as the original tensor, where element 0 means absence, and 1 the presence of anomaly on a given voxel of the scan, and the unlabeled sample π : (π , β¦ , π ) create a classifier π(π) that correctly predicts the binary matrix π β {0, 1} Γ Γ of the new scan tensor π β π Γ Γ utilizing both samples π and π .
Based on the works [9-13] about atlas and shape priors, we propose an improvement of this approach in the form of generalizing pixel distributions and adding more dimensions to the atlas.
To account for anatomical structures and the natural location of anomalies, as well as their color, we propose to create 4D atlas priors based on labeled data segments. The first three dimensions of the atlas are the coordinates of the anomaly location in some section of the scan in 3D. The fourth dimension is the observed color or contrast of anomaly pixels.
To generalize the atlas and remove the limitation of the input data, we propose modeling the pixel distributions of a given atlas through GMMs that would completely cover the atlas with centers at the most frequent locations (Figs. 3b and 5). The use of modeling through GMM will help the models generalize better to new data without restricting them to the existing tumor atlas, as GMM assigns the probability of tumor presence to a wider range of pixels, unlike a conventional atlas.
Let us consider a method of representing segment voxels as GMMs. For this purpose, let's set the set of quadruples of all segment voxels from the dataset π : π is the number of 3D scans in the labeled sample π ; π», π, π· are the 3D scan dimensions. An example of segments is shown in Fig. 2. By aggregating segments from all images, their 2D histogram can be constructed. An example of 2D histogram of the location of segments (tumors in the brain) by image space is presented in Fig. 3a. It is worth noting that the pixel intensities were not used for visualization, but only their 2D locations.
This distribution of location and intensity of all voxels in the dataset is modeled as follows via GMM:
where π is the vector-quadruple from the dataset πΊ; π is the mean vector of the ith normal distribution in the GMM; Ξ£ is the covariance matrix of the ith normal distribution of the model; π€ is the weight of the ith distribution in the model; π is the total number of Gaussian distributions in the model.
The parameters of GMM models can be found using the Expectation-Maximization algorithm [14].
An example of GMM representation of the location of aggregated 2D segments from Fig. 3a is shown in Fig. 3b. In 3D, the tumor atlas may look as in Fig. 4, with its modeling via GMM shown in Fig. 5 (no voxel intensity, only location).
where β¨ is the element-by-element multiplication operation; π is the 3D tensor of the MRI image scan of size π» Γ π Γ π· (height, width, and depth, respectively); π is the 3D tensor of the binary mask of size π» Γ π Γ π· with elements taking values 0 or 1; π is the number of 3D scans in the sample.
We propose the negative log-likelihood (NLL) of the data under the GMM model as a loss function based on the 4D GMM atlas (Eq. 2). The NLL for a single quadruple of a voxel is:
For a dataset with π samples, the total loss is the sum of the NLL over all data points (voxels):
Then to calculate the first option of the loss function the following steps are performed:
ο· Creation of quadruples by Eq. 1.
ο· Calculation of ππΏπΏ for each obtained quadruple (Eq. 4). ο· Summation of ππΏπΏ of all quadruples (Eq. 5).
Then the full proposed loss function would look like Eq. 6, provided that the combined Dice + Binary cross entropy (BCE) loss is used for labeled data: πΏ = πΏ + πΏ , πππ πππππππ πππ‘π πΏ , πππ π’ππππππππ πππ‘π (6)
As a loss function of the second option, MSE is used for voxel intensity:
where β¨ is the element-by-element multiplication operation; π β β Γ Γ is a generated pseudo segment (Eq. 3) from a pseudo mask π β {0, 1}
where π is a future voxel of a validated pseudomask π ; π is a quadruplet of voxel i; π (π π ) is its probability density from the GMM atlas (Eq. 2); π is the probability of an anomaly in a given voxel, calculated by the model segmenter; π is the threshold for selecting voxels into the pseudomask.
The training in this case is only on Dice Loss and BCE loss.
The Training the network on the full dataset by Dice Loss gave a baseline IoU metric of 0.93 for the test sample.
The results of semi-supervised learning of the network by the proposed methods on the test sample are presented in Table 1. The results showed that the proposed SSL methods based on the atlas achieved IoU almost similar to that of training on the full dataset. The best performing method was 3D GMM loss + voxel intensity loss, which achieved an accuracy of 0.9 with only 30% of the used data.
Comparison with existing methods [10,13] showed superior segmentation accuracy of the proposed methods and an overall gain of 0.1 IoU, which confirms the validity of the developed method.
In this paper, we have introduced an advanced semi-supervised learning framework for brain tumor segmentation that leverages 4D atlas priors to utilize both labeled and unlabeled data effectively. Our approach constructs a probabilistic 4D atlas based on labeled segments, generalizes this atlas using GMM, and incorporates additional dimensions to account for contrast variations within the anomalies.
The proposed method addresses the common limitations found in existing research, which often focus on either shape priors, atlas priors, or semi-supervised learning in isolation. By integrating generalized localization of anomalies, their contrast, and precise boundary delineation into a single framework, we achieve a more comprehensive and robust segmentation solution.
Our experiments on a real MRI dataset of brain tumors demonstrate that the proposed method significantly improves segmentation accuracy compared to existing methods. The inclusion of 4D atlas priors enhances the model's ability to generalize across different types of anomalies, ensuring both anatomical plausibility and precise boundary detection.
Future work will explore further enhancements to the atlas generation process and the integration of additional modules into the neural network to extend the applicability of the proposed method to other medical imaging scenarios. This research contributes to the field of medical image analysis by providing a more effective and generalized approach to semi-supervised segmentation, paving the way for improved diagnostic and treatment planning tools in clinical practice.