Cardiac magnetic resonance imaging (MRI) is vital for diagnosing cardiovascular diseases, yet its classification faces significant challenges, including domain shifts across heterogeneous datasets, limited annotated data, and privacy regulations, particularly when deploying deep learning models on resource-constrained edge devices. These issues hinder real-time diagnostics and data protection in edge-computing scenarios. To address these issues, this paper introduces Enhanced Multi-Teacher Knowledge Distillation (EMTKD), an edge-based framework designed to enhance cardiac MRI classification. EMTKD employs adaptive multi-teacher knowledge distillation, utilizing an adaptive weighting mechanism to balance contributions from multiple teacher models trained on diverse domains, ensuring the student model learns from the most reliable sources. It integrates domain adaptation to align feature representations, semi-supervised learning with pseudo-labeling to leverage unannotated data, and diferential privacy to safeguard patient information. Experimental results show EMTKD achieves an accuracy of 88.5% and an area under the curve of 92.5% on target domain images, outperforming state-of-the-art baseline models. Ablation studies highlight the critical roles of adaptive weighting, domain adaptation, and semi-supervised learning in improving performance under edge constraints. These findings demonstrate EMTKD's ability to generalize across domains and utilize limited data efectively. Overall, EMTKD ofers a robust, privacy-preserving solution for cardiac MRI classification on edge devices, ideal for rapid diagnostics in challenging environments.
Cardiovascular diseases remain the foremost cause of mortality worldwide, creating an urgent need for precise, timely diagnoses to improve patient outcomes. Cardiac magnetic resonance imaging (MRI) has become a crucial non-invasive modality for examining cardiac structure, function, and pathology [1]. It delivers the high-resolution information necessary for detecting conditions such as myocardial infarction, cardiomyopathies, and congenital heart diseases [2]. However, interpreting cardiac MRI scans manually is time-intensive and requires substantial expertise [3], which is not always available in every clinical setting. Consequently, automated techniques are needed to streamline MRI data analysis and assist clinicians with accurate interpretations [4].
Despite progress in deep learning (DL) and machine learning for medical image analysis, processing cardiac MRI data still faces two major hurdles. First is the heterogeneity of data sources [5]: scans acquired via diferent imaging protocols and equipment often exhibit domain shifts [ 6]. Consequently, models trained on one dataset tend to underperform when evaluated on unseen data. Second, annotated datasets are scarce; labeling MRI scans demands substantial expert efort [ 7], meaning large-scale labeled data are seldom available [8]. Additionally, privacy regulations prevent the free exchange of clinical data between institutions, further hindering the creation of comprehensive training sets.
Emerging strategies like knowledge integration [9] and knowledge distillation [10] ofer ways to address these challenges. Knowledge integration merges information from varied origins to bolster model learning [11], while knowledge distillation transfers insights from one or more teacher models to a student model [12]. In cardiac MRI, these methods help build models that generalize better and reduce the requirement for extensive annotation [13]. By harnessing knowledge distilled from multiple teachers trained on diverse data, the student model captures a broader range of features [14], boosting overall classification performance on new samples.
In light of these shortcomings – particularly domain generalization [15] and data scarcity [16] – this study focuses on distilling knowledge from multiple teacher models to establish a robust student model capable of operating efectively under heterogeneous conditions. The main contributions of this work are as follows: • Adaptive Teacher Weighting Mechanism: Employs confidence calibration to dynamically balance each teacher’s impact on the student model. • Domain-Adaptive Feature Alignment: Extracts domain-invariant features from teacher models, enhancing the student model’s cross-domain generalization. • Semi-Supervised Learning (SSL) Integration: Utilizes pseudo-labeling for unannotated data, reducing dependence on large annotated datasets. • Privacy-Preserving Techniques: Implements diferential privacy to protect sensitive information during training.
By tackling key limitations in cardiac MRI analysis, this research aims to develop DL models that are not only accurate and generalizable but also aligned with essential privacy requirements.
The rest of the manuscript is structured as follows. Section 2 reviews key studies on DL in medical imaging, highlighting limitations in knowledge distillation, domain adaptation, semi-supervised learning, and privacy preservation—underscoring the impetus for a more holistic framework. Section 3 presents the authors’ proposed EMTKD solution, detailing the integration of domain-adaptive teacher models, an adaptive weighting mechanism, SSL with pseudo-labeling, and diferential privacy to address domain shift, data scarcity, and confidentiality concerns. Section 4 showcases the experimental results and discussion, comparing the new method against baseline and state-of-the-art techniques using standard evaluation metrics. Finally, section 5 concludes on the eficacy and broader applicability of the EMTKD approach.
Over recent times, DL-based solutions have increasingly permeated medical imaging, yielding progress in classification, segmentation, and detection tasks. However, new methods are needed to resolve outstanding issues in cardiac MRI processing.
Knowledge distillation has been a prominent direction: Hinton et al. [10] introduced the idea of transferring a larger model’s knowledge to a smaller student network by matching outputs. Chen et al. [17] later proposed multi-teacher knowledge distillation, where each teacher is trained on a distinct dataset, and the student learns from the collective. Although beneficial, combining knowledge from heterogeneous data sources remains dificult because domain shifts among datasets may create conflicting information.
Domain adaptation has likewise garnered attention. Ganin and Lempitsky [18] explored domainadversarial training to produce domain-invariant representations, and Zeng et al. [19] employed an unsupervised strategy for cross-modality medical image segmentation. Although these methods improve generalization, they often presume direct access to both source and target domains – an assumption complicated by privacy constraints.
To counteract limited annotations, SSL approaches like that of Yuan et al. [20] employ minimal labeled data combined with substantial unlabeled data. Their teacher-student structure uses pseudo-labeling to leverage unannotated samples. Yet, they do not incorporate multi-teacher integration or address domain shifts.
Privacy-preserving methods, such as adding diferential privacy noise to training gradients, have also emerged in medical imaging. Montenegro et al. [21] demonstrated that this approach reduces the likelihood of memorizing individual data points. However, fusing diferential privacy with complex training architectures, including knowledge distillation, remains challenging [22].
A more recent approach by Nabavi et al. [23] applies a multi-teacher, single-student framework to address domain shift and data scarcity. While it preserves privacy by requiring only teacher parameters, its performance may deteriorate if confronted with entirely novel data distributions. Some knowledge distillation gaps in cardiac MRI classification thus remain unsolved. Specifically, persistent obstacles include: • Dificulty Integrating Multiple Teachers: Domain shifts across heterogeneous datasets hinder the student model’s consistency. • Dependence on Source Domain Data: Domain adaptation methods sometimes require both source and target data, which conflicts with privacy standards. • Limited Utilization of Unannotated Data: Scarce labeled samples restrict model capacity, and existing methods do not fully exploit unlabeled data. • Inadequate Privacy Measures: Knowledge distillation frameworks may overlook explicit safeguards for patient confidentiality.
Hence, the proposed solution addresses these gaps by embedding domain adaptation within teacher models to extract domain-invariant features. An adaptive weighting mechanism guided by confidence calibration ensures the student model assimilates knowledge from multiple teachers, and SSL broadens the training set with unlabeled samples. Lastly, diferential privacy is woven into training to keep patient data confidential.
Accordingly, this study proposes an enhanced multi-teacher knowledge distillation (EMTKD) approach for cardiac MRI classification that integrates knowledge from diverse datasets, confronts domain shifts, and alleviates data scarcity. The corresponding tasks are: • Task 1: Train domain-adapted teacher models on diferent cardiac MRI datasets. • Task 2: Devise an adaptive distillation protocol to unify knowledge from multiple teachers into one student. • Task 3: Employ SSL to exploit unlabeled data, advancing the student model’s learning potential. • Task 4: Incorporate privacy-preserving measures, including diferential privacy, to secure sensitive information. • Task 5: Compare the resulting method against existing approaches using standard cardiac MRI datasets.
By merging these techniques, the proposed approach aims to deliver robust, domain-spanning classification performance even under limited annotation conditions.
The proposed approach aims to improve cardiac MRI image classification by efectively distilling knowledge from multiple teacher models into a single student model. The approach is formalized as a step-by-step method that addresses challenges such as domain shift, data heterogeneity, limited annotated data, and privacy concerns.
The method comprises three main blocks: • Block 1: Teacher model training with domain adaptation. • Block 2: Adaptive knowledge distillation. • Block 3: Student model training with privacy preservation.
Inpu.t..Data xi' ... xi'' ... ...
...
F
Attention , k r o w t enn MLP o it a g e r g g a reu Pooling t a e F
MLP s s o l n o iillitt a s D xi'
Finetuning
42–57
T r a n s f e r l e a r n i n g
The schematic representation of the proposed approach is illustrated in figure 1.
Each block involves specific inputs, processes, and outputs that collectively contribute to the final goal of enhancing the student model’s performance.
The input data of block 1 consists of annotated datasets from multiple domains. Each dataset is represented as () = {(x(), (), ())}=1 for = 1, 2, . . . , , where x() is the -th cardiac MRI image from domain , () is the corresponding label (such as “normal” or “pathological”), and () is the domain identifier. The total number of samples in dataset is denoted by , and represents the total number of teacher models or domains involved.
Step 1. Data preparation The first step in the process is collecting and preparing the data. Datasets are gathered from diferent domains, each containing annotated cardiac MRI scans. Once the datasets are collected, preprocessing begins. The images are normalized, resized to a consistent dimension, and undergo necessary augmentations to enhance model performance. These preprocessing techniques ensure uniformity across the datasets and help improve the model’s generalization capabilities. Step 2. Model initialization With the data prepared, the next step is to initialize the teacher models. For each domain , a neural network model is initialized with random weights. Additionally, for each model , a domain discriminator network is defined. This domain discriminator plays a critical role in domain adaptation, allowing the models to learn domain-invariant features that can generalize well across multiple domains. Step 3. Training with domain adaptation During training, two key loss functions are employed. The first is the classification loss for model which is calculated as:
ℒ(CE) = − 1 ∑=︁1 () log (x()), where () represents the true label for sample and (x ()) is the predicted probability distribution over the classes.
The second loss function is the domain adaptation loss, which uses a Gradient Reversal Layer (GRL) to promote learning domain-invariant features. The domain adaptation loss is computed as: ℒ(DA) = − 1 ∑=︁1 () log ((x())), where () is the domain label and represents the domain discriminator network.
These two loss functions, formalized by formulas (
() ()
where DA is a hyperparameter that balances the classification and domain adaptation losses.
42–57
,
(
This optimization process by formula (
z() = (x).
These feature vectors by (
The process of block 2 begins with the input data, which includes trained teacher models {}=1 and their corresponding feature vectors {z()}=1, all derived from multiple domains. Additionally, we have the target dataset samples (), which consist of both annotated and unannotated examples. The input data for the target domain includes samples x(), where = 1, 2, . . . , , with = + , where and represent the number of annotated and unannotated samples, respectively. Step 1. Compute teacher predictions is represented mathematically as: The first step involves computing the predictions from the teacher models. For each teacher model and target sample x (), the SoftMax function is applied to obtain predicted probabilities p(x ()). This
In addition, confidence scores for the predictions are calculated by taking the maximum probability p(x
()) from formula (
()) represents the confidence of the most probable class predicted by the teacher model.
Step 2. Calculate adaptive weights
(
())). (x()) = max p(x ︁(
()))︁ , (x()) = =1 ︁(
︁( exp (x ())/
︁) ∑︀ exp (x ())/ ︁) , sharpens the distribution, placing more emphasis on the higher confidence scores.
Step 3. Weighted feature aggregation vector z()(x
̃︀
()):
In the next step, weighted feature aggregation is performed. For each teacher model , the weighted
feature vector z()(x
()) is computed by multiplying the adaptive weight (x
()) in (
These weighted features, formalized by (
Z(x()) = [︁z(
̃︀
), z(
This concatenated feature vector (
The attention mechanism ensures that the most relevant features from the teacher models are emphasized during the aggregation process.
Step 4. Store aggregated features
The final step involves storing the aggregated feature representations zagg(x()) by (
The output data of block 2 is a set of distilled and aggregated feature representations zagg for each sample x(). These features, now distilled from the teacher models, serve as targets for the student model training in block 3, enabling the student model to learn from the most relevant and informative features across multiple domains.
The input data for this training process includes aggregated features {zagg(x())} derived from previous stages, as well as target dataset samples {x()} from the dataset (). Additionally, there is an annotated subset () = {(x(), ())}=1, which consists of labeled data used for supervised learning. Step 1. Model initialization The first step involves initializing the student model. A neural network is defined with parameters . Depending on the setup, these parameters can either be initialized randomly or use pre-trained weights. This model will learn from labeled and unlabeled data using a combination of techniques. Step 2. Compute student features In this step, the student model processes the target dataset. For each sample x() from the target dataset, a forward pass is conducted through the student model, producing feature vectors z(x()) as follows: ̂︀ ̂z︀(x()) = (x()).
These feature vectors by (
ℒdistill(x()) = ⃦⃦⃦ ̂z︀(x()) − zagg(x())⃦⃦⃦ 22 + regℛ( ), where ℛ( ) is a regularization term (e.g., L2 regularization on the model’s parameters) and reg controls the strength of this regularization.
Step 4. SSL with pseudo-labels For the unlabeled samples x(), where = + 1, . . . , , pseudo-labels are generated by the student model. First, the predicted probabilities are computed using a SoftMax function applied to the student’s classifier: p(x()) = Softmax(( (x ()))).
Pseudo-labels are then assigned to samples where the maximum probability exceeds a confidence threshold : ˆ() = {︃ arg max p(x()), if max p(x()) > ,
unlabeled, otherwise.
For the samples assigned pseudo-labels, a pseudo-labeling loss is computed as:
1 ∑︁ ˆ() log ( (x())),
ℒpseudo = − =1
where is the number of pseudo-labeled samples.
(
Step 7. Privacy-preserving training Diferentiated privacy techniques are applied to ensure privacy during training by adding Gaussian noise to the gradients. The noisy gradients are computed as:
∇ ℒtnootiasyl = ∇ ℒtotal + (0, 2), where (0, 2) is Gaussian noise with mean 0 and variance 2, and controls the level of privacy. Step 8. Parameter update Using the noisy gradients, the student model’s parameters are updated through gradient descent: () represents the true label for sample . where Step 6. Total loss computation The total loss for training the student model is computed by combining the distillation loss, classification loss, and pseudo-labeling loss:
← − ∇ ℒtnootiasyl, where is the learning rate. This ensures the model learns while maintaining diferential privacy. Step 9. Fine-tuning After the initial training phase, the student model is fine-tuned on the annotated dataset () to improve classification performance. The loss used for fine-tuning is: Step 5. Compute classification loss on annotated data Next, the classification loss is computed for the annotated subset (). This loss is based on the cross-entropy between the predicted probabilities and the true labels:
ℒ(CE) = − 1 ∑=︁1 () log ( (x())),
ℒfine = ℒ(CE) + regℛ( ), and the parameters are updated accordingly to refine the model further.
The final output of this process, formalized by formulas (
In this research, two publicly available cardiac MRI datasets that exhibit significant domain diferences were utilized to validate the efectiveness of our proposed approach.
Dataset A originates from the Automated Cardiac Diagnosis Challenge (ACDC) [24], which comprises
cardiac MRI scans collected from multiple clinical centers using diferent MRI scanners and imaging
42–57
(
• Annotations: Diagnostic labels and segmentation masks.
Dataset B is sourced from the Multi-Center, Multi-Vendor, and Multi-Disease Cardiac Image Dataset (M&Ms) [25]. This dataset is characterized by its diversity in terms of scanner vendors, magnetic field strengths, and patient demographics. It includes patients with conditions such as myocardial infarction and healthy controls, ofering a domain diferent from Dataset A. The technical details of Dataset B are as follows: • Number of subjects: 160 patients. • Image resolution: Varies due to diferent scanner settings.
• Annotations: Diagnostic labels provided by experienced radiologists.
Specifically, the primary task is to classify cardiac MRI scans into diagnostic categories using the proposed approach. The significant domain shift between Dataset A and Dataset B poses a challenge for models to generalize across datasets. By treating one dataset as the source domain and the other as the target domain, we aim to evaluate the method’s ability to handle domain heterogeneity and limited annotations.
NumberofCorrectPredictions Accuracy =
TotalNumberofPredictions Next, Precision evaluates the correctness of positive predictions: .
To assess the performance of the classification models, several standard classification metrics [ 26] were employed.
Firstly, Accuracy measures the proportion of correctly classified instances over the total number of instances:
Recall measures the model’s ability to identify all relevant instances:
Precision =
TruePositives TruePositives + FalsePositives
.
Recall =
TruePositives TruePositives + FalseNegatives
.
F1− score = 2 ×
Precision × Recall Precision + Recall .
(
This section presents the evaluation of EMTKD applied to cardiac MRI classification. Here, we also present a detailed analysis of the experimental results obtained using EMTKD on the cardiac MRI classification task. The proposed approach was compared with several baseline and state-of-the-art approaches, providing quantitative results to assess the performance comprehensively.
Computational experiments were conducted using the two cardiac MRI datasets described in subsection 3.4.
• Dataset A (Source Domain): ACDC dataset [24].
• Dataset B (Target Domain): M&Ms dataset [25].
We divided each dataset into training, validation, and test sets, ensuring patient-level separation to prevent data leakage. The teacher models were trained on dataset A, incorporating domain adaptation techniques to learn domain-invariant features. The student model was trained on Dataset B using our proposed EMTKD, leveraging both annotated and unannotated data.
The proposed approach was also compared against the following approaches: • Baseline 1: Single Teacher Model (STM) trained without knowledge distillation. • Baseline 2: Multi-Teacher Knowledge Distillation (MTKD) without adaptive weighting or domain adaptation [17]. • Baseline 3: Domain-Adversarial Neural Network (DANN) [18]. • State-of-the-art method 1: Boosting Medical Image SSL with Adaptive Pseudo Labeling and
Informative Active Annotation (BoostMIS) [22]. • State-of-the-Art method 2: Multiple Teachers-Meticulous Student (MTMS) from recent literature [23].
All models were trained and evaluated under identical conditions for fair comparison.
We evaluated the models using the metrics (
Accuracy (%) Precision (%) Recall (%) F1-Score (%) AUC-ROC (%)
Our proposed EMTKD achieved the highest performance on the source domain, with an accuracy of 95.3% and an AUC-ROC of 97.5%. This indicates that our method efectively captures the essential features necessary for accurate classification.
Table 2 shows the performance of the models on the test set of dataset B, which represents a diferent domain due to variations in imaging protocols and equipment.
EMTKD significantly outperformed all other models on the target domain, achieving an accuracy of 88.5% and an AUC-ROC of 92.5%. This demonstrates the ability EMTKD to generalize across domains efectively.
Based on the obtained numerical results (table 2), STM trained without knowledge distillation performed poorly on the target domain, with an accuracy of 71.0%. This highlights the limitations of models trained solely on source domain data when applied to a diferent domain. Incorporating multiple teachers in MTKD without adaptive weighting or domain adaptation marginally improved performance to 74.2% accuracy. However, the lack of mechanisms to handle domain shifts limited its efectiveness. Furthermore, DANN improved accuracy to 79.0% by learning domain-invariant features. This indicates the importance of domain adaptation in handling domain shifts.
Speaking of state-of-the-art approaches, we may observe the following. BoostMIS achieved an accuracy of 80.5%. While it leveraged unannotated data, it could not efectively integrate knowledge from multiple domains. The MTMS method reached 84.0% accuracy. It combined domain adaptation with knowledge distillation but did not employ adaptive weighting or advanced semi-supervised techniques.
In contrast, our EMTKD surpassed all the methods compared, demonstrating the following advantages. • Adaptive weighting: The student model efectively integrated diverse knowledge by dynamically weighting teacher contributions based on confidence. • Domain adaptation: Incorporating domain adaptation in teacher models enabled the extraction of domain-invariant features, which is crucial for generalization. • SSL: Utilizing pseudo-labels for unannotated data expanded the training set, enhancing the model’s learning capacity. • Privacy preservation: Including privacy-preserving techniques ensured data confidentiality without compromising performance.
Next, the confusion matrices provide insights into the models’ classification performance. Figure 2 demonstrates the confusion matrix for EMTKD on dataset B.
l a u t c A l a m r o N l a c i g o l o h t a P 480 15 20 485 Normal
Pathological
Predicted
Our EMTKD correctly classified 480 out of 500 normal cases and 485 out of 500 pathological cases, indicating high sensitivity and specificity.
To assess the contribution of each component, we conducted ablation studies by systematically removing elements from EMTKD.
Firstly, we evaluated the model without the adaptive weighting mechanism (EMTKD w/o AW). The evaluation results are demonstrated in table 3.
Accuracy (%) Precision (%) Recall (%) F1-Score (%) AUC-ROC (%)
Accuracy (%) Precision (%) Recall (%) F1-Score (%) AUC-ROC (%)
The absence of SSL led to a 3% decrease in accuracy, indicating that leveraging unannotated data enhances model performance.
Finally, we analyzed the computational requirements of our method compared to others (table 6).
While EMTKD required more training time and had a higher parameter count, the performance gains justify the additional computational cost.
Moreover, we used t-distributed stochastic neighbor embedding (t-SNE) [27] to visualize the feature representations learned by the models (figure 3).
MTMS EMTKD Dataset A Dataset B Dataset A
Dataset B 10
0 10 Dimension 1
(a) 30 20 20 30 40 20 10
0 10 Dimension 1 20
30 (b)
The adaptive weighting mechanism based on confidence calibration was pivotal in managing the variability among teacher models. By assigning higher weights to teachers with greater confidence on particular samples, the student model efectively consolidated knowledge from multiple sources, resulting in stronger generalization across diferent domains. Incorporating domain adaptation techniques within each teacher model further supported this process, as domain-invariant features were more readily extracted, ultimately allowing the student model to handle the shift between dataset A and dataset B without requiring annotated data from the target domain. Including SSL, using pseudo-labels for unannotated samples, also proved beneficial, particularly when annotated data were scarce; this expansion of the training set led to marked improvements in classification performance.
Privacy considerations were addressed by introducing noise into model gradients, ensuring that patient-specific details did not become encoded in the model weights. This strategy preserved confidentiality without diminishing classification accuracy or recall. Compared with state-of-the-art methods, EMTKD demonstrated notable gains, particularly regarding accuracy and resilience to domain shifts. The combination of adaptive weighting, domain adaptation, SSL, and privacy-preserving measures provided a multifaceted framework that overcame some of the most pressing challenges in cardiac MRI analysis. Nonetheless, certain constraints remain: the approach demands significant computational resources due to the training of multiple teacher models and a student model; the success of distillation hinges on the caliber of the teacher models themselves; and the confidence threshold used in pseudo-labelling requires meticulous calibration to minimize the risk of noisy labels. Avenues for future exploration could involve optimizing computational requirements, devising methods for on-thelfy teacher selection based on real-time performance metrics, and pursuing more advanced privacy techniques that ofer even stronger protections. In conclusion, the experiments afirmed the value of the proposed EMTKD in elevating classification performance on cardiac MRI data while underscoring its strengths and the potential areas for advancement in real-world clinical applications.
This study aims to overcome major obstacles in cardiac MRI classification by developing a comprehensive approach called EMTKD that seamlessly addresses domain shift, dataset heterogeneity, limited annotated data, and privacy requirements. The proposed step-by-step EMTKD operates by first training domainadapted teacher models to extract consistent features, then employing confidence-based weighting and attention mechanisms to transfer knowledge into a student model, and finally integrating semisupervised and privacy-preserving techniques to refine the student’s learned representations further. Experimental results showed clear gains over baseline and state-of-the-art models, particularly in challenging domain-shift scenarios, with accuracy on the target dataset reaching 88.5% and AUC-ROC values surpassing 92%. Although EMTKD demands more computational power and relies on high-caliber teacher models, it provides a robust mechanism for balancing knowledge across multiple domains and leveraging unannotated data. In its current form, the approach has shown promising results for large-scale clinical deployment, yet certain constraints must be recognized, particularly the elevated computational costs and the reliance on threshold tuning for pseudo-labeling.
Future research will be directed toward reducing computational overhead through model compression or more eficient network architectures, refining strategies for dynamically assessing teacher performance, and exploring advanced privacy-preserving methodologies that maintain high diagnostic accuracy while safeguarding sensitive patient data.
Declaration on Generative AI: During the preparation of this work, the authors used GPT-4o in order to: Drafting content, Generate literature review, Paraphrase and reword, Improve writing style, Abstract drafting, Grammar and spelling check, Content enhancement. After using this service, the authors reviewed and edited the content as needed and takes full responsibility for the publication’s content.