Unlabeled data is often abundant in the clinic, making machine learning methods based on semi-supervised learning a good match for this setting. Despite this, they are currently receiving relatively little attention in medical image analysis literature. Instead, most practitioners and researchers focus on supervised or transfer learning approaches. The recently proposed MixMatch and FixMatch algorithms have demonstrated promising results in extracting useful representations while requiring very few labels. Motivated by these recent successes, we apply MixMatch and FixMatch in an ophthalmological diagnostic setting and investigate how they fare against standard transfer learning. We find that both algorithms outperform the transfer learning baseline on all fractions of labelled data. Furthermore, our experiments show that Mean Teacher, which is a component of both algorithms, is not needed for our classification problem, as disabling it leaves the outcome unchanged. Our code is available online: gitlab.com/Valentyn1997/oct_diagn_semi_supervised.
annotation, etc.) result in exponentially more labelling expenses. Additionally, most clinics lack the tools to In recent years deep learning techniques have taken label vast amounts of data. Secondly, and perhaps more the field of AI by storm. Virtually all state-of-the-art fundamentally, there is an epistemic problem in genersystems in computer vision (CV) rely on some form ating accurate labels. For any given diagnostic problem, of deep learning. This paradigm shift has sparked the the inter-expert agreement is well below 100%. This imagination of many practitioners and researchers in discrepancy stems from the fact that medicine is comthe medical image analysis domain. Computer-aided plex and does not always fit neatly into a classification diagnosis appeared to be next-in-line to benefit from formulation. Additionally, each expert comes with his the advancements made in CV, as the amount of data in or her own set of experiences and knowledge. clinical diagnostics is increasing rapidly. The research Instead of solely relying on supervised learning, community has proposed a plethora of new algorithms semi-supervised learning (SSL) should discover the bulk and systems for the automated diagnosis of a wide of the knowledge required for solving a diagnostic task range of diseases. However, clinical adoption has been on its own, with labels only serving as additional guidslow. One crucial reason is that supervised learning, ance. The idea of SSL is to train a machine learning which forms the basis for the vast majority of deep algorithm on vast amounts of unlabeled data and a learning approaches, is ill-suited to the medical domain. small set of labelled samples. SSL is a much better
This mismatch is two-fold. For one, the labelled data match for the clinical setting, as unlabeled data is
ofneeded for supervised learning is prohibitively costly ten abundant since it is acquired as part of the clinical
to generate for medical applications. With a shortage of routine.
medical practitioners, diverting medical experts’ time In this work, we apply two recently proposed SSL
and energy to labelling eforts becomes exceedingly ex- methods, MixMatch [
Semi-supervised learning. State-of-the-art meth- Transfer learning and semi-supervised learning are two
ods for image classification concentrate on finding the main approaches for predictive modelling when dealing
right combination of SSL paradigms. One of the early with data with few labels. Transfer learning approaches
approaches – Mean-Teacher [
Surprisingly, there exists only a little amount of liter- proach, we freeze all parameters except for the final
ature on SSL applied to ophthalmological data. [
(horizontal flips and random crops) for both labelled and unlabeled data.
FixMatch distinguishes between weak and strong data augmentations. Flip-and-shift augmentations are considered as weak augmentations, whereas afine trasformations and color-jittering are examples of strong augmentations (originally – 14 diferent transformations from RandAugment [22]). makes the predictions for the training data and is updated based on the training loss. Both MixMatch and
of pseudo-labels. Note, that keeping a second model in memory and updating its parameters results in higher memory requirements and computation costs.
nique to avoid overfitting. MixUp linearly combines training instance pairs and their prediction targets.
Pseudo-Labelling. Pseudo-Labeling or self-training Therefore it tries to impose linear behaviour between loss [23] is the process of using the trained model to training samples. MixMatch does not diferentiate beobtain labels for unlabeled instances. The predicted tween pseudo targets predicted for the unlabeled inlabels are used to guide the further learning process, stances and ground truth labels and mixes all possie.g. by using generated labels as new targets. ble target pairs. Therefore a resulting instance used
MixMatch applies diferent augmentations for an un- in training may be a combination of two pseudo tarlabeled instance and computes the class distribution gets, two ground truth labels or of pseudo-target with for each augmentation. Therefore, instead of hard one ground truth label. hot label MixMatch defines a probability distribution as the target. To sharpen the distribution and to reduce its entropy, the temperature of distribution is adjusted 4. Experiments [24].
FixMatch uses a “classic” version of pseudo-labelling with hard labels and fixed confidence. The class probability distribution is taken from model outputs after a weak augmentation. If the probability of the most probable class exceeds a predefined threshold the label is assigned to a strongly augmented version of the same instance and used in the loss calculation.
Our work follows the principles of the fair SSL evaluation framework, defined by [ 27]. The authors highlight the importance of using the same classifying model structure for comparison. The evaluation is also meaningful for the real use-case if SSL methods are compared with well-fine-tuned transfer learning and fully supervised models.
lished by Kermany et al. [
Mean Teacher. Another popular consistency re- Test and validation are balanced, there are 8 and 242
quirement in SSL is a similar prediction over time or images per class respectively (see Fig. 1). In our
experipunishing the behaviour when the model changes its ments, we vary the number of labelled data, which we
decisions rapidly. The Mean Teacher algorithm [
Train subset CNV
E DM
DRUSEN
NORMAL
CNV
E DM
DRUSEN
NORMAL
CNV
E DM
DRUSEN
NORMAL DRUSEN
NORMAL [28] backbone. Since the images are monochrome we Method duplicate the channel three times for RGB channels.
For each model, we perform hyperparameter search,
described in Appendix B.1 and B.2. For all experiments, Alqudah [29]
we report the model performance on test data in the
epoch with the lowest validation loss.
4.1. Comparison of transfer learning
and SSL approaches
Kermany et al. [
Accuracy
Notes
All 97.1% Wu et al. [30] All 97.5% Chetoui et al. [31] All 98.46% Tsuji et al.[32] All 99.6% WideResNet-50-2 All 99.69% (our backbone)
Original paper Extended UCSD with 5 classes With EMA decay ( EMA = 0.999) First, in Table 1 we compare the performance of our He et al. [14] 835 87.25±1.44% * *Average precision backbone model trained with all labelled instances to the results reported previously in the literature for the Table 1 same UCSD dataset. As we can see, the backbone model Reported test accuracies for UCSD dataset. Methods have achieves almost perfect performance when trained with tdhifeerepnrtopboascekdboSnLeLs amnedththoudss.areNnevoetrftuhlelylecsos,mopuarrabbelsetwfiutlhlyenough labels. supervised model outperforms previously reported meth
Next, in Fig. 2b we compare two transfer learning ods. approaches. Note, that the hyperparameter search was done for each number of labels for each approach. We discover that, contrary to our expectations, the fine- almost perfect performance. The Fix-Match algorithm tuning variant outperforms feature extraction approach also outperforms Mix-Match in almost all settings and in all label settings. We believe that a thorough selec- with only 50 labelled points per class achieves the action of hyperparameters with representative validation curacy of 98.14%. We also observe a small SSL perforset reduces the risk of overfitting. Furthermore, since mance drop for 25 labelled images per class – mainly the original models are trained on the dataset with RGB because methods require even more epochs to fit (we channels, we believe that the model can better adapt to employ a heuristical formula for defining the maximum the monochrome setting when all model weights are number of epochs based on the number of labels, see allowed to be changed. Appendix B.2, 3).
In the Fig. 2a we present the results of both SSL Finally, since practitioners have often to deal with algorithms and compare them with the best perform- the resource constraints and actual running times are ing transfer learning setting. We find that the SSL ap- rarely reported in the literature, we report them in proaches outperform transfer learning on all fractions Table 2. Note, that all methods are implemented in of labelled data. The gap between SSL and transfer the same framework and the experiments are done on learning widens significantly for smaller fractions of the same machine with two Tesla V100 Nvidia GPUs. labelled data. With only 10 labelled representatives per To use the same batch size as recommended in the class, the FixMatch achieves an accuracy of over 86%, original publications, we have used both GPUs to train while transfer learning reaches only 59%. We also see, Fix-Match. Other models are trained on a single GPU. that with about 2000-4000 labels all methods achieve 1.0 0.9 y c ra ccu0.8 tea s T0.7 0.6 1.0 0.9 y c a rcu0.8 c ta s eT0.7 0.6 Transfer Learning 10m 9m 12m 15m 24m 39m 1h 39m Mix-Match 1d 16h 5m 9h 12m 6h 13m 2h 30m 2h 37m 2h 24m 2h 26m FixMatch 5d 9h 36m 1d 19h 4m 1d 40m 9h 58m 10h 40m 9h 50m 7h 51m
MixMatch and FixMatch, when applied to an ophthalmological diagnostic problem on OCT data. The two algorithms were able to attain high accuracy, achieving well over 80% on as little as 40 labelled samples (i.e. ten per class). Both algorithms outperformed transfer learning in the few labelled data settings. This study emphasizes the use of SSL methods in the clinical adoption of AI. Although both MixMatch and FixMatch are more computationally expensive than transfer learning, the amount of labelling efort saved by using them is immense. With labelling being one of the biggest factors hindering clinical use of AI methodology, we argue that smarter use of the abundance of unlabeled data already present at the clinic will be a major strategy for overcoming this hurdle.
(a) Best models, maximum performance among 4 runs (Semi-supervised) / 8 runs (Transfer learning) per each 40 100 200 800 2000 4000 nl - number of labelled datapoints, log-scaled 20000 (b) Study of layers freezing for Transfer learning, maximum performance among 4 runs per each 4.2. Mean Teacher
rithm and is also optionally recommended for MixMatch. We observe learning curves to be more stable for both train and validation subsets for all the models when models are trained using it. However, we assume that with the right chosen validation subset, the variability could be advantageous and one can find a better ift. Usage of Mean Teacher causes additional computation and memory costs and as can be seen in Table 3 most of the time models without it perform better. 0.0 0.999 0.0 Acknowledgments denotes the set of weak augmen- do not need the linear ramp-up for . As we use for filtering confident pseudo-labels, we
Let = {( , ), ∈ (1, ..., )} be the batch of laand MixUp [26]. Let tations and (⋅) – strong augmentations. ̂ = M( ; )) is the prediction of backbone classifier, parametrized is unsupervised loss weight. by .
(⋅, ⋅) denotes categorical cross-entropy and MixMatch employs only weak augmentations (⋅) = { , ∈ (1, ..., )} be the unlabeled data batch. The model outputs of dom weak augmentations (⋅) of the same unlabelled sample are treated as soft pseudo-labels . These soft pseudo-labels are averaged and sharpened with the rantemperature for each image in to yield a pseudolabel for that image. Then, images from both randomly augmented and are concatenated and shufled, resulting in set . Afterwards, samples in and are weakly augmented and linearly interpolated with samples from . This results in ̂ and ̂ – "mixed-up" versions of augmented labelled and Coeficients of MixUp are sampled from unlabelled batches. tribution. The final loss is the sum of categorical crossifnal unsupervised part of the loss. They are filtered with the threshold . The loss of FixMatch is then the sum of two categorical cross-entropies for labelled and unlabelled images: = 1
∑ (, )∈ +
∑ (,, ̂ )∈̂
H(, M( ( ); ))+
1max( )> H( ̂ , M(( ); )) (2)
B.1. Transfer learning We took a version of Wide ResNet-50-2 pre-trained on ImageNet from PyTorch.2 Transfer learning was much computational budget: ifne-tuned for every individual , as it did not require • learning rate ∈ {1 ∗ 10−3, 5 ∗ 10−4} • optimizer weight decay ∈ {0.0, 0.0001} • layers freezing ∈ {Fine-tuning,
Feature extraction} (see Section 3.1) Beta( , ) dis- use Adam optimizer [33],
= 32, number of epochs = 50. Additionally, early stopping with the patience of 25
Further hyperparameters are kept fixed, namely we entropy for images from ̂ (supervised part) and Brier epochs was applied to avoid overfitting. score for ̂ images (unsupervised part): = 1
∑ (, )∈̂
H(, M( ; ))+ +
∑ (, )∈̂ || − M( ; )||2, (1)
2 MixMatch linearly ramps up after each batch to reduce the influence of unsupervised
from 0 to its maximum part during early stages of training.
FixMatch is a more simplified method. Unlabeled data batch = { ,
∈ (1, ..., bigger. Given the model’s prediction augmented unlabelled sample , method yields hard pseudo-labels ̂ = argmax( ) and ̂ = {( , , ̂ )}.
Afterwards, the model predicts labels for both a batch of weakly augmented labelled images and a batch of strongly augmented unlabelled images. Only the confident predictions for unlabelled samples are used in the
)} is now -times for a weakly
B.2. MixMatch & FixMatch
two-fold: firstly, we fine-tuned more general parameters on 200 labelled samples ( = 200) with respect to the validation loss (see Table 4). Secondly, for each spe
cific , we tuned subset-size-dependent parameters.
The labeled batch size was = 16 for both algorithms. Additionally, we fix = 4, = 0.7 for FixMatch.
of , each epoch becomes proportionally longer. Thus, we propose the following inverse formula to define the number of epochs:
Number of epochs = round ( div ) (3)
Learning rate Grid-search size Primary hyperparameter search grid for SSL methods. Best value is marked with bold font. SGD – stocastic gradient by maximum number of batches in labelled subset. descent with momentum ( = 0.9) [34]. An epoch is defined where while training.
denotes total number of labelled batches, used
• EMA ∈ {0.0, 0.999} (EMA decay) • ∈ {12000, 15000} for MixMatch / ∈ {24000, 30000} for FixMatch