Training deep neural networks with noisy supervision remains a challenging problem in weakly supervised learning. Mislabeled instances can severely degrade the generalization ability of classification models to unseen data. In this paper, we propose a novel regularization method coined Noise-robust Distillation (NRD) that addresses robust training under noisy supervision. NRD is motivated from a novel learning framework which we name Neural Vicinal Risk (NVR) minimization to improve the estimation quality of the data distribution and handle label noise efectively. Our framework is based upon our observation that a neural network has capability to correctly classify data sampled from vicinal distribution even when the model is overfitted to noisy label. By ensembling the predictions from the neural vicinal distribution, we obtain an accurate estimation of the class probabilities that reflects sample-wise class ambiguity. We validated our method through various noisy label benchmarks and demonstrate significant improvement in robustness to label noise.
Deep learning models have achieved remarkable success in various domains, including image classification, natural language processing, and speech recognition. However, the performance of these models heavily relies on the availability of high-quality labeled data for training. Obtaining accurately annotated labels can be a challenging and time-consuming task, often requiring human annotators to manually label large amounts of data. As a result, noisy labels may arise during the annotation process, leading to suboptimal model performance.
In this paper, we address noisy label learning as a subset
of a more generic type of problem. This encompasses
learning from an over-confident target probability distribution
and image ambiguity [
We propose a noise-robust learning algorithm named Noise-Robust Distillation (NRD) to address the issue of noisy supervision during training. NRD aims to improve the generalization performance of classification models by explicitly considering the noise and ambiguity in the training labels. We motivate NRD by a novel formulation of the noisy supervision learning problem which we name Neural Vicinal Risk (NVR) minimization.
This stems from the observation that deep neural networks have the inherent capability to detect and correct noisy supervision, even when it is trained using noisy supervision. This ability is particularly evident when considering the vicinal distribution, which represents the distribution generated from perturbed versions of the training data. Despite being trained on noisy labels, neural networks can still
AUROC: 0.9935 y t i s n e D 2
1 0
10 20 30
GT class log-likelihood accurately model the vicinal distribution, indicating their potential to correct the noisy supervision.
Our findings suggest that the combination of perturbation-based estimation and ensembling can lead to improved model performance, even in the presence of noisy supervision. Building on these insights, we propose Noise-Robust Distillation (NRD), which is a noise-robust learning method that leverages the neural vicinal risk principle to enhance the generalization performance of classification models trained on noisy labels.
The main contributions of this work are as follows: • We introduce the Noise-Robust Distillation (NRD), a noise-robust learning approach that comprehensively addresses the challenges posed by noisy supervision during training. • NRD is motivated by a novel noise-robust learning framework which we name Neural Vicinal Risk (NVR) minimization. We show that NVR improves the estimation quality of the true class distribution and handles label noise efectively. • We demonstrate the ability of neural networks to horse ship frog ship frog horse truck airplane automobile ship truck airplane label, gt automobile ship label, gt truck airplane automobile ship labbeirl,dgt cat horse frog bird
horse cat frog detect and correct mislabeled examples through sensitivity to perturbations in the input data, leading to improved model predictions and calibration. • We validate the efectiveness of NRD through experiments on benchmark datasets, showing clear improvements in model performance in comparison to standard training methods under noisy supervision.
Noisy label learning Numerous methods tackle the
challenge of training Deep Neural Networks (DNNs) on datasets
that contain a mix of correctly labeled and mislabeled
samples, as discussed in [
Semi-supervised learning (SSL) has emerged as a
powerful method for noisy label learning. Among them,
consistency regularization promotes a model to make consistent
outputs across data augmentations, as in Π -model,
Temporal Ensembling [
Calibration and knowledge distillation Confidence
calibration [
Consider a DNN classification model parameterized by ∈ Θ as (, ) : ↦→ ∆ − 1 which outputs a probability distribution (|; ). The input space is defined as = R× × where , , are the number of height, width, and color channels of the image data. ∆ indicates -simplex. The model takes an image input ∈ and predicts a categorical distribution over = {1, 2, ..., }. We denote an image augmentation operation as () : → , and the training dataset as = {(, )}. The loss function is defined as ℓ(, , ) : × × Θ ↦→ R. (· ) is the Dirac delta function and 1{·} is the indicator function.
The expected risk ( ) is defined as the average loss over (, ), ∫︁
, ( ) = ℓ(, , )(, ) . In practice, a dataset is used to mimic the true distribution (, ), which leads to the empirical risk ˆ( ) = ∫︁ ,
ℓ(, , )ˆ(, ) . where the corresponding empirical distribution ˆ(, ) is a mixture of delta masses using the observed samples, and the class distribution is a one-hot distribution given by annotations, ˆ(, ) =
1 ∑︁ 1{=} ( − ) .
=1
Our goal is to refine the estimation of the data distribution (, ) by utilizing the empirical distribution ˆ(, ).
A pivotal question that arises is how to enhance the
approximation of the true risk ( ) intrinsic to a classification
model. As evidenced by Equation (3), this task
necessitates the accurate estimation of two orthogonal components
present within the true distribution (, ) = (|)():
(1) the input distribution () and (2) the corresponding
conditional distribution (|).
3.3. Neural Empirical Risk
Estimating (|) as a one-hot distribution involves
assigning a single class label per sample, which is vulnerable
to human annotation errors. Unfortunately, it proves
challenging to enhance or secure accurate supervision signals
for (|), as this requires multiple human annotators
reviewing the same image [
Neural Empirical Risk (NER) Instead of using Equation (3), we can choose to parameterize (|) by a neural network (|, ) to further improve the estimation quality. First, we factorize the data distribution as (, ) = (|)(), and denote the corresponding empirical distributions as follows:
ˆ() = 1 ∑︁ ( − )
=1 ˆ (|) = 1{=} .
Instead of using ˆ (|), we choose to use a distribution parameterized by a neural network trained on , (|, ) =
(|, )(|) ,
∫︁
in knowledge distillation, which we view as an instance
of NER minimization. Knowledge distillation is known to
improve generalization and calibration performance due to
the dark knowledge [
However, when the model is over-fitted to the noisy label,
it severely degrades the performance of estimating the class
probabilities. Hence, in order to efectively utilize a neural
network, it is necessary to employ a noise-robust method to
accurately estimate the class probabilities in the presence
of noisy labels.
3.4. Vicinal risk for noise-robust learning
Our motivation is based on the Vicinal Risk Minimization
(VRM) principle [
(9) ∫︁ , ∫︁ where (˜, ˜|, ) is the vicinity distribution around (, ).
For example, [
MixUp [
(˜, ˜|, )ˆ(, ) =1 ,
= 1 ∑︁ (˜, ˜|, ) .
Neural Vicinal Risk (NVR) We propose to further improve by using a neural network to robustly approximate the data distribution by modifying Equation (9). We propose the following approximate vicinal data distribution parameterized by a deep neural network which we name neural vicinal distribution . where (|) is the distribution over the function class parameterized by neural network. By plugging Equation (6) into ˆ(, ) = ˆ (|)ˆ(), we define the neural empirical distribution ˆ and the neural empirical risk ˆ as
ˆ () = ˆ (, |) = (|, )ˆ() ∫︁ ,
ℓ(, , )ˆ (, |) .
Here, we refer to the model (|, ) as the teacher network to distinguish from the model being trained, whose term is borrowed from knowledge distillation. This can provide better estimation quality than ˆ (|) as is often observed ≈ = ∫︁ ∫︁ ∫︁ (˜|˜, )(|)
(˜|)() (˜|˜, ) ( − * )
(˜|)^() (14) = (˜|˜, * ) 1 ∑︁ (˜|)
=1 = 1 ∑︁ (˜|˜, * ) (˜|) .
=1 (˜, ˜|) = (˜|˜; )(˜)
∫︁ Here, * = arg min ˆ() is the maximum-a-posteriori (MAP) model trained on . It is important to note that the samples from the vicinal distribution (˜|) is not shown at the training of the model * . Equation (14) is given by substituting the Bayesian model with the MAP model and also replacing the true distribution () with the empirical distribution. The true neural vicinal distribution is approximated by the ensembled MAP model predictions averaged over the samples from the vicinal distribution.
Therefore, we define the empirical neural vicinal distribution ˆ as,
ˆ (˜, ˜; * ) = 1 ∑︁ (˜|˜, * ) (˜|) .
=1 (17) augmentation
Student augmentation
EMA update stop-grad (student augmentation) is regularized using the predictions generated from unseen views (teacher augmentation). We use asymmetric augmentation policy so that the teacher augmentation generates novel views, and the stop-gradient operation ensures that the model does not memorize the views generated from the teacher augmentation.
Note that (˜|) is distinct from the augmentation strategy applied to the model being trained. Similar to Equation (8), we refer to the (˜|) as teacher augmentation. 3.5. Self-correction for memorized instances We further discuss the behavior of the neural vicinal distribution over a noisy training dataset. Notably, when a training dataset includes mislabeled instances, a teacher neural network can overfit to these noisy labels, where the neural empirical risk minimization fails in mitigating the impact. Interestingly, we observe that the neural vicinal distribution exhibits robustness against label noise, efectively self-correcting incoherent labels within the training set.
To understand this phenomenon, we visualize the
behavior of the neural vicinal distribution in Figure 2a. Here,
we compare the softmax scores from the augmented input
samples, distinguishing between the neural empirical
distribution (red marker) and the neural vicinal distribution
(blue markers). For the transformation policy, the network
was trained using random crop augmentation and
AutoAugment [
The visual analysis contrasts the softmax predictions from both seen and novel views of clean and mislabeled training instances. The self-correction of the neural vicinal distribution is instance-dependent which responds diferently based on if an instance is clean or mislabeled. Notably, while the teacher network’s predictions for the novel views tend to shift misclassified predictions towards ground truth, they remain consistent for clean samples. This suggests that the network outputs corrected predictions by dissociating the novel views from the memorized views.
Next, in Figure 1, we analyzed the label correction behavior of the neural vicinal distribution over the dataset population. Note that the models are trained only using the noisy training set, without access to the ground-truth labels. Applying transformation (blue curve) significantly reduced the GT class cross-entropy loss compared to no transformation (red curve), and we observed a good separation between the two distributions. Also, Table 1 shows the ground-truth accuracy for the training samples where we observed significant improvements for the mislabeled instances when transformation is applied.
We additionally observed that ensembling perturbed predictions enhances the calibration, as depicted in Figure 2b.
While the original model is heavily over-confident due to overfitting (red), vicinal prediction improves accuracy and reflects class ambiguities. (blue)
Motivated from the observation in Section 3.5, we propose a novel learning method for noisy labels named Noise-robust Distillation (NRD). Our method is formulated as a simple loss function which makes it easy to employ in existing training pipeline.
For this, we combine the target loss with the neural vicinal risk loss as a regularization objective. We formulate the combined objectives into a triplet loss. We have found Jensen-Shannon divergence (JSD) to be efective which generalizes to a triplet loss. The JSD for three distributions is,
JSD (p1, p2, p3) = ∑︁ KL (pi||m) ,
(19)
where m = ∑︀ pi. The hyperparameter
chosen to balance the importance weight between the
distributions. Additionally, JS divergence is known to have a
nice robustness property against label noise. [
Next, we derive our NRD objective step-by-step. By applying NVR to JSD loss, we have ℒ( ; , y, ) = JSD (y, ys, y) ys = (, ) yt =
E (˜|)
[ (˜, )] , ¯ assuming that we have a trained teacher network . Here, y is the target label, ys is the model output and yt is the teacher network output. The loss is solved for min ℒ(y, ys, yt).
Equation (20),
To improve noise-robustness, we can further employ an iterative distillation scheme which we repeat the strategy for multiple rounds of training. We set the teacher network as the model obtained from the previous training round, such that = − 1 at the -th training round. Applying to = arg min ℒ( ; , y, − 1) .
A student network obtained from previous training round is switched to the teacher role for next round. However, in practice, we found this to be unstable and dificult to historical models as the teacher and set = ¯ − 1. converge. Instead, we take the exponential average of the ¯ = · − 1 + (1 − ) · .
For the decay rate, we simply set = 0.99 for all experiments. The aggregation reduces the variance of neural vicinal risk estimation caused by stochastic gradient, and we have empirically found that it efectively stabilizes the training and lead to faster convergence.
Finally, we formally define our NRD training objective. To reduce the training cost, we simplify each training round into a single step of stochastic gradient descent. (SGD) This simplifies the algorithm from multi-staged process into a single-staged process, and significantly accelerates the training. The NRD objective is, ℒNRD( ; , y, ¯ ) = JSD (y, ys, yt) ys = (, ) yt =
E (˜|) ︀[ (˜, ¯ )︀] , with a slight abuse of notation for ¯ , which is not an optimization variable but continuously updated after each SGD step. This is implemented by detaching yt from the (20) (21) (22) (23) (24) (25) (26) (27)
Algorithm 1 PyTorch-style pseudocode ema_model = ema(model) optimizer = sgd_optimizer(model) for x, y in dataloader: x_t = teacher_aug(x) x_s = student_aug(x) # disconnect from backprop y_t = ema_model(x_t).detach() y_s = model(x_s) # distance between predictions loss = js_div(y, y_s, y_t) loss.backward() optimizer.step() ema_model.update() backpropagation graph, which prevents the model from memorizing the teacher augmentation views. (stop-grad in step was suficient. The overall architecture is illustrated in
5.1. Experimental settings
Benchmarking datasets For synthetic label noise
benchmarks, we used NoisyCIFAR-10, NoisyCIFAR-100 [
For the real-world benchmark, we used WebVision [
For the WebVision benchmarks, we compared our method
with the state-of-the-art methods including ELR+ [
Models PreActResNet-34 architecture [
Augmentation policy For the CIFAR experiments, we
followed [
Hyperparameters For CIFAR-10/100 benchmarks, we used 400 epochs for each training. We used SGD optimizer with momentum 0.9 and weight decay of 10− 4. Learning rates
Performance on noisy label benchmarks In Table 2, we show the performance of our method in comparison to robust loss functions. While most of the baselines shows inconsistent performance between symmetric and asymmetric noise types, our method shows consistent improvement across a wide range of noise rates and noise types. Notably, we significantly improve performance under high noise rate settings where GJS tend to underperform. For NoisyCIFAR-10 80% noise, we improve by 5%p over SCE, and for NoisyCIFAR-100-80%, we improve by 10%p over GCE.
Furthermore, the results on large-scale real-world noisy label benchmark is shown in Table 3. Notably, we observed that our method outperforms over existing methods that uses two networks.
Performance on clean datasets The proposed method improves model generalization when applied to clean dataset training as seen in Table 4. This is because the training dataset contains visually ambiguous images that make it dificult to draw a clear decision boundary, and therefore the hard target distributions from the annotations serve as a type of noisy supervision signal. We show that applying NRD can regularize and improve the performance of the model.
Comparison to consistency regularization Consistency regularization used in GJS is a powerful technique for noiserobustness. While it is similar to NRD, however, it does not directly prevent memorization of noisy labels. Figure 4 shows that GJS sufers from overfitting when trained for an extended number of steps. This is shown by test accuracy decreasing after reaching a peak at an early epoch. In contrast, NRD significantly mitigates overfitting. Notably, in 80% noise rate setting, we improve GJS by 36%p. The key contributing factor is that our method uses stop-gradient which directly prevents the model from memorizing the views generated by the asymmetric augmentation policy. 0 0 0.2 Co0.n4fide0n.6ce 0.8 1.0
Confidence calibration In Figure 5, we additionally evaluated the calibration performance. We observed that the regularization efect from NRD also improves calibration of the model. Our method shows consistent calibration performance across all noise rates, which aligns with the performance of our method.
Our work proposes Noise-Robust Distillation (NRD) which is a simple regularization objective that is designed to improve a wide range of noisy supervision problems in training. We motivate our method based on the novel formulation of Neural Vicinal Risk (NVR) minimization, which focuses on leveraging deep neural networks to improve empirical risk minimization under noisy supervision scenarios. A key insight of our work is the inherent capacity of deep neural networks to detect and correct mislabeled examples based on vicinal distribution, a feature we exploited to improve model predictions and calibration. We have validated our method on several noisy label learning benchmarks. The results show clear improvements in performance compared to the baselines under noisy supervision. These findings suggest that NRD ofers an efective strategy for handling noisy supervision, leading to enhanced generalization performance of classification models.
Acknowledgement This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. RS-2023-00240379). 1.0 0.8 cy0.6 rau cc0.4 A 0.2
A. Detailed hyperparameter
configurations
A.1. CIFAR-10/100 benchmarks
General training details For the network architecture,
we use PreActResNet-34 [
Augmentation policy For data augmentation, we use
RandAugment [
Hyperparameters See Table 5 for the details. For the
baselines, we follow the same hyperparameter configurations
used by [
A.2. WebVision benchmark
General training details For the network architecture, we
use ResNet-50 with random initialization. For training, we
use SGD optimizer with momentum 0.9, a batch size of 64,
and train for 300 epochs. The initial learning rate was set to
0.1 and reduced by 1/10 after the 100-th and 200-th epoch.
Augmentation policy For data augmentation, we use
random resized crop with size 224, random horizontal flip, and
color jitter. We used the color jitter implementation from
TorchVision [
Hyperparameters For the hyperparameters { 1, 2, 3} in the NRD loss, we used 1 = 3 = 0.1 and 2 = 0.8. The moving average decay rate was set to = 0.99. B. Training dynamics visualization of perturbed inputs In this section, we provide the visualized trajectory of the model prediction of the perturbed inputs throughout training. (See Table 6) These are the same plots presented in Figure 2a, albeit on diferent mid-training epochs. The model is trained using a standard training scheme with the crossentropy loss on the NoisyCIFAR-10-symm-40% dataset. We observe that a significant portion of the predictions perturbed using augmentation unseen at training (AutoAugment) gradually settles to the ground truth class, whereas the predictions perturbed using the same augmentation policy used at training (RandomCrop) eventually converge to the noisy target class. The result shows that predictions from the perturbation identical to the training augmentation (red markers) are non-noise-robust distillation targets, whereas the predictions from the unseen perturbation (blue markers) are noise-robust distillation targets.
horse
horse bird cat bird gt cat bird cat bird bird cat bird gt cat bird cat bird bird cat bird gt cat bird cat bird bird cat bird gt cat bird cat bird horse ship frog ship frog horse horse ship frog gt ship horse ship frog ship frog horse horse ship frog gt ship horse gt
Epoch 150 label truck airplane
automgtobile dog deer ship horse gt dog deer truck airplane
automobile truck airplane
label automobile dog deer truck airplane automobile horse ship frog ship frog horse ship frog gt ship horse gt 150
200 1.0 sscT la0.8 fG0.6 eeodn c0.4 ifn0.2 oC 0.00 1.0 sscT la0.8 fG0.6 eeodn c0.4 ifn0.2 oC 0.00 1.0 sscT la0.8 fG0.6 eeodn c0.4 ifn0.2 oC 0.00 1.0 sscT la0.8 fG0.6 eeodn c0.4 ifn0.2 oC 0.00
ldaobegl deer