Deep learning models have achieved remarkable success in various domains, including image classification, natural language processing, and speech recognition. However, the performance of these models heavily relies on the availability of high-quality labeled data for training. Obtaining accurately annotated labels can be a challenging and time-consuming task, often requiring human annotators to manually label large amounts of data. As a result, noisy labels may arise during the annotation process, leading to suboptimal model performance.

In this paper, we address noisy label learning as a subset of a more generic type of problem. This encompasses learning from an over-confident target probability distribution and image ambiguity [1], human annotation errors, multiple classes in an image, and out-of-distribution training examples [2] that can naturally occur due to, for example, random crop data augmentation. We show that our generic noisy label supervision algorithm can address a combination of these issues using a simple and unified approach.

We propose a noise-robust learning algorithm named Noise-Robust Distillation (NRD) to address the issue of noisy supervision during training. NRD aims to improve the generalization performance of classification models by explicitly considering the noise and ambiguity in the training labels. We motivate NRD by a novel formulation of the noisy supervision learning problem which we name Neural Vicinal Risk (NVR) minimization.

This stems from the observation that deep neural networks have the inherent capability to detect and correct noisy supervision, even when it is trained using noisy supervision. This ability is particularly evident when considering the vicinal distribution, which represents the distribution generated from perturbed versions of the training data. Despite being trained on noisy labels, neural networks can still accurately model the vicinal distribution, indicating their potential to correct the noisy supervision.

Our findings suggest that the combination of perturbation-based estimation and ensembling can lead to improved model performance, even in the presence of noisy supervision. Building on these insights, we propose Noise-Robust Distillation (NRD), which is a noise-robust learning method that leverages the neural vicinal risk principle to enhance the generalization performance of classification models trained on noisy labels.

The main contributions of this work are as follows:

• We introduce the Noise-Robust Distillation (NRD), a noise-robust learning approach that comprehensively addresses the challenges posed by noisy supervision during training. • NRD is motivated by a novel noise-robust learning framework which we name Neural Vicinal Risk (NVR) minimization. We show that NVR improves the estimation quality of the true class distribution and handles label noise effectively.

Noisy label learning Numerous methods tackle the challenge of training Deep Neural Networks (DNNs) on datasets that contain a mix of correctly labeled and mislabeled samples, as discussed in [4]. Some approaches focus on designing a noisy-robust loss to mitigate the impact of mislabeled samples. Mean Absolute Error (MAE) loss [5] demonstrates competitive performance. Following this, the introduction of the Generalized Cross-Entropy (GCE), Symmetric Cross-Entropy (SCE) loss, and active passive loss are proposed with improved noisy-robustness. Generalized Jensen-Shannon divergence (GJS) [6] enforces consistency between predictions from multiple augmented views of a sample to regularize training. Also, the principle of negative learning is emphasized by [7,8]. The strategies inspired by the training dynamics of models [9] such as early stopping [10,11] or over-parameterization [12] exploit the different convergence speeds of clean and noisy samples. Co-teaching [13] involves simultaneous training of two DNNs, where each network learns from the clean samples chosen by its counterpart. Noise identification aims to filter noisy samples from the training dataset. Noisy samples can be filtered by measuring the degree of disagreement between ensemble models, which occurs once the model is overfitted to the noisy samples. Recent algorithms [14,15,16] utilize the power of Semi-Supervised Learning (SSL) by following a two-step process: filtering out noisy labels first, and then treating the detected noisy samples as unlabeled for reducing the noisy learning problem into a SSL task. Semi-supervised learning (SSL) has emerged as a powerful method for noisy label learning. Among them, consis-tency regularization promotes a model to make consistent outputs across data augmentations, as in Π-model, Temporal Ensembling [17] and Mean Teacher [18]. Also, FixMatch [19] integrates pseudo-labeling and and virtual adversarial training [20] utilizes adversarial attacks. MixMatch [21], adopted by DivideMix [14], generates pseudo-label with sharpening for data-augmented unlabeled examples and mixes labeled and unlabeled data using MixUp [22].

Calibration and knowledge distillation Confidence calibration [23] is the process of adjusting a model's predicted probabilities to better reflect the true likelihood. It is demonstrated that training a model with data augmentation like Mixup [22] improves model calibration and robustness to noise [24]. Meanwhile, Knowledge Distillation (KD) [25] enhances the student model by transferring knowledge contained in the prediction of the teacher model, focusing on "dark" or "hidden" knowledge, including its confident and less confident predictions.

The expected risk 𝑅(𝜃) is defined as the average loss over 𝑝(𝑥, 𝑦),

In practice, a dataset 𝒟 is used to mimic the true distribution 𝑝(𝑥, 𝑦), which leads to the empirical risk

where the corresponding empirical distribution 𝑝 ˆ(𝑥, 𝑦) is a mixture of delta masses using the observed samples, and the class distribution is a one-hot distribution given by annotations,

Our goal is to refine the estimation of the data distribution 𝑝(𝑥, 𝑦) by utilizing the empirical distribution 𝑝 ˆ(𝑥, 𝑦). A pivotal question that arises is how to enhance the approximation of the true risk 𝑅(𝜃) intrinsic to a classification model. As evidenced by Equation ( 3), this task necessitates the accurate estimation of two orthogonal components present within the true distribution 𝑝(𝑥, 𝑦) = 𝑃 (𝑦|𝑥)𝑝(𝑥):

(1) the input distribution 𝑝(𝑥) and ( 2) the corresponding conditional distribution 𝑃 (𝑦|𝑥).

Estimating 𝑃 (𝑦|𝑥) as a one-hot distribution involves assigning a single class label per sample, which is vulnerable to human annotation errors. Unfortunately, it proves challenging to enhance or secure accurate supervision signals for 𝑃 (𝑦|𝑥), as this requires multiple human annotators reviewing the same image [1] which is a prohibitively costly process. Nonetheless, enhancing the estimation quality of the true class distribution 𝑃 (𝑦|𝑥) can lead to further improvements in estimating and minimizing the true risk.

Neural Empirical Risk (NER) Instead of using Equation (3), we can choose to parameterize 𝑃 (𝑦|𝑥) by a neural network 𝑃 (𝑦|𝑥, 𝜑) to further improve the estimation quality. First, we factorize the data distribution as 𝑝(𝑥, 𝑦) = 𝑃 (𝑦|𝑥)𝑝(𝑥), and denote the corresponding empirical distributions as follows:

Instead of using 𝑃 ˆ(𝑦|𝑥), we choose to use a distribution parameterized by a neural network trained on 𝒟,

where 𝑝(𝜑|𝒟) is the distribution over the function class parameterized by neural network. By plugging Equation ( 6) into 𝑝 ˆ(𝑥, 𝑦) = 𝑃 ˆ(𝑦|𝑥)𝑝 ˆ(𝑥), we define the neural empirical distribution 𝑝 ˆ𝜌 and the neural empirical risk 𝑅 ˆ𝜌 as

Here, we refer to the model 𝑃 (𝑦|𝑥, 𝜑) as the teacher network to distinguish from the model being trained, whose term is borrowed from knowledge distillation. This can provide better estimation quality than 𝑃 ˆ(𝑦|𝑥) as is often observed in knowledge distillation, which we view as an instance of NER minimization. Knowledge distillation is known to improve generalization and calibration performance due to the dark knowledge [25]. However, when the model is over-fitted to the noisy label, it severely degrades the performance of estimating the class probabilities. Hence, in order to effectively utilize a neural network, it is necessary to employ a noise-robust method to accurately estimate the class probabilities in the presence of noisy labels.

Our motivation is based on the Vicinal Risk Minimization (VRM) principle [26], which is an alternative approximation to 𝑝(𝑥, 𝑦). The vicinal distribution 𝑝𝜈 (𝑥 ˜, 𝑦 ˜) constructed from the data distribution is defined as

where 𝜈(𝑥 ˜, 𝑦 ˜|𝑥, 𝑦) is the vicinity distribution around (𝑥, 𝑦). For example, [26] used additive Gaussian noise 𝒩 (0, 𝜎 2 𝐼).

MixUp [24] and CutMix [27] chose stochastic interpolation between samples which has also shown its effectiveness in noisy label. [24]. Using the dataset, Equation ( 9) is replaced by the empirical distribution as

We propose to further improve by using a neural network to robustly approximate the data distribution by modifying Equation (9). We propose the following approximate vicinal data distribution parameterized by a deep neural network 𝜑 which we name neural vicinal distribution 𝑝𝜋. During training, the predictions from the original views (student augmentation) is regularized using the predictions generated from unseen views (teacher augmentation). We use asymmetric augmentation policy so that the teacher augmentation generates novel views, and the stop-gradient operation ensures that the model does not memorize the views generated from the teacher augmentation.

Label correction behavior for memorized training examples using transformed views. The models are trained to perfectly memorize the noisy labels, then evaluated again for training set with ground-truth labels. Due to memorization, the GT accuracy for mislabeled instances is zero and the overall accuracy is bounded by the noise rate. However, averaging the predictions from the transformed inputs shifts the prediction of the noisy examples to the ground-truth. For the transformation, AutoAugment followed by RandomErasing was used. For the dataset, NoisyCIFAR-10 with symmmetric noise was used. Note that Equation ( 17) is a parameterized version of Equation (11) using a deep neural network. Finally, the neural vicinal risk is,

Note that 𝜈(𝑥 ˜|𝑥) is distinct from the augmentation strategy applied to the model being trained. Similar to Equation ( 8), we refer to the 𝜈(𝑥 ˜|𝑥) as teacher augmentation.

We further discuss the behavior of the neural vicinal distribution over a noisy training dataset. Notably, when a training dataset includes mislabeled instances, a teacher neural network can overfit to these noisy labels, where the neural empirical risk minimization fails in mitigating the impact. Interestingly, we observe that the neural vicinal distribution exhibits robustness against label noise, effectively self-correcting incoherent labels within the training set.

To understand this phenomenon, we visualize the behavior of the neural vicinal distribution in Figure 2a. Here, we compare the softmax scores from the augmented input samples, distinguishing between the neural empirical distribution (red marker) and the neural vicinal distribution (blue markers). For the transformation policy, the network was trained using random crop augmentation and AutoAugment [28] is chosen as the vicinal distribution to generate the novel views. The top row shows clean instances and the bottom row shows mislabeled instances.

The visual analysis contrasts the softmax predictions from both seen and novel views of clean and mislabeled training instances. The self-correction of the neural vicinal distribution is instance-dependent which responds differently based on if an instance is clean or mislabeled. Notably, while the teacher network's predictions for the novel views tend to shift misclassified predictions towards ground truth, they remain consistent for clean samples. This suggests that the network outputs corrected predictions by dissociating the novel views from the memorized views.

Next, in Figure 1, we analyzed the label correction behavior of the neural vicinal distribution over the dataset population. Note that the models are trained only using the noisy training set, without access to the ground-truth labels. Applying transformation (blue curve) significantly reduced the GT class cross-entropy loss compared to no transformation (red curve), and we observed a good separation between the two distributions. Also, Table 1 shows the ground-truth accuracy for the training samples where we observed significant improvements for the mislabeled instances when transformation is applied.

We additionally observed that ensembling perturbed predictions enhances the calibration, as depicted in Figure 2b. While the original model is heavily over-confident due to overfitting (red), vicinal prediction improves accuracy and reflects class ambiguities. (blue)

Motivated from the observation in Section 3.5, we propose a novel learning method for noisy labels named Noise-robust Distillation (NRD). Our method is formulated as a simple loss function which makes it easy to employ in existing training pipeline.

For this, we combine the target loss with the neural vicinal risk loss as a regularization objective. We formulate the combined objectives into a triplet loss. We have found Jensen-Shannon divergence (JSD) to be effective which generalizes to a triplet loss. The JSD for three distributions is,

where m = ∑︀ 𝑖 𝜋𝑖p i . The hyperparameter 𝜋 ∈ ∆ 2 is chosen to balance the importance weight between the distributions. Additionally, JS divergence is known to have a nice robustness property against label noise. [6] showed that JS divergence simulates MAE loss [5] in its asymptote.

Next, we derive our NRD objective step-by-step. By applying NVR to JSD loss, we have

assuming that we have a trained teacher network 𝜑. Here, y is the target label, ys is the model output and yt is the teacher network output. The loss is solved for min 𝜃 ℒ(y, ys, yt).

To improve noise-robustness, we can further employ an iterative distillation scheme which we repeat the strategy for multiple rounds of training. We set the teacher network as the model obtained from the previous training round, such that 𝜑𝑡 = 𝜃𝑡−1 at the 𝑡-th training round. Applying to Equation ( 20

A student network obtained from previous training round is switched to the teacher role for next round. However, in practice, we found this to be unstable and difficult to converge. Instead, we take the exponential average of the historical models as the teacher and set 𝜑𝑡 = 𝜃 ¯𝑡−1.

For the decay rate, we simply set 𝛽 = 0.99 for all experiments. The aggregation reduces the variance of neural vicinal risk estimation caused by stochastic gradient, and we have empirically found that it effectively stabilizes the training and lead to faster convergence.

with a slight abuse of notation for 𝜃 ¯, which is not an optimization variable but continuously updated after each SGD step. This is implemented by detaching yt from the backpropagation graph, which prevents the model from memorizing the teacher augmentation views. (stop-grad in Figure 3) For Equation ( 27), we found single sample per SGD step was sufficient. The overall architecture is illustrated in Figure 3 and the pseudocode is presented in Algorithm 1.

Benchmarking datasets For synthetic label noise benchmarks, we used NoisyCIFAR-10, NoisyCIFAR-100 [29]. For symmetric label noise, we randomly flip the ground truth label with a probability 𝜂 uniformly across all categories.

For asymmetric label noise, we follow the scheme in [30].

For NoisyCIFAR-10-asymm, we flip truck→automobile, bird→airplane, cat→dog, dog→cat, deer→horse. For NoisyCIFAR-100-asymm, within each superclass, we randomly replace a subclass label 𝑦𝑖 to adjacent subclass 𝑦𝑖 + 1 with probability 𝜂.

For the real-world benchmark, we used WebVision [31] dataset. WebVision consists of 2.4M training examples collected via Google and Flickr image search. We used a miniaturized training set following [32] which uses only the first 50 categories in the "Google" image set. Mini-WebVision consists of 66K training and 2.5K validation examples. We additionally evaluated the trained model on ImageNet [33] validation set. The noise rate is known to be around 20%. Baseline methods For the CIFAR benchmarks, we compare against cross-entropy (CE), bootstrapping (BS) [34], label smoothing (LS) [35], symmetric cross-entropy (SCE) [36], generalized cross-entropy (GCE) [37], normalized loss (NCE+RCE) [38], Jensen-Shannon divergence (JS, GJS) [6].

For the WebVision benchmarks, we compared our method with the state-of-the-art methods including ELR+ [10], Di-videMix [14], and GJS [6]. The baseline results were adopted from [6]. Models PreActResNet-34 architecture [39] is used for all experiments conducted on CIFAR-10/100 datasets. For We-bVision experiments, we used ResNet-50. All experiments were trained from random initialization. Augmentation policy For the CIFAR experiments, we followed [6] and used RandAugment [40] chained with Cutout [41] for all methods. For the NRD teacher transformation, we used AugMix [42] in all experiments. Hyperparameters For CIFAR-10/100 benchmarks, we used 400 epochs for each training. We used SGD optimizer with momentum 0.9 and weight decay of 10 −4 . Learning rates were reduced by a factor of 0.1 after 200-th and 300-th epoch.

For WebVision benchmarks, we trained the network for 300 epochs. Learning rate was reduced by a factor of 0.1 after 150 and 250 epochs. Refer to Appendix A for hyperparameter configuration details.

Performance on noisy label benchmarks In Table 2, we show the performance of our method in comparison to robust loss functions. While most of the baselines shows inconsistent performance between symmetric and asymmetric noise types, our method shows consistent improvement across a wide range of noise rates and noise types. Notably, we significantly improve performance under high noise rate settings where GJS tend to underperform. For NoisyCIFAR-10 80% noise, we improve by 5%p over SCE, and for NoisyCIFAR-100-80%, we improve by 10%p over GCE. Furthermore, the results on large-scale real-world noisy label benchmark is shown in Table 3. Notably, we observed that our method outperforms over existing methods that uses two networks.

The proposed method improves model generalization when applied to clean dataset training as seen in Table 4. This is because the training dataset contains visually ambiguous images that make it difficult to draw a clear decision boundary, and therefore the hard target distributions from the annotations serve as ). Enforcing consistency does not fully prevent overfitting because the model memorizes the noisy labels after an extended number of epochs. In contrast, our method (NRD) effectively prevents memorization. NoisyCIFAR-100 dataset is used.

a type of noisy supervision signal. We show that applying NRD can regularize and improve the performance of the model.

Comparison to consistency regularization Consistency regularization used in GJS is a powerful technique for noiserobustness. While it is similar to NRD, however, it does not directly prevent memorization of noisy labels. Figure 4 shows that GJS suffers from overfitting when trained for an extended number of steps. This is shown by test accuracy decreasing after reaching a peak at an early epoch. In contrast, NRD significantly mitigates overfitting. Notably, in 80% noise rate setting, we improve GJS by 36%p. The key contributing factor is that our method uses stop-gradient which directly prevents the model from memorizing the views generated by the asymmetric augmentation policy.

In Figure 5, we additionally evaluated the calibration performance. We observed that the regularization effect from NRD also improves calibration of the model. Our method shows consistent calibration performance across all noise rates, which aligns with the performance of our method.

Our work proposes Noise-Robust Distillation (NRD) which is a simple regularization objective that is designed to improve a wide range of noisy supervision problems in training. We motivate our method based on the novel formulation of Neural Vicinal Risk (NVR) minimization, which focuses on leveraging deep neural networks to improve empirical risk minimization under noisy supervision scenarios. A key insight of our work is the inherent capacity of deep neural networks to detect and correct mislabeled examples based on vicinal distribution, a feature we exploited to improve model predictions and calibration. We have validated our method on several noisy label learning benchmarks. The results show clear improvements in performance compared to the baselines under noisy supervision. These findings suggest that NRD offers an effective strategy for handling noisy supervision, leading to enhanced generalization performance of classification models.

Visualization of the model prediction over training. We randomly selected four distinct noisy samples from the training dataset, which corresponds to the four rows. The model is trained using RandomCrop and tested using RandomCrop-perturbed inputs (red) and AutoAugment-perturbed inputs (blue). Leftmost column shows the predicted confidence of the perturbed inputs with respect to the ground-truth classes. The figures on the right hand-side visualizes the softmax vectors projected onto a decagonal surface, which are analogous to Figure 2a. At the early phase of the training, both red and blue markers predict the ground-truth class. However, as the training progresses and the model overfits to the noisy labels, the red markers predict the target label, whereas a significant portion of the blue markers predicts the ground-truth markers. This shows that unseen perturbation to the input can produce noise-robust learning signal for training.