Defect detection is the task of identifying defects in production samples. Usually, defect detection classifiers are trained on ground-truth data formed by normal samples (negative data) and samples with defects (positive data), where the latter are consistently fewer than normal samples. State-of-the-art data augmentation procedures add synthetic defect data by superimposing artifacts to normal samples to mitigate problems related to unbalanced training data. These techniques often produce out-of-distribution images, resulting in systems that learn what is not a normal sample but cannot accurately identify what a defect looks like. In this paper, we show the research we are carrying out in collaboration with QUALYCO, a startup spin-off of the University of Verona, on multimodal Latent Diffusion Models (LDMs) for accurate anomaly detection in Industry 5.0. Unlike conventional image generation techniques, we work within a human feedback loop pipeline, where domain experts provide multimodal guidance to the model through text descriptions and region localization of the possible anomalies. This strategic shift enhances the interpretability of results and fosters a more robust human feedback loop, facilitating iterative improvements of the generated outputs. Remarkably, our approach operates in a zero-shot manner, avoiding time-consuming fine-tuning procedures while achieving superior performance. We demonstrate its efficacy and versatility on the challenging KSDD2 dataset, achieving state-of-the-art results.
Surface Defect Detection (SDD) is a challenging problem in industrial scenarios, defined as the task of individuating samples containing a defect [1]. In many real-world applications, a human expert inspects every product and removes those defective pieces. Unfortunately, human experts are often inaccurate, and outputs can be inconsistent or biased. Moreover, humans are relatively slow in accomplishing this task, and their performances are subject to stress and fatigue.
Automated defect detection systems [2] can easily overcome most of these issues by learning classifiers on defective and nominal training products. The main drawback is the data collection process required to train a model effectively. Indeed, defective items (i.e., positive samples) are relatively rare compared to nominal items (i.e., negative samples). Thus, the user may need to collect massive amounts of data to have enough positive samples. Moreover, with the rise of the Industry 5.0 [3] and the transition towards flexible manufacturing processes where human operators and production line components actively collaborate, there is an increasing demand for systems that can quickly adapt to new production setups, i.e., customized products manufactured in small batches. Traditional automated systems cannot comply with these demands since data collection could easily involve the whole batch size.
Recent studies on SDD focused on limiting the impact of the labeling process by formulating the problem under the unsupervised learning paradigm [4] or training exclusively on nominal samples [5], possibly using few-shot learning strategies [6]. In both cases, the goal is to generate an accurate model of the nominal sample distribution and predict everything with a low probability score as anomalies. However, due to the limited restoration capability of these models, these approaches tend to generate many false positives, especially on datasets with complex structures or textures [7].
It is worth noting that, in industrial setups, anomalies are not generated by Gaussian processes but are the outcome of specific, often predictable, issues during the production process. Consequently, the anomalous samples are not randomly distributed outside the nominal distribution; they can be modeled as a mixture of Gaussian
"A photo of a scratched surface"
"A photo of a smooth surface"
Anomalous Not anomalous distributions in the feature space instead. While general, unpredictable anomalies can still happen, expert operators can easily define the main problems they can expect from the manufacturing process, such as which kind of defects, in which locations, and how often they wish to appear. Thus, generative AI can represent a powerful tool for SDD, with defect image generation emerging as a promising approach to enhance detector performance. Specifically, in this paper, we report the result of our research on Latent Diffusion Models (LDMs), a powerful class of generative models, to produce fine-grained realistic defect images that can be used as positive samples to train an anomaly detection model. We name our approach DIAG, a training-free Diffusion-based Indistribution Anomaly Generation pipeline for data augmentation in the SDD task. By leveraging pre-trained LDMs with multimodal conditioning, we can exploit domain experts' knowledge to generate plausible anomalies without needing real positive data. When using these augmented images to train an anomaly detection model, we show a notable increase in the detection performance compared to previous state-of-the-art augmentation pipelines. Specifically, this research is being carried out in collaboration with QUALYCO1 , a startup spin-off of the University of Verona. Figure 1 outlines our approach.
The main contributions of our research are as follows:
β’ We present a complete pipeline for training anomaly detection models based on nominal images and textual prompts. We showcase the superior outcomes achieved by utilizing generated defective samples compared to previous state-ofthe-art approaches. β’ We dive into spatial control approaches to enable the synthesis of defect samples incorporating regional information and exhibit enhanced controllability of the image generation through a human feedback loop pipeline, effectively utilizing domain expertise to generate more plausible in-distribution anomalies.
Research on SDD has been conducted according to different setups: unsupervised approaches [8] use a mixture of unlabelled positive and negative sample images for training; supervised approaches require labeled samples in the form of binary masks representing the defects (full supervision) [9] or simply as a tag for the whole image (weak supervision) [10]. Supervised methods demonstrated superior accuracy in the identification of anomalies. Nevertheless, the effort required to provide good annotations is not always justified. Collecting positive samples can be time and resource-consuming due to the low rate of defective products generated by industrial lines. Thus, many recent approaches adopt a "clean" setup, where the training set consists of only nominal samples. Two strategies can be adopted in clean setups: model fitting and image generation. Model fitting approaches aim at generating an accurate model of the nominal distribution, considering an outlier in every sample with a likelihood lower than -or a distance from the nominal prototype higher than-a predefined threshold [11].
On the contrary, data augmentation approaches leverage generative methods to synthesize images of defects and use these images as positive samples for training a supervised model. Specifically, this work focuses on generation-based data augmentation under clean setups.
The most popular data augmentation pipeline for SDD consists of a series of random standard transformations of the input image -such as mirroring, rotations, and color changes-followed by the super-imposition of noisy patches [12].
In MemSeg [12], the pipeline for the generation of the abnormal synthetic examples is divided into three steps: i) a Region of Interest (ROI) indicating where the defect will be located is generated using Perlin noise and the target foreground; ii) the ROI is applied to a noise image to generate a noise foreground ROI; iii) the noise foreground ROI is super-imposed on the original image to obtain the simulated anomalous image. However, all these approaches are based on generating outof-distribution patterns that do not faithfully represent the target-domain anomalies.
More recently, the first work that draws attention to in-distribution defect data is In&Out [13], in which we empirically show that diffusion models provide more realistic in-distribution defects. Here, we significantly improve the generation of in-distribution anomalous samples of [13], incorporating domain knowledge provided by an expert user through textual prompts and localization of salient regions in a training-free setup.
LDMs [14,15] are a class of deep latent variable models that work by modeling the joint distribution of the data over a Markovian inference process. This process consists of small perturbations of the data with a variancepreserving property [16], such that the limit distribution after the diffusion process is approximately identical to a known prior distribution. Starting with samples from the prior, a reverse diffusion process is learned by gradual denoising the sample to resemble the initial data by the end of the procedure.
We leveraged the natural ability of LDMs to incorporate multimodal conditioning in the generation process, taking inspiration from [17,18,19]. Specifically, we use as textual descriptions a prompt and a negative prompt, i.e., a prompt which guides the image generation "away" from its concepts of the desired output, resulting in high-quality images that comply with the given descriptions [20,21].
In particular, we do not do full image generation to effectively enhance spatial control, opting to utilize an inpainting model, as demonstrated in [14,18]. Given an image with a masked region, inpainting seamlessly fills it with content that harmonizes with the surrounding image. Although typically employed to eliminate undesired artifacts, the inpainting process ensures that the masked area incorporates the provided prompt, effectively merging textual and visual content.
To generate an anomalous image π π , the process starts by sampling a random negative image, an anomaly description, and a mask, forming the triplet (π π , π π , π π ). These pieces of information will then be fed to a textconditioned LDM to perform inpainting on image π π using the mask π π .
The anomaly description π π guides the generation, filling the masked region of π π with an anomaly that complies with the prompt. To generate images resembling real anomalous samples, domain knowledge from industrial experts is exploited, providing textual descriptions of the potential anomalies' type, shape, and spatial information.
The LDM is then conditioned on this information to inpaint plausible anomalies on defect-free samples. Formally, given pictures of defect-free (negative) samples πΌ π , domain experts will provide textual descriptions π· π of what different anomalies may look like. At the same time, regions where these anomalies may appear on the defect-free samples will be designated. We define this set of regions as a set of binary masks π π of possible anomalies, shapes, and locations. The result of this operation is π π , an anomalous version of π π , where an anomaly has been inpainted in the masked region π π . Due to the stochastic nature of LDMs, this process can be repeated multiple times to generate an augmented set of anomalous sample images πΌ π . Finally, the set πΌ π can be used as data augmentation for training anomaly detection models, as presented in the following section.
We approach the anomaly detection problem as a binary classification problem, where the objective is to predict whether a sample belongs to one of two classes. Specifically, we utilized a ResNet-50 [22] backbone trained with a binary cross-entropy loss function denoted as β BCE . Mathematically, it is defined as:
(1) where, π¦ represents the ground truth labels, Ε· represents the predicted probabilities, and π is the number of sam-ples. In detail, π¦ π denotes the true label for sample π, which can be either 0 or 1, while Ε· π signifies the predicted probability that sample π belongs to class 1.
Ongoing developments aim to optimize a model through TinyML [23] techniques in order to have an ultraefficient system that can work smoothly in real-time on a production line.
Datasets We use the Kolektor Surface-Defect Dataset 2 (KSDD2) [10], one of the most recent, complex, and realworld SDD datasets. This dataset comprises 246 positive and 2085 negative images in the training set and 110 positive and 894 negative images in the testing set. Positive images are images with visible defects, such as scratches, spots, and surface imperfections. Since the images have different dimensions, we standardize the dataset resolution, resizing all the images to 224 Γ 632 pixels while keeping the number of normal and anomalous samples unchanged.
The anomaly detection performance was evaluated based on Average Precision (AP), Precision, and Recall, following the evaluation protocol defined in [13].
In this section, we specify all the implementation details for reproducibility. All training and inferences were conducted on an NVIDIA RTX 3090 GPU.
We use the pretrained implementation of SDXL [21] from Diffusers as our text-conditioned LDM. Following the procedure outlined in Section 3.2, we use the negative images of KSDD2 as the set πΌ π . As the set of anomaly descriptions π· π , we used the prompts "white marks on the wall" and "copper metal scratches". Instead, "smooth, plain, black, dark, shadow" were used as a negative prompt to improve the performance further. These prompts were chosen after a series of tests, simulating the iterative process of our human feedback loop pipeline until the resulting images resembled plausible anomalies. We used the segmentation masks of positive samples in the KSDD2 dataset to simulate the domain experts' definition of plausible anomalous regions. Then, these data are fed to the pre-trained SDXL model to perform inpainting on the negative images in a training-free process, generating the set of augmented anomalous images πΌ π as described in Section 3.2. Finally, the generated images πΌ π are added to the training set, which will be used to train the anomaly detection model.
For a fair comparison with [13], we use the same PyTorch implementation of the ResNet-50 [22] as our anomaly detection model, in which we substitute the fully connected layers after the backbone to make it a binary classifier. The network is trained for 50 epochs with Adam [24] as an optimizer, a learning rate of 0.0001, and a batch size of 32. To maintain consistency with the training and evaluation procedures of KSDD2, our setup is the same as presented in [10,13], where only the images and ground truth labels are used to train the model.
Zero-shot data augmentation Here, we emulate the situation where no original positive samples are available in the training set. This scenario makes generating augmented positive samples necessary and restricts the users to augmentation procedures that do not rely on positive images. To do this, we build the set of augmented anomalous images πΌ π by generating π ππ’π augmented positive samples with different pipelines, i.e., MemSeg [12], In&Out [13] and DIAG. Then, we train the ResNet-50 model on a dataset that includes the original negative samples πΌ π and the augmented positive samples πΌ π . Finally, we evaluate the model on the original test set. Table 1 reports the comparison between the models trained with MemSeg, In&Out, and DIAG augmented data at different values of π ππ’π . As we can see, our proposed method achieves the highest AP (.801), recorded at 100 augmented images, while also resulting in a consistently higher AP when compared to the MemSeg and In&Out pipelines. These impressive results highlight how, through domain expertise in the form of anomaly descriptions and segmentation masks, it is possible to generate in-distribution images able to meaningfully guide an anomaly detection network, even in a complicated scenario where no real anomalous data is available. Surprisingly, the DIAG performance with π ππ’π = 120 augmented images is lower than using a smaller number of augmented images. We hypothesize this is due to the stochastic nature of the LDMs image generation. While it allows the generation of various images given the same guidance, it can also lower, in some cases, the predictability of the quality of the generated samples, which sometimes may not faithfully comply with the prompt. Future works will focus on studying quality consistency in the image generation pipeline.
To showcase DIAG as a general data augmentation technique, we also explore the scenario where real positive samples are available in the training set. To this aim, we include all the 246 real positive samples πΌ π in the training set, together with the real negative images πΌ π and the π ππ’π augmented positive images πΌ π .
As we can see from Table 2, DIAG achieves the highest average AP yet (.924), surpassing the .782 set by the previous state-of-the-art data augmentation pipeline [13]. When comparing these results to the ones obtained in the "zero-shot data augmentation" scenario, it is clear how more in-distribution images improve model performance during training. This is highlighted by the improvement in performance of all the models when adding the real positive images πΌ π to the training set. At the same time, the inclusion of DIAG augmented images allows the model to explore the anomaly distribution further, resulting in the difference in performance between the different data augmentation pipelines.
The main goal of our data augmentation pipeline is to generate in-distribution synthetic positive images, meaning images that closely resemble the real ones. Figure 2 shows qualitative results. It's evident that the images produced by DIAG are markedly more realistic compared to those generated by MemSeg [12] and In&Out [13].
This work presents DIAG, a novel data augmentation pipeline that leverages visual language models to produce training-free positive images for enhancing the performance of an SDD model. We introduced domain experts in the generation pipeline, asking them to describe with textual prompts how a defect should look and where it can be localized. Then, we adopt a pre-trained LDM to generate defective images and train a binary classifier for isolating the anomalous images. We focus our experiments on the KSDD2 dataset and establish ourselves as the new state-of-the-art data augmentation pipeline, surpassing previous approaches in both the zero-shot and full-shot data augmentation scenarios with an AP of .801 and .924, respectively. These results highlight the potential of in-distribution data augmentation in the anomaly detection field, where training-free generative model pipelines such as DIAG can provide meaningful data for downstream classification, making them appealing solutions in scenarios where real anomalous data is difficult to collect or unavailable. These promising results promote further exploration across various datasets, particularly investigating how robust the image generation is compared to noisy textual prompts.