Currently, AI image analysis research is being conducted on automated cutting, bending, and loading systems, which are the main facilities of rebar processing factories. For automation, various datasets through machine vision cameras are required. However, environmental factors include difficult data collection or high production costs to collect datasets in the production process. To solve these problems, we propose a real-time twist rebar detection system based on GAN (Generative adversarial network), with real rebar datasets collected from 20 rebar videos. In this paper, we generated additional datasets from a deep image generation network and detected rebars' endpoints through YOLO (You Only Look Once) v4, a deep-learning object detection model. In experiments, we generated rebar images corresponding to normal and abnormal, the measured quality between real rebar dataset and generated synthetic rebar dataset by FID (Frechet Inception Distance). As a result, FID measurements showed the normal synthetic rebar dataset 79.363 and the abnormal synthetic rebar dataset 113.973. After that, as a result of training in YOLO v4 by combining the synthetic rebar dataset generated from GAN and the real rebar dataset, we obtained the mean Average Precision (mAP) of 100% and a misdetection rate of 5% compared to the real rebar dataset, the mAP increased by 0.6%, and decreased by 10%. Overall, our results demonstrate a strong effect on rebar twist detection accuracy and misdetection rate.
Recently, in response to the development of the artificial intelligence and robot industry, unmanned operations have been accelerated. Artificial intelligence's creative capacity can create on its own shows innovation in the manufacturing industry. Rebar processing requires an automated intelligent production system that minimizes loss rate, such as automatic correction and optimization of rebars. However, the improvement of calibration work time and accuracy of the machining rebar factory still depends on the worker's skill level, as shown in Figure 1. In addition, rebar processing has quality problems and safety accidents that occur during the machining process. Therefore, an unmanned system that detects the endpoints of the processed rebar and predicts the errors of correction value is needed.
First, to detect the endpoints of rebars, it is necessary to collect datasets of normal and abnormal rebars through a machine vision camera. However, collecting data in response to environmental factors is difficult, and the high production cost is when collecting datasets. These problems can be addressed by augmenting various high-quality images using an image generation model for existing small image data. In addition, the performance of rebar twist detection can be improved by utilizing deep learning detection models through real and synthetic rebar data.
In this paper, our contributions are three for
the rebar detection system.
1. A set of rebar data is generated by extracting
1000 normal and 1000 abnormal rebar
images from the rebar video collected in the
field with a machine vision camera.
2. To generate rebar images in various
situations from the real rebar dataset, 500
images of rebar and 500 images of abnormal
rebar are generated through GAN[
In this paper, this research proposed a
starting point rebar detection method using the
average brightness change of a high-speed
Infrared Ray (IR) camera to reduce errors
according to the environment [
The average value of the pixel matrix had measured by a specific size of standby window at the rebar injection point, which was based on 848x848 grayscale and 90fps with the INTEL RealSense D435 IR camera and performed maximum detection accuracy of about 81%.
To automate the rebar injection system, the rebar detection accuracy must be over 90%, and this research show failed on flicker phenomenon cases, as shown in Figure 3.
In this paper, this research used a feature
point matching algorithm to determine the
twisting of the machining rebar[
This research has resulted that there was a twist with an average accuracy of 96.5%. However, the amount of computation increased significantly during real-time detection, resulting in a significant decrease in fps to 10-20, and accuracy was significantly decreased in twist detection when experimented in a new environment.
In this paper, this research proposed a real-time
system to detect and track rebar endpoints based
on YOLO v3 from the input of rebar images of
the camera and predict rebar endpoints in
advance with non-linear regression of the
obtained coordinates [
In this paper, we introduce generating a real rebar dataset and synthetic rebar dataset for rebar twist detection performance. as shown in Figure 10, we show a system for determining the presence or absence of real-time rebar twist by combining generated images from GAN with real rebar dataset, and training yolov4, famous for real-time detection deep learning model with these datasets.
the HIKVISION MV-CAD13-20GM Machine camera was selected as an environment for collecting a dataset of processed rebars and detecting real-time rebar twists, as shown in Figure 6. After that, to reduce the flicker phenomenon and bright and dark lighting differences in various environments, we set the working distance to 2000mm, the focal length to 50mm, the lighting to 90 Hz with an LED lamp, and the exposure time to 500ms.
The rebar injection videos have an average frame width of 1280, frame height of 720, and fps of 88.38, and a total of 20 data videos were collected. The average length of the videos is about 3 seconds because the speed of rebar injection is very fast. Through the collected images, 351 to 443 images were extracted for each video. But blurred or broken images were removed to generate a rebar dataset. Finally, the rebar dataset consists of a total of 2,000 images, and as shown in Figure 7, it was classified into 1,000 normal rebar images and 1,000 abnormal rebar images.
In this paper, we use the basic GAN, which is famous for its image generation model. As shown in Figure 8, it can be seen that synthetic rebar images similar to the real rebar images were extracted from rebar injection videos. Finally, we combined extracted rebar dataset with synthetic rebar images.
the average detection accuracy and error detection rate by training the real rebar dataset and the combined rebar dataset. Figure 9 shows that the twist detection test was conducted by training the real rebar dataset, and the twist detection test was conducted by training the combined dataset of rebar images generated through GAN.
Finally, in our study, YOLO v4, which is famous for its real-time detection model, was used to detect rebar twists. For performance comparison, the model obtained by training the real rebar dataset and the synthetic rebar images generated through GAN were compared through
First, for rebar image generation, the training environment in GAN is set to generate images of size 20000 for epoch, 0.0002 for learning rate, 1000 for latent dimension, and 416x416 for image size. And finally, the interval is set to 100. Second, For the classification of twist rebar, the training environment of YOLO v4 was set to epoch 2000 to 3000, learning rate 0.0013, saturation 1.5, and exposure 1.5.
In addition, when training in YOLO v4 by applying data augmentation, training was conducted by applying random vertical/horizontal flipping. The twist detection test for the experiment was compared by obtaining results through a total of three training times. 4.2
In order to measure the quality of rebar images generated through GAN, the average value was measured through FID. As can be seen from Table 1, the synthetic normal rebar images generated through GAN were FID 79.3638476, and the synthetic abnormal rebar images were 113.9733602. 4.3
First, for rebar image generation, the training environment in GAN is set to generate images of size 20000 for epoch, 0.0002 for learning rate, 1000 for latent dimension, and 416x416 for image size. And finally, the interval is set to 100.
Second, For the classification of twist rebar, the training environment of YOLO v4 was set to epoch 2000 to 3000, learning rate 0.0013, saturation 1.5, and exposure 1.5.
In addition, when training in YOLO v4 by applying data augmentation, training was conducted by applying random vertical/horizontal flipping. The twist detection test for the experiment was compared by obtaining results through a total of three training times. 4.4
In order to measure the quality of rebar images generated through GAN, the average value was measured through FID. As can be seen from Table 1, the synthetic normal rebar images generated through GAN were FID 79.3638476, and the synthetic abnormal rebar images were 113.9733602.
For normal/abnormal rebar classification, Figure 11 shows that the average mAP obtained by training the real rebar dataset from YOLO v4 was 99.4%. When the epoch reached 2600, the AP showed a 69% drop. And 99.4% of mAP were obtained when data augmentation was applied.
And the results were the same as those learned by applying non-data augmentation. Finally, when training was performed by combining the real rebar dataset with the synthetic(generated) rebar images through GAN, good performance was shown at 100% mAP, as shown in Figure 12.
Finally, in this paper, a misdetection test was performed on 20 rebar injection videos to confirm the twist detection performance. Table 2 shows that three misdetections occurred when data augmentation was applied, showing a 15% misdetection rate. When the GAN-based generated dataset was combined with the real rebar dataset, one misdetection occurred, showing a 5% misdetection rate.
In this paper, we introduced the development of an AI image analysis-based processing rebar productivity improvement system. First, videos of rebar injection were collected through a machine vision camera. After that, images of real rebar were extracted to create a rebar dataset. And they were classified into normal rebars and abnormal rebars. However, only the relevant rebar image has limitations in improving the detection accuracy and classification performance of rebars, and there is a problem that there is a significant cost problem in collecting additional datasets.
To solve this problem, we proposed various types of rebar images generated through GAN to improve the performance of the real-time twist detection system. After that, the detection accuracy and misdetection rate were tested by YOLO v4 by combining the synthetic rebar images with the extracted real rebar images; In experiments, FID measurements showed the normal synthetic rebar dataset 79.363 and the abnormal synthetic rebar dataset 113.973. After that, as a result of training in YOLO v4, we obtained the mAP by 100% and the misdetection rate by 5% compared to the real rebar dataset, the mAP increased by 0.6%, and decreased by 10%. Overall, our results demonstrate a strong effect on rebar twist detection accuracy and misdetection rate.
Based on this study, various regression prediction models will be used to improve the accuracy performance of predicting rebars' endpoints and recognizing the rebars' shape.
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education, Science and Technology (NRF-2021R1G1A1006381).