This study detected the defected chocolate packages by image processing methods and separated them from the conveyor by a robotic arm. In the system, it was assumed that a conveyor belt system was set at the output of the packaging machine. The products transferred from the packaging machine to the conveyor were photographed in real-time from a fixed point with a camera while the conveyor belt was operating. The packages in the images acquired were classified as non-defected / defected. When improperly packaged chocolate is detected, the robot arm separated the product from the conveyor belt. The proposed method can detect the packaging performance of the machine with the camera quality control system and ensure that the necessary improvements can be made depending on the machine's performance. In this way, the performance of the produced packaging machine can be increased.
The mistakes in the production line in the industry are generally detected by human eyes. Therefore, its efficiency is low and the margin of error is high due to eye fatigue. Therefore, an automatic inspection system is required for mass production. Industry 4.0 is one of the terms that has been frequently heard recently, and it is explained as the combination of information technologies and industrial activities. Artificial intelligence pioneers information technologies. While machines become smarter with artificial intelligence, machines that interact with each other in this direction cause a wide range of changes, especially in the business world. While the technologies dreamed of in the past years come true; Robots, smart computers, and advanced automation systems that perform humanspecific tasks have ceased to be abstract concepts and are perceived as normal. The effect of the upward acceleration captured by technology should not be ignored in this area. In particular, it has enabled it to happen in the last 10 years. Although it is not easy to spread the advanced technologies introduced in the previous industrial revolutions to the base, especially due to cost, new generation technologies are very easily integrated into our lives. Today, it is observed that many of the new generation technologies have become an integral part of our lives. Intelligent automation systems in factories, and smart robots that perform human-specific tasks in factories and interact with each other, ensure the advancement of technology.
[1] used image processing method in the quality control stage of ceramic tile manufacturing. They performed color analysis, size verification, and surface defect detection on ceramic tiles using image processing and morphological methods techniques before packaging to improve homogeneity. [2] determined the deformed patterned fabrics with Fisher Criterion Based Deep Learning method in their study. They have successfully classified the fabrics to be flawless and imperfect. [3] applied quality control in smart factory prototype using Deep Learning method. In their study, visual quality control automation with a camera placed on the assembly line in a smart factory model is proposed. The image obtained from the camera was detected, then they were classified as "okay" or "not ok" using deep learning methods. [4] have conducted quality control studies with deep learning methods in the printing industry. They have created a Deep Neural Network (DNN) to minimize the errors that occur during the production of engraving cylinders. Their study used a high-resolution optical quality control camera. [5] mounted a camera on the robot arm and checked the quality of the inverters that move on the conveyor belt. Their study first detected the inverter, then the robot arm moved to the quality control position, automatically. The proposed method checked if the braking resistance was mounted on the inverter or not via deep learning methods.
In this study, a Logitech C270 camera was used to capture images. The images acquired were processed with the OpenCV library in Python environment. Mitsubishi FX5U PLC is used to control the conveyor system and robot arm. PLC and Python were communicated via MODBUS TCP/IP communication. The defected packages were taken from Memak Machinery and used during the training of the system. Figure 1 presents the structure of the system used.
Median filter [6], edge detection [7], and morphological closing [8] methods were used in the image processing stage.
A 5x5 size Median Filter was first applied to the colored images that were acquired from the camera to remove noise. The operation of the Median Filter is presented in Figure 2.
The closing process tries to turn off the text on the object and increase the white area and reduce the noise. At the end of the closing process, the points in the image close each other, the main lines in the image become more stable. Gaps between points that are close to each other were filled and the dots merge.
Edges can be explained as a curve connecting all continuous points (along the border) of the same color and density. Finding the edges of an image significantly reduces most of the data and filters out unnecessary information while preserving important structural features in the image. It is a useful tool for edge detection, shape analysis, and object recognition-detection. Figure 3 shows the matrix representation of a simple vertical edge finding method.
The median filtered colored image was transformed into HSV color space. After the transition to the HSV color space, a mask was applied by determining the lower and upper limits for the H, S, and V values to eliminate the background and obtain the necessary image for processing.
A morphological closing filter was applied to close the gaps in the HSV masked image. A 5x5 matrix with all elements of “1” was used as the kernel for the filter. By applying an edge detection algorithm on the final image, the defected chocolate packages were detected.
The methods used in the image processing stage and the visual results obtained are shown in Figure 4. The area of the region whose edges are detected was determined in pixels² with the help of the function. Figure 5 shows the area of defected and non-defected chocolate packages.
Considering the average area values of the defected packages and non-defected packages, a limit value was determined for the area, and the chocolates with an area above this limit value were incorrectly packaged, the chocolates underneath were labeled as correctly packaged. As a result of the image processing, the package information and the number of packages belonging to the two classes were sent to the PLC via MODBUS TCP / IP communication to be used in the automation system. In this study, defected packages separation process was done without stopping the conveyor. The robot arm follows the conveyor in the system. For this purpose, an encoder is connected to the asynchronous motor driving the conveyor. The encoder is also connected to the PLC and its value is transferred to the robot arm via the HMI. There is no direct communication between the PLC and the robot arm. Therefore, HMI is used for PLC and robot arm communication. The robot arm separates only the defected packages from the system according to the encoder information received.
The extraction system is presented in Figure 6.
It is assumed that the system shown in Figure 6 will be added to the output of the packaging machine presented in Figure 7.
The robot arm was moved in x, y, and z axes to grip the product. Movement in the Z-axis is always the same. Because it is the position to grip the product. Movement in the Y-axis is the movement in the flow direction of the conveyor belt. Here, a connection was found between the encoder pulse value and the mm movement of the robot arm (1 pulse = 0.05 mm) and the robot arm followed the conveyor according to this connection. One of the most important parts here is the movement on the X -axis. Packages come in random locations. Therefore, the robot arm must go to the center point of the product correctly. For this, the robot was trained in the first stage. During the training phase, the robot x point corresponding to the package in the x point in the camera was recorded. This process was applied for seven different points and the relationship between the robot arm x position corresponding to the x position on the camera was obtained by the simple regression method. The regression equation obtained is presented in Figure 8.
The point where the robot arm should go on the x-axis was found when the x value of the package with the center point of the camera was multiplied by -0.477 and added by 498.3. The R2 value was obtained as 0.999. This expression shows that the equation calculates position with very high accuracy.
The motion of the robot according to the package received as a result of all these operations is presented in Figure 9. 3.1.
The system in Figure 9 has been given 12 products including 9 non-defected and 3 defected packages. The system separated 3 defective packages from the conveyor belt and ensured that the non-defected packages continued over the conveyor. The confusion matrix of the results is presented in Table 1.
As seen in Table 1, TP was found as 9, FP and FN were found as 0, TN was found as 3. There was no error in detecting non-defected or defected packages. The proposed system works with 100% accuracy.
In this study, the encoder and the robot arm are communicated via PLC. At this stage, while the encoder data was transmitted to the robot arm via PLC and HMI, approximately 10millisecond time loss occurred. The encoder can be connected directly to the robot arm driver. This will increase hardware speed and the robot will be able to work more synchronously with the conveyor belt. At the same time, instead of writing an extra formula in the robot program, it will be able to extract the formula for the robot encoder itself. Besides, if the products move over the conveyor are passed through an illuminated indoor environment, the effect of the external environment can be minimized in image processing methods.
This project was supported by TÜBİTAK (The Scientific and Technological Research Council of Turkey) 2209A Research Funding Program for University Students.
[1] Hussam Elbehiery, Alaa A. Hefnawy, and M. TarekElewa. "Surface Defects Detection for Ceramic Tiles Using Image Processing and Morphology Operation Techniques." Egyptian Informatics Journal 6.1 (2005): 123-133. [2] Yundong Li, Weigang Zhao, and JiahaoPan. "Deformable Patterned Fabric Defect Detection with Fisher CriterionBased Deep Learning." IEEE Transactions on Automation Science and Engineering 14.2 (2016): 1256-1264. [3] Ozdemir Ridvan and Koc Mehmet, A Quality Control Application on a Smart Factory Prototype Using Deep Learning Methods, in: International Conference on Computer Sciences and Information Technologies, CSIT ’14, IEEE, pp. 46–49. doi: 10.1109/STC-CSIT.2019.8929734. [4] Daniel Schmidt, Javier Villalba Diez, Joaquín Ordieres-Meré, Roman Gevers, Joerg Schwiep, and Martin Molina. "Industry 4.0 Lean Shopfloor Management Characterization Using EEG Sensors and Deep Learning." MDPI Sensors 20.10 (2020): 28-60. [5] Korkmaz Mehmet and Barstuğan Mücahid. "A Deep Learning-Based Quality Control Application." European Journal of Science and Technology Special Issue (2020): 332-336. [6] Desai Smita and Rajendra Kanphade. "Image Processing Using Median Filtering for Identification of Leaf Disease." Nanoelectronics, Circuits and Communication Systems (2021): 17-23. [7] Romualdas Bausys, Giruta KazakeviciuteJanuskeviciene, Fausto Cavallaro, and Ana Usovaite. "Algorithm Selection for Edge Detection in Satellite Images by Neutrosophic WASPAS Method." MDPI Sustainibility 12.2 (2020): 1-24. [8] Romualdas Bausys, Giruta KazakeviciuteJanuskeviciene, Fausto Cavallaro, and Ana Usovaite. "Algorithm Selection for Edge Detection in Satellite Images by Neutrosophic WASPAS Method." MDPI Sustainibility 12.2 (2020): 1-24. [9] Mondal, Ranjan, Moni Shankar Dey, and Bhabatosh Chanda. " Image Restoration by Learning Morphological OpeningClosing Network." Mathematical Morphology-Theory and Applications 4.1 (2020): 87-107.