=Paper=
{{Paper
|id=Vol-2353/paper82
|storemode=property
|title=Method of Operative Search of Solder Joints During Printed Circuit Boards Development
|pdfUrl=https://ceur-ws.org/Vol-2353/paper81.pdf
|volume=Vol-2353
|authors=Galina Shcherbakova,Viktor Krylov,Anatoliy Sachenko,Pavlo Bykovyy,Diana Zahorodnia,Myroslav Komar,Ivan Kit,Dmytro Lendiuk,Andriy Kaniovskyi,Mykola Dacko
|dblpUrl=https://dblp.org/rec/conf/cmis/ShcherbakovaKSB19
}}
==Method of Operative Search of Solder Joints During Printed Circuit Boards Development==
https://ceur-ws.org/Vol-2353/paper81.pdf
Information Technologies based on Wavelet
Transform for Soldered Joints Diagnostic of Printed
Circuit Boards
Galina Shcherbakova1[0000-0003-0475-3854], Viktor Krylov1[0000-0003-1950-4690],
Anatoliy Sachenko2[0000-0002-0907-3682], Pavlo Bykovyy3[0000-0002-5705-5702],
Diana Zahorodnia3[0000-0002-9764-3672],, Myroslav Komar3[0000-0001-6541-0359],
Ivan Kit 3, Dmytro Lendiuk3, Andriy Kaniovskyi3, Mykola Dacko3
1
Odessa National Polytechnic University, Odessa, Ukraine
2
Kazimierz Pulaski University of Technology and Humanities in Radom, Radom, Poland
3
Ternopil National Economic University, Ternopil, Ukraine
galina_onpu@ukr.net
Abstract.In modern instrumentation, the number of soldered joints in printed
circuit boards can reach several thousand. Diagnostics of the soldered joints
defects within the optical wave length range is carried out using automated
diagnostic systems. A number of stages of the existing information technologies
for such systems are implemented on the basis of target functions extremum
search using the gradient estimation. In the large batches of products
production, the use of expensive automated diagnostic systems within lighting
subsystems and high cost positioning are justified. These subsystems can
provide improved noise immunity. However, in conditions when small batches
of products are produced, and at some stages (for example, when positioning by
comparison with a standard/prototype images) in general, such objective
functions can be noisy and can be multi-extremes. For such cases, information
technologies based on methods of enhanced noise immunity are required. Such
an increase in noise immunity can be provided by methods using wavelet
transformation. For this purpose, information technologies were proposed using
wavelet transformation-based procedures that improve noise immunity and
reduce the error of procedures in the diagnostic systems of printed circuit
boards and their soldered joints.
Keywords: Automated Diagnostic Systems, Solder Joint, Printed Circuit
Boards, Wavelet Transformation
1 Problem description
In modern instrument production, the number of solder joints (SJ) in the printed cir-
cuit boards (PCB) can reach several thousand [1-14]. Diagnostics of solder joint de-
fects in the optical wavelength range is carried out in automated diagnostic systems
(ADS) of the solder joints (Fig. 1) [15-26]. In the existing ADS, the detection of PCB
assembly defects is carried out by comparing their images with the images without the
defects (standard/prototype images). Such a comparison in the ADS is carried out
using systems of the precision mechanics and high cost optics.
Reducing the requirements for such systems in order to reduce the cost of small
batches of products production leads to uneven illumination of objects and the
appearance of multiplicative and additive interference in images. In conditions with
interferences and signal-to-noise ratio (power) for additive interference ranging from
8 to 11 and from 7 to 10 for multiplicative interference, such automated diagnostic
systems are not operational. In the existing automated diagnostic systems, the
combination of images for comparison is carried out on the basis of gradient methods
searching for the minimum of the multimodal function and this way to determine the
error in combining with the standard/prototype image. Such methods are not
functional within the signal / interference ratio being less than 15. Therefore, for such
a combination of images, information positioning technologies are needed because it
can provide necessary reliability in the conditions of interference and a priori
uncertainty due to small sets of training data. To this end, it is proposed to implement
these IT on the basis of the developed multi-start optimization methods of the wavelet
transformation [27-32].
Often the reduction of the resource consumption in automated diagnostic systems
and reduction the operational speed are increased by conducting diagnostics on the
conveyor for PCB panels (not separate PCB’s) (Fig. 2). In this case, several auto-
mated diagnostic systems are present at different stages of the PSB assembly. For
example, when assembling a PCB during mounting components on a surface, these
are the stages of pattern printing, resistors and integrated circuits (ICs) placement
(with a small output range) (Table 1).
Fig. 1. Fragment of the automated diagnostic system (ADS) block diagram for soldered joints
within the optical range
Fig. 2. PCB Panel
A generalized structure of the automated diagnostic system (ASD) in printed
circuit board assemblies has been developed within the optical range (Fig. 3).
Fig. 3. Generalized structure of the automated diagnostic system for the printed circuit
assemblies diagnostics within the optical range
Such an ASD may contain the following subsystems:
− subsystem influencing the object, which contains device impact on the object
sources of illumination and / or penetrating fields (with fluoroscopy) and scanning
devices;
− transportation mechanical subsystem, which contains the object delivery
devices (loading and unloading elevators, conveyor), coordinate table, positioning
devices;
− subsystems of signal acquisition and information conversion, which contain
receiver-transducers of penetrating fields, devices for a signal pickup from receiver-
transducers, devices for primary conversion and normalization of signals, image
sensors, video processing devices (amplification, quantization and normalization of
the video signal);
− subsystems of the computer-based processing and information storage, that
are based on applied IT implemented signal and image processing.
Table 1. Types of defects at various PCB assembly stages
Stage Type of defects detected during optical inspection
Screen printing soldering Skip of the paste, improper application, lintels
paste
Component positioning The absence/presence of components, bevel, shear, wrong
orientation and/or polarity.
Printed circuit board (PCB) warping
Melting solder Presence/absence, shear, bevel, polarity of components,
correctness of soldered joints, defects in paste failure, solder
bridges, raised connectors.
Practically, it is necessary to control the soldered joints for all PCB after the solder
melting step. Such diagnostics in the existing integrated circuits are carried out by
comparing the solder joints on the PCB with the reference images of the solder joints,
so it is important to carry out the exact combination of such images.
When choosing the number of automated diagnostic systems, the cost of repairs af-
ter performing this operation, the type of equipment for which these launchers are
assembled for are taken into consideration. It is also taken into account that up to half
of all defects of the PCB is concentrated in soldered joints of integrated circuits, as
well as the fact that the cost of the repairs after soldering averages up to 60% of the
repair cost of the PCB. Therefore, for medical equipment and security systems (for
example, air bags in cars), it is recommended to perform diagnostics at the stage of
placement of the integrated circuits. Such diagnostics is carried out by comparing
with reference images of PCB, therefore the exact combination of images of such type
is important. Therefore it is necessary to carry exact positioning of such images (in
order to reduce costs of the automated diagnostic systems, the cost of PCB and to
increase the performance).
2 Main stages of the developed information technologies of the
automated system for printed circuit boards and soldered
joints diagnostics
The developed information technology for PCB automated diagnostic systems in-
cludes the following steps.
Stage 1. Positioning of the PCB panel image along the contours of the reference
marks (fig 4).
Fig.4. Processing sequence when information technology for positioning and defects selection
on the print nodes are being used
This IT allows for automated positioning to use images of reference marks (RM) -
marks of a familiar shape on the surface of printed circuit assemblies PCBs (Fig. 5)
and has two modes: training mode and operating mode.
1 2 3 4 5 6 7
Fig. 5. Classes of reference marks
The training mode includes the following main stages (Fig. 6).
Stage 1. Pre-processing of the image of the product obtained with the help of a
video camera in order to reduce the level of noise.
Stage 2. Binarization of the product image.
Stage 3. Localization of the reference marks. With the localization of reference
marks, their coordinates on the image are determined, and their contours are selected
and traced. The result of this stage is the coordinates of the contours of the reference
signs.
Stage 4. Determination of the identification vector of the reference marks. As a
result, a training sample is formed to classify reference marks.
Stage 5. Training of a multilayered neural network (MNN), as a result of which
the surface is built, dividing the reference signs in the feature space. This dividing
surface is used on the next stage in the operating mode during the classification.
Fig. 6. Basic IT of the reference marks positioning procedures
In operating mode, the following steps are implemented.
Stage 1. Classification of reference marks. The operator of the ASD receives
information about the belonging of the reference marks to a certain class - that is, the
result of recognition - a classification decision.
Stage 2. Positioning of the object image on the reference marks is relative to the
standard, coming from the database (Fig. 7).
Fig. 7. Fragment of the PCB image and the result of localization of reference marks
The implementation specifics of the IT procedures for the positioning of printed
circuit assemblies of control stations by the reference marks are described below:
a) image pre-processing. In order to reduce the level of additive interference
arising in a television sensor at the stage of forming a digital image of an object, is
proposed to be conducted with the application of the median filtering. It implements a
non-linear noise removal procedure. When the procedure is implemented, a window is
created that moves through the image, covering an odd number of its pixels. The
value of the current pixel is replaced by the median of the pixel values in the window.
The intensity drops with such processing in contrast to the smoothing filters. In order
to reduce the level of multiplicative noise caused by uneven illumination,
homomorphic filtering was applied.
b) binarization. To highlight the defects of the printed circuit assemblies of the
PCB in the ASD, binarization of their images and references is performed. This
procedure should ensure invariance to intensity transformations and reduce the
amount of information processed by the system. In this case, the error of binarization
should not exceed the 50% of the track width of printed circuit assemblies of the
PCB. During binarization, the following decision is made: is there a track (useful
signal) or substrate (background) among the pixels of the image? Depending on this,
the pixel is assigned the value "0" or "1".
c) localization. For the localization of reference marks, a method of localization of
small-sized objects in the image using the wavelet transform was chosen. As a result
of the localization procedure, a traced contour of the reference marks of PCB and the
coordinates of their position (centers of gravity) are obtained (Fig. E). This reduces
the amount of information for the identification procedure.
d) identification. Based on the calculation of geometric moment features using
information about the characteristic points - the most informative part of the contour
of the reference marks. The first and second coefficients of the discrete cosine
transform of the moment-signs, the geometric sign of compactness, and the Euler
number are included in the feature set. The selected features provide: scale, shift, and
rotation invariances sufficient for the requirements of the practice.
e) reference marks classification. It is carried out with the help of MNN
(perceptron). The result of MNN learning is stored in the database with the
parameters of the MNN (see Fig. 6). Then, at the recognition stage, a set of reference
marks obtained under the noisy conditions is supplied to the input of the perceptron.
As a result of the calculations, the class of the relay is determined in the network
output.
IT for defect allocation involves the following steps.
Stage 1. Binarization of the reference images and controlled products.
Stage 2. Median filtering of the product image. In order to exclude false defects
from consideration (relative shift results by 1 pixel on images, for example, current
PCB and standard one), median filtering is used to ensure the noise immunity of the
decision on the presence of a defect. By varying the length of the median filter, you
can identify defects with a given accuracy.
Stage 3. Determination of the resistors and capacitors solder joints defect locations
on the image of the PCB.
Selection of such defects is performed by combining the binarized images of the
PCB relative to the standard ones based on their pixel-by-pixel comparison. In Fig. 8
a fragment of the PCB image is presented with identified defects.
Fig.8. Fragment of the PCB assembly with defects in the capacitors soldered joints
To assess the recognition accuracy, a database was used which consisted of 100
photographs of printed circuit assemblies of various sizes, rotation and tilt angles, and
various levels of illumination. As a result of testing such an ASD, the recognition
accuracy was 0.85. The noise immunity of the system was increased on average by up
to 20% compared to the existing systems through the use of developed information
technology that meets the requirements of practice. The reliability of diagnostics in
terms of interference on the signal-to-noise ratio range 10 ... .20 (in power) increased
up to 1.2 times in comparison with the existing ASD.
The next step is to determine the defects of components with a small lead pitch
(Fig. 9). Information technology for components positioning in automated diagnostic
systems for soldered joints with a small range of component conclusions involves the
implementation of two main stages.
The first stage is the positioning of the PCB by the reference marks, the second
stage is the positioning of the resistors, capacitors and integrated circuits on the PCB
image (see Fig. 1).
At the second stage, in order to determine the defects of the PCB of the integrated
circuit, it is necessary to identify the region of these integrated circuits location on the
image of the PCB. To solve this problem, the following methods are used, the ones
that use information about the entire halftone image, and methods that use informa-
tion about specific areas of the image - characteristic features.
For the methods of the first group, the computational costs are high, since the pa-
rameters of the soldered joints positioning are calculated from the information about
the entire half-tone image of the PCB. The characteristic features method is easy to
implement and has higher speed. Therefore, this method was chosen as the baseline
for the positioning on the second stage (fig. 3). When searching for the position of the
integrated circuits on the PCB, the minimum of the multimodal function should be
found, which determines the error of combining the integrated circuit component and
a sample (fig 3.a), with a signal-to-noise ratio below 6. Gradient methods of the min-
imum search that are traditionally used at this stage are not functional Signal / inter-
ference ratio is less than 15. To search for a minimum in such conditions, a multi-start
optimization method in the wavelet transformation space has been developed [27].
This method was used in the work to search for the location of the integrated circuits
and to align the reference image and controlled integrated circuits components in the
ADS with the processing of information on soldered joints obtained in the optical
range [27-29].
Within this the binarization of the PCB image by converting a halftone image (with
a pixel intensity ranging from 1 to 256) into a binary one according to the intensity
threshold is carried out. For this, two thresholds of binarization were accepted in the
paper T1 and T2. Pixel of the binarized image B (x, y) takes the value of 1
if A(x, y) < T1 or A(x, y) > T 2 and the value of 0 if T1 < A(x, y) < T 2 .
Here A (x, y) is a halftone image T1 77 and T2 156 - binarization thresholds.
The result of binarization is shown in Fig. 9, c.
a)
b) c)
d) e)
Fig. 9. Error of PCB combining function and a standard (a); a fragment of the PCB (b); bi-
narized image (c); masks with the designation PCB (U1) (d)
and with soldered joints (e).
Than the image of the PCB is scanned using a mask with the marking of U1 (the
label of this IS on the PCB) (see Fig. 9, d). The minimum of the combining image
error (see Fig. 9, a) during such a scanning is found with the help of the multi-start
optimization method with WT. The carrier length of the wavelet function (WF) in the
search for a minimum in the coordinate x is 17, in y it is 8, the discretization step of
the WF is 1, and in the implementation of the iterative search with the Haar‘s
WF γ1 = 0,1 .
The minimum of the image alignment error was found in 3 iterations (at the start of
the search in [1; 1], the coordinates of the minimum (25; 14), (19; 13), (17,13) were
found). At the second stage of the search in the region of this minimum, scanning of
the image PCB is performed with an image mask of the SJ of this IS (see Fig. 9, e).
The minimum of this error function is also determined by using the optimization
method with the WT for three iterations.
Than the positioning of the integrated circuit on the image PCB is made.
The developed information technology allows a fast search of soldered joints while
the components are being mounted to the surface with a signal-to-noise ratio in
amplitude starting from 5.
3 Conclusions
Thus, it can be concluded that during testing of the developed information
technologies for soldered joints and automated diagnostic systems of the PCB based
on procedures using wavelet transformation, the recognition accuracy was 0.85. In
terms of interference in the signal-to-noise ratio range 10 to 20 (in power) the
reliability of diagnostics increased up to 1.2 times compared to the existing automated
diagnostic systems. The noise immunity was increased on average by up to 20%
compared to the existing systems due to the use of developed information
technologies that meet the requirements of practice.
References
1.Gu, Jian, Lei, Yongping, Lin, Jian, Fu, Hanguang, Wu, Zhongwei: Design of the printed
circuit board for board level drop impact base on the JEDEC standard. 17th International
Conference on Electronic Packaging Technology (ICEPT). pp. 588 – 591 (2016).
2.Krylov, V., Shcherbakova, G.: The combination of images in the optical control systems of
the printed circuit boards. Technology and design in electronic equipment, № 1, pp. 23–25
(2000).
3.Panchak, R., Kolesnyk, K., Lobur, M.: Topology editing strategies in the subsystem of
printed circuit boards manufacturability improvement. 11th International Conference The
Experience of Designing and Application of CAD Systems in Microelectronics (CADSM),
pp. 246 – 247 (2011).
4.Hu, Chaohui, Zhou, Lina: Failure analysis for micro-short circuit between two pins in
printed circuit board assembly. 11th International Conference on Reliability, Maintainabil-
ity and Safety (ICRMS), pp. 1-4 (2016).
5.Moganti, M., Ercal, F., Dagli, C., Tsunekawa, S.: Automatic PCB inspection algorithms: a
survey. Computer vision and image understanding. Vol. 63, No.2. pp. 287–313 (1996).
6.Zhang, Yong, Yang, Peng, Ji, Mingjiang, Liu, Guanjun: Mechanism of Solder Joint Inter-
mittent Faults and Its Detection. In: IEEE International Symposium on the Physical and
Failure Analysis of Integrated Circuits (IPFA), pp. 1 – 5 (2018).
7.Weibo, Huang, Peng, Wei: A PCB Dataset for Defects Detection and Classification. Jour-
nal of Latex Class Files, Vol. 14, No. 8, August 2018. рp.1-9 (2018).
8.Bangshu, Xiong, Qiaofeng, Ou: A measurement system for printed circuit board parame-
ters based on image. 4th IEEE Conference on Industrial Electronics and Applications, pp.
2396 - 2400 (2009).
9.Borba, J.F., Facon, J.: A printed circuit board automated inspection system. Proceedings of
the 38th Midwest Symposium on Circuits and Systems. Vol.1, pp. 69 – 72 (1995).
10.Pedro, M.A., Vitoriano, Tito, G.: Amaral: Improved Pattern Matching Applied to Surface
Mounting Device Components Localized in Automated Optical Inspection. World Acad-
emy of Science, Engineering and Technology International Journal of Computer and In-
formation Engineering, vol:11, no:4, pp. 435-439 (2017).
11.Yao, Bin, Lu, Yudong, Luo, Daojun: Key failure modes of solder joints on HASL PCBs
and root cause analysis. 14th International Conference on Electronic Packaging Technol-
ogy, Dalian (China), 11-14 Aug. 2013, pp 742-745 (2013).
12. Švecová, O., Kosina, P., Šandera, J., Szendiuch, I.: Reliability model for assessment of
lifetime of lead-free solder joints. 18th European Microelectronics & Packaging Confer-
ence, Brighton (UK), 12-15 Sept. 2011, pp 1-5 (2011).
13.Lou, Haohuan, Qu, Xin, Chen, Zhaoyi, Wang, Jiaji, Taekoo, Lee, Hui, Wang: Lifetime
Assessment of Solder Joints of BGA Package in Board Level Drop Test. 6th International
Conference on Electronic Packaging Technology, Shenzhen (China), 30 Aug.-2 Sept.
2005, pp 1- 5 (2005).
14.Carroll, D., Bates, C., Zampino, M., Jones, K.: A novel technique for modeling solder
joint failure during system level drop simulations. Thermal and Thermomechanical Pro-
ceedings 10th Intersociety Conference on Phenomena in Electronics Systems (ITHERM),
San Diego (CA, USA), 30 May-2 June 2006, pp. 861-868 (2006).
15.Belbachir, A.N., Lera, M., Fanni, A., Montisci, A.: An automatic optical inspection system
for the diagnosis of printed circuit based on neural networks. IEEE 40th Industry Applica-
tions Society Annual Meeting. – Hong-Kong : China, 2005. pp. 680–684 (2005).
16.Garcia, H.C., Villalobos, J.R., Runger, G.C.: An Automated Feature Selection Method for
Visual Inspection Systems. IEEE Transactions on Automation Science and Engineering.
Vol. 3, Iss. 4, pp. 394 - 406 (2006).
17.Garcia, H.C., Villalobos, J.R.: Automated Refinement of Automated Visual Inspection Al-
gorithms. IEEE Transactions on Automation Science and Engineering., Vol. 6 , issue 3,
pp. 514 – 524 (2009).
18.Zahorodnia, D., Pigovsky, Y. Bykovyy, P., Krylov, V., Sachenko, A., Molga, A.: Auto-
mated Video Surveillance System based on Hierarchical Object Identification. 14th Inter-
national Conference on Development and Application Systems (DAS), Suceava (Roma-
nia), 24-26 May 2018, pp. 194-199 (2018).
19.Garcia, H.C., Villalobos, J.R., Pan, R., Runger, G.C.: A Novel Feature Selection Method-
ology for Automated Inspection Systems. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence. Vol. 31, Issue 7, pp. 1338 – 1344 (2009).
20.Park, J.-S., Tou, J.T.: A solder joint inspection system for automated printed circuit board
manufacturing. Proceedings of IEEE International Conference on Robotics and Automa-
tion. Vol.2, pp. 1290 – 1295 (1990).
21.Park, J.-S., Tou, J.T.: Automatic inspection of assembled PC board via highlight separation
and dual channel processing. Proceedings of IEEE International Conference on Robotics
and Automation. vol.3, pp. 2702 – 2707 (1991).
22.Pierce, B.L., Shelton, D.J., Longbotham, H.G., Baddipudi, S., Yan, P.: Automated inspec-
tion of through hole solder joints utilizing X-ray imaging. IEEE Aerospace and Electronic
Systems Magazine, vol. 9, issue 2, pp. 28 – 32 (1994).
23.Asaad, F. Said, Bonnie, L., Bennett, L.J., Karam, Pettinato, J.S.: Automated Detection and
Classification of Non-Wet Solder Joints. IEEE Transactions on Automation Science and
Engineering, vol. 8, issue 1, pp. 67 – 80 (2011).
24.Kuk, Won Ko, Young, Jun Roh, Hyung, Suck Cho, Hyung, Cheol Kimn: A neural network
approach to the inspection of ball grid array solder joints on printed circuit boards. Pro-
ceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks.
(IJCNN 2000). Neural Computing: New Challenges and Perspectives for the New Millen-
nium, vol. 5, pp. 233-238 (2000).
25.Ko, K.W., Cho, H. S.: Solder joint inspection using a neural network and fuzzy rule-based
classification method . IEEE Transaction on electronics packaging manufacturing. No. 23
(2). pp. 93–103 (2000).
26.Ibrahim, Z., Aspar, Z., Al-Attas, S.A.R., Mokji, M.M.: Coarse resolution defect localiza-
tion algorithm for an automated visual printed circuit board inspection. IEEE 2002 28th
Annual Conference of the Industrial Electronics Society. (IECON 02), Vol. 4, pp. 2629 -
2634 (2002).
27.Shcherbakova, G., Gerganov, M., Antoshchuk, S., Polyakova, M., Sachenko, A., Krylov,
V.: Areal Multistart Method of Optimization for Image Recognition. IEEE Second Interna-
tional Conference on Data Stream Mining & Processing, Lviv (Ukraine), 21-25 August
2018, pp.605-608 (2018).
28.Krylov, V., Antoshchuk, S., Shcherbakova, G.: Photomasks Diagnostic in the Automated
Optical Inspection System. Modern problems of radio engineering, telecommunication and
computer science: Int. conf. TCSET’2008, Lviv – Slavske, Ukraine, 19–23 February 2008,
pp. 533–534 (2008).
29.Shcherbakova G., Krylov V., Abakumov V., Brovkov V., Kozina I.: Sub Gradient Iterative
Method for Neural Networks Training. In: The 6th IEEE International Conference on In-
telligent Data Acquisition and Advanced Computing Systems: Technology and Applica-
tions - Proceeding of Conference, Prague, 15-17 September 2011, pp.361 – 364 (2011).
30.Paliy, I., Dovgan, V., Boumbarov, O., Panev, S., Sachenko, A., Kurylyak, Y., Zagorodnya,
D.: Fast and Robust Face Detection and Tracking Framework. Proceedings of the IEEE 6th
International Conference on Intelligent Data Acquisition and Advanced Computing Sys-
tems: Technology and Applications (IDAACS’2011), Prague (Czech Republic), 15-17
September 2011, vol.1, pp. 430-433 (2011).
31.Zahorodnia, D., Pigovsky, Y., Bykovyy, P.: Canny-based Method of Image Contour Seg-
mentation. International Journal of Computing, Vol. 15(3), pp. 200-205 (2016).
32.Zahorodnia, D., Pigovsky, Y., Bykovyy, P., Krylov, V., Sachenko, A.: Information tech-
nology for structural and statistical identification of hierarchical objects. Proceedings of
the 14th International Conference on Advanced Trends in Radioelecrtronics, Telecommu-
nications and Computer Engineering (TCSET), Lviv-Slavske (Ukraine), 20-24 February
2018, pp. 272-275 (2018).