To predict the quality parameters of products (in particular, the assembly parameters) mathematical models were implemented in the form of computer models. To ensure the adequacy of calculations, it is necessary to have information about the actual geometry of the parts, which can be obtained using noncontact measurements of parts of the assembly. As a result of measuring parts and components using optical or laser scanner, a large dimension array of measured points is formed. After standard processing (e.g. noise removal, combining the scans, smoothing, creating triangulation mesh), the recognition of individual surfaces of parts becomes necessary. This paper presents a neural network model that allows the recognition of elements based on an array of measured points obtained by scanning.
The least automated step in the industry is the assembly of single and serial products characterized by medium and high complexity. These products include aircraft engines. The considered products are not made in large quantities as cars; they are characterized by a high degree of optionality and increased requirements for complexity and accuracy. The share of labour-consuming assembly in the total labour-intensiveness of products is up to 25% and largely determines their quality. There are several reasons that make it difficult to fully automate the assembly of these products. One of the significant reasons is the difficulty of determining the parameters of the operations performed by robots, which are guaranteed to ensure the specified accuracy and quality of products. The assembly of medium and high complexity products is a unique operation, during which the course of operations is changed according to the results of measurements and the geometric analysis of the assembled parts. Measurement of geometry is made by both contactless and contact methods.
To partially automate engine assembly processes, it is necessary to recognize both the individual
parts and the surfaces of the parts along which the assembly will take place. Face recognition is
possible using computer vision approaches [
To test the model, a real engine parts simulator was designed and manufactured: spacers in the turbine of an aircraft engine. A detail drawing is shown in Figure 1. The part contains cylindrical and flat edges, as well as threaded holes.
Automated element (surface) recognition of measured parts using neural networks solves two tasks: 1) segmentation of a part’s components into types of surfaces (plane, cylinder, cone, etc.) 2) additional refinement of the boundaries of triangulation surfaces based on deviations of the facet normal vectors. To solve the first problem, a convolutional neural network was used.
Convolutional neural networks (CNN) are a very wide class of architectures which main idea is to
reuse parts of the neural network to work with different small, local input areas [
At present, many approaches have been developed for recognizing three-dimensional objects in
works devoted to computer vision. These approaches can be divided into two groups: recognition of
elements, directly working on their own three-dimensional representations of objects, such as
polygonal grids, voxel representations and arrays of points, and approaches based on signs and metrics
that describe the shape of a three-dimensional object, "what it looks like" in the collection 2D
projections [
Except for the recent work by Wu et al. [
One of the latest works on the problems of object classification and the segmentation of individual
parts, in which an array of measured points is used directly at the entrance to the network, is [
Based on the literature review and the specifics of the problem being solved, the decision was made
to use an approach based on the use of 2D projections of objects in solving the current problem. In this
approach, a convolutional neural network, U-net [
We will reveal the main idea and stages of the developed model for the segmentation of individual faces. The main idea is to create projections (pictures) for the faceted model of measured data, segmentation of faces on projections and identification of facets by segmented images. The stages of the approach are shown in Figure 2.
1. Loading measured points and creating a triangulation grid—
loading stl file
As noted in the introduction, the model is designed to recognize geometry after measurement using optical and laser scanners. After measurement using the scanner and preliminary data processing, a file is created with the coordinates of points united into a facet surface of the *.stl format. The file contains the following data: Vg×3 (matrix of coordinates of the vertices of the grid stl-model), Fm×3 (matrix of combinations of three vertices forming the facets of the surfaces), Nm×3 (matrix of coordinates of facet normals).
To enable semantic segmentation of the facets into separate surfaces using deep neural networks, it is
necessary to create projections of 3D surfaces on coordinate planes. To prepare the projections,
Roberts’ algorithm was used [
Roberts’ algorithm is the first known solution to the problem of removing invisible lines. This is a mathematically elegant method that works in object space. The algorithm primarily removes from each body the edge or edges that are screened by the body itself. Then each of the visible edges of each body is compared with each of the remaining bodies to determine which part or parts, if any, are shielded by these bodies. Therefore, the computational complexity of Roberts’ algorithm grows, theoretically, with the square of the number of objects.
The operation of Roberts’ algorithm takes place in two stages: 1. Definition of non-face faces for each body separately. 2. Identify and remove invisible edges.
To prepare the data, only the first stage of the algorithm was used. The second stage is not necessary for further decision; it is more complex and requires additional facets.
When creating projections on coordinate axes, orthogonal projections in the visual plane are obtained. For example, when projected on the XOY plane, the structure Fm×3 is preserved, and the matrix of the vertices Vg×3 is converted into Vg×2, having only coordinates along the x- and y-axes. Bypassing the vertices of the obtained projections of the facets in the same sequence as in the original, you can divide them into two types: those that are oriented counter-clockwise, which means that we are looking at the facet from the outside of the body and those that are oriented clockwise, which means that we are looking at the facet from the inside. The order of the vertices determines the direction of the normal. Thus, if the component of the normal vector of the projection plane (in this example, the component along the z-axis) is negative, we look at the facet from the inside. Since the object is bounded by a closed surface, we cannot observe the faces from the inside—they are invisible. Thus, it is necessary to exclude the facets identified by the above method from the Fm×3 structure, obtaining the projection structure Fm1×3, where m1<m.
Image saving is performed using STL work libraries (functions of the stltools package by Pau Micó) and MATLAB graphics saving tools. The sizes of the pictures are saved, as are all projections. Using the same tools, training projection images are saved, where objects for recognition are highlighted in different colours (Figures 3 and 4). (1)
Nevertheless, segmentation requires clear boundaries, so images stored in the *.png format are imported into MATLAB (variable loading matrix RGB-colour) and converted to grayscale using the expression:
I gray =0, 299 ⋅ R + 0, 587 ⋅ G + 0,114 ⋅ B , where I gray is the grayscale image matrix and R, G, B are the matrix components of the RGB system. Multidimensional matrixes of images are saved to a *.mat file. A total of six projections remain. The object is placed as if in a cube whose faces are parallel to the planes of the coordinates. The dimensions of the cube are such that it includes all the measured objects in the sample. Projections are accordingly made on the faces of the cube: two on the faces parallel to the XOY plane; two parallel to XOZ and two parallel to YOZ.
The U-Net network architecture is shown in Figure 5.
The network architecture is a sequence of layers of convolution and pooling, which first reduce the spatial resolution of the image, and then increase it by first merging it with the image data and passing it through other layers of the convolution. Thus, the network serves as a kind of filter.
The first half of the network contains layers of convolution with the activation function ReLu, normalization by mini-batch and layers of pooling (sub-sampling) and it is called a compressing path. The second part is an expanding path.
The upsampling layer is a reverse pooling layer that expands the feature map, followed by a convolution, which reduces the number of feature channels. Then comes the concatenation (“pasting” of linear objects) with an appropriately cut map of features from the compressive path and two convolutional layers.
On the last layer, convolution with a 1x1 core is used to bring each 64-component feature vector to the required number of classes. The activation function on the last layer is “softmax”.
The network was reproduced in the Python software environment.
After the image is segmented, its pixels are compared to the coordinates of the corresponding projection. For matching, pixels are converted to points on the corresponding face of the cube described in Section 3.3. The values of the image pixels lie in the interval from 0 to 255. Accordingly, in order to identify the pixels responsible for a certain edge, values of a certain intensity are searched for in the image matrix. In this case, the search is made with a certain tolerance. The position of the found pixels in the matrix (row-column) is translated, taking into account the scale, into space coordinates (two coordinates from the image, the third—the coordinate of the corresponding cube face). In addition, in Section 3.2, the faces closed by others were not deleted. Therefore, when matching, a point of an object in an image can fall on two or more projection facets. In this case, the facet that is closest to the viewpoint is selected.
The previous steps were necessary for preliminary automated recognition of only a fraction of the
measured points. After identifying the vertices and facets to different surfaces, they are compared with
the full set of vertices (search by equality of point coordinates). Points close to the geometric centre of
the surface are selected. On the top in the geometric centre, a facet is selected that belongs to a specific
face of the body. At the next stage, the algorithm for searching and refining facets of one face is used.
We briefly describe the steps of the algorithm for searching for facets belonging to a specific face
[
Facets are selected, from all the facets found that are suitable in the direction of the normal, that are associated with the first and among themselves common vertices.
To measure the details an optical 3D scanner RANGEVISION Pro2M was used. Figures 6 and 7 show photographs of the process of measuring the “spacer simulator” part (drawing in Figure 1). (2)
To assess the quality of segmentation, you can use the modified loss function given in [
In our case, we work with body facets, so instead of squares, we can operate on the number of facets. Therefore, a coefficient δ segm is calculated that is equal to the ratio of the number of facets N р∩д that are the intersection of the set of facets of the surface, obtained as a result of recognition, and actually belong to the surface of the facet to the N р∪д set of facets, which is their union: δ segm = N р∩д / N р∪д .
Thus, the value of the coefficient lies in the interval [0; 1]. If there are several recognizable surfaces, then a generalized coefficient can be calculated δ segm for all recognizable body faces.
The number of measured facets of the part m was 622130. For training the neural network, a sample of 1000 cases of the stl-model of the considered part, aligned in different ways in space, corresponding to a total of 6000 projections, was formed. For formation of the training set, a nominal model of the part was used, which was saved in the *.stl format.
Recognized flat and cylindrical faces of the “spacer simulator” detail measured with the scanner using all six projections are shown in Figures 8 and 9.
Table 1 shows the calculated coefficients δ segm of the measured cylindrical, flat and combined recognizable faces.
The recognition error of cylindrical faces is higher than flat ones. Although all cylindrical faces were identified, there were facets belonging to flat faces, as well as some facets from the threaded holes. The overall ratio exceeded 90%, which is associated with a much larger number of facets of flat faces compared to cylindrical.
The model presented in this paper allows you to quickly recognize cylindrical and flat surfaces of parts
using a trained neural network and face facet search algorithm from previously recognized facets. The
developed model is needed for further prediction of the assembly parameters of the product based on
computer modelling [
The work was supported by the Ministry of Education and Science of the Russian Federation in the framework of the program to improve the competitiveness of the Samara University among the world's leading research and educational centers for 2013–2020 and partly by the Russian Federation President's grants (project code СП-262.2019.5). Experimental studies were carried out on the equipment of the center for the collective use of CAM technologies of the Samara University (RFMEFI59314X0003).