Abstract

-The paper suggests the complex approach to solve the problem of segmentation of STM-images for the detection of nanoparticles, which consists in using curvature detectors as well as convolution neural networks of different architectures. The developed information system is implemented using modern methods of machine learning, computer vision and visual programming. Evaluation of the proposed segmentation algorithms is performed by calculating the number of found particles on segmented images and IoU metric. Also the results of their operation in processing real STM-images are given.

Keywords

• information system
• STM-image
• segmentation
• nanoparticles
• neural network

-

INTRODUCTION

In spite of significant achievements in the field of creating algorithms and programs for processing STMimages, development of new effective approaches for their processing still remains a relevant task. At present, the level of STM research has progressed significantly. From simple visualization of surfaces with nanometer resolution the researchers have proceeded to a serious analysis of data obtained using the scanning tunneling microscope (STM) and a more complete study of the sample surface [ 1-4 ]. Measuring the geometric parameters of nanoparticles is an important research task for creating materials with predicted properties. At the same time, there are increased requirements for the digital processing of measuring information.

The following applications are used as a software for scanning and filtering STM images: Callisto, Gwyddion, nSurf, SPIP, Nova. The analysis of the existing software for processing and visualization of the measuring information of the scanning tunneling microscope has revealed that the main drawback of all the considered software (except for SPIP) is insufficient detail of the visualization of images in threedimensional space due to the use of OpenGL technology. General drawback of the software is a lack of possibility to use several branches of image processing with possible simple change of filter parameters at different stages. Such tools can be provided by programs that use visual programming tools. This paper considers GraphMIC software complex for medical image processing that uses such tools. Interactive user interface components enable to change settings (e.g., reorder the filter sequence for processing or manage filter parameters). GraphMIC is best suited for processing experimental data and creating individual image processing pipelines. In general, analysis of existing software has identified the need to create a program combining all the positive features of existing STM-image processing software.

II.

NANOPARTICLE EXTRACTION IN STM-IMAGES USING MACHINE LEARNING, COMPUTER VISION AND SURFACE

In the work, the complex approach to solution of the problem of segmentation of STM-images for detection of nanoparticles is offered, which consists in using the curvature detectors as well as convolution neural networks of different architectures. Fig. 1 shows a scheme of segmentation of nanoparticles in the proposed information system.

Firstly, segmentation is carried out by neural network methods that require a large database for network training. Therefore, a step-by-step method of modeling the STMimage has been implemented. In the first stage, various methods (such as Perlin noise, Diamond-Square, Worley noise, fractal Brownian motion) are used as substrate generation algorithms. Then ellipsoids with different diameters are placed on the substrate by random law. The base of the obtained images serves as a training dataset for neural networks. Besides STM-image, a mask is needed to recreate the network. The algorithm used for their generation has been developed based on the calculation of an approximate value of the brightness gradient using the Sobel operator with G x and G y cores:

 values

G x   0   1  2  1 

0 0    1  2  1  

  1 0  1 G y    2 0  2    1 0  1  where G x and G y convolution gives approximate derivatives in the x and y axes. The gradient module is calculated from the G x and G y  are two matrices with which the   and direction of the gradient

G  ( G x )2  ( G y )2 ,  



G   a r c tg ( y ).

G x       The generation algorithm is described in more detail in [ 5 ]. As a result of the analysis of the convolution networks, several modern architectures, well-proven in image segmentation, have been chosen [ 6-20 ] and adapted for the developed information system. Then a real STM-image is applied to the input of the trained network, and segmented STM-image is formed at the output. The network results are displayed as the segmented image and the numerical value of the number of found nanoparticles in the image.

Secondly, segmentation task is carried out by the combined method including several stages. Initially, the centers of the particle nuclei are determined using the curvature detector [ 21 ]. The detector operation is based on the notions of surface convexity and concavity, the function of local curvature and its extremes, i.e., on the semantic characteristics of its structural elements. Then the received coordinates of particles are transmitted to the processing block of the "Watershed" algorithm of OpenCV library. After that, this block receives the centers of "troughs" for the areas to be filled in. The segmented image is submitted to the input of the findContours algorithm that draws and counts the number of found particles.

III. INFORMATION SYSTEM FOR AUTOMATIC

SEGMENTATION OF STM-IMAGES

The structural scheme of the information system is shown in Fig. 2. The information system for segmentation of nanoparticles on the STM-image is implemented as a program complex and includes the following structural elements:      

Input/output subsystem;

Connected external modules;

Primary processing subsystem;

A subsystem for generating model STM-images;

User block;

User block link map.

To process data obtained from different microscopes it is often necessary for the system to "know how" to handle different types of data obtained from STM. Therefore, a subsystem has been created for reading common formats used in these studies (.mdt, .pc).

The connection of external modules to the information system is an important addition that entails advantages as follows:     

Simplified extension of the complex functionality by other developers as there is no need to study the source code of the complex to make changes in it; Possibility to use previously written, debugged and tested program code in different programming languages with minor changes (C#, Python, Java, Delphi); Using capacities of different programming languages, libraries and development tools for them (OpenCV, Keras); Improving the stability of the complex due to localization of faults inside the module; Creating of module libraries used in the work of an individual user.

The primary processing subsystem is responsible for filtering the input image – it is important for eliminating interference in the STM-image. Methods of classification, clustering, segmentation are used to analyze data after primary processing and extract useful information about the material under study.

The model image generation subsystem consists of two blocks. Firstly, modeling of the substrate is performed. Its implementation in the form of polynomial function does not reflect the random nature of the surface on the real image. Therefore, different methods of fractal surface generation are used to obtain a model close to the real conditions. The height map, received as an outcome of the algorithms operation, creates such a "noise" that is difficult enough to be filtered by classical methods of computer filtration. Secondly, the simulation of nanoparticles is performed in the same block. Ellipsoids are distributed on the resulting surface of the substrate by the uniform law. Due to the difference in size and height the ellipsoids are superimposed on each other, which makes the segmentation process complicated. Masks for the obtained model STM-images are also formed.

The user block represents a multifunctional structure. For each created block there is a field with an image and fields necessary for the internal algorithm of the block. It is possible to select the input file, the primary processing filter and its parameters, the number of inputs and outputs (the number of images submitted to the input and the required number of outputs for further processing chains), to connect external modules, in which the segmentation of the STMimage was carried out, and to control the final processing result (the segmented STM-image).

The primary image convenient methods for manipulation of image contours, it is also possible to work with data in different programming languages (C, C++, C#, Python, etc.).

Fig. 3 shows a map of links of user's blocks in more detail as the implemented program module using the library NodeEditor. It performs several functions: control, linking the user's blocks and visualization of the results of each step of image processing.

Thus, the created information system has the following advantages:     

Convenient interface for the researcher's work, visible image processing chain, possibility to change parameters of this or that filter at any step; Support of various filters for primary image processing; Modeling of STM-images with different parameters (the number of particles, noise for substrate generation); Possibility to add program modules from other programming languages (Python, C#, Java, Delphi);

Support for detection methods.

RESULTS AND DISCUSSION INFORMATION

The studies have yielded a wealth of information and evaluated the proposed methods. To assess the quality of segmentation the metric IoU (Intersection over Union) is used which is calculated by the formula:

Io U 

T P (T F  F P  F N )    where TP, TF, FP, FN – the number of pixels correctly assigned to the class «particle», correctly assigned to the class «background», incorrectly assigned to the class «particle» and incorrectly assigned to the class «background», respectively. Table 1 shows the results of segmentation by different methods.

Evaluation of segmentation algorithms has been carried out and showed that methods using curvature detectors better determine the number of particles on the scanned surface. However, neural networks trained on model images using the DiamondSquare method are more resistant to various distortions and artifacts in the STM-image, detect the boundaries of particles more accurately and also are able to separate "sticky" particles from each other.

The result of the joint work of the Watershed algorithm and the Horde detector is shown in Fig. 4. Fig. 5 illustrates the results of segmentation of real STM-images after training networks based on model STM-images (generation of the substrate by DiamondSquare method). The networks have been trained by the method of stochastic gradient descent based on input images and corresponding masks. 512 training images of 256x256 pixels were supplied to the input of the networks. In total, the networks were trained for about 12 hours on a GeForce GTX 1070 graphics card.

Neural network methods and methods based on the application of curvature detectors have been aggregated, and a new integrated approach has been created on the basis of the hierarchy analysis method. The method of hierarchy analysis (proposed by T.L. Saati) [ 22, 23 ] consists in using a hierarchical structure incorporating the purpose of choice, criteria, alternatives and other factors influencing the choice of the solution. The construction of such a structure helps to analyze all aspects of the problem and to penetrate deeper into the essence of the problem. The top of the hierarchy is the main goal – segmentation of nanoparticles; elements of the lower level represent many variants of achieving the goal (alternatives) – segmentation methods; elements of the intermediate levels – nodes that meet the criteria or factors that link the goal with the alternatives. The target is the parent node for all criterion nodes.

V.

CONCLUSION

As a result of the research carried out, the information system was developed using modern methods of machine learning, computer vision and visual programming. A new complex approach to solve the problem of segmentation of nanoparticles on STM-images is proposed, which allows to increase the reliability of control of particles on the surface consisting in the use of curvature detectors as well as convolution neural networks of different architectures. The proposed segmentation algorithms have been evaluated and have proved to be highly efficient in operation.

References

[1] T. Fukuma , K. Onishi , N. Kobayashi , A. Matsuki and H. Asakawa , “ Atomic-resolution imaging in liquid by frequency modulation atomic force microscopy using small cantilevers with megahertz-order resonance frequencies , ” Nanotechnology , vol. 23 , pp. 12 , 2012 .

[2] C. Gerber and H.P. Lang , “ How the doors to the nanoworld were opened,” Nature Nanotechnology , vol. 1 , no. 1 , pp. 3 - 5 , 2006 .

[3] E. Yu . Shelkovnikov, “ Hybrid tunneling microscope with integrated biotentiostat for investigation of the nanoparticles,” Chemical physics and mesoscopy , vol. 13 , no. 3 , pp. 448 - 451 , 2011 .

[4] K. Bui , “ Segmentation of scanning tunneling microscopy images using variational methods and empirical wavelets , ” Pattern Analysis and Applications , pp. 20 , 2018 .

[5] E. Yu. Shelkovnikov , K.A. Shlyakhtin , T.E. Shelkovnikova and S.F. Egorov , “ U-Net architecture neural network application for STMimages segmentation ,” Chemical physics and mesoscopy , vol. 21 , no. 2 , pp. 330 - 336 , 2019 .

[6] P. Goldsborough , N. Pawlowski , J. C Caicedo , S. Singh and A. Carpenter , “Cytogan: Generative modeling of cell images,” 2017 [Online] . URL: bioRxiv 227645.

[7] O. Ronneberger , P. Fischer and T. Brox , “U-Net: Convolutional networks for biomedical image segmentation,” International conference on medical image computing and computer-assisted intervention , pp. 234 - 241 , 2015 .

[8] Y. LeCun , “Backpropagation Applied to Handwritten Zip Code Recognition,” Neural Computation , vol. 1 , no. 4 , pp. 541 - 551 , 1989 .

[9] J. Hu , L. Li , Y. Lin , F. Wu and J . Zhao , “ Comparison and Strategy of Semantic Segmentation on Remote Sensing Images,” Advances in Natural Computation, Fuzzy Systems and Knowledge Discovery . Advances in Intelligent Systems and Computing , pp. 21 - 29 , 2020 .

[10] V. Badrinarayanan , A. Kendall and R. Cipolla , “ SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation ,” PAMI, p. 14 , 2017 .

[11] H. Zhao , J. Shi , X. Qi , X. Wang and J. Jia , “ Pyramid scene parsing network , ” CVPR , 2017 , 11 p.

[12] Y. LeCun , Y. Bengio and G. Hinton , “Deep learning, ” Nature , vol. 521 , no. 7553 , pp. 436 - 444 , 2015 .

[13] S. Jesse , “ Big data analytics for scanning transmission electron microscopy,” Ptychography Sci. Rep . vol. 6 , no. 26348 , 2016 .

[14] N. Lubbers , T. Lookman and K. Barros , “ Inferring low-dimensional microstructure representations using convolutional neural networks , ” Phys. Rev. E , vol. 96 , 052111 , 2017 .

[15] A. Belianinov , “ Big data and deep data in scanning and electron microscopies: deriving functionality from multidimensional data sets , ” Adv. Struct. Chem. Imaging , vol. 1 , no. 6 , 2015 .

[16] A. Chowdhury , E. Kautz , B. Yener and D. Lewis , “ Image driven machine learning methods for microstructure recognition , ” Comput. Mater. Sci. , vol. 123 , pp. 176 - 187 , 2016 .

[17] W. Li , K.G. Field and D. Morgan, “ Automated defect analysis in electron microscopic images,” Npj Computational Materials , vol. 4 , no. 36 , 2018 .

[18] T.E. Shelkovnikova , K.A. Shlyakhtin and S.F. Egorov , “ Analysis of the convolution neural networks for the nanoparticles segmentation on the STM-image,” Polzunovsky almanah , vol. 4 , pp. 10 - 13 , 2019 .

[19] A.V. Nikonorov , M.V. Petrov , S.A. Bibikov , V.V. Kutikova , A.A. Morozov and N.L. Kazanskiy , “ Image restoration in diffractive optical systems using deep learning and deconvolution ,” Computer Optics , vol. 41 , no. 6 , pp. 875 - 887 , 2017 . DOI: 10 .18287/ 2412 -6179- 2017-41-6- 875 -887.

[20] T.A. Pashina , A.V. Gaidel , P.M. Zelter , A.V. Kapishnikov and A.V. Nikonorov , “ Automatic highlighting of the region of interest in computed tomography images of the lungs ,” Computer Optics , vol. 44 , no. 1 , pp. 74 - 81 , 2020 . DOI: 10 .18287/ 2412 -6179-CO- 659 .

[21] P.V. Gulyaev , Yu.K. Shelkovnikov , A.V. Tyrikov and S.R. Kiznertsev , “ Application of the curvature detectors for a coordinate binding of the overlapping images in a probe microscopy,” Polzunovsky vestnik , vol. 2 , pp. 122 - 125 , 2016 .

[22] T.L. Saati , “ Decision-making at dependences and feedback: Analytical networks ,” Moscow: LKI Publishing House, 2008 .

[23] T.L. Saaty , “ The Hierarchon: A Dictionary of Hierarchies Pittsburgh,” Pennsylvania: RWS Publications , 1992 .