Information system for segmentation of nanoparticles in STM-images Stanislav Egorov Igor Arhipov Tatyana Shelkovnikova Institute of Mechanics Kalashnikov Izhevsk State Technical Institute of Mechanics Udmurt Federal Research Center University Udmurt Federal Research Center Izhevsk, Russia Izhevsk, Russia Izhevsk, Russia stos.mitm@mail.ru aio1024@mail.ru shelktan@udman.ru Abstract—The paper suggests the complex approach to II. NANOPARTICLE EXTRACTION IN STM-IMAGES USING solve the problem of segmentation of STM-images for the MACHINE LEARNING, COMPUTER VISION AND SURFACE detection of nanoparticles, which consists in using curvature CURVATURE DETECTORS detectors as well as convolution neural networks of different architectures. The developed information system is In the work, the complex approach to solution of the implemented using modern methods of machine learning, problem of segmentation of STM-images for detection of computer vision and visual programming. Evaluation of the nanoparticles is offered, which consists in using the proposed segmentation algorithms is performed by calculating curvature detectors as well as convolution neural networks of the number of found particles on segmented images and IoU different architectures. Fig. 1 shows a scheme of metric. Also the results of their operation in processing real segmentation of nanoparticles in the proposed information STM-images are given. system. Keywords—information system, STM-image, segmentation, Firstly, segmentation is carried out by neural network nanoparticles, neural network. methods that require a large database for network training. Therefore, a step-by-step method of modeling the STM- I. INTRODUCTION image has been implemented. In the first stage, various In spite of significant achievements in the field of methods (such as Perlin noise, Diamond-Square, Worley creating algorithms and programs for processing STM- noise, fractal Brownian motion) are used as substrate images, development of new effective approaches for their generation algorithms. Then ellipsoids with different processing still remains a relevant task. At present, the level diameters are placed on the substrate by random law. The of STM research has progressed significantly. From simple base of the obtained images serves as a training dataset for visualization of surfaces with nanometer resolution the neural networks. Besides STM-image, a mask is needed to researchers have proceeded to a serious analysis of data recreate the network. The algorithm used for their generation obtained using the scanning tunneling microscope (STM) has been developed based on the calculation of an and a more complete study of the sample surface [1-4]. approximate value of the brightness gradient using the Sobel Measuring the geometric parameters of nanoparticles is an operator with G x and G y cores: important research task for creating materials with predicted properties. At the same time, there are increased 1  2  1 1 0  1 requirements for the digital processing of measuring     Gx  0 0 0 G y  2 0 2 information.       1  2  1    1 0  1  The following applications are used as a software for     scanning and filtering STM images: Callisto, Gwyddion, n- where G x and G y are two matrices with which the Surf, SPIP, Nova. The analysis of the existing software for convolution gives approximate derivatives in the x and y processing and visualization of the measuring information of axes. The gradient module is calculated from the G x and G y the scanning tunneling microscope has revealed that the main drawback of all the considered software (except for SPIP) is values insufficient detail of the visualization of images in three- G  (Gx )  (Gy ) , 2 2 dimensional space due to the use of OpenGL technology.      General drawback of the software is a lack of possibility to and direction of the gradient use several branches of image processing with possible simple change of filter parameters at different stages. Such Gy tools can be provided by programs that use visual   a r c tg ( ). G programming tools. This paper considers GraphMIC   x     software complex for medical image processing that uses The generation algorithm is described in more detail in such tools. Interactive user interface components enable to [5]. As a result of the analysis of the convolution networks, change settings (e.g., reorder the filter sequence for several modern architectures, well-proven in image processing or manage filter parameters). GraphMIC is best segmentation, have been chosen [6-20] and adapted for the suited for processing experimental data and creating developed information system. Then a real STM-image is individual image processing pipelines. In general, analysis of applied to the input of the trained network, and segmented existing software has identified the need to create a program STM-image is formed at the output. The network results are combining all the positive features of existing STM-image displayed as the segmented image and the numerical value of processing software. the number of found nanoparticles in the image. Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0) Image Processing and Earth Remote Sensing Fig. 1. Scheme of nanoparticles segmentation on the STM-image in the information system. Secondly, segmentation task is carried out by the different types of data obtained from STM. Therefore, a combined method including several stages. Initially, the subsystem has been created for reading common formats centers of the particle nuclei are determined using the used in these studies (.mdt, .pc). curvature detector [21]. The detector operation is based on the notions of surface convexity and concavity, the function The connection of external modules to the information of local curvature and its extremes, i.e., on the semantic system is an important addition that entails advantages as characteristics of its structural elements. Then the received follows: coordinates of particles are transmitted to the processing  Simplified extension of the complex functionality by block of the "Watershed" algorithm of OpenCV library. other developers as there is no need to study the After that, this block receives the centers of "troughs" for the source code of the complex to make changes in it; areas to be filled in. The segmented image is submitted to the input of the findContours algorithm that draws and counts  Possibility to use previously written, debugged and the number of found particles. tested program code in different programming languages with minor changes (C#, Python, Java, III. INFORMATION SYSTEM FOR AUTOMATIC Delphi); SEGMENTATION OF STM-IMAGES  Using capacities of different programming The structural scheme of the information system is shown languages, libraries and development tools for them in Fig. 2. The information system for segmentation of (OpenCV, Keras); nanoparticles on the STM-image is implemented as a program complex and includes the following structural  Improving the stability of the complex due to elements: localization of faults inside the module;  Input/output subsystem;  Creating of module libraries used in the work of an individual user.  Connected external modules; The primary processing subsystem is responsible for  Primary processing subsystem; filtering the input image – it is important for eliminating interference in the STM-image. Methods of classification,  A subsystem for generating model STM-images; clustering, segmentation are used to analyze data after  User block; primary processing and extract useful information about the material under study.  User block link map. The model image generation subsystem consists of two To process data obtained from different microscopes it is blocks. Firstly, modeling of the substrate is performed. Its often necessary for the system to "know how" to handle implementation in the form of polynomial function does not VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 131 Image Processing and Earth Remote Sensing reflect the random nature of the surface on the real image. manipulation of image contours, it is also possible to work Therefore, different methods of fractal surface generation are with data in different programming languages (C, C++, C#, used to obtain a model close to the real conditions. The Python, etc.). height map, received as an outcome of the algorithms operation, creates such a "noise" that is difficult enough to be Fig. 3 shows a map of links of user's blocks in more filtered by classical methods of computer filtration. detail as the implemented program module using the library Secondly, the simulation of nanoparticles is performed in the NodeEditor. It performs several functions: control, linking same block. Ellipsoids are distributed on the resulting surface the user's blocks and visualization of the results of each step of the substrate by the uniform law. Due to the difference in of image processing. size and height the ellipsoids are superimposed on each Thus, the created information system has the following other, which makes the segmentation process complicated. advantages: Masks for the obtained model STM-images are also formed.  Convenient interface for the researcher's work, The user block represents a multifunctional structure. For visible image processing chain, possibility to change each created block there is a field with an image and fields parameters of this or that filter at any step; necessary for the internal algorithm of the block. It is possible to select the input file, the primary processing filter  Support of various filters for primary image and its parameters, the number of inputs and outputs (the processing; number of images submitted to the input and the required  Modeling of STM-images with different parameters number of outputs for further processing chains), to connect (the number of particles, noise for substrate external modules, in which the segmentation of the STM- generation); image was carried out, and to control the final processing result (the segmented STM-image).  Possibility to add program modules from other programming languages (Python, C#, Java, Delphi); The primary image processing unit implements convenient methods for filtering, segmentation and  Support for detection methods. Fig. 2. The structural diagram of the information system. IV. RESULTS AND DISCUSSION INFORMATION where TP, TF, FP, FN – the number of pixels correctly assigned to the class «particle», correctly assigned to the The studies have yielded a wealth of information and class «background», incorrectly assigned to the class evaluated the proposed methods. To assess the quality of «particle» and incorrectly assigned to the class segmentation the metric IoU (Intersection over Union) is «background», respectively. Table 1 shows the results of used which is calculated by the formula: segmentation by different methods. TP Evaluation of segmentation algorithms has been carried Io U     (T F  F P  F N ) out and showed that methods using curvature detectors better VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 132 Image Processing and Earth Remote Sensing determine the number of particles on the scanned surface. distortions and artifacts in the STM-image, detect the However, neural networks trained on model images using the boundaries of particles more accurately and also are able to DiamondSquare method are more resistant to various separate "sticky" particles from each other. Fig. 3. Block link map in the information system. The figure shows the possibilities of primary processing of STM-images and also the possibility of importing the Python module. The result of the joint work of the Watershed algorithm that link the goal with the alternatives. The target is the and the Horde detector is shown in Fig. 4. Fig. 5 illustrates parent node for all criterion nodes. the results of segmentation of real STM-images after training networks based on model STM-images (generation of the TABLE I. RESULTS BASED ON NEURAL NETWORK AND COMBINED substrate by DiamondSquare method). The networks have METHODS PROPOSED IN THE INFORMATION SYSTEM ON 100 MODEL STM- IMAGES WITH THE NUMBER OF PARTICLES EQUAL TO 50, WITH THE been trained by the method of stochastic gradient descent GENERATION OF THE SUBSTRATE BY DIFFERENT METHODS. IOU METRIC, based on input images and corresponding masks. 512 ITS STANDARD DEVIATION (SD) AND MATHEMATICAL EXPECTATION (ME) ARE CALCULATED training images of 256x256 pixels were supplied to the input of the networks. In total, the networks were trained for about Method of Metric IoU Number of particles 12 hours on a GeForce GTX 1070 graphics card. segmentation ME SD ME SD with neural Neural network methods and methods based on the network application of curvature detectors have been aggregated, and methods a new integrated approach has been created on the basis of U-Net 0.9047 0.0112 50.1156 5.3675 the hierarchy analysis method. The method of hierarchy SegNet 0.9112 0.0156 53.6353 6.5744 PSPNet 0.9202 0.0101 52.1156 5.9822 analysis (proposed by T.L. Saati) [22, 23] consists in using a Method of hierarchical structure incorporating the purpose of choice, segmentation by criteria, alternatives and other factors influencing the choice combined of the solution. The construction of such a structure helps to methods analyze all aspects of the problem and to penetrate deeper Horda 0.6724 0.1314 50.0221 4.9564 into the essence of the problem. The top of the hierarchy is Sector 0.7268 0.2217 51.4215 5.2156 the main goal – segmentation of nanoparticles; elements of Circle 0.7255 0.1514 50.8899 5.5958 Sphere 0.7648 0.1634 50.2156 4.9648 the lower level represent many variants of achieving the goal (alternatives) – segmentation methods; elements of the intermediate levels – nodes that meet the criteria or factors Fig. 4. Segmentation results of the real STM-image: a) real STM-image; b) particle centers extraction using the Horda curvature detector (the trajectories of local minimums calculated by the set of functions are presented); c) particles selected by the Watershed algorithm; d) histogram of the found nanoparticles, their number and size. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 133 Image Processing and Earth Remote Sensing Fig. 5. Segmentation results of the real STM-image: a) real STM-image; b) U-Net segmentation result; c) SegNet segmentation result; d) PSPNet segmentation result. V. CONCLUSION Natural Computation, Fuzzy Systems and Knowledge Discovery. Advances in Intelligent Systems and Computing, pp. 21-29, 2020. As a result of the research carried out, the information [10] V. Badrinarayanan, A. Kendall and R. Cipolla, “SegNet: A Deep system was developed using modern methods of machine Convolutional Encoder-Decoder Architecture for Image learning, computer vision and visual programming. A new Segmentation,” PAMI, p. 14, 2017. complex approach to solve the problem of segmentation of [11] H. Zhao, J. Shi, X. Qi, X. Wang and J. Jia, “Pyramid scene parsing nanoparticles on STM-images is proposed, which allows to network,” CVPR, 2017, 11 p. increase the reliability of control of particles on the surface [12] Y. LeCun, Y. Bengio and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436-444, 2015. consisting in the use of curvature detectors as well as [13] S. Jesse, “Big data analytics for scanning transmission electron convolution neural networks of different architectures. The microscopy,” Ptychography Sci. Rep. vol. 6, no. 26348, 2016. proposed segmentation algorithms have been evaluated and [14] N. Lubbers, T. Lookman and K. Barros, “Inferring low-dimensional have proved to be highly efficient in operation. microstructure representations using convolutional neural networks,” Phys. Rev. E, vol. 96, 052111, 2017. REFERENCES [15] A. Belianinov, “Big data and deep data in scanning and electron [1] T. Fukuma, K. Onishi, N. Kobayashi, A. Matsuki and H. Asakawa, microscopies: deriving functionality from multidimensional data “Atomic-resolution imaging in liquid by frequency modulation atomic sets,” Adv. Struct. Chem. Imaging, vol. 1, no. 6, 2015. force microscopy using small cantilevers with megahertz-order [16] A. Chowdhury, E. Kautz, B. Yener and D. Lewis, “Image driven resonance frequencies,” Nanotechnology, vol. 23, pp. 12, 2012. machine learning methods for microstructure recognition,” Comput. [2] C. Gerber and H.P. Lang, “How the doors to the nanoworld were Mater. Sci., vol. 123, pp. 176-187, 2016. opened,” Nature Nanotechnology, vol. 1, no. 1, pp. 3-5, 2006. [17] W. Li, K.G. Field and D. Morgan, “Automated defect analysis in [3] E.Yu. Shelkovnikov, “Hybrid tunneling microscope with integrated electron microscopic images,” Npj Computational Materials, vol. 4, biotentiostat for investigation of the nanoparticles,” Chemical physics no. 36, 2018. and mesoscopy, vol. 13, no. 3, pp. 448-451, 2011. [18] T.E. Shelkovnikova, K.A. Shlyakhtin and S.F. Egorov, “Analysis of [4] K. Bui, “Segmentation of scanning tunneling microscopy images the convolution neural networks for the nanoparticles segmentation using variational methods and empirical wavelets,” Pattern Analysis on the STM-image,” Polzunovsky almanah, vol. 4, pp. 10-13, 2019. and Applications, pp. 20, 2018. [19] A.V. Nikonorov, M.V. Petrov, S.A. Bibikov, V.V. Kutikova, A.A. [5] E.Yu. Shelkovnikov, K.A. Shlyakhtin, T.E. Shelkovnikova and S.F. Morozov and N.L. Kazanskiy, “Image restoration in diffractive Egorov, “U-Net architecture neural network application for STM- optical systems using deep learning and deconvolution,” Computer images segmentation,” Chemical physics and mesoscopy, vol. 21, no. Optics, vol. 41, no. 6, pp. 875-887, 2017. DOI: 10.18287/2412-6179- 2, pp. 330-336, 2019. 2017-41-6-875-887. [6] P. Goldsborough, N. Pawlowski, J. C Caicedo, S. Singh and A. [20] T.A. Pashina, A.V. Gaidel, P.M. Zelter, A.V. Kapishnikov and A.V. Carpenter, “Cytogan: Generative modeling of cell images,” 2017 Nikonorov, “Automatic highlighting of the region of interest in [Online]. URL: bioRxiv 227645. computed tomography images of the lungs,” Computer Optics, vol. [7] O. Ronneberger, P. Fischer and T. Brox, “U-Net: Convolutional 44, no. 1, pp. 74-81, 2020. DOI: 10.18287/2412-6179-CO-659. networks for biomedical image segmentation,” International [21] P.V. Gulyaev, Yu.K. Shelkovnikov, A.V. Tyrikov and S.R. conference on medical image computing and computer-assisted Kiznertsev, “Application of the curvature detectors for a coordinate intervention, pp. 234-241, 2015. binding of the overlapping images in a probe microscopy,” [8] Y. LeCun, “Backpropagation Applied to Handwritten Zip Code Polzunovsky vestnik, vol. 2, pp. 122-125, 2016. Recognition,” Neural Computation, vol. 1, no. 4, pp. 541-551, 1989. [22] T.L. Saati, “Decision-making at dependences and feedback: [9] J. Hu, L. Li, Y. Lin, F. Wu and J. Zhao, “Comparison and Strategy of Analytical networks,” Moscow: LKI Publishing House, 2008. Semantic Segmentation on Remote Sensing Images,” Advances in [23] T.L. Saaty, “The Hierarchon: A Dictionary of Hierarchies Pittsburgh,” Pennsylvania: RWS Publications, 1992. VI International Conference on "Information Technology and Nanotechnology" (ITNT-2020) 134