Technology for Cast Iron Microstructure Analysis in SIAMS Software Using Neural Networks Tatyana Sivkova 1, Aleksandr Gusev 2 and Artem Syropyatov 1 1 SIAMS Ltd, Kominterna, 16, office 604, Ekaterinburg, 620078, Russia 2 Ural Federal University, Technical Physics Department, 19 Mira St., Ekaterinburg, 620002, Russia Abstract The paper covers key issues of metal and alloys’ microstructure control using cast iron microstructure examples, and ways of resolving these issues by integration of neural networks into algorithms of SIAMS software. Paper lists key specifics of using the technology and training neural network, aimed at improving algorithm reproducibility, analysis acceleration and simplification. The method for training neural network models as part of the SIAMS software includes functionality for assessing the quality of training. The described method allows you control the model error using the value of the loss function. Developed algorithms in form of ready solutions were integrated into the SIAMS software package, and can be recommended for serial microstructure control in industrial laboratories. Keywords 1 Microstructure control, comparison with reference scales, cast iron microstructure evaluation, microstructure image segmentation, neural network for microstructure analysis, GraphiCon, SIAMS software, GOST 3443-87. Introduction Materials microstructure control is a complex process that requires special skills and knowledge from a specialist. However, the necessity to provide quantitative and qualitative description of microstructure in question requires a professional to make a decision on material compliance or non- compliance with requirements. In order to simplify decision making, most of regulatory documents referring to quality control of metallurgy products include variants of microstructures that can appear in finished products under certain conditions, including certain range of chemical component content, processing stages, and heat treatment. Russian GOST standards include reference charts where images are arranged in ascending or descending order with regard to the number of phases, number of inclusions, or dimensional characteristics for the objects of interest. These references really assist in decision making; however, it is impossible to preselect and preliminarily evaluate all versions of microstructures that can occur in course of production. Thus, reference scales include limited number of variants, and it is not always possible to find a suitable image. Besides that, tightening quality requirements for metallurgy products and operational characteristics of the goods, require production technology changes that cause new microstructures not covered by standards. As there is no systematic work in the field of actualizing national standards [1, 2], standard reference scales are not enough for metallographic control. For example, according to GOST 3443-87 [3] it is necessary to assess shares for each graphite type in cases where cast iron microstructure contains several types of graphite. Figure 1 demonstrates results GraphiCon 2021: 31st International Conference on Computer Graphics and Vision, September 27-30, 2021, Nizhny Novgorod, Russia EMAIL: sivkova@siams.com (T.Sivkova); aogusev@siams.com (A.Gusev); artems@siams.com (A.Syropyatov) ORCID: 0000-0002-0621-2712 (T.Sivkova); 0000-0002-4948-6166 (A.Gusev); 0000-0001-5676-0525 (A.Syropyatov) ©️ 2021 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). CEUR Workshop Proceedings (CEUR-WS.org) of comparing grey cast iron sample microstructure with reference scale 1A “Shape of graphite inclusions” from GOST 3443-87 [3]. One can see that it is hard to pick the best match. a b c d e Figure 1: a –analyzed image; b – scale 1, PGf1 form (lamellar rectilinear graphite); c – scale 1, PGf2 form (lamellar exploded graphite); d – scale 1, PGf3 form (spiky graphite); e -scale 1, PGf4 form (chunky graphite) Analytical method using reference scales is considered to be a semi-quantitative method. In order to improve precision of metallographic analysis, companies implement image analysis systems featuring automated quantitative analysis methods. These methods include software-aided identification of structural contents with following computation of necessary parameters [4, 5, 6]. Image analysis systems are doing well in the field of calculating necessary quantitative characteristics of the structure and inclusion, including number, size, area, shares, form factors and microstructure grades according to a given standard. Figure 2 demonstrates results of automated graphite microstructure description performed for grey iron with SIAMS 800 software [4]. In this case automated analysis prevails over the semi-quantitative assessment based on comparison due to calculation of necessary parameters with minimum of subjective assessment. At the same time precision of results obtained with automated methods directly depends upon the recognition quality for microstructure objects that are further assessed for quantitative characteristics. Object recognition in image analysis is performed using various segmentation types [7, 8], where the simplest one is binary thresholding. In complex cases correct object selection is possible only using adoptive segmentation or graph-based image segmentation. Publications [9, 10] describe the use of said methods in SIAMS 800 software. Despite positive results of applying the said segmentation algorithms, they are not universal and are unsuitable for solving more complex image analysis problems. It is often necessary to recognize (select) of classify objects on the same image that are similar in terms of brightness and color, texture and internal gradients, and also in terms of shape and size, while not having visible separation boundaries. Such microstructures can be referred to as “complex” in terms of automated image analysis. In an example shown on figure 3 it is necessary to select graphite particles from pearlite matrix, and distinguish structurally free cementite from ferrite and lamellar cementite present in pearlite. a b Figure 2: a – analyzed image; b – results of software-assisted graphite recognition Lamellar graphite -31 %, swirling graphite – 69 %, particle length – 120-250 micron, number of particles – 10% а б Figure 3: a – lamellar graphite and cementite in pearlite matrix of cast iron; b – globular graphite and cementite in ferrite-pearlite matrix of cast iron Use of neural networks in SIAMS software 2.1. Examples of using neural networks, results and discussion In order to analyze “complex” microstructures, SIAMS software uses semantic segmentation method using neural network [11]. Recognition criteria used by neural network imitate image recognition performed by humans. Figures 4-6 show fragments of cast iron microstructure after chemical etching. Graphite particles are barely visible upon the pearlite background (figures 4, 5). Arrows show fragments, where there are no visible boundaries separating phases. One can see that cementite particles are identical to ferrite fragments in terms of brightness (figure 6). Color contours show results of automatically separating structural components, demonstrating that the models are performing well. Figure 7 shows microstructure of Ni-resist type cast iron, with metal matrix consisting of austenite with particles of excess cementite, after mild etching. Cementite particles have similar brightness parameters with austenite and are barely distinguishable. Automated recognition task was further complicated by excluding phase of final polishing, and image was captured with disabled equalization of microscope illumination. That caused increase in brightness gradient for objects belonging to both phases. Figure 7b demonstrates that the change in specimen preparation and capturing conditions did not cause degradation of recognition results. Figure 7c demonstrates results of simultaneous selection of graphite and cementite particles from the soft austenitic matrix with remaining sample preparation scratches and stains. Final results of recognizing cast iron structural components are satisfactory and suitable for further calculation of quantitative characteristics required by standards. Analytical algorithms used in the examples were integrated in the SIAMS software solutions package and can be recommended for serial microstructure control in industrial laboratories. a b Figure 4: a – analyzed image of lamellar graphite in pearlite matrix of grey cast iron; b – results of software-assisted recognition of graphite particles a b Figure 5: a – analyzed image of cast iron with globular graphite; metal matrix consists of pearlite, cementite, and ferrite; b – results of software-assisted recognition of typical graphite forms a b Figure 6: a – analyzed image – cast iron with globular graphite, metal matrix consists of pearlite, cementite, and ferrite; b – results of software-assisted recognition for the particles of graphite, cementite, and ferrite a b c Figure 7: a – a nalyzed image – Ni-resist-type cast iron with lamellar graphite, metal matrix consists of austenite and cementite; b – results of software-assisted recognition of cementite particles; c - results of software-assisted recognition of graphite and cementite particles 2.2. Evaluating quality of training neural network models Mechanism for training neural network models used in SIAMS to perform automated recognition of structural elements includes functionality for assessing training quality. Quality is assessed by measuring trained system recognition bias with the value of loss function. Minimum value of the loss function indirectly informs about minimum bias of analysis results for a model. Figures 8 and 9 show dependency graphs for loss function upon the number of training epochs that were formed in course of training with training and validation sets. Loss function value Number of training epochs Figure 8: Dependency of loss function value upon the number of training epoch while training the model on the training set Loss function value Number of training epochs Figure 9: Dependency of loss function value upon the number of training epoch while training the model on the validation set Graph for a training set (figure 8) demonstrates that value of loss function decreases while the number of epochs grows, and the model learns to recognize specific cases. When the number of epochs exceeds 50, the decrease in the value of loss function becomes insignificant. Minimum for distribution of loss function built using validation set (figure 9) is located in the region close to 10 learning epochs, which demonstrates that model ends generalization of useful data and starts analyzing specifics for a given set [12], which lowers model stability with regard to image variations. Analysis of the graphs allows understanding that it makes no sense to continue training a given model after a certain number of epochs. Long-term training allows improving object-of-interest recognition for similar cases, but change of conditions (i.e., specifics of specimen preparation, etching, and image capturing) causes increaseв number of errors. Therefore, the training mechanism for models used in SIAMS allows both determining an optimal point to end training, and returning system to the conditions of minimum bias. 2.3. Specifics of integrating neural network into SIAMS software After complex efforts aimed at integration of neural network technology into SIAMS software, the following applicable practical results were obtained:  stability of algorithms with regards to instability of external factors [13]: microscope illumination, etching duration, camera settings, presence of image artefacts (i.e., scratches, pores, particles of different phases, etc.);  ability to fast-train algorithm using minimum number of images, which is not typical for the majority of software solutions using neural networks;  iterative approach to training procedure that allows multifold decrease in the number of errors for each following iteration;  acceptable image processing speed (close to 2 seconds per 4 megapixels);  taking into account image scale, which is extremely important for using algorithm within a measurement tool, including equipment set built around an optical microscope and equipped with SIAMS software. Conclusions Paper demonstrates microstructure specifics for high-tensile cast iron, grey cast iron, and alloyed cast iron of Ni-resist type that require use of novel image analysis approaches, and create new opportunities to control quality of different materials. The use of described methods for cast iron microstructure analysis creates the following advantages for SIAMS software users:  ready solution of a specific task in form of a trained algorithm that is stable with regards to conditions of specimen preparation, etching, and image capture;  lack of necessity to customize sensitivity thresholds that are usually used in major segmentation methods;  ability to use one-step analysis instead of two-step one (graphite analysis before etching and metal matrix analysis after etching);  shortened analysis process duration due to abovementioned factors. Complex approach to research used by developers in order to integrate neural networks into SIAMS software along with development of universal tools for developing new models (or algorithms) allows extending the number of problems solved in the area of image analysis. Analytical algorithms used to demonstrate solutions on examples used in this paper were integrated into SIAMS software solutions package, and can be recommended for mass control of cast iron microstructure in industrial laboratories. References [1] S. Andreeva, Improving quality of metal products on the basis of developing new improving existing methods of quantitative metallography, Saint Petersburg State Polytechnic University, Thesis abstract, Saint-Petersburg, Russia, 2009. [2] N. Vakunov, Development and research of multiscale algorithms for image processing and analysis in industrial quality control systems, Thesis abstract, Vladimir State University, Russia, Vladimir, 2005. [3] GOST 3443-87, Cast iron castings with graphite of different form, Methods of structure determination, Moscow, 2005. [4] SIAMS Ltd (site), URL: https://siams24.ru/solutions. [5] Olympus company (site), URL: https://www.olympus-ims.com/ru/microscope/stream. [6] Zeiss: (site), URL: https://zeiss-solutions.ru/equipment/mikroskopy/programmnoe- obespecheniye/zeiss-zen. [7] Bauman National Library, Osnovnye tipy metodov segmentatsii izobrazheniy (in Russian), Key types of image segmentation methods, URL: https://ru.bmstu.wiki. [8] R. Gonzales, R. Woods, Digital image processing, Third Edition, University of Tennessee. [9] T. Sivkova, S. Gubarev, Computer-aided Metals' Microstructure Analysis, A non-standard approach to image analysis, in: Proceedings of GraphiCon 2019, Bryansk State Technical University, Bryansk, 2019, pp. 255–259, URL: https://www.graphicon.ru/conference/2019/proceedings. [10] T. Sivkova, S. Gubarev, I. Kamenin, Computer-aided Metals' Microstructure Analysis, Analysis of images with sample preparation defects, in: Proceedings of GraphiCon 2020, Saint-Petersburg, 2020, pp. 15-26, URL: https://www.graphicon.ru/node/211. [11] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional Networks for Biomedical Image Segmentation, Computer Science Department and BIOSS Centre for Biological Signalling Studies, University of Freiburg, Germany, 2015, URL: https://arxiv.org/pdf/1505.04597.pdf. [12] L. Prechelt, Early Stopping - but when? Faculty for Computer Science; University of Karlsruhe D- 76128 Karlsruhe, Germany, URL: https://page.mi.fu- berlin.de/prechelt/Biblio/stop_tricks1997.pdf [13] C. Shorten, T.M. Khoshgoftaar, A survey on Image Data Augmentation for Deep Learning, Journal of Big Data 6(60) (2019). doi: 10.1186/s40537-019-0197-0.