Rendering Semisynthetic FIB-SEM Images of Rock Samples

Abstract

Digital rock analysis is a prospective approach to estimate properties of oil and gas reservoirs. This concept implies constructing a 3D digital twin of a rock sample. Focused Ion Beam Scanning Electron Microscope (FIB-SEM) allows to obtain a 3D image of a sample at nanoscale. One of the main specific features of FIB-SEM images in case of porous media is pore-back (or shine-through) effect. Since pores are transparent, their back side is visible in the current slice, whereas, in fact, it locates in the following ones. A precise segmentation of pores is a challenging problem. Absence of annotated ground truth complicates fine-tuning the algorithms for processing of FIB-SEM data and prevents successful application of machinelearning-based methods, which require a huge training set. Recently, several synthetic FIBSEM images based on stochastic structures were created. However, those images strongly differ from images of real samples. We propose fast approaches to render semisynthetic FIBSEM images, which imply that intensities of voxels of mineral matrix in a milling plane, as well as geometry of pore space, are borrowed from an image of rock sample saturated by epoxy. Intensities of voxels in pores depend on the distance from milling plane to the given voxel along a ray directed at an angle equal to the angle between FIB and SEM columns. The proposed method allows to create very realistic FIB-SEM images of rock samples with precise ground truth. Also, it opens the door for numerical estimation of plenty of algorithms for processing FIB-SEM data. FIB-SEM, digital rock, pore-back effect, ground truth generation, semisynthetic image.

1. Introduction

The construction of a precise digital twin of a rock sample is one of the cornerstones of digital rock

concept in the oil and gas industry. It is used for reservoir evaluation in addition to traditional laboratory experiments. Digital rock workflow implies mathematical simulations of fluids flow in the digital twin and estimations of numerous physical and chemical characteristics of oil-bearing rocks [1]. The advanced methodology consists of simulation on various scales including the nanoscale [2].

Focused Ion Beam - Scanning Electron Microscope (FIB-SEM) is a powerful device for 3D serial

imaging at the nanoscale. The FIB column has a source of ions, which are accelerated and focused into the beam. The ions mill a thin layer of substance from the surface of a sample, and after that SEM scans the surface to produce current slice. The angle α between FIB and SEM columns usually equals 52°.

Multiple repetitions of these two operations produce a stack of slices of a specimen. Figure 1 illustrates the FIB-SEM image acquisition procedure.

The construction of accurate digital twin of a rock sample from FIB-SEM data is a challenging problem. There are a lot of peculiarities of FIB-SEM images: misalignment of slices, curtaining effect, instable intensity across a stack of slices, charging, etc. One of the main specific features of FIB-SEM images of porous media is, so-called, pore-back or shine-through effect. Pores are transparent, and we

2021 Copyright for this paper by its authors. see lower sides of pores in the current slice due to material from deeper slices being visible through the voids. For each slice, the intensity distributions of pixels of mineral matrix and pores are partially overlapped. The situation becomes even more complicated in the case of several phases of solid matrix or organic inclusions in a sample. That is why segmentation of FIB-SEM images of porous specimens is rather difficult [3, 4, 5].

A lack of annotated ground truth for FIB-SEM data prevents fine-tuning the algorithms for preprocessing and segmentation of FIB-SEM images as well as an application of machine-learningbased techniques due to absence of representative datasets. A manual annotation is long, tiresome and inaccurate work. Typical FIB-SEM image has several hundreds of slices, and the size of each slice can be up to 3000x2000 pixels. Painting regions of interest in 3D without breaks, discontinuities, and ragged edges between the regions in adjacent slices requires huge efforts. In addition, there is uncertainty in labeling of some fragments even for a human. There is no essential number of available datasets containing a real FIB-SEM image accompanied by high-quality segmentation outcomes. For example, digital twin [6] of multiphase sandstone [7] is far to be perfect. A use of such inaccurate results of segmentation as ground truth cannot provide any progress in algorithms development and adjustment.

In the case of lack annotated real data, a common approach in computer vision is usage of synthetic

images. For instance, synthetic image rendering helps to solve annotation problem in deep learning nanoparticle segmentation [8]. Recently, several methods for generation of synthetic FIB-SEM images of various materials appeared [9, 10, 11]. Though such synthetic images fill the gaps of large data demand, the more realistic datasets are still needed due to the variances between synthetic data and real images of rock samples.

In this paper, we propose novel approaches to render of 3D FIB-SEM images with pore-back effect, in which intensities of voxels of mineral matrix in a milling plane as well as geometry of pore space are taken from a real image of a rock sample, and intensities of voxels in pores are rendered depending on the distance from the milling plane to the given voxel along a ray directed at an angle equal to the angle between FIB and SEM columns. The proposed methods allow to generate very realistic FIB-SEM images of rock samples with precise ground truth. Following [12], we call such images as semisynthetic, because the image is a combination of the real data and synthetic one.

The paper is organized as follows. In Section 2, we briefly consider existing approaches for creation of synthetic FIB-SEM images. Section 3 contains a description of the proposed methods. The results are presented and discussed in Section 4. Finally, in Section 5, we make conclusions and outline future works.

2. Previous work

There are a couple of papers where effectiveness of FIB-SEM segmentation is demonstrated by

synthetic images of stochastic geometrical objects such as unions of independently identically uniformly distributed random grains, packed spheres, and straight circular cylinders. It is claimed, those simple models look like nanoporous FIB-SEM images of electrodes of fuel cells [4] and membranes from a zirconium dioxide [9, 10]. The simulation based on the physical model was used to generate

SEM slices in concordance with the method described in [13]. To compute the electrons diffusion,

method from [13] uses the Monte-Carlo approach, which simulates one electron at a time. For the generation of track of each electron, MONSEL 2 algorithm [14] with several optimization tricks is applied. A tracking of about 1000 electrons is necessary for simulation intensity of a voxel. So, it is a time-consuming approach, requiring at least several hours for simulation of a 3D FIB-SEM image.

Despite of usage of a well-grounded physical model and good reproduction of pore-back effect,

simulated images look rather artificial due to the absence of typical defects of FIB-SEM data such as accumulation of charge in pores, curtaining effect, intensity irregularities, etc. In addition, images of stochastic geometrical objects significantly differ from images of rock samples.

In contrast to physical-based simulation, Python library PoreSpy [15] uses fast simple heuristic

approach to create pore-back effect: pixels of a slice are colored according to their depth into the image, darker pixels are further away. For modelling an arbitrary binary image, coded pore space can be applied. A small similarity with real pore-back effect takes place, but in general a simulated image looks unnatural.

Outcome of multiclass segmentation of sandstone image is used to generate synthetic data in [11].

Intensities of voxels for both solid phase and pores set to be equal to average value for a given phase

obtained from initial grayscale image. To simulate pore-back effect, a 1D convolutional ramp kernel is applied in Z direction towards the front of the image. The angle between FIB and SEM columns is not taken into account. Finally, additive Gaussian noise is added. An advantage of that approach is more or less adequate geometry of the rock sample. Shortcomings are unnatural pore-back effect and absence of typical peculiarities of FIB-SEM data.

FIB and SEM columns are inclined to each other, therefore, there is a view angle α in the images.

To take this fact into account, it is reasonable to use of one of well-known 3D rendering engine. The paper [16] describes an application of in Avizo® software (Thermo Fisher Scientific) for rendering 3D image of pores media. Avizo uses Open Inventor® engine for 3D visualization by ray-casting. The software allows to cut off (i.e., not to show) required number of slices and to emulate ion milling process in such a way. Angle of camera view can be set close to the tilt angle of the electron column relative the ion one. X-ray microtomography image of a real rock sample is employed as initial data in [16] because such source of data provides perfectly aligned slices. Figure 2 demonstrates the same visualization approach for segmented real FIB-SEM image. That approach provides correct (that is identical to real images) movement of structures located in pores during playback slices as video frames.

However, a darkening in pores depending on its depth looks unnatural. Also, such synthetic images have no typical defects of FIB-SEM.

3. Approaches for rendering of semisynthetic FIB-SEM image

It is required to generate synthetic FIB-SEM image that is as more similar to a natural one as possible. For this purpose, it is reasonable to take characteristics and fragments of a natural image, so the image becomes semisynthetic. If there is an accurate segmentation of the pore space, then all the voxels of the mineral matrix can be transferred to a semisynthetic image. Sometimes the pore space of the sample can be filled with epoxy. In this case, there is no pore-back effect in the image, and precise segmentation can be performed by intensity thresholding. Figure 3 demonstrates a slice of FIB-SEM image of such sample.

Unfortunately, only sometimes epoxy intrusion is technically possible. The existing methods of

segmentation of FIB-SEM images with pore-back effect do not provide a high-quality separation into solid and void phase, so taking solid voxels directly from segmentation results leads to mistakes in the semisynthetic image. Nevertheless, the geometry of the pore space in these segmentations is quite adequate. Filling solid voxels with textures manually extracted from fragments of natural images allows to get a good reproduction in the part of mineral matrix. dil dist(xp, yp, zp) Inear(xp, yp, zp) solid phase, which are taken from segmented image of the sample with epoxy; dark cells belong to pores and have zero intensity; is the angle between ion and electron beams; ( , , ) is current pore voxel where pore-back is calculated;

( , , ) is an intensity of the nearest voxel of solid phase laying on the ray from SEM; ⃗ is a normal to local edge between solid and voids; is an angle between ⃗ and Z-axis 1) Obtaining image

( , , ), for which voxels of pores equal zero: ( , , ) = { ( , , ), 0, ( , , ) = 1 ( , , ) = 0 , where ( , , ) is an initial grayscale FIB-SEM image of natural rock sample or an image constructed from tiled fragments of textures related to solid; ( , , ) is binary segmentation result of ( , , ), where 0 designates voxels of voids, and 1 designates voxels of solids; ( , , ) and ( , , ) have size ×

× . 2) Dilation of ( , , ) with aperture

ℎ for suppression of unwanted structures on the edge between voxels of solid phase and pores: for each voxel of pore { performed for each ( , , ) ∈ : ( , , ) = ( , , ) ⋅ ( ⊕ ℎ)( , , ).

3) Finding intensity

( , , ) of the nearest voxel of solid phase laying on the ray from SEM ( , , )| ( , , ) = 0}. To do that, the following steps should be

SEM rays.

a) Forming array of coordinates from (0,0,0) to (0, − 1, − 1) by Bresenham algorithm [17]: {( , , ) | = 0, … , − 1}. That array is auxiliary to find coordinates for tan b) Incrementing while coordinate ( + , + , + ) is inside bounding box ((0,0,0), ( − 1, − 1, − 1)) and

( + , + , + ) = 0. The aim is finding ( , , ), that is intensity of the nearest voxel of solid laying on the ray. If coordinate ( + , + , + ) is out of bounding box ((0,0,0), ( − 1, − 1, − 1)), then ( , , ) = 0 and items c)-g) can be omitted for this voxel. c) Calculating the distance from coordinate ( , , ) to ( + , + , + ): ( , , ) =

sin . d) Obtaining mask of edge between solid and void based on morphological erosion and exclusive ( 1 ) ( 2 ) ( 3 ) ( , , ) = ( , , ) XOR ( ⊖ e) Calculation of inertia tensor for cubic local region of image with center in ( + , + , + ): = [ where 011, 101, 110, 002, 020, and 200 are central second order moments. f) Calculation of eigen values and vectors of matrix . We need to find eigen vector ⃗ = ( , , ) corresponding maximal eigen value. That vector is normalized as: ⃗′ = { − ⃗ ⃗ ‖ ⃗‖ ‖ ⃗‖ , ≥ 0 , < 0 .

g) Obtaining the angle between normal to local edge between solid and voids and Z axis: 4) Blurring slices of images 5) Calculation of semisynthetic FIB-SEM image with pore-backs as: ( , , ) = − arccos ′. ( , , ) and ( , , ) by Gaussian blur.

, ′ ( 4 ) ( 5 ) ( 6 ) ( 7 ) (8) (9) (10) (11) where is function for darkening of voxel intensity depending on distance to the nearest voxel of solid; is function for lightening of voxel intensity depending on angle ; ( , 2) is additive white gaussian noise with mean

= 0 and variance 2.

We use the following function :

, where , ℎ ℎ are minimal and maximal values of for entire image; Function is given by: ( ) =

1.1 1 + 0.5(40 ℎ − ℎ−

−5) ( ) = |1 − 2 ∙ | − 0.5|| (

− 1) + 1, 2 where

is the maximum value by which the intensity is multiplied when the angle between the normal vector and the axis Z is equal , it is reasonable to use = 1.5.

The variance 2 is estimated based on the analysis of variance of uniform fragment of solid phase.

6) Blurring edges between pores and solid by alpha-blending of

with its copy blurred by Gaussian filter

. Alpha-channel is calculated as morphological gradient of image . It is necessary to avoid unnatural too sharp edges between solid and pores.

There is small modification of the algorithm described above. Instead of vector ⃗′ it is possible to

SEM, then reflected vector is calculated as: use a vector from SEM column reflected from the local surface. If ⃗ ⃗ ′ = ⃗ ′ − ( ⃗ ′ , ⃗′) ⃗′.

is a unit vector directed from

Accordingly, the angle is calculated in (7) based on z-coordinate of ⃗

′ instead of ′.

4. Results and discussion

these slices of synthetic and semisynthetic images can be compared with the natural one from Figure 6.

The image in Figure 6 has stronger curtaining effect (that is vertical stripes) in comparison with the initial image from Figure 3, but it is not an issue.

The image rendered with Avizo using ray-casting (Figure 5) looks visually good, but the intensity changes in the pores are not similar to the real ones. Even deep pores do not become dark. Intensity of voxels of mineral matrix is unnaturally uniform.

Images created by PoreSpy (Figure 5b) are very rough approximation of the natural pore-back.

Visually slices by PoreSpy are far from any practical usefulness. Synthetic image created by method

from [11] looks much better (Figure 5c). However, intensity in pores is always less than that of the voxels of the mineral matrix, therefore, using such images, for example, to select a segmentation method leads to the fact that simple thresholding gives the best results, which does not work for real images.

Both proposed approaches allow to render FIB-SEM images, which are very similar to the real ones.

On closer inspection, one can see that in the texture-filled image (Figure 5d) this texture is repeated. In the Figure 5e, all solid voxels are taken from the real image of the sample and all defects of mineral matrix such as noise and curtaining are natural. The disadvantages of the proposed approaches include more blurred surface in the pores compared, for example, to Figure 5a, as well as simulation of charge accumulation in the pores depending on the orientation of the local surface, whereas this effect has a random nature. We are going to overcome enumerated shortcomings in the nearest future. d) e) Figure 5: A slice of FIB-SEM image rendered by different techniques: a) ray-casting by Avizo; b) visualization with usage of PoreSpy for simulation of pore-back effect; c) synthetic image by [11]; d) proposed approach with filling of solid phase by texture; e) proposed approach with usage of all voxels of solid phase from initial image

Proposed approaches for rendering of semisynthetic FIB-SEM image are extremely fast. CPU-based parallelized code for creation image with size 1870×860×801 voxels takes about 2 minutes on the workstation with CPU Intel® Xeon® E5-2630 v3 @ 2.40 GHz 2.40 GHz (32 logical cores) and 128 GB RAM.

5. Conclusion and future work

The proposed approach for creating semisynthetic FIB-SEM images of rock samples allows to obtain

realistic images with ground truth. Additionally, typical FIB-SEM defects such as geometrical distortions between slices [18], instable intensity, etc., can be added to these images. Based on these data, it is possible to adjust the parameters of the correction and segmentation algorithms.

The proposed approach opens the door to the automatic generation of a huge dataset for using machine learning methods to solve FIB-SEM image segmentation problems. To generate such a dataset, the pore space geometry is taken from one subset of real images and subjected to random elastic distortions. Intensity of solids is taken from another subset of real images and is applied with random intensity distortions to the resulting image with random geometry. The pore-back effect is created using the proposed method.

6. References

[8] L. Mill, D. Wolff, N. Gerrits, P. Philipp, L. Kling, F. Vollnhals et al., Synthetic Image Rendering Solves Annotation Problem in Deep Learning Nanoparticle Segmentation, Small Methods (2020) 2100223. doi: 10.1002/smtd.202100223. [9] C. Fend, A. Moghiseh, C. Redenbach, K. Schladitz, Reconstruction of highly porous structures from FIB‐SEM using a deep neural network trained on synthetic images, Journal of Microscopy 281( 1 ) (2021) 16-27. doi:10.1111/jmi.12944. [10] D. Roldán, C. Redenbach, K. Schladitz, M. Klingele, M. Godehardt, Reconstructing porous structures from FIB-SEM image data: Optimizing sampling scheme and image processing, Ultramicroscopy 226 (2021) 113291. doi: j.ultramic.2021.113291. [11] M. Andrew, A quantified study of segmentation techniques on synthetic geological XRM and FIBSEM images, Computational Geosciences 22( 6 ) (2018) 1503-1512. doi:10.1007/s10596-0189768-y. [12] J. He, E. Zhou, L. Sun, F. Lei, C. Liu, W. Sun, Semi-synthesis: A fast way to produce effective datasets for stereo matching, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021) 2884-2893. doi: 10.1109/CVPRW53098.2021.00323. [13] T. Prill, K. Schladitz, Simulation of FIB-SEM images for analysis of porous microstructures,

Scanning 35 (2013) 189–195. doi: 10.1002/sca.21047. [14] J.R. Lowney, Monte Carlo simulation of scanning electron microscope signals for lithographic metrology, Scanning 18 ( 4 ) (1996) 301–306. doi:10.1002/sca.1996.4950180406. [15] J. T. Gostick, Z. A. Khan, T. G. Tranter, M. D. Kok, M. Agnaou, M. Sadeghi, R. Jervis, PoreSpy:

A python toolkit for quantitative analysis of porous media images, Journal of Open Source

Software 4(37) (2019) 1296. doi:10.21105/joss.01296. [16] I. Reimers, I. Safonov, I. Yakimchuk, Construction of 3D Digital Model of a Rock Sample Based on FIB-SEM Data, 24th Conference of Open Innovations Association (FRUCT) IEEE (2019) 351359. doi: 10.23919/FRUCT.2019.8711959. [17] J. E. Bresenham, Algorithm for computer control of a digital plotter, IBM Systems journal (1965) 4( 1 ) 25-30. doi:10.1147/sj.41.0025. [18] I. Reimers, I. Safonov, A. Kornilov, I. Yakimchuk, Two-Stage Alignment of FIB-SEM Images of Rock Samples, Journal of Imaging 6(10) (2020) 107. doi:10.3390/jimaging6100107.

References

[1] I. Yakimchuk , N. Evseev , D. Korobkov , O. Dinariev , O. Ridzel , V. Khan , V. Semkov , A. Zhonin , D. Kravets , Digital Core Analysis - Innovative Approach for EOR Agent Screening at Pore-Scale for Achimov Rocks , SPE Russian Petroleum Technology Conference , 2020 .

[2] O. Dinariev , N. Evseev , Modeling of nanoscale liquid mixture transport by density functional hydrodynamics , Physical Review E 95 ( 2017 ) 063307 . doi: 10 .1103/PhysRevE.95.063307.

[3] M. Salzer , S. Thiele , R. Zengerle , V. Schmidt , On the importance of FIB-SEM specific segmentation algorithms for porous media , Materials Characterization 95 ( 2014 ) 36 - 43 . doi: 10 .1016/j.matchar. 2014 . 05 .014.

[4] T. Prill , K. Schladitz , D. Jeulin , M. Faessel , C. Wieser , Morphological segmentation of FIB-SEM data of highly porous media , Journal of Microscopy 250 ( 2013 ) 77 - 87 . doi: 10 .1111/jmi.12021.

[5] R. Moroni , S. Thiele, FIB/SEM tomography segmentation by optical flow estimation , Ultramicroscopy 219 ( 2020 ) 113090 . doi: 10 .1016/j.ultramic. 2020 . 113090 .

[6] M. Andrew , Vaca Muerta FIB-SEM, Digital Rocks Portal, 2019 . URL: http://www.digitalrocksportal.org. doi: 10 .17612/P7038S.

[7] M. Andrew , Comparing organic-hosted and intergranular pore networks: topography and topology in grains, gaps and bubbles , Geological Society , London, Special Publications 484 ( 1 ) ( 2020 ) 241 - 253 . doi: 10 .1144/SP484.4.