Permeability Prediction of Porous Media using Convolutional Neural Networks with Physical Properties Hongkyu Yoon1, Darryl Melander2, and Stephen J. Verzi2 1Geomechanics Department, Sandia National Laboratories, Albuquerque, NM 87123 2Complex System for National Security, Sandia National Laboratories, Albuquerque, NM 87123 hyoon@sandia.gov Abstract Multiphysics behaviors in complex porous media (e.g., Permeability prediction of porous media system is very im- Yoon et al., 2013 and 2015). Although it is now largely pos- portant in many engineering and science domains including sible to understand how pore topology, structure, and com- earth materials, bio-, solid-materials, and energy applica- position impact various processes affecting flow patterns, tions. In this work we evaluated how machine learning can transport process, and evolution of porous media by com- be used to predict the permeability of porous media with physical properties. An emerging challenge for machine bining a suite of imaging techniques and advanced numeri- learning/deep learning in engineering and scientific research cal methods, integration of these techniques requires tre- is the ability to incorporate physics into machine learning mendous computational powers and expenses. process. We used convolutional neural networks (CNNs) to Recent advances in machine learning provide a great op- train a set of image data of bead packing and additional phys- portunity to enhance image-based property estimation and ical properties such as porosity and surface area of porous media are used as training data either by feeding them to the modeling capabilities (e.g., Raissi et al., 2018; Wu et al., fully connected network directly or through the multilayer 2018). In addition, combination of image data with other nu- perception network. Our results clearly show that the optimal meric and categorical data has improved the prediction of neural network architecture and implementation of physics- various quantities such as house prices (e.g., Rosebrock, informed constraints are important to properly improve the 2019) and image classification as well (Aimone and Severa, model prediction of permeability. A comprehensive analysis of hyperparameters with different CNN architectures and the 2017). data implementation scheme of the physical properties need In this work, we explore how machine learning can be to be performed to optimize our learning system for various used to predict the permeability of porous media with phys- porous media system. ical properties. An emerging challenge for machine learn- ing/deep learning in engineering and scientific research is the ability to incorporate physics into machine learning pro- Introduction cess. We used convolutional neural networks (CNNs) to Recent advances in multiscale imaging techniques for the train a set of image data of bead packing and additional analysis of complex pore structures and compositions have physical properties such as porosity and surface area of po- revolutionized our ability to characterize various porous me- rous media are used as training data either by feeding them dia systems (Bultreys et al., 2016). Applications of imaging to the fully connected network directly or through the mul- for porous media systems have been expanded for multi-in- tilayer perception network. We evaluated the effect of hy- terdisciplinary areas including fractured and porous natural perparameters with different training dataset on permeabil- media, biofilm, human bones/bodies, and various materials ity prediction. among many others. Flow and transport properties in porous media are very important to control and impact a variety of Earth science applications. Imaging methods have been tre- Related work mendously advanced to produce 2D/3D structures and com- Convolutional neural network (CNN) has been very suc- positions of porous media over a range of scales, and numer- cessful for image classification and segmentation and has ical methods also have been advanced to fully understand been adopted for various scientific and engineering Copyright © 2020, for this paper by its authors. Use permitted under Crea- tive Commons License Attribution 4.0 International (CCBY 4.0). problems including permeability estimation in network sys- and surface area) as shown in Figure 3. For the MLP the tems (Wu et al., 2018), physics-informed reduced order activation function was Relu. model combined with high fidelity turbulence simulations (Ling et al., 2016), and extraction of flow features (Ströfer et al., 2018). In particular, recent works (Ling et al., 2016, Raissi et al., 2018) demonstrated that deep neural network architectures have an ability to account for underlying phys- ics behind the data. Dataset and Physical Properties First, a set of images as shown in Figure 1 was generated to represent a two-dimensional (2-D) porous media system with binary phase using an open source PoreSpy (Gostick et al., 2019) where a sphere packing module was used and po- rosity (fraction of void space as black in Figure 1) and sur- face area of porous media (i.e., beads as white in Figure 1). To represent a range of permeability which accounts for the capability of porous media system to allow fluid to flow through, different sizes of spheres were used to arrange the packing as shown in Figure 1. The set of images were used to compute the directional permeability of images using an open source OpenPNM (Gostick et al., 2016). The size of image is 192 x 192 and void and solid phases are shown in black and white, respectively. All physical data (permeabil- Figure 1. Examples of porous media generated with different sizes of sphere. Fluid flows through the void space in black. The ity, porosity, and surface area) were normalized from 0 to 1. black and white pixels have one and zero values in a Boolean Since the logarithmic scale of the permeability is more cor- type, respectively. Permeability decreases from upper left to related with porosity, we use a logarithmic permeability in lower right and ranges over two orders of magnitude in m2. this work. Figure 2 shows the relationship between permea- bility and porosity-surface area. As seen, the permeability has positive and negative correlations with porosity and sur- face area, respectively. Methods Additional physical information can provide physical con- straints for training the model. The combination of image and numerical data allows us to build and train a hybrid physics-informed machine learning model. To handle pro- cessing of the porous media images, we have developed Figure 2. Normalized permeability in x-direction (log10Kx) vs. convolutional neural networks (CNNs) whose input consists porosity (left) and surface area. of binary phase image. The first CNN used in our work (CNN1) include four convolutional layers with the number of kernels from 16, 32, 64, and 128, each followed by batch normalization, leaky Relu activation, and a max pooling. The second CNN (CNN2) follows the CNN architecture Each convolutional layer has a kernel of size 3 x 3 to extract from Wu et al. (2018) where 2 CNN layers with 10 channels, the features from the corresponding input, and the max pool- each of size 5x5 were followed by the three FC layers with ing with a kernel of size 2 x 2 were used. The two fully- 10,32, and 10 neurons. The porosity and surface area data connected (FC) layers have 36 and 12 neurons and a dropout are directly combined into the second FC layer. For the of 0.4 between two FC layers. The 12 neurons are combined CNN2+Num model, two direction permeability values (in x with either the MLP output with the dense layers with 32 and y directions) are trained. For the CNN1, a total number hidden nodes and 4 output or two numerical data (porosity of 345 images are used with 80% and 20% of training and testing data. For the CNN2, a total number of 250 images (out of 345 images) are used with 70% and 30% of training with features extracted from image data. Although there is and testing data. In particular, the CNN2 was trained with need to study what features are extracted from image and two directional permeability values compared to one hori- how two input data can be used to learn the underlying fea- zontal permeability in the CNN1. ture to the permeability, Figure 4 shows that the CNN1 mod- Key hyper parameters are the following: the els with both numeric data tend to predict the lower and up- (Leaky)ReLU activation function, the dropout of 0.4 for the per ranges of permeability better than the CNN1 models CNN1, a batch size of 16 for the CNN1, Adam optimizer with single numeric data. This may imply that the physical with a learning rate of 0.0005 and the decay of 0.0001. The constraints from the numeric data would influence the learn- number of epochs was 250 for most of cases with an obser- ing process of the features that impact either high and low vation of apparent no learning after 250 epochs based on permeability systems. For example, the high and low per- cases with 750 and 5000 epochs. The loss function is the meability (see an example in Figure 1) contains larger and mean-squared error (MSE). A number of network sizes were smaller space (or cross-sectional distance) between spheres, evaluated, but in this work we focus on the impact of addi- respectively. The fact that the CNN1 with image data only tional data and different CNN+Numeric data structure on tends to predict the permeability over a narrow range (be- permeability prediction. tween ~0.3 and ~0.7) may indicate that without physical property information the CNN tends to learn more common features rather than critical features for low and high perme- Convolutional Neural Network (CNN) FC + ability patterns. Permeability Linear Prediction [0.92626 Multilayer Perception Activation Table 1. Summary of results with six different models. 0.09189] (MLP) CNN1- Figure 3. Schematics of convolutional neural networks and addi- CNN1 CNN2 modified CNN1 CNN1 CNN1 tional information stream to construct a physics-informed model MSE +Poro +Poro +Poro +Poro +SA only architecture. +SA +SA +SA Training 0.00176 0.00171 0.00307 0.00185 0.00360 0.00789 Results and Discussion Valida- 0.00689 0.00708 0.00684 0.00720 0.01060 0.01030 tion The mean squared error (MSE) values for training and vali- MSE – Mean Squared Error with normalized permeability values. dation data are reported in Table 1. Testing results with val- Poro and SA stand for porosity and surface area. CNN1-modified idation data sets for six different case are shown with a linear has a variation from CNN1 with two fully connected dense layers regression fitting. First, it is very clear that all four cases with 36 and 4 neurons. The 4 outputs from the CNN1 are combined with the CNN1 and numeric data outperformed the CNN2 with one MLP output with the dense layer of 16 hidden nodes. with numeric data in training and validation. Although we need to compare the results with the same training data, the CNN architecture significantly influences the learning pro- Conclusions cess of the features of porous media image and numeric data. We evaluated how additional physical information can en- The CNN1 with image only performed similarly to the hance the permeability prediction with the CNN models. As CNN2 with numeric data. it is now well accepted in the community that a physics-in- Second, all CNN1 models with additional physical nu- formed machine learning model can overcome overfitting to meric data performed better than two other models (Table the training data and improve the features underlying the 1). To compare the prediction with validation data, the pre- physical processes, there is a strong need to improve how dicted and validation data are plotted with the linear regres- the physical constraints and/or additional information (e.g., sion fitting and a R2 value in Figure 4. As a reference, the equations and theory) can enhance the learning process in single perfect line is also shown. The slope shows the over- machine learning. Our results clearly show that the optimal all performance of each model with a better performance neural network architecture and implementation of physics- closer to one, while the R2 value shows the proximity of informed constraints are important to properly improve the predicted data along the linear regression line. As expected, model prediction of permeability. The analysis of the fea- the CNN1 with both porosity and surface area performed tures learned through each layer and the output data from better than the CNN1 with either porosity and surface area. the MLP will reveal a better mechanistic understanding of As shown in Figure 2, the porosity and surface area are cor- the machine learning processes. A comprehensive analysis related with the permeability, so both information would of hyperparameters with different CNN architectures and provide additional physical constraints that are combined the data implementation scheme of the physical properties will be performed to optimize our learning system for vari- pathways to limit network size. In Proceedings of 2017 NIPS Cog- ous porous media system. nitive Influenced Artificial Intelligence Workshop. arXiv preprint arXiv:1711.09876. Bultreys, T.; De Boever, W.; Cnudde, V. 2016. Imaging and im- age-based fluid transport modeling at the pore scale in geological materials: A practical introduction to the current state-of-the-art. Earth-Science Reviews 155: 93-128. Gostick, J.; Khan, Z.A.; Tranter, T.G.; Kok, M.D.R.; Agnaou, M.; Sadeghi, M.A.; Jervis, R. 2019. PoreSpy: A Python Toolkit for Quantitative Analysis of Porous Media Images. Journal of Open Source Software. doi:10.5281/zenodo.2633284. Gostick, J.; Aghighi, M.; Hinebaugh, J.; Tranter, T.; Hoeh, M.A.; Day, H.; Spellacy, B.; Sharqawy, M.H.; Bazylak, A.; Burns, A.; Lehnert, W. 2016. OpenPNM: a pore network modeling package. Computing in Science & Engineering 18(4):60-74. Ling, J.; Kurzawski, A.; Templeton, J. 2016. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics 807: 155-166. Raissi, M.; Karniadakis, G.E. 2018. Hidden physics models: Ma- chine learning of nonlinear partial differential equations. Journal of Computational Physics 357: 125-141. Rosebrock, A., Keras: Multiple Inputs and Mixed Data, https://www.pyimagesearch.com/2019/02/04/keras-multiple-in- puts-and-mixed-data/, accessed on July 15, 2019. Ströfer, C.M.; Wu, J.; Xiao, H.; Paterson, E. 2018. Data-driven, physics-based feature extraction from fluid flow fields. arXiv pre- print arXiv:1802.00775. Wu, J.; Yin, X.; Xiao, H. 2018. Seeing permeability from images: fast prediction with convolutional neural networks. Science bulle- tin 63(18): 1215-1222. Yoon, H.; Dewers, T.A. 2013. Nanopore structures, statistically representative elementary volumes, and transport properties of chalk. Geophysical Research Letters 40(16): 4294-4298. Figure 4. Comparison of the permeability prediction with six dif- Yoon, H.; Kang, Q.; Valocchi, A.J. 2015. Lattice Boltzmann-based ferent models listed in Table 1 for the validation data. The linear approaches for pore-scale reactive transport. Rev. Mineral. Geo- regression fitting is also shown. The black dot line represents the chem 80: 393–431. http://dx.doi.org/10.2138/rmg.2015.80.12. perfect predicted case. Acknowledgments This work was supported by the Laboratory Directed Re- search and Development program at Sandia National Labor- atories. Sandia National Laboratories is a multimission la- boratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. De- partment of Energys National Nuclear Security Administra- tion under contract DE-NA-0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. References Aimone, J.B.; Severa, W.M. 2017. Context-modulation of hippo- campal dynamics and deep convolutional networks: Using parallel