Equivariant graph neural network for crystalline materials Astrid Klipfel1,2,3 , Zied Bouraoui2 , Yaël Frégier1 and Adlane Sayede3 1 Laboratoire de Mathématiques de Lens (LML)-UR 2462, Artois University, France 2 Centre de Recherche en Informatique de Lens (CRIL)-CNRS UMR 8188, Artois University, France 3 Unité de Catalyse et Chimie du Solide (UCCS)-CNRS UMR 8181, Artois University, France Abstract Materials generation is an essential task in material science that aims to discover new materials. While most of the existing models have shown interesting results in simulation, they struggle to produce new original and stable materials. This paper discusses the salient properties required for material generation and studies the difficulties related to material pattern repetition, which impacts the stability of the generated structures. 1. Introduction Energy Crystalline materials are involved everywhere in our modern society. From metal alloy to semi-conductor, sev- eral technological objects contain crystalline materials. Discovering new materials remains a difficult task in ma- Materials space terial science. While existing algorithms can search for new structures in the materials space [1], searching for Figure 1: The energy of a structure is given by its geometry a new material with a given set of desirable properties and chemistry. Stable structures are the local minima of the is not a trivial task. The set of potential candidate ma- energy (green dots). Relaxing a structure is the process of terials is not countable by a computer, and the portion modifying the geometry of a structure to minimize its energy (an unstable structure represented as a red dot converge to of stable materials (i.e. materials that can exist with- the nearest stable minima) out self-destructing) is small. Moreover, estimating the properties of a single material with chemical simulation as Density Functional Theory (DFT) is computationally expensive. To this end, methods based on evolution- terial. It permits the discovery of new materials since ary algorithms (e.g. [2]) are introduced for generating minimizing the total energy of the structure allows it to new materials from existing datasets composed of stable be more stable. Consequently, the local minima of the materials. However, most of these approaches work by energy corresponding to stable materials and relaxation hybridization and mutation of existing materials. As a lead to producing stable crystalline structures from an consequence, these methods are not able to find complex initially unstable crystal around the stable structures in materials. the materials space. This is illustrated in Figure 1. Discovering new materials is a challenging problem. Several methods were recently introduced for chem- But contrary to most generation problems, theoretical ical simulation either by enhancing current algorithms chemistry provides a powerful set of tools to analyse with machine learning techniques (e.g. [3]) or by using synthetic and real data. As a matter of fact, simulation end-to-end models (e.g. [4]). Most of these approaches techniques like hartree-fock or DFT are able to estimate focus on accelerating DFT by approaching relaxation the properties of a given structure by applying physics as a supervised task. End-to-end models generally re- laws. Consequently, the stability of the materials can be quire expensive labels to be trained as interaction forces. estimated through simulation. These methods can also However, these labels are generally not available and be used to perform the relaxation of crystalline materials. need to be produced through DFT calculation. Thus, the Relaxation is the process of minimizing the energy of benefits of end-to-end models is limited because such a crystalline structure by deforming it. This process is models required data from atomistic simulation. Finally, very common in material science to study a given ma- even if the relaxation may lead to a crystalline struc- ture, there is no guaranty on its stability. In fact, most of STRL’22: First International Workshop on Spatio-Temporal Reasoning the random structures do not result in stable structures. and Learning, July 24, 2022, Vienna, Austria Consequently, approaches based on simulation for new $ klipfel@cril.fr (A. Klipfel); zied.bouraoui@cril.fr (Z. Bouraoui); yael.fregier@univ-artois.fr (Y. Frégier); materials discovery are limited. adlane.sayede@univ-artois.fr (A. Sayede) Another direction consists in directly generating stable © 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). crystalline structures using machine learning techniques. CEUR Workshop Proceedings http://ceur-ws.org ISSN 1613-0073 CEUR Workshop Proceedings (CEUR-WS.org) In this case, the generative process can be performed have multiple edges between a pair of nodes and between with successive actions applied to the structure [5]. This a node with itself. process doesn’t require strictly following physics laws as long as the stable points (i.e. local minima of the energy) learned by the machine learning models remain the same as the real stable points. This fact can be advantageous in some cases because the realistic relaxation of a ran- dom structure is not guaranteed to converge to a stable material. As a result, learning realistic stable points with- out realistic physics can help machine learning models Figure 2: The periodic structure of a material is represented produce stable structures. as a lattice (in dotted lines). The multi-graph associated with Indeed, graph neural networks (GNN) have already a material (blue arrow) can overlap on the adjacent repetition been used for the representation and generation of or- of the lattice and a pair of nodes can be connected multiple ganic molecules [6]. However, crystalline materials are time. known to be more difficult to generate since they gener- ally have more complicated chemistry. Also, they contain repetitive patterns defined by the lattice of the crystals, Material deformation To tackle the crystalline sys- which make them harder to process. In this paper, we tem generation problem, we should define action on the present how graphs representation of materials are de- geometry of the material. This action can be seen as an fined. We also discuss some important properties re- action on the lattice of the material 𝜌 resulting in the quired for generative models. updated lattice 𝜌′ and action on the atomic positions 𝑥𝑖 resulting in the updated atomic position 𝑥′𝑖 . 2. Problem statement {︃ 𝜌′ = 𝑔𝜌 . (2) Crystalline systems As molecules, crystalline sys- 𝑥′𝑖 = [𝑥𝑖 + 𝑔𝑖 ] tems can be defined as coloured point clouds. However, as crystals are periodic structures, additional information In this case, the goal is to predict the action 𝑔 ∈ about how the point could is repeated in the space is re- GL3 (R) on the lattice and the actions 𝑔𝑖 ∈ R on the 3 quired to represent it. The periodicity of the material can atomic position. The atomic positions are brought back be then represented as a network where a group of points into the lattice of the crystal by truncation. is repeated by discrete translation, which is equivalent to tiling space with a parallelepiped containing a cloud 3. Method of atoms as illustrated in Figure 2. As a result, a periodic system can be describe as atomic Equivariance To obtain meaningful actions applied to positions 𝑥𝑖 ∈ [0, 1[3 with an associated feature space a material, we should satisfy the equivariance property. representing the chemical information of each atom 𝑧𝑖 ∈ Indeed, translations and rotations have no impact on the ℱ and a lattice 𝜌 ∈ GL3 (R) representing the periodicity material’s properties. Consequently, a machine learning of the material. The infinite point cloud generated by model acting on the material should not be dependent this representation can be defined as on the orientation and position of the structure but only be dependent on its geometry as shown in Figure 3. 𝜌·(𝑥𝑖 +𝜏 ), 𝑧𝑖 |𝜏 ∈ Z , 1 ≤ 𝑖 ≤ 𝑛 ⊆ R ×𝐹 (1) 3 3 {︀(︀ )︀ }︀ ℎ 𝑀 ℎ·𝑀 Where 𝜏 act like a Z3 vector that translate the point 𝑔 𝑔 cloud. 𝑀′ ℎ · 𝑀′ ℎ Material graph There are multiple ways to define Figure 3: Equivariance between the action applied on the the graph of a material. Chemical bonds can be used material 𝑔 and the actions of translation and rotation group to build the set of edges, but generally, edges are built ℎ ∈ R3 ∪ SO(3). 𝑀 and 𝑀 ′ denoting the original material from the atoms under a given threshold distance of from and the material after having applied the action 𝑔 . the 𝑘 nearest neighbourhood [7]. The resulting graph is a multi-graph because the local environment of an atom can be on a translated point cloud near the border of the lattice as depicted in Figure 2. As a result, an edge can Actions on the lattice When 𝑔 acts on 𝑟ℎ𝑜 the lattice of the crystal, we would restrain 𝑔 to the group of actions deforming the structure. However, any matrix in R3×3 can be decomposed as a sum of a symmetric and an anti- symmetric matrix. As we define the action of 𝑔 in GL(3), the anti-symmetric part of the transformation can be seen as a rotation. But as we discussed earlier, applying rotations on a crystal doesn’t act on a material as the rotated material is equivalent to the original material. This is illustrated in Figure 4. Consequently, it is better Figure 5: Actions defined on the edges of the graph can be to restrain 𝑔 to GL(3) ∖ SO(3). In other words, to restrain decomposed into action on the atomic positions and action 𝑔 to the subset of matrices of GL(3) that are symmetric. on the lattice of the crystalline material. References [1] C. J. Pickard, R. J. Needs, Ab initio random struc- Figure 4: The SO(2) group doesn’t act on the materials but ture searching, Journal of Physics: Condensed Mat- only rotates them without deformation. ter 23 (2011) 053201. URL: https://doi.org/10.1088/ 0953-8984/23/5/053201. doi:10.1088/0953-8984/ 23/5/053201. [2] A. R. Oganov, A. O. Lyakhov, M. Valle, How Acting on the crystalline structure To act on both evolutionary crystal structure prediction the lattice and the atomic position, we can define actions works—and why, Accounts of Chemical Re- on the edge of the graph. These actions can be then search 44 (2011) 227–237. URL: https://doi. decomposed into a global action on the lattice of the org/10.1021/ar1001318. doi:10.1021/ar1001318. crystal and local action on the atomic position. In order arXiv:https://doi.org/10.1021/ar1001318, to define these actions, the contributions of all edges are pMID: 21361336. aggregated to compute the action on the lattice. To act [3] Y. Yang, O. A. Jiménez-Negrón, J. R. Kitchin, on the atomic positions, only the actions of the edges Machine-learning accelerated geometry optimiza- connected to the node are taken into account. tion in molecular simulation, The Journal of Chem- ical Physics 154 (2021). URL: https://par.nsf.gov/ 4. Conclusion biblio/10250472. doi:10.1063/5.0049665. [4] F. Ekström Kelvinius, R. Armiento, F. Lindsten, Crystalline materials are difficult to process because of Graph-based machine learning beyond stable ma- the complexity and the variety of their chemistry, but also terials and relaxed crystal structures, Phys. Rev. Ma- because of their repetitive structure. However, materials terials 6 (2022) 033801. URL: https://link.aps.org/doi/ can be represented as graphs containing both chemical 10.1103/PhysRevMaterials.6.033801. doi:10.1103/ and geometrical information about the structures. Con- PhysRevMaterials.6.033801. sequently, geometric machine learning techniques such [5] T. Xie, X. Fu, O.-E. Ganea, R. Barzilay, T. S. Jaakkola, as graph neural networks can be used in a wide variety Crystal diffusion variational autoencoder for peri- of tasks including supervised and unsupervised learning. odic material generation, in: International Con- But in order to enhance the generalization capability of ference on Learning Representations, 2022. URL: machine learning models, some properties such as the https://openreview.net/forum?id=03RLpj-tc_. symmetry of the actions on the lattice or the equivariance [6] V. G. Satorras, E. Hoogeboom, M. Welling, E(n) equiv- can be beneficial. ariant graph neural networks, in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceed- Acknowledgments ings of Machine Learning Research, PMLR, 2021, pp. 9323–9332. URL: https://proceedings.mlr.press/v139/ This work has been supported by HPC resources from satorras21a.html. GENCI-IDRIS (Grant 2021-[AD011013288]) and FEI INS2I [7] T. Xie, J. C. Grossman, Crystal graph convolu- 2022-EMILIE. tional neural networks for an accurate and inter- pretable prediction of material properties, Phys. Rev. Lett. 120 (2018) 145301. URL: https://link.aps.org/ doi/10.1103/PhysRevLett.120.145301. doi:10.1103/ PhysRevLett.120.145301.