Advancing Materials Property Prediction: A Comparative Study of Graph Neural Network Models

Abstract

This study investigates the efficacy of Graph Neural Networks (GNNs) in predicting material properties by comparing a baseline GraphSage model with a hybrid model incorporating Convolutional Graph Neural Network (CGCNN), Graph Attention Net work (GAT), and GraphSage layers. Both models secure positions on leaderboards, but the proposed hybrid model significantly outperforms the baseline across diverse tasks. The baseline struggles in 4 out of 9 tasks, emphasizing limitations in capturing intricate dependencies. Conversely, the hybrid model consistently excels, ranking in the top 10 for 5 tasks and top 5 in the critical dielectric task. Insights highlight the importance of holistic approaches, considering structural and edge-related features. Future research aims to refine models, addressing materials science intricacies and fostering advancements in predictive accuracy. Overall, our findings contribute to the evolving landscape of materials property prediction, emphasizing the need for sophisticated models at the intersection of machine learning and materials science.

Keywords

• GNNs
• GraphSage
• GAT
• CGCNN
• Matbench

1. Introduction

against the existing benchmark leaderboard. Through this work, we aim to contribute nuanced insights to the realm of generalization of material properties prediction.

2. Methodology

Materials property prediction has seen a paradigm shift with the advent of Graph Neural Networks (GNNs), offering a unique approach to understanding complex relationships in materials datasets [ 1 ]. GNNs, as a subset of neural networks designed to handle graph structured data, have shown promise in capturing intricate structural patterns within materials, making them particularly apt for applications in materials science [ 6 ]. Our methodology revolves around the careful construction and comparison of two distinct Graph Neural Network (GNN) models, each tailored for materials property prediction on the Materials Project benchmark datasets.

2.1. Model Architectures

Baseline model (GraphSage): This baseline model exclusively incorporates GraphSage layers [ 3 ] (Fig. 1). GraphSage is renowned for its ability to capture local structural information, making it a suitable candidate for materials datasets where intricate crystallographic patterns are prevalent. GraphSage is considered a sampling method and the main idea behind it is to uniformly sample a set of nodes from its neighborhood, aggregate the feature information and perform graph/node classification. The model’s architecture consisted of ten GraphSage layers and a model head of four dense layers.

Proposed model (CGCNN+GAT+GraphSage): In contrast, our proposed model rep resents a more intricate architecture, combining Convolutional Graph Neural Network (CGCNN) [ 4 ] (Fig. 2), Graph Attention Network (GAT) [ 5 ] (Fig. 3), and GraphSage layers [ 3 ].

GAT and CGCNN layers are specifically chosen for their ability to leverage edge features (in contrast to the GraphSage layers), enhancing the model’s capability to capture non-local dependencies and relationships within the materials. GATs stand out for their integration of the attention mechanism, a concept extensively employed in domains like natural language processing, into graph neural networks. The distinctive feature of GATs lies in utilizing the attention mechanism to assign weights to nodes within a graph. This approach enables the model to prioritize certain nodes over others during information processing, a critical aspect for effectively capturing the intricate nuances present in graph-structured data. Moreover, regarding CGCNNs, the key idea behind them is to adapt the convolutional operation to the irregular and non-grid nature of graphs, enabling the model to learn and understand the inherent connectivity and relationships present in the data, both in the node as well as the edge level. The model’s architecture included three CGCNN layers, followed by one twoheaded and three one-headed GAT layers, leading to four GraphSage layers. The model head consisted of four dense layers.

2.2. Featurization, Data and Setup

The featurization of materials is a critical step in ensuring that our models encapsulate both structural and edgerelated features. We employ the CGCNN method for featurization, utilizing the deepchem library [ 7 ], drawing inspiration from convolutional neural networks to capture spatial relationships within crystal structures. Starting from a typical pymatgen crystal structure form [ 8 ], the deepchem library enables the user to easily featurize and produce graphs with a standard number of node and edge features, namely 92 and 41, respectively. We utilise standard training techniques, such as stochastic gradient descent, and evalu ate model performance using appropriate metrics for materials property prediction. Hyper parameters were kept the same throughout the training and testing of all datasets, as the focus of this study was the comparative performance of the models and not the achievement of good performance in a single task. The training set was split into training and validation sets at 80/20 ratio as no validation set was included in the data, the training lasted for 200 epochs, using Adam optimizer [ 9 ] with a 0.001 learning rate and a constant random seed for reproducibility. The pytorch framework [ 10 ] was used for training and the pytorch geometric library was used for the graph dataloader construction [ 11 ].

3. Results and Discussion

Our results present a comprehensive analysis of the performance of the baseline (GraphSage) and proposed (CGCNN+GAT+GraphSage) models on the Materials Project benchmark datasets. The comparative evaluation highlights the impact of incorporating edge features through CGCNN and GAT layers in the proposed model. (a) Task: dielectric (b) Task: jdft2d (c) Task: gvrh (d) Task: kvrh (e) Task: mp-gap (f) Task: mpe-form (g) Task: is-metal (h) Task: perovskites (i) Task: phonons Both models successfully secured positions on the leaderboards, signifying the effectiveness of Graph Neural Networks (GNNs) in predicting material properties. However, as we scrutinize their performance, notable distinctions emerge. The baseline GraphSage model, while making a commendable entry, consistently lags in 4 out of 9 tasks, as it is next to last in the gvrh, kvrh, mp-gap and phonons. This pattern suggests limitations in capturing intricate dependencies within certain materials when relying solely on local node information. In stark contrast, the proposed hybrid model significantly outperforms the baseline across all evaluated tasks. This performance disparity underscores the advantages of integrating diverse GNN layers, especially those capable of leveraging edge features, in materials property prediction. The consistent excellence of the proposed model is evident as it se cures a position within the top 10 models in 5 out of 9 tasks. Notably, in the dielectric task, the proposed model emerges among the top 5 performers, emphasizing its exceptional predictive capabilities. Lastly, it should be noted that the proposed model outperforms the CGCNN [ 4 ] model in 6 out of 8 benchmark tasks.

These results carry implications for materials science, affirming the relevance of machine learning, particularly GNNs, in advancing our understanding of material behaviors. The challenges faced by the baseline model underscore the complexities inherent in materials science tasks and highlight the need for more sophisticated models.

The successes, challenges, and observed disparities provide valuable insights for re searchers seeking to harness the power of machine learning in the intricate realm of materials science. Future considerations may involve fine-tuning hyperparameters, exploring interpretability, and potentially incorporating additional GNN layers to further enhance predictive accuracy. This ongoing research aims to refine and advance the capabilities of models in materials property prediction.

In conclusion, our comprehensive examination of the baseline and proposed models sheds light on the intricate landscape of materials science. The successes and challenges observed underscore the evolving role of machine learning, encouraging continued exploration and refinement to unlock new possibilities in materials discovery and understanding.

4. Conclusions

In the culmination of our study, we reflect on the insights gained from comparing the baseline GraphSage model and the proposed hybrid model (CGCNN+GAT+GraphSage) across a spectrum of tasks within the Materials Project benchmark datasets. Both models demonstrated their competence by securing positions on the leaderboards, affirming the applicability of Graph Neural Networks (GNNs) in predicting material properties. However, a nuanced examination revealed distinct performance characteristics that unveil valuable considerations for the field of materials science.

The baseline GraphSage model, although making an entry, consistently faced challenges in capturing intricate dependencies within materials, particularly evident in its lower standings in 4 out of 9 tasks. This emphasizes the limitations associated with relying solely on local structural information for materials property prediction. In contrast, the proposed hybrid model emerged as a formidable solution, showcasing significant performance advantages across all evaluated tasks. The incorporation of Convolutional Graph Neural Network (CGCNN) and Graph Attention Network (GAT) layers, along with GraphSage layers, demonstrated a remarkable ability to leverage edge features and non-local dependencies, thereby vastly outperforming the baseline model. The proposed model’s consistent excellence, securing positions in the top 10 for 5 out of 9 tasks and achieving a top5 status in the critical dielectric task, highlights its robust predictive capabilities. This success underscores the importance of a holistic approach, considering both structural and edge-related features, in materials property prediction.

Moving forward, research will concentrate on refining hyperparameters, enhancing interpretability, and integrating additional GNN layers to improve predictive accuracy. These efforts aim to advance machine learning models in tackling the complexities of materials science tasks.

In conclusion, our study underscores the importance of sophisticated models capable of understanding both local and non-local relationships within materials. As we navigate the intersection of machine learning and materials science, our findings propel ongoing exploration, fostering advancements in materials discovery and understanding.

Acknowledgements

This work was supported by the Horizon Europe Framework Programme and the EC-funded project DiMAT under grant agreement No 101091496.

Declaration on Generative AI

The author(s) have not employed any Generative AI tools.

References

[1] J. Cheng , C. Zhang, and L. Dong, “ A geometric-information-enhanced crystal graph network for predicting properties of materials,” Communications Materials , vol. 2 , no. 1 , p. 92 , 2021 .

[2] A. Dunn , Q. Wang , A. Ganose , D. Dopp , and A. Jain , “ Benchmarking materials property prediction methods: the matbench test set and automatminer reference al gorithm , ” npj Computational Materials , vol. 6 , no. 1 , p. 138 , 2020 .

[3] W. Hamilton , Z. Ying , and J. Leskovec , “ Inductive representation learning on large graphs , ” Advances in neural information processing systems , vol. 30 , 2017 .

[4] T. Xie and J. C. Grossman , “ Crystal graph convolutional neural networks for an ac curate and interpretable prediction of material properties,” Physical review letters , vol. 120 , no. 14 , p. 145301 , 2018 .

[5] P. Veliˇckovi´c, G. Cucurull, A. Casanova , A. Romero , P. Lio , and Y. Bengio , “ Graph attention networks , ” arXiv preprint arXiv:1710.10903 , 2017 .

[6] V. Fung , J. Zhang , E. Juarez, and B. G. Sumpter , “ Benchmarking graph neural net works for materials chemistry , ” npj Computational Materials , vol. 7 , no. 1 , p. 84 , 2021 .

[7] B. Ramsundar, Molecular machine learning with DeepChem . PhD thesis , Stanford University, 2018 .

[8] S. P. Ong , W. D. Richards , A. Jain , G. Hautier, M. Kocher , S. Cholia , D. Gunter , V. L. Chevrier , K. A. Persson , and G. Ceder, “ Python materials genomics (pymatgen): A robust, open-source python library for materials analysis , ” Computational Materials Science , vol. 68 , pp. 314 - 319 , 2013 .

[9] D. P. Kingma and J. Ba , “ Adam: A method for stochastic optimization , ” arXiv preprint arXiv:1412.6980 , 2014 .

[10] A. Paszke , S. Gross , F. Massa , A. Lerer , J. Bradbury , G. Chanan, T. Killeen , Z. Lin , N. Gimelshein , L. Antiga , A. Desmaison , A. Kopf , E. Yang , Z. DeVito , M. Raison , A. Tejani , S. Chilamkurthy , B. Steiner , L. Fang , J. Bai , and S. Chintala , “ Pytorch: An imperative style, high-performance deep learning library ,” in Advances in Neural Information Processing Systems 32 , pp. 8024 - 8035 , Curran Associates, Inc., 2019 .

[11] M. Fey and J. E. Lenssen , “ Fast graph representation learning with pytorch geometric . arxiv 2019 ,” arXiv preprint arXiv: 1903 .02428, 1903 .