In this work we compare the performance of Pascal P100 GPUs vs POWER8 CPU on OpenPOWER HPC system by VASP calculation of energy and magnetic characteristics of Fe/Cu(111)/Fe and Co/Cu(100)/Co multilayer magnetic nanostructures. We revealed that the VASP code demonstrates a maximum performance on OpenPOWER System with the GPUs.
i=1 From the variational principle, the Kohn-Sham equations are obtained: −
2 + Veff + B xc (r ) − v v (r ) = 0,
N m (r ) = v* (r ) v (r )
v=1 B xc = E xc (r ), m(r ) m(r ) (1) (2) where Bxc is the effective magnetic field arising from the exchange-correlation energy.
The main problem associated with the density functional theory method is that exact analytical expressions for exchange and correlation functionals are known only for the particular case of a gas of free electrons. Nevertheless, the existing approximations allow us to calculate a number of physical quantities with sufficient accuracy.
In this work we used GGA (generalized gradient approximations) approximations in terms of Perdew–Burke–
Ernzerhof (PBE) [
E xGcGA n (r ), n (r ) = n (r ), n (r ), n (r ), n (r )dr (3)
The essence of the projection augmented wave (PAW) method is to transform the pseudowave functions, obtained
in the pseudopotential method into all-electron wave functions, thereby restoring the information lost when
considering pseudowave functions. The number of plane wave components is limited by the Cut-off Energy. To
describe the first Brillouin zone we used standard method Monkhorst–Pack with the parameter K-points
characterizing regular grid in k-space [
Official support of the GPU calculations appeared in VASP from version 5.4.1 and in the our work we used version 5.4.4. VASP has one precompiled configuration file, named makefile.include, with a lot of parameters. Showing all parameters is redundancy and we present the main part of it in table 1. We use Intel Parallel Studio XE C/C++ with Intel MKL library to compile the VASP package on X86_64 architecture. The optimization flags were choosen -O1 and -O2 because compilation with harder optimization was not complete successfully. It was used Ubuntu 16.04 with 4.4.0-137 kernel. We used XlC 13.1.5 and Xlf 15.1.5 with including ESSL library on CentOS 7 with 3.10.0-514 kernel to compile VASP on IBM Power System S822LC.
IBM Power System S822LC is two-socket HPC system with two POWER8 CPUs with 20 cores running at 4
GHz and interconnected with two Nvidia Pascal P100 GPUs with a high bandwidth (80GByte in and 80GByte out)
NVLink 1.0 interface (Fig. 1). It is very important for exchange data between multiple GPUs and fast load data
from CPU. The major goal of this system is efficiently use GPU units and accelerating calculations. A large part of
HPC resources installed during the last decade are based on Intel CPUs. Novel generations of Intel CPUs present a
wide spectrum of multicore processors [
In this work, the results of numerical first-principles calculations of the energy and magnetic characteristics for cobalt and iron films on a copper surface obtained by using VASP software package by means of the Projector Augmented Wave (PAW) method are presented. The values of the total energy of collinear spin configuration, the total magnetic moment and the magnetic moments of Co and Fe atoms are calculated. We investigated a system consisting of a copper slab and adsorbed on it from both sides by a ferromagnetic film with the thickness of the films in three monoatomic layers. The multilayer structure was simulated using a periodic 2×2 36-atom supercell with the lattice constant corresponding to the copper substrate a = 3.6367 (5) Å, which we obtained as a result of calculations taking into account the optimization of the lattice parameters. The surface face orientation is (100) for Co/Cu system and (111) for Fe/Cu system.
For Fe/Cu system the calculations of the total energy were realized for ferromagnetic and two different antiferromagnetic spin configurations. The antiferromagnetic spin configurations for which the calculation was carried out are shown in Fig. 3. The magnetic moment of the atoms is directed along the z axis.
VASP INCAR file has several adjusting parameters that can increase GPU performance. The main of those are NCORE, NPAR, NSIM, LPLANE.
− NCORE determines how many cores work on individual orbital; − NPAR depend on NCORE as NCORE = number of the cores / NPAR; − − − −
If NPAR is equal to the number of cores than NCORE = 1, therefore, one orbital will treat by one core. In the INCAR file we need to set only NCORE or NPAR parameter because NPAR have precedence over NCORE and in the relatively modern version of VASP using NCORE instead of NPAR is recommended. NSIM is an important parameter to get calculation on GPU faster. It changes the number of bands treated simultaneously. There is an opinion that for GPU NSIM parameter needs to be increased while we have free memory on the GPU.
LPLANE is a useful parameter for optimization which can reduce intercommunication time, but it is actual firstly for massively parallel systems, according to VASP documentation.
For CoCu system we used three configurations to compare VASP calculation times with similar INCAR parameters for IBM POWER8 CPU and Intel Haswell CPU. We set LREAL=.TRUE. as described at VASP official documentation and use NCORE=1 with one MPI thread for GPU calculations. We do not use NVIDIA MPS system and do not set NSIM parameter clearly, but we know that it is important for large tasks especially. Core i7 has only 4 real cores and we run VASP with 4 processes only. On POWER8 we run VASP on 8 cores for using most of one CPU. The results of magnetization and free energy calculations (Table 3) are well correlated. The calculations with the GPU provide less accuracy, but the value of the error is not so sufficient. The times of calculations are different for POWER8 system and Intel Core i7 (Table 4). The one POWER8 thread was more efficient than one Intel thread for the VASP calculations. If we use GPU only with one MPI thread we have much better performance (Table 4) than Intel or POWER CPUs even without optimizations of VASP parameters in INCAR file.
For FeCu systems we used NCORE = 4 and NSIM = 32 to get better performance for GPU calculations We perform simulation of the ferromagnet FeCu system on ten POWER8 cores to compare execution times with GPU (Fig. 4). As we can see in Table 5 antiferromagnet spin configuration need sufficient more memory than a ferromagnet. Execution time on GPU highly depends on spin configuration too. It makes sense to note, that GPU utilization is not full and floating from about 20% to 70% during calculation for both systems.
40 35 )30 s r u o25 h n i ( e20 m i t c e15 x E 10 5 0
VASP is widely used by researchers to get characteristics of solids and multilayer magnetic structures. NCORE and NSIM parameters can be very useful to maximize performance on GPU. The values of acquired quantities and accuracy of GPU calculations are in a good agreement with CPU results. To use VASP efficiently with GPUs more memory and calculation time is required in comparing with calculations on CPU, especially for antiferromagnet spin configurations.
We would like to thank the IBM experts, who help us to optimize the VASP package for IBM Power Systems
S822LC. This research was supported by the grants 17-02-00279, 18-42-550003 of Russian Foundation of Basic
Research and by the grant MD-6868.2018.2 of the President of the Russian Federation. The simulations were
supported by the computational resources of Shared Facility Center ”Data Center of FEB RAS” (Khabarovsk) [