About VASP : VAMP/VASP is a package for performing ab-initio quantum-mechanical molecular dynamics (MD) using pseudopotentials and a plane wave basis set. The approach implemented in VAMP/VASP is based on a finite-temperature local-density approximation (with the free energy as variational quantity) and an exact evaluation of the instantaneous electronic ground state at each MD-step using efficient matrix diagonalization schemes and an efficient Pulay mixing. These techniques avoid all problems occurring in the original Car-Parrinello method which is based on the simultaneous integration of electronic and ionic equations of motion. The interaction between ions and electrons is described using ultrasoft Vanderbilt pseudopotentials (US-PP) or the projector augmented wave method (PAW). Both techniques allow a considerable reduction of the necessary number of plane-waves per atom for transition metals and first row elements. Forces and stress can be easily calculated with VAMP/VASP and used to relax atoms into their instantaneous groundstate.

Official website : http://cms.mpi.univie.ac.at/vasp/


This document explains how to build VASP-4.6.36 and VASP-5.2.8 on Intel Westmere with Infiniband network, using the following software:

  • Intel Compiler Suite 11.1.072 (also includes MKL, and FFTW wrapper library for Intel MKL)
  • OpenMPI-1.4.2*

* OpenMPI was built with ICS-11.1.072, BLCR-0.8.2, OFED-1.5.1 and with SGE flags.

If you want to build it with Intel-MPI, you only have to change two lines in the makefile. The performance results for OpenMPI and Intel-MPI, are almost the same with VASP-5.2.8. Finally, we chose OpenMPI because of the easy SGE integration and the "checkpointing & restart" options.
It is important to know that this build it is highly optimised for our environment, and obviously, if you have other network or architecture, you will have to investigate what kind of compilers, libraries and parallel environments offers you the best performance.
VASP can do several types of calculations, and the scalability of this kind of processes can be improved if you use some particular optimisations when you build the binary. In our clusters, we have 6 binaries for VASP-5.2 and 6 more for VASP4.6.

  • vasp MPI parallel, charge density and wavefunction complex
  • vasp_cd MPI parallel, charge density: half grid mode (-DNGZhalf)
  • vasp_gamma MPI parallel, gamma-point only (-DwNGZhalf)
  • vasp_vtst MPI parallel, charge density and wavefunction complex, includes VASP TST Tools
  • vasp_vtst_cd MPI parallel, charge density: half grid mode (-DNGZhalf), includes VASP TST Tools
  • vasp_vtst_gamma MPI parallel, gamma-point only (-DwNGZhalf), includes VASP TST Tools


Figure 1 - Performance and scalability of VASP-4.6.36 on Intel Westmere (X5650) comparing three binaries.

Table 1 - Walltime of XRQTC Benchmarks on Intel Westmere (X5650) with Infiniband QDR.

Previous

We have evaluated other compilers, libraries and compiling options, and we have obtained the best performance with this proceeding.

Environment Set Up

For long term builds it is recommended to use a screen session. This software allows to reattach a background session.

# screen -S VASP-5.2.8_ics_mkl_ompi
# tar -zxvf vasp.5.2.tar.gz
# tar -zxvf vasp.5.lib.tar.gz

First of all, we load the modules needed to build this software. It is very popular to integrate the dependencies inside the module files. In this case, when we load the OpenMPI environment, this module also loads the Intel Compiler Suite, BLCR and OFED modules.

# module load OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2
# module load intel_mkl/11.1.072
# module list
Currently Loaded Modulefiles:
1) intel_compiler_suite/11.1.072
2) blcr/0.8.2
3) OFED/1.5.1
4) OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2
5) intel_mkl/11.1.072

Build FFTW wrapper library for Intel MKL

You can also build VASP with FFTW3, but we find more performance using the FFTW wrapper library for Intel MKL. Change directory to $MKL_PATH/interfaces/fftw3xf/

# cd /opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf/

Build the fftw3.x Fortran wrapper library for Intel 64 (previously known as em64t) platform using the Intel compilers.

# make libem64t compiler=intel

This will create the fftw wrapper library "libfftw3xf_intel.a" in

# file ../../lib/em64t/libfftw3xf_intel.a

Build libdmy.a

Change directory to vasp.5.2.lib and change the fortran compiler ifc for ifort.

# cd vasp.5.lib/
# vi makefile.linux_ifc_P4
# make -f makefile.linux_ifc_P4

After a successful compilation, you will find libdmy.a in the same directory.

Build VASP

 

Now, change directory to vasp.5.2.

# cd ../vasp.5.2

In order to compile with fftw3 included in MKL, just copy fftw3.f in the main directory.

# cp /opt/intel/Compiler/11.1/072/mkl/include/fftw/fftw3.f .

We will create a makefile template from makefile.linux_ifc_P4:

# cp -p makefile.linux_ifc_P4 ../makefile.linux_ifc_X5650_charge-density

Now you will have 2 options:

  • If you have exactly the same architecture, you can download all 6 makefiles from the HPCKP GITHUB repository (and maybe, you will have to change some path's).
  • The other way is to edit the makefile as the following:

Edit the makefile:

# vi  ../makefile.linux_ifc_X5650_charge-density

FFLAGS

You will need to add the -i_dynamic flag in order to preserve compatibility with older libraries included on previous versions of MKL.

FFLAGS=-I/opt/intel/Compiler/11.1/072/mkl/include/fftw -FR -lower_case -i_dynamic
FFLAGS_F77= -i_dynamic

OPTIMISATION

The new Intel processors series X5600 (Intel Nehalem) are SSE4.2 capable, that is why we change the standard -xW for -xSSE4.2

OFLAG=-O3 -xSSE4.2
OFLAG2=-O1 -xSSE4.2
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =
DEBUG = -FR -O0
INLINE = $(OFLAG)

BLAS and LAPACK

If you try to compile with the default description of BLAS and LAPACK for MKL, you will find some errors. We suggest to follow the Intel instructions and examples and visit the official web site of Intel, where you can find a web form with a MKL link line advisor.

MKLINCLUDE=/opt/intel/Compiler/11.1/072/mkl/include
MKLPATH=/opt/intel/Compiler/11.1/072/mkl/lib/em64t
BLAS=-L${MKLPATH} -I${MKLINCLUDE} -I${MKLINCLUDE}/em64t/lp64 -lmkl_blas95_lp64 -Wl,--start-group ${MKLPATH}/libmkl_intel_lp64.a ${MKLPATH}/libmkl_sequential.a ${MKLPATH}/libmkl_core.a -Wl,--end-group -lpthread
LAPACK=-L${MKLPATH} -I${MKLINCLUDE} -I${MKLINCLUDE}/em64t/lp64 -lmkl_lapack95_lp64 -Wl,--start-group ${MKLPATH}/libmkl_intel_lp64.a ${MKLPATH}/libmkl_sequential.a ${MKLPATH}/libmkl_core.a -Wl,--end-group -lpthread

COMPILER

Obviously, it is mpif90 (OpenMPI previously built with ICS-11.1.072).

FC=mpif90
FCL=$(FC)

For the first build, we will use NGZhalf (charge density reduced in Z direction), but we will do the same with wNGZhalf (gamma point only reduced in Z direction) and without any reduction.

#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (usually slower on 100 Mbit Net)
#-----------------------------------------------------------------------

CPP = $(CPP_) -DMPI -DHOST=\"LinuxIFC\" -DIFC \
-Dkind8 -DCACHE_SIZE=12000 -DPGF90 -Davoidalloc -DNGZhalf \
-DMPI_BLOCK=1000 -DscaLAPACK \
-DRPROMU_DGEMV -DRACCMU_DGEMV

SCALAPACK

The ScaLapack version included on the MKL works better than our own build version. You may pay attention to the BLACS library, because it is MPI dependant. In our case, we take the OpenMPI version, so, we use ${MKLPATH}/libmkl_blacs_openmpi_lp64.a

SCA=${MKLPATH}/libmkl_scalapack_lp64.a ${MKLPATH}/libmkl_solver_lp64_sequential.a -Wl,--start-group ${MKLPATH}/libmkl_intel_lp64.a ${MKLPATH}/libmkl_sequential.a ${MKLPATH}/libmkl_core.a ${MKLPATH}/libmkl_blacs_openmpi_lp64.a -Wl,--end-group -lpthread

Libraries for MPI

LIB     = -L../vasp.5.lib -ldmy  \
../vasp.5.lib/linpack_double.o $(LAPACK) \
$(SCA) $(BLAS)

FFTW wrapper library for Intel MKL

In order to use the previously compiled FFTW wrapper for MKL, just add at the end of FFT3D this library libfftw3xf_intel.a

FFT3D   = fftmpiw.o fftmpi_map.o fftw3d.o fft3dlib.o  /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libfftw3xf_intel.a

fft3dlib.o

And last, you need to modify fft3dlib.o prescription as the follow:

fft3dlib.o : fft3dlib.F
$(CPP)
$(FC) -FR -lowercase -O1 -xSSE4.2 -prefetch- -unroll0 -vec_report3 -c $*$(SUFFIX)

Nevertheless, if you want to try to compile with Intel-MPI, you only have to change FC and BLACS library:

FC=mpif90 for FC=mpiifort
${MKLPATH}/libmkl_blacs_openmpi_lp64.a for ${MKLPATH}/libmkl_blacs_intelmpi_lp64.a

MAKE

Now, You can close the makefile and run the make instruction in order to obtain the final binary.

# make -f ../makefile.linux_ifc_X5650_charge-density | tee -a ../vasp-5.2.8_ics-11.1.072_mkl_fftw-wrapper_ompi-1.4.2