ABOUT NWCHEM:

NWChem provides many methods for computing the properties of molecular and periodic systems using standard quantum mechanical descriptions of the electronic wavefunction or density. Its classical molecular dynamics capabilities provide for the simulation of macromolecules and solutions, including the computation of free energies using a variety of force fields. These approaches may be combined to perform mixed quantum-mechanics and molecular-mechanics simulations.
NWChem is an open-source computational chemistry package distributed under the terms of the Educational Community License (ECL) 2.0
Copyright (c) 1994-2010 Pacific Northwest National Laboratory, Battelle Memorial Institute.
This software and its documentation were developed at the EMSL at Pacific Northwest National Laboratory, a multiprogram national laboratory, operated for the U.S. Department of Energy by Battelle under Contract Number DE-AC05-76RL01830. Support for this work was provided by the Department of Energy Office of Biological and Environmental Research, Office of Basic Energy Sciences and the Office of Advanced Scientific Computing.

Official website: http://www.nwchem-sw.org/index.php/Main_Page

This document explains how to build Nwchem 5.1.1 on Intel Westmere with Infiniband Network using the following software:

  • Intel Compiler Suite 11.1.072 *1)
  • OpenMPI 1.4.2 *2)


*1) We don't use MKL libraries because it seems that there are more problems with the compilation of nwchem with this library.
*2) OpenMPI was build with intel 11.1.072, BLCR 8.2, OFED-1.5.1 and with SGE flags.

The reason of using this version of Intel Compiler is because we tested the compilation with different versions (intel 8.1, 9.1, 11.1.072) that provokes a high number of failed tests in the Test phase.

Environment Set Up


For long term builds is is recommended to use a screen session. This software allows to reattach a background session.

# screen -S nwchem-5.1.1
# bzip2 -dc nwchem-5.1.1.tar.bz2| tar -xv


First of all, we load the modules needed to build this software. It is very popular to integrate the dependencies inside the module files. In this case, when we load the OpenMPI environment, this module also loads the Intel Compiler, BLCR and OFED modules


# module load OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2
# module list
Currently Loaded Modulefiles:
1) intel_compiler_suite/11.1.072
2) blcr/0.8.2
3) OFED/1.5.1
4) OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2

 

Build Nwchem

First of all we must enter to the source directory:

cd nwchem-5.1.1/src


Then, we change the file's permissions of the directory:

find . -exec chmod go-rwx "{}" ";" 


And we load the following environment variables:

#Compilation settings


export CC=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpicc
export CXX=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpicxx
export F77=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpif77
export F90=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpif90
export FC=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpif90
export FL=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpif90

export FOPTIMIZE="-O3 -xSSE4.2"
export COPTIMIZE="-O3 -xSSE4.2"



# Our nodes have a maximum of 48GB of memory, in this variable we set the max memory to use with nwchem, in our case we reserve 3GB for the Operative System

export LIB_DEFINES="-DDFLT_TOT_MEM=45327376"


#Parallelization settings

export USE_MPI=y  
export MPI_LOC=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/
export MPI_LIB=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/lib/
export MPI_INCLUDE=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/include/
export LIBMPI="-lmpi -lopen-pal -lopen-rte -lmpi_f90 -lmpi_f77"


#Nwchem settings

 

export LARGE_FILES=TRUE
export NWCHEM_MODULES="all venus geninterface"
export NWCHEM_TARGET="LINUX64"
export TARGET="LINUX64"
export NWCHEM_TOP=/scratch/nwchem-5.1.1


#Infiniband settings

 

export IB_HOME=/usr
export IB_INCLUDE=/usr/include/infiniband/
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"
export ARMCI_NETWORK=OPENIB


Then, it's time to generate the Makefile with this command:

make nwchem_config 2>&1 | tee kkconfig


We compile it:

make FC=mpif90 2>&1 | tee kkmake


Once compiled, we must test the compilation:

 cd $NWCHEM_TOP/QA
export NWCHEM_EXECUTABLE=/scratch/nwchem-5.1.1/bin/LINUX64/nwchem
export MPIRUN_PATH=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpirun
export PARALLEL_PATH=/aplic/MPI/OpenMPI/1.4.2_ics-11.1.072_ofed-1.5.1_blcr-8.2/bin/mpirun

./doqmtests 4 >& doqmtests.log &

./runtest.md >& runtest.md.log &


And check that all is fine