Skip to content

platforms

Jump to bottom
Steven Hahn edited this page Apr 3, 2024 · 22 revisions

Specific build instructions

OSX (Catalina - tested on AWS)

Assumes homebrew is installed, which it is by default on the AWS instances.

brew install cmake
brew install openblas
git clone https://github.com/project-asgard/asgard.git
cd asgard
mkdir build
cd build
cmake .. -DCMAKE_CXX_COMPILER=g++-10 -DASGARD_BLAS_PATH=/usr/local/opt/openblas
make
ctest
./asgard

OSX (Mojave)

The homebrew based install of openblas in the Catalina instructions above also likely work here - but I've not tested it.

git clone https://github.com/project-asgard/asgard.git
cd asgard
mkdir build
cd build
cmake ../ -DASGARD_BUILD_OPENBLAS=1
make
ctest
./asgard

Build with Optional ScaLAPACK dependency

git clone https://github.com/project-asgard/asgard.git
cd asgard
mkdir build
cd build
cmake -DASGARD_USE_MPI=ON -DASGARD_USE_SCALAPACK=ON ../
make
ctest
./asgard

Fusiont5 Development Machine (ORNL)

prepare environment:

source /home/dg6/spack/share/spack/setup-env.sh
module load openmpi-3.1.2-gcc-7.3.0-zz5232w
module load cuda-10.0.130-gcc-7.3.0-s6ervyw
module load cmake-3.12.2-gcc-7.3.0-duokccd

build asgard:

git clone https://github.com/project-asgard/asgard.git
cd asgard && mkdir build && cd build
cmake ../ -DASGARD_USE_CUDA=1
make
ctest
./asgard

Fusiont6 Development Machine (ORNL)

prepare environment:

module load cmake-3.15.3-gcc-7.4.0-i44ddvb
module load cuda-10.2.89-gcc-7.4.0-hzby2dv
module load openmpi-3.1.5-gcc-7.4.0-s4r4ogv

build asgard:

git clone https://github.com/project-asgard/asgard.git
cd asgard && mkdir build && cd build
cmake ../ -DASGARD_USE_CUDA=1
make
ctest
./asgard

Ascent (OLCF) (belongs to Summit)

  1. Access the Ascent login node:
ssh [email protected]
  1. Load modules, other combinations may work, this one has been tested
module load cmake/3.17.3 cuda/10.1.243 gcc/7.4.0 openblas/0.3.9-omp spectrum-mpi/10.3.1.2-20200121

  1. No need of allocation to compile the code

  2. Make a build directory, compile the code. For a GPU version of the code without MPI:

mkdir build
cd build
cmake ../ -DASGARD_USE_CUDA=1
make -j 10

For a GPU version with MPI:

cmake ../ -DASGARD_USE_CUDA=1 -DASGARD_USE_MPI=1 -DCMAKE_CXX_COMPILER=mpicxx

You should find that ASGarD is able to detect an existing installation of BLAS on the Cori GPU nodes:

Found BLAS: /autofs/nccsopen-svm1_sw/ascent/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-7.4.0/openblas-0.3.9-cjxfkk67xpigoo4qo77tzvigloabwuvr/lib/libopenblas.so  
-- LINALG libraries found: /autofs/nccsopen-svm1_sw/ascent/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-7.4.0/openblas-0.3.9-cjxfkk67xpigoo4qo77tzvigloabwuvr/lib/libopenblas.so;-lm;-ldl;/autofs/nccsopen-svm1_sw/ascent/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/gcc-7.4.0/openblas-0.3.9-cjxfkk67xpigoo4qo77tzvigloabwuvr/lib/libopenblas.so

so it is no longer required to use Intel MKL, or to download and build OpenBLAS by hand. In fact it is recommended to avoid Intel MKL altogether, because the way Intel MKL uses OpenMP threads will cause Nsight Systems to hang when one attempts to profile the code.

  1. Use bsub to submit jobs
bsub launch.bsub

launch.bsub:

#!/bin/bash
# Begin LSF Directives
#BSUB -P GEN138
#BSUB -W 1:00
#BSUB -nnodes 1
#BSUB -T 44
#BSUB -alloc_flags "gpumps smt1"
#BSUB -J cont6_prof
#BSUB -o cont6_prof.%J
#BSUB -e cont6_prof.%J
module load cmake/3.17.3 cuda/10.1.243 gcc/7.4.0 openblas/0.3.9-omp spectrum-mpi/10.3.1.2-20200121

## /!\ UNCOMMENT which one you want to use /!\

## To run ASGarD using a single task (either 1 MPI rank, or without MPI at all), and with 1 OpenMP thread per core on the 2 sockets:
#export OMP_NUM_THREADS=42
#export OMP_PROC_BIND=spread
#export OMP_PLACES=cores
#jsrun -n 1 -a 1 -c 42 -g 1 -d packed -b rs ${PWD}/build/asgard -p continuity_6 -l 3 -d 4

##To run the code using multiple MPI ranks, with 1 GPU per rank:
#export OMP_NUM_THREADS=1
#export OMP_PROC_BIND=spread
#export OMP_PLACES=cores
## use 6 resource set, 1 core per resource set, 6 GPUs (all resource in one node)
#jsrun -n1 -a6 -g6 -c42 -d packed -b packed:7 ${PWD}/buildtmp/asgard -p continuity_6 -l 3 -d 4

GEN138 should be replaced with the project identity related to your account. For more documentation please look into: https://docs.olcf.ornl.gov/systems/summit_user_guide.html#logging-in-to-ascent

  1. To run in interactive mode look into: https://docs.olcf.ornl.gov/systems/summit_user_guide.html#running-jobs
Clone this wiki locally