Spack is a package manager for supercomputers, Linux, and macOS. It makes installing scientific software easy. Spack isn’t tied to a particular language; you can build a software stack in Python or R, link to libraries written in C, C++, or Fortran, and easily swap compilers or target specific microarchitectures.
First, on the head node - which we connected to via SSM or DCV we’ll run:
export SPACK_ROOT=/shared/spack git clone -b v0.19.0 -c feature.manyFiles=true https://github.com/spack/spack $SPACK_ROOT echo "export SPACK_ROOT=$SPACK_ROOT" >> $HOME/.bashrc echo "source \$SPACK_ROOT/share/spack/setup-env.sh" >> $HOME/.bashrc source $HOME/.bashrc
This script assumes you have a
/shared filesystem that’s accessible to all the compute nodes. I reccomend configuring this with either EFS or FSx Lustre so you can persist installed packages across multiple cluster. This makes doing version upgrades much easier.
This script is also written with a single-user environment in mind. If you have multiple users, each of them will need to add
export SPACK_ROOT=/shared/spack && source \$SPACK_ROOT/share/spack/setup-env.sh to their
Spack Build Cache
v0.18.0, we introduced the concept of a public build cache. This allows users to install packages from pre-built binaries, dramatically lowering build time, up to 20x faster for certain packages.
You can also browse the contents of this caches at cache.spack.io.
To get started, add the binary mirror:
spack mirror add binary_mirror https://binaries.spack.io/develop spack buildcache keys --install --trust
Then you’re able to install packages from the binary cache. For example the package WRF can be installed like so:
spack install wrf
When you get a cache hit you’ll see
Extracting [package] from binary cache like so:
To check with packages you have installed, run:
So how much faster is this?
For WRF, this improves the installation time from 40 minutes to 10 minutes. A 4x speedup.