Isambard 3

Contributed by Jeremy Pike (Exeter)

• Load the system Cray GNU programming environment and Cray python

module load PrgEnv-gnu
module load cray-python
module list

Currently Loaded Modules:
  1) brics/userenv/2.4   3) brics/tmux/3.4-r0   5) craype/2.7.32      7) libfabric/1.22.0     9) cray-libsci/24.07.0  11) PrgEnv-gnu/8.5.0
  2) brics/default/1.0   4) gcc-native/13.2     6) craype-arm-grace   8) craype-network-ofi  10) cray-mpich/8.1.30    12) cray-python/3.11.7

• Make a directory for the install:

mkdir ~/firedrake
cd ~/firedrake

• Download firedrake utility script. Get the latest version from the docs.

• Clone the version of PETSc specified by firedrake-configure. Note the --no-package-manager flag.

git clone --depth 1 --branch $(python3 firedrake-configure --show-petsc-version --no-package-manager) https://gitlab.com/petsc/petsc.git
cd petsc

• Run PETSc configure, passing in the flags generated by firedrake-configure. Note the download-cmake flag.

python3 ../firedrake-configure --show-petsc-configure-options --no-package-manager | xargs -L1 ./configure --download-cmake

This pessimistically assumes that no external packages are available and runs with the following flags. Should perhaps investigate if its a good idea to load some additional modules and link in the packages. Also need to try with -mcpu=neoverse-v2 flag.

--with-c2html=0 --with-debugging=0 --with-fortran-bindings=0 --with-shared-libraries=1 --with-strict-petscerrorcode PETSC_ARCH=arch-firedrake-default --COPTFLAGS='-O3 -march=native -mtune=native' --CXXOPTFLAGS='-O3 -march=native -mtune=native' --FOPTFLAGS='-O3 -march=native -mtune=native' --download-bison --download-fftw --download-hdf5 --download-hwloc --download-metis --download-mumps --download-netcdf --download-pnetcdf --download-ptscotch --download-scalapack --download-suitesparse --download-superlu_dist --download-zlib --download-hyprep --download-cmake

Note we've had to add --download-cmake, without it I get the error:

--dowload-cmaA package requires CMake version 3.22.0 (detected version is 3.20.4): use --download-cmake

Now run make command as promted:

make PETSC_DIR=~/firedrake/petsc PETSC_ARCH=arch-firedrake-default all

Then finally check it working as promted:

make PETSC_DIR=~/firedrake/petsc PETSC_ARCH=arch-firedrake-default check

Outputs:

Running PETSc check examples to verify correct installation
Using PETSC_DIR=~/firedrake/petsc and PETSC_ARCH=arch-firedrake-default
C/C++ example src/snes/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/tutorials/ex19 run successfully with 2 MPI processes
C/C++ example src/snes/tutorials/ex19 run successfully with HYPRE
C/C++ example src/snes/tutorials/ex19 run successfully with MUMPS
C/C++ example src/snes/tutorials/ex19 run successfully with SuiteSparse
C/C++ example src/snes/tutorials/ex19 run successfully with SuperLU_DIST
C/C++ example src/vec/vec/tests/ex47 run successfully with HDF5
Completed PETSc check examples

• Install Firedrake

Create virtual environment for firedrake and use firedrake-configure to set some environment variables:

cd ~/firedrake
python3 -m venv venv-firedrake
. venv-firedrake/bin/activate
# remember the --no-package-manager
export $(python3 firedrake-configure --show-env --no-package-manager) 

Now install firedrake. Check the docs for the latest pip commands:

pip cache remove petsc4py
pip cache remove firedrake
pip install --no-binary h5py 'firedrake[check]'

As mpiexec isnt available on Isamabard 3 for firedrake-check successfully I did the following hack

# Maybe this isnt the most sensible place for this dummy file
echo 'srun --mpi=cray_shasta "$@"' > ./venv-firedrake/bin/mpiexec
chmod +x ./venv-firedrake/bin/mpiexec

Also need the following command:

export LD_LIBRARY_PATH=$PETSC_DIR/$PETSC_ARCH/lib:$LD_LIBRARY_PATH

Without I get the following error:

Traceback (most recent call last):
  File "~/gusto/venv-gusto/bin/firedrake-check", line 5, in <module>
    from firedrake._check import main
  File "~/gusto/venv-gusto/lib/python3.11/site-packages/firedrake/__init__.py", line 16, in <module>
    petsc4py.init(sys.argv)
  File "~/gusto/venv-gusto/lib/python3.11/site-packages/petsc4py/__init__.py", line 42, in init
    PETSc = petsc4py.lib.ImportPETSc(arch)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/gusto/venv-gusto/lib/python3.11/site-packages/petsc4py/lib/__init__.py", line 33, in ImportPETSc
    return Import('petsc4py', 'PETSc', path, arch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/gusto/venv-gusto/lib/python3.11/site-packages/petsc4py/lib/__init__.py", line 100, in Import
    module = import_module(pkg, name, path, arch)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/gusto/venv-gusto/lib/python3.11/site-packages/petsc4py/lib/__init__.py", line 77, in import_module
    module = importlib.util.module_from_spec(spec)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Then running firedrake-check gives the following output:

    Running serial smoke tests
...........................                                                                                                                                                                            [100%]
27 passed in 2.95s
    Serial tests passed
    Running parallel smoke tests
..                                                                       [100%]
2 passed, 1 deselected in 18.59s
..                                                                       [100%]
2 passed, 1 deselected in 18.59s
..                                                                       [100%]
2 passed, 1 deselected in 18.59s
    Parallel tests passed