🎓 Intranet

Tips using Fugaku

3 min read

Hardware

48 cores per node in ARM64 architecture However, the login node is an Intel machine. Compilation uses Fujitsu compiler, and the target mode should be set to ARM.

Job Scheduler

Fugaku uses PJM job scheduler Equivalent command:

PBS ProPJM
qstatpjstat
qsubpjsub
qdelpjdel

Example submission script for VASP

#!/bin/sh
#PJM -L "rscgrp=small"
#PJM -g "PROJECTID"
#PJM -L node=2
#PJM -L "elapse=24:00:00"
#PJM --mpi  max-proc-per-node=48
#PJM -x PJM_LLIO_GFSCACHE=/vol0004
#PJM -S
#PJM -m b
#PJM -m e
#PJM -N jobname
 
export OMP_NUM_THREADS=1
 
export LD_LIBRARY_PATH=/lib64:${LD_LIBRARY_PATH}
export vasp_std=/PATH/vasp_std
export vasp_gam=/PATH/vasp_gam
vasp_ncl=/PATH/vasp_ncl
export PATH=$PATH:/PATH
 
echo `date` "Starting relaxation ..." >> status.txt
if [ ! -f vasprun.xml.relaxed.gz ]
then
  if [ ! -s POSCAR.orig ]
  then
    echo `date` "Backuping POSCAR to POSCAR.orig ....." >> status.txt
    cp POSCAR POSCAR.orig
  fi
  if [ -s CONTCAR ]
  then
    echo `date` "Found CONTCAR... Copying CONTCAR to POSCAR and continuing relaxation ...." >> status.txt
    cp CONTCAR POSCAR
  fi
  echo `date` "Running VASP" >> status.txt
  mpirun -n 96 vasp_std >> vasp.out
  echo `date` `tail -n -1 OSZICAR` >> status.txt
 
  rel=`tail -1 OSZICAR | cut -c 1-4`
 
  grep "Inconsistent Bravais lattice types found for crystalline" OUTCAR
 
  if [ $? -eq 0 ]
  then
    echo "SYMPREC = 1e-08" >> INCAR
  fi
 
 
  while [ ${rel} -gt 10 ]; do
    cp CONTCAR POSCAR
    rm WAVECAR CHG*
    echo `date` "Running VASP " >> status.txt
    mpirun -n 96 vasp_std >> vasp.out
    echo `date` `tail -n -1 OSZICAR` >> status.txt
    rel=`grep -v '^$' OSZICAR| tail -1 | cut -c 1-4`
    if [ ${rel} -gt 10 ]
    then
            echo `date` "No. of ionic steps ${rel} doesn't meet the requirement (<10), restarting ....." >> status.txt
    fi
  done
 
 
  if [ ${rel} -le 10 ]
  then
    echo `date` "Relaxation finished!" >> status.txt
    mv CONTCAR CONTCAR.relaxed
    mv DOSCAR DOSCAR.relaxed
    mv EIGENVAL EIGENVAL.relaxed
    mv IBZKPT IBZKPT.relaxed
    mv INCAR INCAR.relaxed
    mv KPOINTS KPOINTS.relaxed
    mv OSZICAR OSZICAR.relaxed
    mv OUTCAR OUTCAR.relaxed
    mv POSCAR POSCAR.relaxed
    mv PROCAR PROCAR.relaxed
    mv vasprun.xml vasprun.xml.relaxed
    mv vasp.out vasp.out.relaxed
    rm WAVECAR CHG*
    echo `date` "Ending job ..." >> status.txt
    gzip -f  *
    exit 0
 
  else
    echo `date` "Error or unfinished job!" >> status.txt
    exit 1
  fi
 
else
  echo `date` "vasprun.xml.relaxed.gz exist, skip this calculation ....." >> status.txt
  exit 0
fi

Module system

Fugaku uses spack to manage modules Add following to ~/.bashrc to load spack (need relogin)

alias loadspack=". /vol0004/apps/oss/spack/share/spack/setup-env.sh"

Some commands to use spack

  • spack list show all installed modules
  • spack find xx find installed module, e.g. spack find gcc
  • spack load xx load a module, e.g. spack load gcc@12.2.0
  • spack unload xx unload a module

Tips for running VASP

Optimal NCORE is 12 for VASP

Graph View