Project

General

Profile

Debug SURFEX-Crocus

If you are implementing new developments or if you meet unexpected troubles during a run execution, please check the following instructions before contacting the support team :

Always compile SURFEX with the DEBUG option :

export OPTLEVEL=DEBUG
./configure

Once the development is finished,recompile with O2 optimization level before running your long experiments.

For problems occurring inside the snow scheme, it is helpful to use a special debugging mode for Crocus, please read the following documentation:
http://www.cnrm.meteo.fr/surfex-lab/IMG/pdf/doc_crodebug.pdf

Meteo-France supercomputers
To understand unexpected crashes during belenos runs, two options are possible:

1/ For complex bugs, it is highly recommended to use use the ddt debugger from ARM-FORGE.
A local installation of the debugger is much easier to use from Grenoble than the remote installation on the supercomputers.
Unfortunately Meteo-France DSI does not currently provide any access to the debugger for teleworking.

Local installation procedure:

First check the remote version of DDT on the server:

module avail

Download the same version of ddt client on your machine at https://developer.arm.com/downloads/-/arm-forge/older-versions-of-remote-client-for-arm-forge
Then untar and run the install

tar xvf arm-forge-20.0.2-Redhat-7.0-x86_64.tar
arm-forge-20.0.2-Redhat-7.0-x86_64/textinstall.sh

Follow the installation procedure and specify an appropriate path for installation (example: /home/username/arm19.1/forge)

Then start locally the debugger:

/home/username/arm/forge/bin/forge

Below "remote launch" click on "configure", "add", and fill the different fields :

Connection name: belenos
Host name: username@belenos
Remote installation directory: /opt/softs/arm/20.0.2/forge

and click "ok"

Now, you are ready to use the debugger from your PC.

Run the debugger job on the server

Remember, that the binaries must have be compiled in DEBUG mode.
Then, prepare a simple debug job job_debug.sh following this example:

#!/bin/bash
#SBATCH --verbose
#SBATCH --job-name=debug_offline
#SBATCH --nodes=1
#SBATCH --ntasks=80
#SBATCH --ntasks-per-core=1
#SBATCH --time=01:00:00

module load arm/forge/20.0.2
cd /scratch/mtool/username/abort/dump_XXXXXX_reas2m/root/surfex_vortex_task
ulimit -s unlimited
ddt --connect srun --ntasks 80 /home/cnrm_other/cen/mrns/lafaysse/SURFEX/cen_dev/exe/OFFLINE-LXifort-SFX-V8-1-1-MPIAUTO-OMP-DEBUG-X0

(!) Tricky segmentation faults are likely to happen if you forget "ulimit -s unlimited"

Run the job:

sbatch job_debug.sh

or in case of saturation:

sbatch --partition=debug256 --qos=debug job_debug.sh

Once the job is running, a pop-up opens in ddt : you must accept the remote connection in ddt on your PC.
You can start using the debugger normally.

(!) Don't forget to tick the OpenMP box and sectect 1 threads!!

Full documentation available here: http://intradsi.meteo.fr/spip.php?article2007