Project

General

Profile

Data assimilation of snow observations with CrocO

Implementation of a Particle Filter to assimilate spatialized snow observations within SURFEX platform
Development of vortex task to perform the assimilation sequence on beaufix-hendrix, launched through a s2m (snowtools) command.
User documentation : see CrocO user doc

Publication:
Cluzet et al., CrocO : a Particle Filter to assimilate snowpack observations in a spatialised framework (in prep)

Git projects :
  • Surfex_Git2, branch cen_dev
  • Vortex, branch cen_assim
  • snowtools_git, branch cen_assim

Main developments

  • Spatialized PF algorithms developped into SURFEX/SODA in global, k-localized and r-localized configurations
  • Use of ESCROC ensemblist version of Crocus (Lafaysse et al. 2017) and perturbed meteorological forcings (Charrois et al., 2016)
  • Use of Vortex (cen_assim branch) /snowtools_git to launch the assimilation sequence on beaufix and store data on hendrix (similarly as described in Run a SURFEX-Crocus experiment with vortex)

Developer name

Bertrand Cluzet

Date of start of development

January 2018

Commit number before the development

  • Surfex_Git2 : b469f610
  • Vortex : 4ab65aa3

Current status

  • Surfex_Git2: Phased in SURFEX V8.1 branch cen_dev, tag CrocO_v1.0
  • Vortex : Phased with recent (late october 2018 (commit 87922088)) cen_assim branch, tag CrocO_v1.0
  • snowtools_git : commit 53e2e16, cen_assim branch, tag CrocO_v1.0

Key technical points (SURFEX)

  • new diag variable SPECMOD to represent sow reflectance in MODIS bands, sampled from SPEC_ALB
  • writing of this diag variable into PREP file (mandatory for soda assimilation)
  • new dimension in PRO file : MODIS_bands
  • in SODA, use of a SURFEX_LIST (vector of SURFEX_TYPE) to store all members (PREP files) before the assimilation mix
  • mixing of PREP is done "manually" in PUT_ALL_VAR_PF.F90. Some fields (..._ISBA) are not mixed at the moment (no need for us.)
    Need to actualize this routine after any change in the SURFEX types and variable names and for new prognostic/cumulated variables
  • LFIX_OUTPUT (see below)
  • reading of obs. in NETCDF.
  • computing Snow Cover Fraction from SWE in PREP : implementation of a depletion curve (Zaitchik et al., 2009, De Lannoy et al., 2012)

Evaluated against SURFEX test database ?

No, in progress.

New test added in the database

No, in progress

Changes in namelist

&NAM_ASSIM

LASSIM = .TRUE.                       ! Activate Surfex data assimilation  
LCROCO = .TRUE.                       ! Activate CrocO
CASSIM_ISBA = 'PF   '                 ! Particle Filter activation in SODA  (/!\ don't forget the '   ' ) 
CPF_CROCUS = 'GLOBAL' (default)       ! select the PF algo to perform ('GLOBAL', 'RLOCAL', 'KLOCAL')
XDLOC_PF = 1E-8                       ! localisation angle (degrees) for the RLOCAL PF. 
                                      ! Let to default or < 1e-7 to consider only local observations.
NEFF_PF = 1                           ! set the targer effective sample size for the PF with inflation.
NPRINTLEV=1,                          ! verbosity
LEXTRAP_SEA=.FALSE.,                  !  |
LEXTRAP_WATER=.FALSE.,                !  |  Prevent SODA 
LEXTRAP_NATURE=.FALSE.,               !  |  from being too fancy
LWATERTG2=.TRUE.,                     !  |

(!) the following option has been deleted in code versions posterior to #5734b28e
NLOC_PF = 0 (default)                 ! set the maximal number of obs (>0) to select in the 'RLOCAL' and 'KLOCAL'.


About the algorithm of the Particle Filter:
- 'GLOBAL' means that all the observations are used globally for all the points of the simulation.
- 'RLOCAL' means that for each point of the simulation, only observations around this point (if any) are used.
- 'KLOCAL' means that for each point of the simulation, a correlation with other points is calculated. Only observations on best correlated points are used.

&NAM_OBS

  
NOBSTYPE = 8                  ! number of different obs. variables (up to 9)
CFILE_FORMAT_OBS="NC    "     ! daily obs. files on NETCDF file format (/!\ don't forget the '    ' ) 
COBS_M = 'B1','B2','B3','B4',\! Names of MODIS bands and Snow cover fraction variables 
         'B5','B6','B7','SCF' ! in the observation files. (/!\ same length as NOBSTYPE)
                              ! other possible values : 'DEP' (snow depth),
                              ! 'R51', R52', 'R53', 'R54' (band ratios)
NNCO = 1,1,1,1,1,1,1,1        ! select which variables to assimilate among COBS_M 
                              ! (/!\ same length as NOBSTYPE)
XERROBS_M = 0.00071, 0.00046,\! default obs.error *variances* for MODIS bands 
            0.00056, 0.00056,\! (charrois PhD manuscript pp 104.,140.).
            0.002, 0.0015,\   ! last one is a percentage of value for SCF. 
            0.00078, 0.05,\   ! For DEP, it's also an error variance [m^2]
XERROBS_FACTOR_M = 1,1,1,1,1,\! If necessary multiply the prescribed errors by some factors
                   1,1,1 

&NAM_VAR

NVAR=8                        ! number of different model control variables (up to 9)
CVAR_M='PB1','PB2','PB3',\    ! keywords for the control variables to be read. 
       'PB4','PB5','PB6',\    ! here, 7 MODIS spectral bands 
       'PB7','PSB'            ! + bulk SWE, which is the control variable for SCF obs.
                              ! other possible values : 'DEP' (snow depth),
                              ! 'R51', 'R52', 'R53', 'R54' (band ratios).
NNCV = 1,1,1,1,1,1,1,1        ! select which variables to assimilate among CVAR_M
XTAUSCF =4                    ! value for the tau parameter of the depletion curve 
                              !used to retrieve model SCF from bulk SWE (default=4)
XCEILSCF = 20                 ! value for the ceil SWE of the depletion curve (default=20mm)

&NAM_ENS

NENS = 1                      ! ensemble size (overwritten by snowtools_git argument --nmembers)

&NAM_DIAG_ISBAN

LSPECMOD = .TRUE.  ! write MODIS spectral into PREP files, mandatory for assimilation
LPROBANDS = .TRUE. ! mandatory too for MODIS assimilation

&NAM_DIAG_SURFN

LRAD_BUDGET = .TRUE.

&NAM_WRITE_DIAG_SURFn

LRESETCUMUL = .TRUE.
LSNOWDIMNC = .TRUE. ! mandatory + activate layer/bands dimensions in PRO
CSELECT = 'SPECMOD' ! add MODIS bands to PRO file

&NAM_IO_OFFLINE

LFIX_OUTPUT = .TRUE. !(default : .FALSE.) : force hours of output in PRO at 00h+n*timestep 
                     ! instead of timestart+ n*timestep (/!\ possible edge effects on cumulated diags)

local testing of SODA

SODA is the executable performing the assimilation. Provided a set of background PREP files and an observation, it produces a set of analyzed PREP files.
It is possible to test the performance of this algorithm locally, (without the use of vortex, snowtoools_git etc.)
For that, simply execute SODA in a repertory where all files are given (Namelist, PGD, PREP files, observations, etc.)
For more details, ask for support.

Other required changes

Namelist

Forcing files

Spinup files

  • mandatory
  • /!\ V8.0 files are not compatible with V8.1, but it is possible to upgrade it (ask for support).

New files

Assimilation configuration file

  • mandatory (see an example at s2m_postes_12_csv.ini)
  • named s2m_<region_id>.ini
    - region_id is the safran id of the region (for ex. 12 for Grandes-Rousses)
    - region_name is the safran name of the region (for ex. grandes_rousses for Grandes-Rousses)
  • format: python configuration file (text file):
    • first line : section header set to DEFAULT:
      [DEFAULT]
      
    • use the assimdates keyword to specify the *assimilation dates. ex:
      assimdates=2013110106,2013120106
      
    • use the members_id keyword to specify the ids of the ESCROC members you'd like to run (if you are using ESCROC E1* ensemble). Otherwise they will be randomly drawn (and written down in the configuration file). This is necessary for twin experiments and reproducibility.ex with 3 members:
      membersid=505,402,705
      
  • some additional information on the experiment are appended to it during the run. It is then archived (see below)

Observation files

  • daily files named obs_<your_sensor>_<region_name>_yyyymmddhh.nc
    ex : obs_MODIS_grandes_rousses_20341231.nc
  • NETCDF format(see attached example)
    1. same geometry as PREP/PGD
    2. variables must be double
    3. variables _FillValue must be set to 1.e+20.
      • for synthetical observations, <your_sensor> is free to define.

Particle Filter auxiliary output files

  • Output from the assimilation sequence.
  • One file per assimilation date, named PART_YYYYMMDDHH.txt
    This last file give the ids(starting at 1)of the replicated members, separated by spaces. Moreover:
    - if DISTRIBUTED assimilation, assimilation per grid point as CPF_CROCUS = 'KLOCAL': one line per grid point
    - if SEMI-DISTRIBUTED assimilation, CPF_CROCUS = 'GLOBAL': only one line (for all points, same ids of replicated members)

Additionally, depending on the PF configuration the following files are produced

  • ALPHA : (NEFF_PF >0) inflation factor of the background correlation matrix for each point. Only one value if CPF_CROCUS = 'GLOBAL'.
  • BG_CORR : (CPF_CROCUS = 'KLOCAL') background correlation matrix for the assimilation variables (NVAR lines per grid point, variables written in the order of NAM_VAR).
  • IMASK : (CPF_CROCUS = 'KLOCAL') klocalisation mask (FORTRAN IDs of the selected points) for the different variables (NVAR lines per grid point, variables written in the order of NAM_VAR).

Simulation framework

Simulations are performed on beaufix-hendrix through an s2m command (see added options below).
Vortex requires a strict file storing and naming. The following lines are detailing it.

Forcing files

  • stored in a vortex path on hendrix :
    <yourhome>/vortex/safran/<region_id>/<xpid_forcing>/mbxxxx/meteo/
    - xpid_forcing is a user-defined name (usually the name of the experiment that generated the forcings), given to s2m via -f option (see below)

Spinup files

  • named PGD_<region_name>.nc and PREP_<starting_date>.nc
  • stored in a vortex path on hendrix (xxx is "pgd" and "prep" respectively):
    <yourhome>/vortex/s2m/<region_id>/spinup/<xxx>/

Namelist

  • Stored on beaufix home.

Assimilation Configuration file

  • Stored on beaufix home.

Observation files

  • stored in a vortex path on hendrix :
    <yourhome>/vortex/s2m/<region_id>/obs/<your_sensor>/
  • for synthetical observations : do the same.

Simulation outputs storing

If the simulation has succeded :
  • all files are deleted from beaufix
  • PREP and PRO files are stored on hendrix :
    <your_home>/vortex/s2m/<region_id>/<xpid>/<mbxxxx>/<xxx>
    - xpid is specified through s2m -o option, see below
    - xxx is bg, an or pro for backgroud PREP, analysis PREP and PRO files respectively
  • Assimilation configuration file (useful for twin experiments and traceability) is stored in <your_home>/vortex/s2m/<region_id>/conf/
  • Base namelist is also stored in <your_home>/vortex/s2m/<region_id>/conf/ (name namelist.surfex.foo)
  • PF sample files are stored in <your_home>/vortex/s2m/<region_id>/workSODA/ (name PART_YYYYMMDDHH.txt.foo)
  • For post-processing and dev, PREP files and the configuration file can also be fetched to sxcen (in /cnrm/cen/users/NO_SAVE/...), which has no backup and limited space (see the options below)

New s2m options/modifications

- w option is useless
- o xpid : output repertory on hendrix
--escroc E1Tartes or E1noTartes : Escroc big ensembles with Tartes only (1944 members) or no Tartes (the rest). nmembers will be randomly drawn from this ensemble between each assimilation date
--croco your_path_to_assimilation_configuration_file : activate CrocO and give the path to your conf file.
--nforcing : numbers of different forcings to use
- f xpid_forcing: xpid for your forcing files on hendrix
--openloop : activate openloop sequence (no assimilation but output at each assimilation date)
--synth mbid : assimilation of synthetic data. Remove and replace the <mbid> member (synthetic truth)
--real : assimilation of real data
--walltime : estimate for execution time on beaufix
--writesx : fetch output data directly to NO_SAVE/ on sxcen.
--sensor your_sensor: specify the name of the sensor used to generate the observations (default :MODIS), or of your synthetical obs.