PDAF manual

josephzhang8 · josephzhang8 · commit 8fe76cebf5d3 · 2026-03-10T11:10:58.000-04:00
diff --git a/docs/modules/pdaf.md b/docs/modules/pdaf.md
@@ -1 +1,212 @@
-We are working on a user manual; meanwhile this [paper](https://ccrm.vims.edu/yinglong/SVN_large_files/Yu_etal_OM_2025-SCHISM-Data-Assimilation.pdf) serves as interim manual. 
+#Theory
+Refer to this [paper](https://ccrm.vims.edu/yinglong/SVN_large_files/Yu_etal_OM_2025-SCHISM-Data-Assimilation.pdf).
+
+#Compilation
+
+Pre-requirements: ESMF, PDAF, SCHISM as libraries; the following sections show how to build for each. 
+
+## Build ESMF 
+
+ESMF version must be newer than version 8.1, users can download from https://earthsystemmodeling.org/download/ 
+
+Specify macros depending on your platform resources, the following use intel with intelmpi as example, more options can be found from https://earthsystemmodeling.org/docs/release/latest/ESMF_usrdoc/node10.html#SECTION000106000000000000000 : 
+
+```
+export ESMF_DIR= where you untar ESMF 
+export ESMF_COMPILER=intel 
+export ESMF_OPENMP=OFF 
+export ESMF_PTHREADS=OFF 
+export ESMF_COMM=intelmpi 
+export ESMF_NETCDF=split 
+export ESMF_NETCDF_INCLUDE="$NetCDF_FORTRAN_DIR/include -I$NetCDF_FORTRAN_DIR/include $NetCDF_C_DIR/include -I$NetCDF_C_DIR/include" 
+export ESMF_NETCDF_LIBPATH="$NetCDF_FORTRAN_DIR/lib -L$NetCDF_FORTRAN_DIR/lib $NetCDF_C_DIR/lib -L$NetCDF_C_DIR/lib" 
+export ESMF_NETCDF_LIBS="-lnetcdff -lnetcdf" 
+export ESMF_PIO=OFF 
+export ESMF_ABI=x86_64_medium  #(optional) 
+```
+ 
+Then do the following: 
+
+```
+cd $ESMF_DIR 
+make info >& tmp #(inspect config info) 
+make clobber 
+make 
+```
+ 
+
+This may take some time depending on your system. If successful, the library & esmf.mk will be located in `$ESMF_DIR/lib/libO/Linux`.intel.x86_64_medium.intelmpi.default/;  the actual path depends on compile option. 
+
+If this is a fresh install, and you intend to build schism_pdaf after, remember to unset the macro to avoid conflict when building schism_pdaf, like the following: 
+
+```
+unset ESMF_DIR 
+unset ESMF_COMPILER 
+unset ESMF_OPENMP 
+unset ESMF_PTHREADS 
+unset ESMF_COMM 
+unset ESMF_NETCDF 
+unset ESMF_PIO 
+unset ESMF_ABI 
+```
+ 
+
+##Build PDAF 
+
+PDAF version 3.0 is not compatible with previous version (2.3.1 and older), so if you have previous version, please update it to the latest one. 
+
+Get PDAF 3.0 from github 
+
+git clone https://github.com/PDAF/PDAF.git 
+
+Choose build option from folder make.arch or make one and put into it. 
+
+Common choices are linux_ifort_impi.h, linux_ifort_mvapich2.h, linux_mpiifort_impi.h.  
+
+Change compiling option if necessary. PDAF requires BLAS and LAPACK library. If your platform provides compiler optimized math library like Intel-MKL, OpenBLAS, AOCL (AMD), these can also be used and you need to config corresponding options in the *.h file. 
+
+Here we use linux_ifort_impi.h as example, define marco as following then build the lib 
+
+export ARCH=linux_ifort_impi 
+
+export PDAF_ARCH=$ARCH 
+
+make 
+
+If successful, libraries & module files are located in lib and include respectively. 
+
+ 
+
+1.3 Build SCHISM 
+
+Follow https://schism-dev.github.io/schism/stable/getting-started/getting-sourcecode.html and https://schism-dev.github.io/schism/stable/getting-started/compilation.html#cmake to build schism libs. 
+
+Note: schism_pdaf only support scribe-IO under fully parallel mode (all ensembles run concurrently), and OLDIO (schout output nc files) under flexible mode (ensembles members run in batches, for under-resourced platform).  
+
+If users wish to use scribe-IO, build schism with OLDIO=OFF. If OLDIO is preferred, set OLDIO=ON.  
+
+schism_pdaf also supports static domain decomposition without ParMetis (requires partition.prop as input). If this feature is used, set NO_PARMETIS=ON. 
+
+ 
+
+1.4 Build schism_pdaf 
+
+Get schism_pdaf code with the following: 
+
+git clone https://github.com/schism-dev/schism-esmf.git   
+
+Define macros for the following: 
+
+export ESMFMKFILE= where ESMF lib is located 
+
+export SCHISM_BUILD_DIR= where schism build is located 
+
+export PDAF_LIB_DIR=where PDAF lib is located 
+
+export SCHISM_NO_PARMETIS=off  (If you wish to use static decomposition, then set it on) 
+
+If any library like Intel-MKL, OpenBLAS, AOCL (AMD)… is used when compiling PDAF, remember to check LDFLAGS within corresponding compiler section in the beginning of Makefile, make sure linking path & compile options are correct. 
+
+Then we can build schism_pdaf with the following: 
+
+make distclean 
+
+make pdaf  
+
+If successful, an executable schism_pdaf will be there. 
+
+ 
+
+2. Prepare the inputs  
+
+The input files include: 
+
+i. standard schism inputs, these files depend on user case setups 
+
+ii. schism_pdaf control files 
+
+schism_pdaf.cfg: control ensemble size, # of concurrent members, scribe cores (if used), ics_set (same as ics in param.nml: coordinate system) 
+
+ 
+
+     global.nml: control parameters for ESMF & PDAF coupling info, set starting date and run hours and DA steps. Note that the time origin info, duration and SCHISM time step need to be consistent with param.nml (under any ihot setting). Specify the DA interval with ‘num_schism_dt_in_couple’ (# of SCHISM steps when calling PDAF), and this value should always be identical as delt_obs in pdaf.nml. 
+
+ 
+
+pdaf.nml: control pdaf parameters such as filter type, localization range…etc. 
+
+ 
+
+ 
+
+iii. schism_pdaf ensemble inputs  
+
+We follow suggestions from PDAF and use Pham’s method (2001) to generate ensemble information. More details can be found from https://pdaf.awi.de/trac/wiki/EnsembleGeneration . 
+
+In summary, users can do a freerun (no DA) and extract its snapshots for EOFs analysis. Snapshots can be extracted with ncl scripts (extract2bin.ncl or extract2bin_scrio.ncl), and use generate_covar.F90 to generate required inputs as follows: 
+
+eofs.dat: number of EOFs 
+
+eofs.bin: EOFs matrix 
+
+svd.bin: svd values from analysis 
+
+meanstate.bin (optional): usually only used for twin experiments.  
+
+Note: generate_covar.F90 is serial program. If any large case needs to be analyzed, large memory is required for analysis and long execution time should be expected. These utility scripts are located at src/PDAF_bindings/utility/gen_cov/  
+
+ 
+
+iv. DA_data: observation inputs, file naming is data_????????.dat (?s with i8.8 format with padding 0s, representing the SCHISM time steps). If the file does not exist, the code still runs with no DA done at that DA step. 
+
+Currently, it supports 6 types of observation including: elevation(z), sea-level anomaly(a), temperature (t), salinity(s), u-velocity(u), and v-velocity(v). 
+
+These are ascii files and their format are as follows:  
+
+ 
+
+1st row is total number of observations, then followed by 
+
+obs-type, X, Y, Z, obs-values,  
+
+If rms_type=3, one extra column is needed for observation errors. Users can also specify uniform observation error with rms_type=1,2 in pdaf.nml. 
+
+ 
+
+Fig: Working folder structures 
+
+3. How to run it 
+
+When inputs are ready, users need to create each ensemble folder with naming schism_??? (i3.3 format), and link schism input file inside. makedir.pl (under src/PDAF_bindings/utility) can be used to create each member outputs dir and link input files in each dir.  
+
+Users also need to create folder DA_output with two subfolder a and f for storing DA outputs after/before the DA step, respectively. 
+
+    DA_output/a : Analysis (After DA) 
+
+    DA_output/f : Forecast (Before DA) 
+
+To launch the run, users may need to add esmf lib path into LD_LIBRARY_PATH in the job script or environment, like the following: 
+
+export LD_LIBRARY_PATH=$LD_LIBRARY_PATH: /where_esmf_installed/lib/libO/Linux.intel.64.openmpi.default/  
+
+ 
+
+  The following configs are mandatory! 
+
+  For scribe-IO: set nc_out=1 in param.nml 
+
+  For OLDIO: set nc_out=0 in param.nml 
+
+ 
+
+4.Check the outputs 
+
+During the run, users can check job screen output which will show PDAF config information and DA status, as well as timing information. 
+
+If users choose scribe-IO, scribe output will be located in each schism_???/outputs. 
+
+DA_output/ will store ensemble-mean files in OLDIO format, and users need to combine them, and link local_to_global* from schism_001/outputs/ to perform the combine task. 
+
+DA effects in schism_???/output/*.nc (either scribe or schout format) only can be seen after 2nd DA (if DA frequency = output frequency), since model state update always happens at the beginning of next DA cycle. If users need to check instantaneous DA result, it will be only under DA_output/a 
+
+If DA frequency > output frequency, DA effects will be seen after 1st DA + 1 output step.    
