GFZ/feature/snow

.github/workflows/ci_env.yml

@@ -1,6 +1,11 @@
 name: Lisflood OS Unit Tests
 
-on: [push]
+# Controls when the workflow will run
+on:
+  push:
+    branches: [ "master", "development" ]
+  pull_request:
+    branches: [ "master", "development" ]
 
 jobs:
   tests:

.gitignore

@@ -25,3 +25,5 @@ lisflood_model.egg-info
 .coverage
 *.tox.ini
 .vscode/
+*.ipynb
+.ipynb_checkpoints/

docs/3_step3_preparing-setting-file/index.md

@@ -680,6 +680,7 @@ Within the 'lfoptions' element of the settings file, each option is defined usin
 + **NetCDFTimeChunks**: chunking size in the time dimension. Recommended value is “auto" but chunking size can be specified manually or set to “-1" to load the whole time series into memory (very fast but expensive in terms of memory). 
 + **MapsCaching** (True or False): option designed for the lisflood calibration. If set to True, all the static maps and forcings will be stored in a cache so that they don't have to be loaded by each lisflood instance. This option sets the value of NetCDFTimeChunks to "-1", meaning that the whole time series in the NetCDF inputs is loaded into memory. 
 + **OutputMapsChunks**: this option is used to dump outputs to disk every X steps (default 1).  
++ **OutputMapsDataType**: this option sets the output data type and may take the following values: "float64" or "float32" (default float64)
 
 ### Reference settings file
 In order to facilitate the preparation of the settings file, a complete example is provided [here](https://github.com/ec-jrc/lisflood-code/tree/master/src/lisfloodSettings_reference.xml). The user is encouraged to update the paths, the names of the maps and of the tables in the provided template.

docs/3_step4_preparing-input-files/index.md

@@ -6,14 +6,14 @@ In the current version of LISFLOOD, all the model inputs are provided as either
 
 LISFLOOD requires that all maps must have *identical* location attributes (number of rows, columns, cellsize, upper x and y coordinates).
 
-The input maps can be classified according to two main cathegories:<br>
+The input maps can be classified according to two main categories:<br>
 + **meteorological forcings**. These maps provide time series of values for each pixel of the computational domain. More specifically, the meteorological forcings provide the values of precipitation, temperature, reference values of evaporation from water surfaces, reference values of evaporation from open water bodies, reference values of evapotranspiration for each pixel of the modelled area. For each meteorological forcing, one map is required for each computational time step. <br>
 + **static maps**. These maps provide information of morphological, physical, soil, and land use properties for each pixel of the computational domain. 
 
 
 ### Meteorological forcings
 
-The meteorological forcing variables are defined in *map stacks*. A *map stack* is simply a series of maps, where each map represents the value of a variable at an individual time step.<br>It is recommented to use the netcdf format. <br> The users that prefer to prepare the meteorological forcings maps in pcraster format, must name the files according to the following rules: the name of each map is made up of a total of 11 characters: 8 characters, a dot and a 3-character suffix. Each map name starts with a *prefix*, and ends with the time step number. All character positions in between are filled with zeros ("0"). <br>
+The meteorological forcing variables are defined in *map stacks*. A *map stack* is simply a series of maps, where each map represents the value of a variable at an individual time step.<br>It is recommented to use the netcdf format. <br> LISFLOOD is capable of reading meteorological forcings split into multiple files (e.g. yearly chuncks). To use this functionality, it is enough to add the symbol '\*' after the file name (e.g. ET0_\*) <br> The users that prefer to prepare the meteorological forcings maps in pcraster format, must name the files according to the following rules: the name of each map is made up of a total of 11 characters: 8 characters, a dot and a 3-character suffix. Each map name starts with a *prefix*, and ends with the time step number. All character positions in between are filled with zeros ("0"). <br>
 
 Generally used prefixes for the meteorological forcings maps are: <br>
 + tp : total precipitation; units: mm/day.<br>
@@ -77,14 +77,14 @@ Calibrated parameters are optimised for a specific model set up. It is often req
 
 ### INPUT TABLES
 
-The geographical location of lakes and reservoirs is identified by the two maps described [here](../4_Static-Maps_reservoirs-lakes/). These maps provide the location of lakes and reservoirs. Each lake and each reservoir is identified by its ID (a in integer number). LISFLOOD requires additional information for the adequate modelling of [lakes](https://ec-jrc.github.io/lisflood-model/3_02_optLISFLOOD_lakes/) and [reservoirs](https://ec-jrc.github.io/lisflood-model/3_03_optLISFLOOD_reservoirs/). These additional pieces of information are supplied to the numerical code by using tables in *.txt* format. Each table has 2 colums: the first column is the ID of the lake or of the reservoir, the second column is the quantity required by LISFLOOD. The table below provides the list of the pieces of information which are required for the adequate modelling of lakes and reservoirs.
+The geographical location of lakes and reservoirs is identified by the two maps described [here](../4_Static-Maps_reservoirs-lakes/). These maps provide the location of lakes and reservoirs. Each lake and each reservoir is identified by its ID (an integer number). LISFLOOD requires additional information for the adequate modelling of [lakes](https://ec-jrc.github.io/lisflood-model/3_02_optLISFLOOD_lakes/) and [reservoirs](https://ec-jrc.github.io/lisflood-model/3_03_optLISFLOOD_reservoirs/). These additional pieces of information are supplied to the numerical code by using tables in *.txt* format. Each table has 2 columns: the first column is the ID of the lake or of the reservoir, the second column is the quantity required by LISFLOOD. The table below provides the list of the pieces of information which are required for the adequate modelling of lakes and reservoirs.
 ##### Table: LISFLOOD input tables
 
 | **Table**             | **Default name**      | **Description**       |
 |----------------------------|-----------------------|--------------------------|
-| Lake area   | Lakearea.txt          | Lake syrface area in m2 |
-| Lake alpha parameter   | lakea.txt          | Lake parameter alpha: a detailed descrpition can be found [here](https://ec-jrc.github.io/lisflood-model/3_02_optLISFLOOD_lakes/)  |
-| Lake average inflow   | lakeaverageinflow.txt          | Average inflow to the lake: a detailed descrpition can be found [here](https://ec-jrc.github.io/lisflood-model/3_02_optLISFLOOD_lakes/) |
+| Lake area   | Lakearea.txt          | Lake surface area in m2 |
+| Lake alpha parameter   | lakea.txt          | Lake parameter alpha: a detailed description can be found [here](https://ec-jrc.github.io/lisflood-model/3_02_optLISFLOOD_lakes/)  |
+| Lake average inflow   | lakeaverageinflow.txt          | Average inflow to the lake: a detailed description can be found [here](https://ec-jrc.github.io/lisflood-model/3_02_optLISFLOOD_lakes/) |
 | Reservoir storage   | rstor.txt          | Volume in m3, total reservoirs storage capacity |
 | Reservoir minimum outflow   | rminq.txt          | Discharge in m3/s.   |
 | Reservoir normal outflow   | rnormq.txt          | Discharge in m3/s.  |

docs/4_annex_input-files/index.md

@@ -116,19 +116,5 @@ To this end a dedicated pre-processing application has been developed (LISVAP),
 
 
 
-## Tables
-
-In the previous version of LISFLOOD a number of model parameters are read through tables that are linked to the classes on the land use and soil (texture) maps. 
-Those tables are replaced by maps (e.g. soil hydraulic property maps) in order to include the sub-grid variability of each parameter. 
-
-Therefore only one default table is used in the standard LISFLOOD setting. The following table gives an overview:
-
-***Table:*** *LISFLOOD input tables.*                      
-
-| Table                  | Default name | Description                              |
-| ---------------------- | ------------ | ---------------------------------------- |
-| **LAND USE**           |              |                                          |
-| Day of the year -> LAI | LaiOfDay.txt | Lookup table: Day of the year -> LAI map |
-
 [🔝](#top)
 

docs/4_annex_settings_and_options/index.md

@@ -219,7 +219,6 @@ param| lfuser          | IRRIGATION AND WATER ABSTRACTION
 | lfuser          | IRRIGATION AND WATER ABSTRACTION                             | WATER ABSTRACTION                  |        | LeakageReductionFraction        | 0                                                 | 0                                                 | value           | input                  | Leakage reduction fraction   (e.g. 50% = 0.5 as compared to current Leakage) (baseline=0, maximum=1) |
 | lfuser          | IRRIGATION AND WATER ABSTRACTION                             | WATER ABSTRACTION                  |        | WaterSavingFraction             | 0                                                 | 0                                                 | value           | input                  | Water savings fraction (e.g. 10% = 0.1 as   compared to current Use (baseline=0, maximum=1)        scenwsav.map |
 | lfuser          | IRRIGATION AND WATER ABSTRACTION                             | WATER ABSTRACTION                  |        | WaterReUseFraction              | 0                                                 | 0                                                 | value           | input                  | Fraction of water re-used in industry   (e.g. 50% = 0.5 = half of the water is re-used, used twice (baseline=0,   maximum=1         scenruse.map |
-| lfuser          | INPUT WATER USE MAPS AND PAR                                 | LAI                                |        | FirstDayOfMonth                 | $(PathTables)/firstdayofmonth.txt                 | $(PathTables)/firstdayofmonth.txt                 | table           | input                  | NO LONGER USED table with number of first day for each month |
 | lfuser          | INPUT WATER USE MAPS AND PAR                                 | CALC INDICATOR                     |        | Population                      | $(PathMapsLanduse)/pop                            | $(PathMapsLanduse)/pop                            | map             | input                  | Population per pixel                                         |
 | lfuser          | INPUT WATER USE MAPS AND PAR                                 | CALC INDICATOR                     |        | PopulationMaps                  | $(PathWaterUse)/pop                               | $(PathWaterUse)/pop                               | map             | input                  | Population map for   TransientLandUseChange                  |
 | lfuser          | INPUT WATER USE MAPS AND PAR                                 | CALC INDICATOR                     |        | LandUseMask                     | $(PathMaps)/lusemask.map                          | $(PathMaps)/lusemask.map                          | map             | input                  | Land use mask      map to mask out deserts and high mountains (to cover ETdif map, otherwise   Sahara etc would pop out; meant as a drought indicator |
@@ -345,8 +344,6 @@ param| lfuser          | IRRIGATION AND WATER ABSTRACTION
 | lfbinding       | INPUT METEO AND VEG MAPS                                     | LAI                                |        | LAIOtherMaps                    | $(PathLAI)/$(PrefixLAIOther)                      | $(PathLAI)/$(PrefixLAIOther)                      | 0               | input                  | leaf area index [m2/m2]                                      |
 | lfbinding       | INPUT METEO AND VEG MAPS                                     | LAI                                |        | LAIForestMaps                   | $(PathLAI)/$(PrefixLAIForest)                     | $(PathLAI)/$(PrefixLAIForest)                     | 0               | input                  | leaf area index forest [m2/m2]                               |
 | lfbinding       | INPUT METEO AND VEG MAPS                                     | LAI                                |        | LAIIrrigationMaps               | $(PathLAI)/$(PrefixLAIIrrigation)                 | $(PathLAI)/$(PrefixLAIIrrigation)                 | 0               | input                  | leaf area index irrigation   [m2/m2]                         |
-| lfbinding       | INPUT METEO AND VEG MAPS                                     | LAI                                |        | LAIOfDay                        | $(PathTables)/laiofday.txt                        | $(PathTables)/laiofday.txt                        | table           | input                  | table with days for each LAI   maps      1st column: range of days; 2nd column: suffix of LAI map |
-| lfbinding       | INPUT WATER USE MAPS AND PAR                                 | WATER ABSTRACTION                  |        | FirstDayOfMonth                 | $(PathTables)/firstdayofmonth.txt                 | $(PathTables)/firstdayofmonth.txt                 | table           | input                  | NO LONGER USED table with   number of first day for each month |
 | lfbinding       | INPUT WATER USE MAPS AND PAR                                 | WATER ABSTRACTION                  |        | DomesticDemandMaps              | $(PathWaterUse)/$(PrefixWaterUseDomestic)         | $(PathWaterUse)/$(PrefixWaterUseDomestic)         | map             | input                  | Domestic water abstraction   daily maps [mm]                 |
 | lfbinding       | INPUT WATER USE MAPS AND PAR                                 | WATER ABSTRACTION                  |        | LivestockDemandMaps             | $(PathWaterUse)/$(PrefixWaterUseLivestock)        | $(PathWaterUse)/$(PrefixWaterUseLivestock)        | map             | input                  | Livestock water abstraction   daily maps [mm]                |
 | lfbinding       | INPUT WATER USE MAPS AND PAR                                 | WATER ABSTRACTION                  |        | EnergyDemandMaps                | $(PathWaterUse)/$(PrefixWaterUseEnergy)           | $(PathWaterUse)/$(PrefixWaterUseEnergy)           | map             | input                  | Energy water abstraction daily   maps [mm]                   |

src/lisflood/Lisflood_initial.py

@@ -47,7 +47,7 @@
 from .hydrological_modules.routing import routing
 from .hydrological_modules.groundwater import groundwater
 from .hydrological_modules.surface_routing import surface_routing
-from .hydrological_modules.reservoir import reservoir
+from .hydrological_modules.reservoir import Reservoir
 from .hydrological_modules.lakes import lakes
 from .hydrological_modules.polder import polder
 from .hydrological_modules.waterabstraction import waterabstraction
@@ -136,7 +136,7 @@ def __init__(self):
         self.routing_module = routing(self)
         self.groundwater_module = groundwater(self)
         self.surface_routing_module = surface_routing(self)
-        self.reservoir_module = reservoir(self)
+        self.reservoir_module = Reservoir(self) # get_reservoir(option['reservoirHanazaki'])
         self.lakes_module = lakes(self)
         self.polder_module = polder(self)
         self.waterabstraction_module = waterabstraction(self)

src/lisflood/global_modules/add1.py

@@ -271,6 +271,10 @@ def compressArray(map, pcr=True, name=None):
         mapnp = pcr2numpy(map,np.nan)
         mapnp1 = np.ma.masked_array(mapnp, maskinfo.info.mask)
     else:
+        if map.mask is not np.bool_(0):
+            if (map.mask[maskinfo.info.mask==False].any()==True):
+                # warning: fill values masking is different from the area mask map, and some values in the area mask map contains invalid fill values
+                warnings.warn(LisfloodWarning("Warning in compress array: map '{}' has fill values inside the area mask map!".format(name)))
         mapnp1 = np.ma.masked_array(map, maskinfo.info.mask)
     mapC = np.ma.compressed(mapnp1)
 
@@ -489,6 +493,10 @@ def loadmap_base(name, pcr=False, lddflag=False, timestampflag='exact', averagey
         # masking
         try:
             maskinfo = MaskInfo.instance()
+            if mapnp.mask is not np.bool_(0):
+                if (mapnp.mask[maskinfo.info.mask==False].any()==True):
+                    ## warning: fill values masking is different from the area mask map, and some values in the area mask map contains invalid fill values
+                    warnings.warn(LisfloodWarning("Warning: map {} (binding: '{}') has fill values inside the area mask map!".format(filename, name)))
             mapnp.mask = maskinfo.info.mask
         except (KeyError, AttributeError):
             pass

src/lisflood/global_modules/default_options.py

@@ -1424,9 +1424,9 @@
                        'TotalRunoffTS': TimeSeries(name='TotalRunoffTS', output_var='TotalRunoff', where='Sites',
                                                    repoption=['repRateSites'], restrictoption=['nonInit'],
                                                    operation=['']),
-                    #    'UZAvUpsTS': TimeSeries(name='UZAvUpsTS', output_var='UZtotal', where='Gauges',
-                    #                            repoption=['repStateUpsGauges'], restrictoption=['nonInit'],
-                    #                            operation=['total']),
+                       'UZAvUpsTS': TimeSeries(name='UZAvUpsTS', output_var='UZtotal', where='Gauges',
+                                               repoption=['repStateUpsGauges'], restrictoption=['nonInit'],
+                                               operation=['total']),
                        'UZOutflowAvUpsTS': TimeSeries(name='UZOutflowAvUpsTS', output_var='UZOutflowPixel',
                                                       where='Gauges', repoption=['repRateUpsGauges'],
                                                       restrictoption=['nonInit'], operation=['total']),
@@ -1456,7 +1456,7 @@
                        #                         restrictoption=['nonInit'], operation=['']),                                                                   
                        'actETPUpsTS': TimeSeries(name='actETPUpsTS',
                                                  output_var='ESActPixel+self.var.TaPixel+self.var.TaInterceptionAll+self.var.EvaAddM3*self.var.M3toMM',
-                                                 where='Gauges', repoption=['repRateUpsGauges'], restrictoption=['nonInit'],
+                                                 where='Gauges', repoption=['repRateUpsGauges'], restrictoption=['nonInit','openwaterevapo'],
                                                  operation=['total']),
                        'Theta1AvUpsTS': TimeSeries(name='Theta1AvUpsTS',
                                                  output_var='Theta1aPixel',

src/lisflood/global_modules/netcdf.py

@@ -216,10 +216,10 @@ def __init__(self, data_path, time_chunk, dates, indexer=None, climatology=False
         # read maps using always a standard x and y reference system using x in ascending and y in descending order
         if (y_flipped):   # y in in ascending order
             warnings.warn(LisfloodWarning("Warning: map {} (var_name: '{}') has y coordinates in ascending order and will be flipped vertically".format(data_path, var_name)))
-            func_y = lambda y : np.flipud(y).copy()
+            func_y = lambda y : np.flip(y, axis=1).copy()   # y axis is the number 1, as zero is the time axis
         if (x_flipped):   # x in in descending order
             warnings.warn(LisfloodWarning("Warning: map {} (var_name: '{}') has x coordinates in descending order and will be flipped horizontally".format(data_path, var_name)))
-            func_x = lambda x : np.fliplr(x).copy()
+            func_x = lambda x : np.flip(x, axis=2).copy()  # y axis is the number 2, as zero is the time axis
 
         # compress dataset (remove missing values and flatten the array)
         maskinfo = MaskInfo.instance()