Merge branch 'develop' into develop

.github/workflows/ci.yml

@@ -17,17 +17,17 @@ jobs:
     strategy:
       matrix:
         monet: [cf, dev]
-      fail-fast: false  # just until cf versions are working
+      fail-fast: false  # always both
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v15
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: docs/environment-docs.yml
-          cache-env: true
-          extra-specs: |
+          cache-environment: true
+          create-args: >-
             pytest
 
       - if: ${{ matrix.monet == 'dev' }}
@@ -53,8 +53,7 @@ jobs:
           cd docs/examples
           for f in *.ipynb; do
             if [ "$f" == 'idealized.ipynb' ]; then
-              jupytext --to py $f -o t.py
-              python t.py
+              jupytext --to py $f -o t.py && python t.py || exit 1
             fi
           done
           cd -
@@ -96,13 +95,13 @@ jobs:
         shell: bash -l {0}
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Set up Python (micromamba)
-        uses: mamba-org/provision-with-micromamba@v15
+        uses: mamba-org/setup-micromamba@v1
         with:
           environment-file: docs/environment-docs.yml
-          cache-env: true
+          cache-environment: true
 
       - name: linkcheck
         run: sphinx-build -b linkcheck docs docs/_build/linkcheck
@@ -125,7 +124,7 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
 
       - name: Check that .py files have the license header
         run: python3 ci/check-for-license-header.py

docs/appendix/machine-specific-install.rst

@@ -1,11 +1,11 @@
 Machine-specific Install
 ========================
 
-NCAR HPC Cheyenne/Casper
+NCAR HPC Derecho/Casper
 ------------------------
 
 Below are specific recipes for getting started with MELODIES MONET
-on the NCAR HPC, Cheyenne/Casper.
+on the NCAR HPC, Derecho/Casper.
 
 **Official NCAR JupyterHub kernel**
 

docs/appendix/troubleshooting.rst

@@ -0,0 +1,18 @@
+Troubleshooting
+===============
+
+Installation problems
+---------------------
+* Conda installation fails:
+    * Often the problem is in the installation of wrf-python. Check that your computer does not have an Apple Silicon CPU (Apple Intel should be fine) and that the Python version is compatible with the wrf-python Conda package.
+
+Runtime issues
+--------------
+* Pairing (:meth:`~melodies_monet.driver.analysis.pair_models`) takes too long:
+    * ``analysis.pair_models()`` is one of the most expensive functions in MELODIES MONET, and you might be running out of memory. If you have access to more RAM, try it with that. A Time Chunking functionality is being developed to deal with this issue. Check :doc:`/users_guide/time_chunking`.
+* The plots are not produced. The error message references ``leg.legendHandles``:
+    * You are probably using Matplotlib 3.9+ with pandas 1.x. Downgrade Matplotlib to 3.8 (upgrading pandas should also work, but you might run into some incompatibilities for some specific functionalities, especially those related to downloading observational data with MONETIO. Check :doc:`/getting_started/installation`).
+* NaN does not exist, or pandas not detecting NaN values, etc:
+    * In NumPy 2.x, ``np.NaN`` has been completely replaced by ``np.nan``. ``np.NaN`` is required by some versions of MONETIO. There are two solutions: (i) downgrading NumPy to previous versions (``'numpy<2'``) or (ii) upgrading MONETIO (``'monetio>=0.2.7'``). If you go with option (ii), you might need to use the _develop_ branch of MONETIO (this should make it into _stable_ soon).
+* Failure downoloading maps:
+    * MELODIES-MONET uses Cartopy for mapping. Cartopy downloads shapefiles automatically to create the maps, and if you are offline or working on a machine with download restrictions, this might fail. Check :doc:`/appendix/machine-specific-install`. This is the situation for *NOAA HPC Hera*, and the solution discussed there should work.

docs/appendix/yaml.rst

@@ -185,7 +185,7 @@ Generalizing this to include other surface observations is under development.
 **filename:**  The file directory location and name. These observations need 
 to be preprocessed prior to incorporating them into MELODIES MONET.
 Shell variables prefixed with the ``$`` symbol, such as ``$HOME``, will be expanded.
-See :doc:`../tutorial/downloading_obs` for more details.
+See :doc:`../getting_started/downloading_obs` for more details.
 
 **obs_type:** The observation type. Options are: "pt_sfc" or point surface. Adding 
 options for Aircraft and Satellite observations are under development.
@@ -231,7 +231,7 @@ nan_values are set to NaN first and then the unit conversion is applied.
      (in %) is used to calculate the percentile (e.g., 5, 50, 95). Currently only
      used for "spatial_bias" plots. Will work with data as is and regulatory metrics.
    * **regulatory:** If false (default), use data as is. If set to true, the
-     regulatory metric is calculated as explained under :doc:`/background/supported_analyses`.
+     regulatory metric is calculated as explained under :doc:`/users_guide/supported_diagnostics`.
      Only works for "OZONE" and "PM2.5" variables.
    * **ylabel_reg_plot:** String to use as ylabel in plot for regulatory calculation.
      Useful for adding units or instrument information. Only used if regulatory = True.
@@ -276,15 +276,15 @@ Plots
 -----
 All input for each plotting group. A plotting group consists of one plotting 
 type. The plotting types are described in 
-:doc:`/background/supported_plots`. All model /
+:doc:`/users_guide/supported_plots`. All model /
 observational pairs and domains specified for the plotting group will be 
 included. You may include as many plotting groups as you like.
 
 For each plotting group, update the label and include the following information.
 Note: the labels need to be unique, but otherwise are not used.
 
 **type:** The plot type. Options are: "timeseries", "taylor", "spatial_bias",
-"spatial_overlay", "spatial_bias_exceedance", "boxplot", "multi-boxplot"
+"spatial_overlay", "spatial_bias_exceedance", "boxplot", "multi-boxplot","csi"
 Note: "spatial_bias_exceedance" plots only work when regulatory = True.
 
 **fig_kwargs:** This is optional to provide a dictionary with figure 
@@ -323,13 +323,29 @@ data will be used and the domain_name is used only in the plot title. If
 domain_type is not equal to all, MELODIES MONET will query all of the data 
 where domain_type is equal to domain_name.
 
-**region_name:** multi-box plot only. user specified variable from which the region information
-will be used to plot. if use 'epa+region' then a total of 61 regions can be choose from.
+**region_name:** list of source of regions used in title.
+(e.g., ['epa_region'])
 
-**region_list:** multi-box plot only. list of regions in the above 'region_name' variable 
-user want to plot.
+**region_list:** list of regions we will calculate for scorecard. 
+(e.g., ['R1','R2','R3','R4','R5','R6','R7','R8','R9','R10']
 
-**model_name_list:** multi-box plot only. list of observation and model names user choose to set as x-labels
+**urban_rural_name:** list of only one string input, which is variable used to
+determine wheter urban or rural site. (e.g., ['msa_name'])
+
+**urban_rural_differentiate_value:** string of value used to determine whether 
+variable is rural or urban. (e.g., '').
+
+**better_or_worse_method:** string of method used to determine which models 
+is better compared to observations. (e.g., 'RMSE', 'IOA' ,' NMB', 'NME'). choose
+one only for each time scorecard code run.
+
+**model_name_list:** 
+for multi-box plot, list of observation and model names user choose to set as x-labels; 
+for csi plot, list of model names (only) user choose to set as labels.
+
+**threshold_list:** csi plot only. list of values used as x variables. example: [10,20,30,40,50,60,70,80,90,100] 
+
+**score_name:** csi plot only. list of scores user can choose to plot. examples are "Critical Success Index' 'False Alarm Rate' 'Hit Rate'.
 
 **data:** This a list of model / observation pairs to be plotted where the 
 observation label is first and the model label is second 
@@ -381,7 +397,7 @@ Stats
 -----
 All input needed to calculate the statistics. The supported statistics available 
 in MELODIES MONET are described in 
-:doc:`/background/supported_stats`. All model /
+:doc:`/users_guide/supported_stats`. All model /
 observational pairs and domains specified will be included. You may include as 
 many statistics as you like. Note however that the calculation of the statistics 
 is relatively slow right now. Optimizing this code is under development.
@@ -391,7 +407,7 @@ use Kelvin. Wind direction has special calculations for AirNow if the observatio
 name is 'WD'. 
 
 **stat_list:** List of acronyms of statistics to calculate as defined in 
-:doc:`/background/supported_stats`. (e.g., ['MB', 'MdnB',
+:doc:`/users_guide/supported_stats`. (e.g., ['MB', 'MdnB',
 'NMB', 'NMdnB','R2', 'RMSE']). A dictionary of definitions is also included in 
 MELODIES-MONET/melodies_monet/stats/proc_stats.py. 
 

docs/cli.rst

@@ -15,6 +15,7 @@ any Python code::
 * |run|_ -- run a control file
 * |get-airnow|_ -- get AirNow data
 * |get-aeronet|_ -- get AERONET data
+* |get-aqs|_ -- get AQS data
 * |get-ish|_ -- get ISH data
 * |get-ish-lite|_ -- get ISH-Lite data
 
@@ -27,6 +28,9 @@ any Python code::
 .. |get-aeronet| replace:: ``get-aeronet``
 .. _get-aeronet: #melodies-monet-get-aeronet
 
+.. |get-aqs| replace:: ``get-aqs``
+.. _get-aqs: #melodies-monet-get-aqs
+
 .. |get-ish| replace:: ``get-ish``
 .. _get-ish: #melodies-monet-get-ish
 

docs/conf.py

@@ -40,8 +40,8 @@
 ]
 
 extlinks = {
-    'issue': ('https://github.com/noaa-csl/melodies-monet/issues/%s', 'GH'),
-    'pull': ('https://github.com/noaa-csl/melodies-monet/pull/%s', 'PR'),
+    'issue': ('https://github.com/noaa-csl/melodies-monet/issues/%s', 'GH %s'),
+    'pull': ('https://github.com/noaa-csl/melodies-monet/pull/%s', 'PR %s'),
 }
 
 autosummary_generate = True  # default in Sphinx v4
@@ -88,7 +88,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.

docs/develop/contribute.rst

@@ -11,8 +11,8 @@ the contributions and support from you.
 - Please check out 
   `GitHub Projects <https://github.com/NOAA-CSL/MELODIES-MONET/projects>`__ 
   to learn about current development plans.
-- Join an :ref:`Upcoming Workshop <develop/workshops:Upcoming>`
-  or check out :ref:`Past Workshops <develop/workshops:Past>`.
+- Join an :ref:`Upcoming Workshop <develop/other_resources:Upcoming>`
+  or check out :ref:`Past Workshops <develop/other_resources:Past>`.
 - See our developers guide, to learn 
   :ref:`develop/developers_guide:How to incorporate updates to MELODIES MONET`.
 - Email [email protected] with questions.

docs/develop/developers_guide.rst

@@ -55,10 +55,10 @@ In order to prepare for developing MELODIES MONET,
 we clone the repositories and create a conda environment that references them.
 
 .. note::
-   If you are installing MELODIES MONET on NCAR Cheyenne or NOAA Hera
+   If you are installing MELODIES MONET on NCAR Casper or NOAA Hera
    follow these machine specific instructions instead.
 
-   - :ref:`NCAR Cheyenne <appendix/machine-specific-install:NCAR HPC Cheyenne/Casper>`
+   - :ref:`NCAR Casper <appendix/machine-specific-install:NCAR HPC Derecho/Casper>`
    - :ref:`NOAA Hera <appendix/machine-specific-install:NOAA HPC Hera>`
 
 .. important::
@@ -74,7 +74,7 @@ these instructions:
 
        $ conda create --name melodies-monet python=3.9
        $ conda activate melodies-monet
-       $ conda install -y -c conda-forge pyyaml pandas=1 monet monetio netcdf4 wrf-python typer rich pooch jupyterlab
+       $ conda install -y -c conda-forge pyyaml pandas=1 'matplotlib-base<3.9' monet monetio netcdf4 wrf-python typer rich pooch jupyterlab
 
 (b) Clone [#clone]_ and link the latest development versions of MONET and MONETIO from GitHub to
     your conda environment::
@@ -140,8 +140,8 @@ changes.
     changes. 
 
 
-User's Guide Development
-------------------------
+Contributions to the Docs
+-------------------------
 
 If you add a component to MELODIES MONET, please follow the instructions below 
 to update the readthedocs user guide. 

docs/develop/workshops.rst → docs/develop/other_resources.rst

@@ -1,5 +1,5 @@
-Tutorials & Workshops
-=====================
+Other Resources
+===============
 
 Check out our upcoming and past tutorials, workshops, surveys, etc. below.
 

docs/environment-docs.yml

@@ -7,6 +7,7 @@ dependencies:
   - python=3.9
   #
   # melodies_monet deps
+  - matplotlib-base<3.9  # for pandas v1 compat
   - monet
   - monetio
   - netcdf4