Basin Analysis Report Batch Tool - Alaska basins

[jdduh]

There is no SNODAS data for Alaska. We still don't have a timeline when these reports will be generated. Here are some notes for further discussion.

1. There are different Data Sources (i.e., different webservice URLs) for PRISM (1981-2010) and DEM in Alaska. We need a design (or just some instructions) to handle the report parameter/settings file so that we can easily switch between the batch generation of the conterminous US and Alaska reports.
2. The tool needs to check for the availability of SNODAS data and inserts blanks pages for basins that don't have SNODAS data.
3. There is a total of 16 AOIs in Alaska that will have Basin Analysis reports.

[lbross]

I recommend adding a flag to the maps\analysis.xml file that indicates if a basin is in Alaska. Ideally we would make changes in BAGIS V3 to automatically or deliberately set this flag when the basin is created, but that is extra work for a tool that isn't used much now and for only 16 basins. Instead, we can add the flag and set it to 'No' (if it doesn't exist) when the basin is opened in BAGIS-PRO. Then it would be a manual process for the analyst to close Pro, update the analysis.xml to 'Y', and reopen the basin.

We can add the different data sources for PRISM and DEM to the central configuration file on basins. BAGIS-PRO will need to be updated to use the appropriate data sources if the Alaska flag is set to 'Y' for the basin.

The current tool does a good job of skipping the maps if the data layers aren't in the basin, but I will need to verify this. This will impact the Snodas Mean and Snodas Mean Delta SWE maps. I will need to ensure the clipping and Snodas calculation tools fail gracefully if the extent of the Snodas data source does not include the basin boundary.

Edit: Even if we don't have a timeline for the Alaska reports, I can work on BAGIS-PRO being able to gracefully handle the lack of Snodas data if I have a sample AOI from that geography to work with. I don't know how difficult an ask this is.

[lbross]

We will need an NLCD layer for Alaska. The current one doesn't cover that area.

[jdduh]

Here is the NLCD image service. Please double check the URL so that it works with the scripting convention.

http://bagis.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin/NLCD_2016_land_cover_AK/ImageServer

[jdduh]

Here is the 1981-2010 PRISM precipitation service for Alaska

http://bagis.geog.pdx.edu/arcgis/rest/services/BAGIS_PRISM_1981_2010_AK/Ak_Prism_1981_2010_Precipitation_01/ImageServer

[jdduh]

New data source entries for Alaska. Please update the settings file on the server. See #21 for relevant information.

{
"units": "Inches",
"description": "Alaska 30-year (1981-2010) monthly/quarterly/annual average precipitation data created by the PRISM Climate Group. See https://prism.oregonstate.edu/ for more information.",
"shortDescription": "PRISM 1981-2010 annual precipitation averages",
"shortDescriptionMonthly": "PRISM 1981-2010 monthly precipitation averages",
"uri": "http://bagis.geog.pdx.edu/arcgis/services/BAGIS_PRISM_1981_2010_AK/",
"layerType": "Precipitation",
"heading": "Precipitation"
},

{
"description": "Alaska 2016 NLCD Land Cover. See https://www.mrlc.gov for more information.",
"shortDescription": "2016 National Land Cover Database (NLCD)",
"uri": "http://bagis.geog.pdx.edu/arcgis/rest/services/BAGIS_Databin/NLCD_2016_land_cover_AK/ImageServer",
"layerType": "Land Cover",
"heading": "Land Cover – for Potential Site Location Analysis"
},

[lbross]

@jdduh What do you think about maintaining the Alaska indicator in the Master AOI List rather than in the client-side analysis.xml file? We already query the Master AOI List when an AOI is loaded in BAGIS-Pro so getting the Alaska indicator from there would be straightforward. It also means that no action would need to be taken by the analyst to identify an AOI as Alaskan. And it doesn't have to be an 'Alaska' indicator. We could come up with a more general purpose/name as discussed previously.

[jdduh]

Sounds good. Does the code have to be binary (i.e., 0/1 or yes/no)? We could use HUC2 (https://water.usgs.gov/GIS/regions.html) values as an indicator if the code is none-binary.

[jdduh]

Here is the HUC2 information for NWCC active forecasts. https://www.wcc.nrcs.usda.gov/ftpref/gis/basin_defs/site_lists/HUC2_FCST.json

[lbross]

That could also work and it doesn't have to be any particular value. Where would I find the HUC2? Will you add it to the Master List or do I have to query the .json service that you provided?

[jdduh]

huc2 is added to the Master AOI List Feature Service. There are two huc2 columns, huc2 and huc2code. huc2 is a text field and huc2code is an integer field. The only difference is for 6 stations bordering US and Canada with the huc2 value of '09' or huc2code of 9. The other stations all have a two digits huc2 code with a non-zero leading number.
https://services.arcgis.com/SXbDpmb7xQkk44JV/arcgis/rest/services/master_aoi_list_fs/FeatureServer

[lbross]

Do you care which huc2 field I use? Either one would work fine. I noticed that huc2code and huc2region are null for inactive AOIs.

[jdduh]

How about using huc2? It aligns better with the "2" (2-digit) nomenclature used in USGS publications. huc2 values were derived from huc that was in the master AOI list feature service. The huc2code and huc2region were from the AWDB .json file. That explains why inactive AOIs have null values for huc2code and huc2region.

[lbross]

Would it be possible to rename the Alaska Prism webservices to match the North America ones? I store the list of service layers in an array of Constants (Precipitation_01, Precipitation_02, etc.) It would be easiest to only have to have the parent folder name vary for Alaska. If we need to indicate at the lowest level that it is an AK service, can we at least not include the years? If the years are part of the name, we will need to update the code when the years change. Otherwise, we just update the base URI on the data sources server document.

[jdduh]

The current lower 48 PRISM base name is Prism_Precipitation_XX. I can change the AK ones to use the same base name. I thought there is a function to remove the prefix (i.e., Prism) in place. Please confirm that the name could include "Prism_".

[lbross]

Yes. Prism_Precipitation_XX would be perfect to use the same base name. The output clipped layer names are a separate array and do not have the Prism_Precipitation prefix.

[lbross]

Unfortunately I can't get clipping against the NLCD_2016_land_cover_AK to work. I wonder if it's because the clip envelope extends beyond the boundary of the web service? Do you want to try it? I am testing with 15056210:AK:USGS. Here is the input rectangle: -2395645.74407369 4502666.68000478 -2367558.52180903 4535803.04206261. This is what the clip polygon looks like against the web service layer:

[jdduh]

The renaming of AK Prism services is done. http://webservices.geog.pdx.edu/arcgis/rest/services/BAGIS_PRISM_1981_2010_AK

[jdduh]

The AK land cover image covers a much bigger area. The area outside the buffered AOI contains zeros. I had difficulty clipping the AK land cover. After testing the CLIP RASTER tool on the lower48 land cover with a different AOI, suddenly the clipping on AK stated to work without any problem (even after restarting ArcGIS Pro).

The image below shows the clipped land cover. Blue indicates NODATA, pink for zeros.

Here is the clip message.

Clip Raster

Parameters

Input Raster BAGIS_Databin\NLCD_2016_land_cover_AK
Rectangle -2395645.74407369 4502666.68000478 -2367558.52180903 4535803.04206261
Output Raster Dataset C:\Users\jduh\AppData\Local\Temp\ArcGISProTemp6632\e1d83f9c-439b-428f-99f4-2dcc294e4695\Default.gdb\test
Output Extent p_aoi_v
NoData Value 256
Use Input Features for Clipping Geometry ClippingGeometry
Maintain Clipping Extent NO_MAINTAIN_EXTENT

Messages

Start Time: Tuesday, February 14, 2023 9:15:17 PM
Building Pyramids...
Succeeded at Tuesday, February 14, 2023 9:15:20 PM (Elapsed Time: 3.82 seconds)

[lbross]

I still can't get the clipping for the Alaska NLCD layer to work. When I clip manually from Pro, it seems like it works but then it hangs indefinitely when it is saving the raster. It never moves beyond 0%. When I try to clip using the add-in, I get this error message (both on my laptop and CS3):

Start Time: Wednesday, February 15, 2023 10:46:59 AM
ERROR 160326: The table already exists.
Failed to execute (Clip).
Failed at Wednesday, February 15, 2023 10:47:00 AM (Elapsed Time: 1.03 seconds)

If you google the error message online, it says "While this error can occur, it occurs so rarely that the typical causes have not been identified so no solution is available at this time."

I wonder if it is because it is multiple band? The PRISM layer clips fine.

[jdduh]

I tried the clip raster tool on another computer and the tool worked in the first attempt without any problem. The Lower 48 land cover imageservice is also from a 3 bands image. We probably can rule out the multiple bands concern.

[jdduh]

Just checked the lower 48 land cover service. It's not a multiple band. I will see if I can compress the bands for the AK land cover.

[lbross]

I'm not sure if you've made any changes for the AK land cover, but I finally got it working through the tool. It appears that the layers.gdb was corrupted on both my laptop and CS3. I kept deleting the nlcd_land_cover layer but ArcMap thought it was still there. I finally created a new layers.gdb and moved the files over that I wanted. Now I'm able to clip the land cover for Alaska.

Edit: I can clip the Alaska NLCD data now but it doesn't display correctly. The lower 48 NLCD has an attribute table with the land cover classes specified, but the Alaska data is displaying as a 3 band layer without any attribute table. See image below from Pro:

[jdduh]

I will need to reprocess the AK NLCD.
[update] - The AK NLCD imageservice is republished and the 3-band issue is fixed. The source data has a colormap. I need to tell the imageservice publication tool not to convert the colormap into 3 bands imageservice.

[jdduh]

Ran the first AK report. The land cover map doesn't show up correctly.
Here is the source land cover data in the AOI.

Here is the map in the report.

The site analysis map looks correct.

[jdduh]

Just for the record. AK's huc2 (huc2code) is 19.

[lbross]

The problem is the field name that contains the land cover value. It is NLCD_Land on the lower 48 webservice and NLCD_Land_Cover_Class on the Alaska webservice. The easiest fix would be to change the Alaska webservice to match the lower 48. Alternatively I could change the layer template to key off of the value instead of the NLCD_Land field, but that will take longer and mean distributing a new .lyrx file.

And I double-checked that BAGIS-PRO is using the correct huc2 code.

[jdduh]

Thanks for troubleshooting this. I will update the field name of the AK NLCD imageservice.

[jdduh]

We do need a new layer file to reflect the updates on the NLCD data.

• Additional land cover types in AK land cover: 51, 72, 74
• Updated land cover type descriptions for 12, 71, 81

The updates are in bold.

Value,Red,Green,Blue,Opacity,NLCD_Land
0,0,0,0,0,Unclassified
11,70,107,159,255,Open Water
12,209,222,248,255,Perennial Ice/Snow
21,222,197,197,255,"Developed, Open Space"
22,217,146,130,255,"Developed, Low Intensity"
23,235,0,0,255,"Developed, Medium Intensity"
24,171,0,0,255,"Developed, High Intensity"
31,179,172,159,255,Barren Land
41,104,171,95,255,Deciduous Forest
42,28,95,44,255,Evergreen Forest
43,181,197,143,255,Mixed Forest
51,172,146,57,Dwarf Shrub
52,204,184,121,255,Shrub/Scrub
71,223,223,194,255,Grassland/Herbaceous
72,209,209,130,Sedge/Herbaceous
74,130,184,158,Moss
81,220,217,57,255,Pasture/Hay
82,171,108,40,255,Cultivated Crops
90,184,217,235,255,Woody Wetlands
95,108,159,184,255,Emergent Herbaceous Wetlands

[jdduh]

Here is the layer file. Rename .txt to .lyrx.

NLCD_2016_land_cover_AK.txt

[lbross]

I posted the new .lyrx file to ArcGIS online. There is no way to version these files. You'll need to delete the existing nlcd_land_cover.lyrx from the BAGIS folder in your AppData directory. Then Pro will pull the new copy in from AGOL.

[jdduh]

The good news: the updated NLCD legend file works.
The not-so-good news:

1. (I think this is a normal behavior based on our design) The batch tool failed to generate any report when the AOI's stationtriplet/stationname data is missing. The error message in the batch_tool_log.txt is none-descriptive. We probably can provide a more descriptive error message (e.g., missing station ID data!). It seems that the batch tool continued to "complete" the processing of the report layers but failed to generate the report at the end.
2. The batch tool GUI showed this (see the screenshot below) when the reports were actually generated correctly (probably due to that SWE data weren't available). We probably need a different status message in the GUI (e.g., a red exclamation mark following "success").

Here are the entries in the batch_tool_log.txt file.

03/30/23 0:48:05 Starting batch tool to publish in E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\maps_publish
03/30/23 0:48:12 Starting batch PDF export for 15258000_AK_USGS_03272023
03/30/23 1:01:44 Copied report to E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\maps_publish\15258000_AK_USGS_Watershed-Report.pdf
03/30/23 1:01:44 Completed batch PDF export WITH ERRORS for 15258000_AK_USGS_03272023. The output is located at E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15258000_AK_USGS_03272023\maps_publish
Check for errors in the logs at E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15258000_AK_USGS_03272023\logs!
03/30/23 1:02:00 Starting batch PDF export for 15276000_AK_USGS_03272023
03/30/23 6:26:16 Copied report to E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\maps_publish\15276000_AK_USGS_Watershed-Report.pdf
03/30/23 6:26:16 Completed batch PDF export WITH ERRORS for 15276000_AK_USGS_03272023. The output is located at E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15276000_AK_USGS_03272023\maps_publish
Check for errors in the logs at E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15276000_AK_USGS_03272023\logs!
03/30/23 6:26:22 Starting batch PDF export for 15290000_AK_USGS_03272023
03/30/23 6:38:53 Copied report to E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\maps_publish\15290000_AK_USGS_Watershed-Report.pdf
03/30/23 6:38:53 Completed batch PDF export WITH ERRORS for 15290000_AK_USGS_03272023. The output is located at E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15290000_AK_USGS_03272023\maps_publish
Check for errors in the logs at E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15290000_AK_USGS_03272023\logs!
03/30/23 6:39:07 Starting batch PDF export for 15292700_AK_USGS_03272023

=====================================
Here are the last several lines in the AOI's log file.

2023-03-30 01:01:36.4971 DEBUG GenerateSitesTableAsync: Sites table E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15258000_AK_USGS_03272023\maps_publish\sites_table.pdf created!!
2023-03-30 01:01:37.6173 DEBUG GetPortalFile: The requested file was successfully downloaded from the Portal
2023-03-30 01:01:37.7214 DEBUG QueryAnnualRunoffValue: Found Runoff value of 2197.79
2023-03-30 01:01:39.5920 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snolite_sites
2023-03-30 01:01:39.6319 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class coop_pillow_sites
2023-03-30 01:01:39.7046 DEBUG CountFeaturesAsync: Feature class snolite_sites not found. Returning 0 features
2023-03-30 01:01:39.7325 DEBUG CountFeaturesAsync: Feature class coop_pillow_sites not found. Returning 0 features
2023-03-30 01:01:40.9708 DEBUG GenerateMapsTitlePageAsync: Title page created!!
2023-03-30 01:01:40.9957 DEBUG GenerateMapsTitlePageAsync: Data sources page created!!
2023-03-30 01:01:42.9669 DEBUG GenerateBlankPage: Blank page E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15258000_AK_USGS_03272023\maps_publish\Kenai_R_at_Cooper_Landing_snodas_swe.pdf created!!
2023-03-30 01:01:42.9759 DEBUG GenerateBlankPage: Blank page E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15258000_AK_USGS_03272023\maps_publish\Kenai_R_at_Cooper_Landing_snodas_swe_delta.pdf created!!

[lbross]

1. Regarding the NLCD .lyrx file: I remembered last night that we shortened some of the classification descriptions on the previous version so that they didn't run into the map. I don't know if we need to take a look again at this with the new version? If works for you, it's fine for me.
2. Currently the batch tool runs and keeps a count of any errors generated by the functions (for each aoi). There is only one level of error (success/failure). If there is at least one failure, the status will report errors. This doesn't necessarily mean that the report wasn't generated. I've tried to make it as resilient as I can. The errors status means that the user needs to look at the aoi-level log and see if the errors are significant. I believe you reported at one time that if there are no sites, it shows up as an error. It sounds like we may need another error category (fatal error) that means a .pdf wasn't generated? I don't want to write too many specific errors to the batch tool log because it becomes difficult to manage. I use a logging framework at the AOI level, but the batch tool log is more manual. I could add something to indicate that the .pdf wasn't generated.
3. I assumed that you wanted the batch tool to continue generating the layers even if it couldn't create a report due to the missing station triplet. We can change this if desired.
4. It's difficult to add color to that gridbox. Do we want to show 'warnings' if one of the functions failed ie: No SWE data and 'errors' if there was no report generated? And if there are any functions that you don't want reported as a warning (ie: no sites found), let me know.

It may be a process of trial and error before we get the errors/warnings like you want them. It's a complex question.

[lbross]

I am running into a related question while validating the 3.x build so I will add it here. I stumbled upon an AOI that has its station triplet populated. This station triplet is in the AOI Master List but not in the Master FCST List. When we implemented to Master FCST list, I switched all calls to the AOI Master List to the Master FCST list. I'm running into issues with this AOI because I can't retrieve the nwcc name or the winter months to create the PDF. So my question(s):

1. Did I do the right thing by swapping the FCST list for the older master list? This means that we can only create reports for FCST points. Or should I only use the FCST list for the Forecast station data tool?
2. Having an AOI with a station triplet that isn't in the master list seems like it should get a similar response to not having a station triplet at all. That is, not trying to create the report and writing an error message to the main log.

[jdduh]

Questions:

1. What the status categories do we have in the gridbox? Success/Failure/Errors? Now that I know what errors means, it's fine to leave them as they are.
2. What prevents the generation of reports when stationTriplet and stationName values are missing? Can we just put "Not specified" in the places where the data are shown in the report? We could implement a two- (three-)stage checking logic: check active FCSH first, then USGS GAUGE STATION (i.e., the original master AOI list), and finally "Not specified."

If we do implement the checking on multiple lists, then we should have some labels to differentiate them in the reports or in the log file. (For example, put "(not an active forecast station)" after the Streamgage station value on the summary page) NWCC currently only needs FCSH reports, but hasn't excluded the possibility of generating reports for some random locations or for new USGS gauge stations.

[Additional notes]
See PSU-CSAR/awdb-retrieve#5 for the request to automatically update the FCSH list. According to Beau, NWCC still wants the active USGS stations. Ideally, the batch report/update station data tool should check this yet to be created FCSH service for station data.

[lbross]

1. The status values are Waiting, Started, Failed, Completed, Errors. Failed is a critical error where the report could not be generated. Given the changes I've made the last couple of days, a missing stationTriplet should now be handled better. The biggest problem with it being null is not being able to look-up the supporting information from a master list. Could you post one of the aois that has a null stationTriplet so that I can be sure it doesn't cause problems elsewhere?
2. Currently nothing intentionally prevents report generation when stationTriplet is missing. I ran into 4 or 5 null exceptions when a record couldn't be located on the master FCST list. I have fixed these. I like the idea of putting "Not specified" in place of the places where the nwcc name is shown on the report. Another option is to use the stationName off of the pourpoint since that may be there, depending on how the AOI was created?

I'm in favor of not checking the multiple lists for now. The items that will be missing are the nwccName, the runoff value, and the winter months (thus no winter map). We could use default values in the code to create the winter precipitation map (and log it) to avoid that last error. I'm against having to maintain multiple master lists if we don't have to.

Perhaps when the new web service is created, we could add something to check some of the attributes on that list to indicate if it's a forecast point (or not) and annotate the report accordingly.

[jdduh]

"Regarding the NLCD .lyrx file: I remembered last night that we shortened some of the classification descriptions on the previous version so that they didn't run into the map. I don't know if we need to take a look again at this with the new version? If works for you, it's fine for me."

Please use the shortened descriptions (per #28).

Snow/Ice
Developed, Open
Developed, Low
Developed, Medium
Developed, High
Herbaceous Wetlands

We might not need to shorten the followings.
Grassland/Herbaceous
Sedge/Herbaceous

[lbross]

I've updated the NLCD .lyrx file on AGOL with the requested changes. You'll need to delete the local version in the AppData/BAGIS folder and BAGIS-Pro will pull this latest copy down. Let me know if you need any more changes.

[jdduh]

This is a bug specific to Alaska AOIs. It seems that the reclipped DEM was clipped with a buffer. Some of the sites don't have the elevation values. Also, the slope and aspect (and probably aspect too) values were not correctly retrieved from the data. Here is an example:

Watershed Characteristics Report
Kenai R at Cooper Landing
Publisher: nwcc_nrcs
Comments: Basin Analysis Report Version 1 (2023)
Local file path:
E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15258000_AK_USGS_03272023
Exported on March 30, 2023 at 1:01 AM (UTC-08:00) Pacific Time (US & Canada)
For more information see the Basin Analysis Reports Users Manual at https://nwccnrcs.
hub.arcgis.com/documents/basin-analysis-reports-users-manual
The boundary of a basin is the watershed delineated upstream of the streamgage station listed below:
Streamgage station: 15258000:AK:USGS

[jdduh]

Here is another AOI.

Watershed Characteristics Report
Ship Ck nr Anchorage
Publisher: nwcc_nrcs
Comments: Basin Analysis Report Version 1 (2023)
Local file path:
E:\NWCC\Alaska_AOIs_2023\Alaska\East_Kenai_Peninsula_Basin\15276000_AK_USGS_03272023
Exported on March 30, 2023 at 6:26 AM (UTC-08:00) Pacific Time (US & Canada)
For more information see the Basin Analysis Reports Users Manual at https://nwccnrcs.
hub.arcgis.com/documents/basin-analysis-reports-users-manual
The boundary of a basin is the watershed delineated upstream of the streamgage station listed below:
Streamgage station: 15276000:AK:USGS

[lbross]

Can you post one of these AOIs to basins for me to work with? The sites information is working correctly on my only Alaska example.

[jdduh]

I put two new Alaska AOI on the server (in the BAGIS_aois\Alaska folder). The one that has the problem is 15258000_AK_USGS_03272023.zip. The file size is 16 GB. The other smaller AOI seems to not have the problem.

[lbross]

This is fixed in the latest 3.x AddIn