Basin Analysis Report - forecast AOI level statistics

[jdduh]

This is being spec'ed. We have a master FCST AOI polygon webservice (https://nwcc-nrcs.hub.arcgis.com/datasets/nrcs::master-fcst-aoi-polygons). NWCC is considering adding AOI-level data to the layer as attributes.
Here are some of attributes being considered. We will need a tool to either extract the values from the reports or recalculate the values. The output can be a .CSV file with stationTriplet as the key.

• AOI area in square meters - from shape
• AOI area in square miles - from shape
• x and y coordinates (in DD) of the centroid of the AOI polygon - from shape
• states that AOI boundaries intersect with (2-character state abbriv.)
• annual runoff ratio (%) - from the report
• elevation range min and max (in feet) - from the report
• median elevation (in feet)
• site counts for SNOTEL, SNOLITE, Coop Pillow, Snow Courses within AOI and within 5 km buffer - from the report
• % area represented by automated and snow course site - from the report
• % area by aspect - from the report
• % area by elevation zones - from the report
• % forested area
• % wilderness
• % public non-wilderness
• major precipitation zones (to be clarified)

User interface: similar to the batch tool or the update station data tool. Users select a folder and the tool will loop through the sub-folders and perform the calculation on the AOIs.

[lbross]

Look at Geoprocessing tool for GUI example. Put list of AOIs one tab and tool choices on another.

[lbross]

I am starting work on the spec for this. Re: x and y coordinates (in DD) of the centroid of the AOI polygon, there is a 'Calculate Geometry Attributes' tool that seems to do a good job with this and writes the values in DD to the feature class where we can query them. Did you mention that the centroid we report needs to be inside the polygon? There are some additional options for the tool (INSIDE_X, INSIDE_Y) that may accomplish this. The documentation for the tool wasn't clear to me.

[jdduh]

The centroid coordinates are for placing labels. The centroid needs to be inside the polygon. We need to find the centroid of the largest AOI polygon for the AOIs that have multiple polygon features in the aoi_v.

[lbross]

It appears the only other option is to 1) use the FeatureToPoint GP tool with inside option selected 2) Use the AddXY Coordinates tool to get the coordinates of the point. But it returns the coordinates in the annotation of the source feature class which is not DD. I've been trying to use the ConvertCoordinateNotation tool to convert these coordinates to WGS_1984 (DD) but it keeps failing. Any ideas?

[lbross]

median elevation (in feet): Median is not an option with floating point data like we have in the dem. ESRI suggests this workaround:

1. multiply your raster with for instance 100
2. use raster statistics to calculate the median
3. divide the result with the same factor (e.g. 100)
   Thoughts?

[jdduh]

How about using the Project tool on the FeatureToPoint output first, then AddXY Coordinate?

[jdduh]

The zonal statistics as table tool can generate median statistics for floating point rasters. We can use aoi_v as the input zone data.

[lbross]

Those were both good suggestions. I am attaching an .xlsx with 2 tabs. The first is sample output for the user and the second is the methodology we will use to calculate the values. We could put the units in a separate column to the right of the values if you think that is better. I vote against putting them in the same cell with the values because it is harder to do with C# and because it is harder for the analyst to include the values in calculations. This is a check-in to see what you think of the format. I know there are more statistics to add.
issue_45_forecast_statistics.xlsx

[jdduh]

The output is a table for all the AOIs. Imagine it as the attribute table of the AOI polygon featureclass (i.e., https://nwcc-nrcs.hub.arcgis.com/datasets/nrcs::master-fcst-aoi-polygons/explore). The metadata (i.e., specification) probably will be documented in the featureclass. It's very likely that the output will be joined to the polygon featureclass so that users can access them through the polygon geometry (spatial query) as well as stationTriplet (attribute query).

The "states" is one attribute field containing all the state abbreviations. The centroid coordinate should be two attribute fields (e.g., centroid_x and centroid_y). The elevation stats should be, elevation_min, elevation_max, elecation_range (i.e., max minus min), elevation_median.

[lbross]

So the output is an ArcGIS table in a geodatabase? Maybe the same geodatabase as the merged polygons? I think a standalone table has more restrictions on column names if you can even have one outside of a shapefile.

I suggest that we put the units in the field names for clarification:
states: the state abbreviations are separated by a comma if more than one
area_sq_meters
area_sq_miles
ann_runoff_ratio_pct
centroid_x_dd
centroid_y_dd
elev_min_ft
elev_max_ft
elev_range_ft
elev_median_ft

[jdduh]

Good idea with the naming suggestion. Either a geodatabase table, excel file, or csv file will do. I have created a Google doc for the "specification" metadata. https://docs.google.com/document/d/1k8dA8Qm9dhhrUBotb-NKLzUSolIUAuQpTrqdKl3GDio/edit?usp=sharing

[lbross]

Do you have any thoughts on how to represent the % area by aspect and % area by elevation zones in tabular format. It seems to make the most sense to have a column for each zone, but these columns would need to be dynamic. The aspect zones vary on the number of aspects selected and the elevation zones varies too.

It looks like I don't have permissions for the Google doc link you posted. I've continued work on my spreadsheet and will post when I finish working today so you can review it for our meeting tomorrow.

[jdduh]

I will bring this up with NWCC. We can treat this attribute table as an extension (or supplemental document) to the reports that were generated, which was based on the "default" settings. However, some of the intervals (e.g., elevation zones) were varied among the AOIs. Please document your questions in the shared Google doc.

[lbross]

I don't have permissions to the shared Google doc. Attached is the current spec and sample.
issue_45_forecast_statistics.xlsx

[jdduh]

AOI area in square meters - from shape
AOI area in square miles - from shape
x and y coordinates (in DD) of the centroid of the AOI polygon - from shape
states that AOI boundaries intersect with (2-character state abbriv.)
annual runoff ratio (%) - from the report
elevation range min and max (in feet) - from the report
median elevation (in feet)
site counts for SNOTEL, SNOLITE, Coop Pillow, Snow Courses within AOI and within 5 km buffer - from the report
% area represented by automated and snow course site - from the report
% area by aspect - from the report (4 directions are fine, but could make the tool to have the options to generate stats for 8 and 16)
% area by elevation zones - from the report (create 500 feet intervals for all possible elevation range 0 to 14,000 feet)
% forested area
% wilderness
% public non-wilderness
major precipitation zones, list all the elevation range (in text) for CPZ
critical precipitation zones percent area of AOI (would be close to 66% for most AOIs)

Additional attributes:
site density, total # of sites / AOI area (unit # site / 100 square miles)
site count, tallied by elevation zones (spec needs further discussion with NWCC)
% area in Canada (outside US)

[lbross]

I've been thinking about the % area by elevation zones. The current suggestion of 500 foot intervals may be confusing because the intervals will be disconnected from the maps and also from the major precipitation zones statistic. We could put the % area by elevation zones into 2 columns using a similar format to the major precipitation zones:
elev_zones_ft: 6500-6750, 6750-7000, 7000-7250, 7250-7500 ...
elev_zones_pct: 5.6, 10.1, 25, 13, ...

This also keeps the table narrower because we aren't always adding 13 columns.

[lbross]

I am adding this to the Admin Tools form. Please supply text for the tooltip.

[lbross]

Title for gridbox: Generate AOI Summary. Tooltip TBD.

[jdduh]

Tooltip for Generate AOI Summary: Extract AOI report data to generate a .csv summary table that can be linked to AOI geometries using stationTriplet values.

[jdduh]

A few updates on the AOI summary table.

1. Please add % area by slope zones to the table.
2. % public non-wilderness does not include AIR. Please add a new % AIR area to the table.
3. Please combine multiple values in one attribute field for the following summary data. NWCC has concerns for this approach if the table is used in yet-to-be-identified spatial analyses. They might ask us to implement their proposed design if the summary table is restrictive to analysis.

• states that AOI boundaries intersect with
• % area by aspect (could vary by 4, 8 or 16 directions)
• % area by elevation zones
• number of sites by elevation zones
• major precipitation zones (critical precipitation zones)

[lbross]

Seeking some clarification. These questions may also be in the google doc.

1. For the zonal fields (% area by aspect, % area by elev zones, # sites by elev zones, critical precip zones) is NWCC okay with putting the zones definition in a second field? Actually, 2 of these zonal fields use the full set of elevation zones so we could only record those once.
2. The Slope zones are fixed. Do we follow the pattern of the zone definitions in one column and the percentages in another? Or do you want a separate column for each slope zone?
   I added the new fields to the requirements document on the google site.

[jdduh]

1. They have concerns about the ease of use of putting the zones definition in an attribute field, but have yet to come up with a real scenario for not doing so. Given that the output is in CSV format, it won't be too difficult to reformat the % area by aspect. The elev zones attribute has different values for different AOIs. We have to wait until they get a hold of the CSV file to see what additional usage might surface.
2. The Slope zones is similar to % area by aspect. It's very likely that all AOIs have the same slope/aspect definition. We can easily reformat the CSV file afterwards. Please implement them in the same manner.

[lbross]

The initial release for this enhancement is done and a new AddIn is available on the bagis FTP server. This release includes the warning messages requested under issue #48. I updated the date on this AddIn to today's date because this feels like a major release.
I have also updated the design spec on the google site as I worked so it should be current. I have indicated dependencies on the batch tool layers and earlier statistics where appropriate.

[jdduh]

The AOI summary tool crashed when working on an AOI that hasn't been processed with the Batch Tool. Please improve the error handling routine. Also, problematic AOI should be captured in both the log and output CSV files. Please prevent popup messages that might interrupt the continuing execution of the statistics extraction process. The CA AOI (10346000_CA_USGS_08222023) that caused the crash is now available on the ftp server under BAGIS_aois\debugging_AOIs folder.

[lbross]

I found a problem with the site_count_elev_zone statistic that causes ArcGISPro to crash. I'm not sure if this is the same problem that is causing your crash. We calculate the elevation zones based on the elevations of the AOI boundary. The site buffer distance often includes sites outside of the AOI boundary that may have elevations outside of the elevation zones boundaries. Should we exclude these sites from the site_counts? Or should we add two new columns for sites with no elevation zone?

[jdduh]

The tool works fine on AOIs with all the analysis layers generated by the batch tool. I noticed some issues with the output csv file.

1. It seems that there is a mix-up with the stationName and stationTriplet fields. First, I noticed that all the previously processed AOIs are now "NotReady". I had to run the Forecast Station Data tool to make them ready. In the output csv file, the stationName heading is associated with the stationTriplet values and the stationTriplet heading is associated with the stationName values.
2. The area_outside_us has two different values in the output csv file for the AOI I processed. One is Not Found, the other is 0. What is the difference? All the AOIs are completely within the US.

The other data fields seem to be correct.

[lbross]

1. The stationName and stationTriplets were swapped in the output .csv file. This was a display problem only, and I fixed it
2. I'm not sure what to say about the "NotReady" status, since I'm unsure of the order of operations. The logic that is in the 'Forecast Station Data' tool to set the required station triplet and associated fields is separate from the Batch Tool. We agreed that this should be a 2-step process. Run the 'Forecast Station Data' tool and then run the Batch Tool. The AOIs should be showing as 'Waiting' going forward if the 'Forecast Station Data' tool was successful. (An aside is that the status doesn't update correctly when the Statistics tool runs in the version that you have. This is fixed).
3. For this statistic, I subtract the polygon area from a re-projected, clipped layer from the area of the aoi_v. It seems in some cases this can result in a small negative number (if the aoi is completely within the USA) which I wasn't expecting. This is fixed.

I am ready to post a new AddIn for you with these fixes, but I'm awaiting a response to my question from 2 days ago about what to do with sites that are outside of the AOI elevation zones.

[jdduh]

Please add the columns to keep track of the sites that are outside the elevation zone range. We can separate the ones above from the ones below the zones.

An alternative approach is to relax the upper and lower bound definitions in the elev_zones_def column. Here is an example:
Original def: "2209.35-3000,3000-4000,4000-5000,5000-6000,6000-7000,7000-8000,8000-9000,9000-10000,10000-11000,11000-11420.94"
Modified def: "<3000,3000-4000,4000-5000,5000-6000,6000-7000,7000-8000,8000-9000,9000-10000,10000-11000,11000<"
Then lump the sites outside the elevation range to their associated zones.

[lbross]

Updated algorithm for calculating elev_median_ft. Using station_name as zone field on aoi_v instead of aoiname. Station_name should always be populated going forward because of the forecast station information tool.

[lbross]

Digging into the return value for ann_runoff_ratio_pct. Not sure if this should be a -1 (error) or null (missing). Here are the possible conditions that I am currently trapping that prevent this value from being calculated:

1. Unable to download the reference .xls file from AGOL
2. Not finding the annual runoff value in the .xls file for the current AOI
3. This statistic depends on the batch tool being run because that tool calculates/records the PrecipVolumeKaf value. If that is missing, we can't calculate the ann_runoff_ratio_pct
   Should these issues return a null or -1? We can handle them all the same or differently.