Migrate create AOI from a shapefile process to BAGIS-Pro

[lbross]

Start with create AOI from shapefile.
Clips all the layers and arrange them into an AOI structure. Creates the flow/accumulation layers. New step for tool to create a pourpoint. Intersection with stream network should be the pourpoint. Find location inside shapefile that has the highest allocation value.
Interactive tool user needs to digitize a pourpoint. Make decision on which gauge station to use.

[lbross]

@jdduh: Can you provide a sample shapefile for me to work with, hopefully not too big?

[jdduh]

Here is a shapefile for the Animas AOI.

animas_aoi.zip

[lbross]

Thank you. I'm pasting a screenshot of the UI for this function. I plan to replicate the screen as much as possible. Let me know if you would like to make any changes as we proceed with the transition.

[jdduh]

We need a new BAGIS tool 'group' for 'AOI Creation'. All tools, including the 'create AOI from a shapefile' tool, in the Basin Analyst pulldown menu can be moved to the tool group.

[lbross]

This is what I'm working with right now. I created the Basin Analyst tool group a couple of years ago when we started working with ArcGIS Pro. I need a way to trigger the tool. I'm going to focus on the tool for now, but will update the menu before release.

[lbross]

This is what the UI looks like converted to BAGIS-Pro. No code behind it yet.

[lbross]

Just noticed that the height and width of the filter size is always disabled in BAGIS v3. ie: if you choose to filter, it will always be 3x7. Is this intentional?

[jdduh]

The Basin Analysis tool group looks good. In this case, we can keep the "About" menu item and "hide" the other menu items that I crossed out. We probably will convert some of the AOI utilities functions to BAGIS Pro later.

The filter size should be customizable by the users. 3 x 7 is the recommended size. The default for DEM smoothing is disable. Here is the description on the setting.

[lbross]

Per our conversation on Tuesday, I'd already made the changes to the menu that you requested earlier. I can make additional changes as needed. This is what I currently have:

I've migrated the 'Why Smooth DEM' MessageBox to BAGIS-Pro and made the filter size customizable.
I'm running into some logic that is dependent on the 'Generate AOI Only' checkbox on the BAGIS V3 settings screen. Is this something we still need to support? It seems like an AOI isn't very usable with the BAGIS-Pro reporting if it doesn't have all of the layers.

[lbross]

We need to think about where this tool should get the uri's for the clipping. We read the bagis-pro data sources and their uri's from the GitHub server. There is no way to override these, but we do keep a record of them locally for each aoi to include in the pdf report. We also have the default values for the basinanalyst.def online via GitHub and use these a few places in bagis-pro. Then there is the local basinanalyst.def file. BAGIS V3 gets its uris from there. This allows the analyst to override the default settings by using the 'Options' menu. They can also synchronize the options with the default settings at any time.

[lbross]

There is a function to add the aoiname, basin, and awdb_id fields to the aoi_v feature class? Are these still the correct fields? Should we be adding station_triplet?

[jdduh]

When creating an AOI from a shapefile, the 'station' information is not required. When creating an AOI based on station information, users must create a BASIN first to subset the DEM so that the flowdir and flowacc can be calculated efficiently. Once the BASIN exists, users can create an AOI (within the BASIN) based on an existing station or a location (non-station) that they picked. The stationName and stationTriplet data are available in one of these situations. For other situations, users still need to provide a name for the AOI. This name is not a stationName. When creating an AOI from a BASIN, the BASIN name is propagated to basin attribute of the AOI.

I suggest that, for the create an AOI from a shapefile tool, we keep the original program logics. The AOIs created with this tool need additional methods to find their pourpoints. Once we have the pourpoints, we can then integrate the 'Update forecast station data' tool to see if any of the pourpoints falls close to existing forecast stations.

For AOIs created from a BASIN, we can use the station data or the default station data, when the station doesn't exist, to set the stationName and stationTriplet values and ignore the aoiname and awdb_id fields. Let me know if you have any thoughts.

[lbross]

The original program logic for this tool is to add the aoiname, basin, and awdb_id fields to the aoi_v feature class immediately after the feature class is created. As you indicated, there is only enough information available to populate the aoiname field. The other two remain null. An aoi_v feature class should have the same attribute fields regardless of how it was created. They may not all be populated.
My only suggestion is to stop creating awdb_id and start creating station_triplet since we no longer use awdb_id. We might also want to start adding station_name at this time.

[jdduh]

Let's ditch aoiname and awdb_id field and keep basin. We will set Nodata (or whatever we decided on the default value) on stationTriplet and the AOI name users entered as the stationName for the create an AOI from a shapefile tool. We can just leave the basin name blank.

[lbross]

To summarize: For this tool we will add 3 fields to aoi_v when it is created: stationName, stationTriplet, and basin. For this tool, StationName will be set to the value the user enters in the 'Output AOI Name" field. This could be overwritten by the forecast station tool or an integrated version of that logic. The remaining 2 fields will be left null.

[lbross]

Add new fields to batch_settings_tool.json for dem uri, the dem units, and the gauge station layers. Rename BatchSettings to BagisSettings. Give these 3 new data points a unique prefix so that they can be easily found. Make sure we are recording the buffer distance and units for the dem (AoiBufferUnits, AoiBufferDistance). Remove references to PRISM buffer and units from UI.

Add validation to tool to check to see if layers exist (hard stop) and if stationName and stationTriplet exist.

Edit: Regarding validation. For the create AOI from a shapefile tool, we only need the DEM. We don't use the gauge station layer so we don't check for stationName or stationTriplet.

[lbross]

I am in the process of renaming BatchSettings to BagisSettings. This will break BAGIS-Pro. Let me know if you need to use BAGIS-Pro and I will post a new version

[jdduh]

Please post a new version when you get a chance. I need to generate some new basin reports. Thanks!

[lbross]

There is a new version posted to the basins ftp server. It is lightly tested, so let me know if you run into any issues. If I can't fix them quickly, we can change the server back until you generate the new reports. The names of the fields with the years in the fire statistics tool are probably still not right. I am waiting for confirmation from you on issue #55

[jdduh]

The batch tool reporting function works normally without a glitch. The pdf files were created correctly. I haven't tried the summary tool yet.

[jdduh]

Except some scrambled codes on the cover page, probably related to #54.

and this (the scrambled codes show up in different places in different documents)

The pdfs can be found at: ftp://webservices.geog.pdx.edu/public/reports/Oct2024_NewForecastStations/

[lbross]

Yes. This is related to #54 and I don't know how soon or if it can be fixed. The other free PDF generators I've explored have issues of their own. Maybe we can come up with an alternate to the dash (-)? This is Chrome trying to localize when we haven't built a profile.

[jdduh]

We can replace the long horizontal bars with dashes. I'm not sure about the carriage return characters that appear between the paragraphs.

[lbross]

So this is interesting. It looks fine on my laptop. What computer are you running the tool on? Do you have some of the international features enabled for Chrome? What version of Chrome is installed?

[jdduh]

Windows 11 22H2 with Chinese language pack installed, but all the default language settings are English (US); Chrome version 130.0.6723.70 (just updated, but I check for Chrome updates fairly frequently). Chrome doesn't have new extensions installed recently. All the extensions were disabled.

[lbross]

How important is it that the ArcGIS Pro output match the BAGIS v3 output? The numbers are close, but not exact. I think this may be part of the difference. The raster to polygon tool simplifies the polygons by default. BAGIS v3 didn't do this. The red line is pro and the black line is v3. This is a sample of the aoi_v file.

Also, should I be setting the snap raster for the surfaces tools? I don't think we did this in BAGIS v3 so when I compare the layers they aren't lining up exactly.

[jdduh]

I believe there is an option to not generalize the outline. Please enable it. It's critical to not generalize the AOI boundaries and that the "snap raster" option is activated when clipping the source DEM so the that there won't be any slivers when merging neighboring AOI polygons.

Please check the standard AOI creation subroutines for these settings. We might not have used these settings in the Create AOI from a shapefile tool.

[lbross]

Yes. We do have the option NOT to simplify when converting the raster to polygon. I will make that change.
What snap raster should we use when clipping the source DEM? Throughout BAGIS-PRO we use the filled_dem as the snap raster but we don't have that yet when we are clipping the source DEM!

I have run into a problem with the flow accumulation tool. The tool runs and the range of values looks close to BAGIS v3, but when the raster is displayed, I don't see the channel like I do with the BAGIS v3 output. I posted a flow_accumulation.gdb.zip in the For_Geoffrey folder on basins that has the surfaces.gdb output up until this step.

[jdduh]

We use the source DEM in the settings file as the raster to be snapped to. This option is activated when a user selects a BASIN folder - given that almost all AOIs were created based on a BASIN. The flow accumulation raster looks correct. We need to change its symbolgy to the "histogram equalize" stretch type and set the Statistics to DRA (dynamic range adjustment) in BAGIS Pro whenever a flow_acc layer is added to the map.

[jdduh]

Just want to document this potential bug for future reference. BAGIS is able to handle aoi_v that has multiple dangle polygons. However, one extremely large AOI (09379900_AZ_USGS_Colorado_R_at_Glen_Canyon_Dam) seems to have issues with these dangle polygons. Here is a screenshot of the AOI's aoi_v and aoib_v (the red circle). Obviously, the aoib_v is incorrect. You can find a copy of this AOI on VSCB113-C1 under the NWCC\2023_NWCC_ActiveForecast_AOIs_HUC2\HUC14_Upper_Colorado folder.

This particular aoi_v has 8 polygons.

[lbross]

Seeking clarification. Is BAGIS-Pro able to handle aoi_v that has multiple dangle polygons? It's my understanding that we want to leave these dangles in place for the aoi_v so that we have an accurate representation of the aoi boundary.
Is the problem with the generation of the aoib_v by BAGIS V3? I don't think BAGIS-Pro is generating the aoib_v yet. Do we want to ensure that aoib_v has no dangles when we start generating it in BAGIS-Pro?
Is BAGIS-Pro using the aoib_v incorrectly somewhere? I can't remember where BAGIS-Pro uses aoib_v.

[jdduh]

This is the only AOI (out of almost 600) that has the buffer issue in both ArcMap (and probably ArcGIS Pro too). It's obvious that the buffer was created using just one of the dangle polygons. This has not been an issue for other AOIs that have multiple aoi_v polygons. Some of the reclipped layers generated by BAGIS Pro also have the wrong extent. These include the road and land cover layers (and maybe some others).

[lbross]

Upon further examination, it appears that the aoib_v is used extensively for clipping if the requested buffer distance is the same as the aoi buffer distance. It makes sense that I did this because it saves processing time not having to create the buffer file for every clip. How do you want to handle it? Is it okay to buffer aoib_v by 0.5m to get rid of the dangles and then re-save the file so we don't have to do this every time we clip to aoib_v?

Also I found that there is some old code that overwrites p_aoi_v with aoib_v if there is no prism buffer distance selected. We haven't run into it because we always buffer prism, but this should probably come out. I will regenerate p_aoi_v with the correct buffer distance independent of aoib_v. But we have the same question here, is we can modify the cartography slightly and persist it to avoid the dangles.

[lbross]

We agreed at our December 2 2024 call that we will take no action on aoib_v because it is only a problem with this one AOI.

[lbross]

I'm generating the pourpoint class but noticed that BAGIS v3 adds/populates several fields in this feature class that we don't use yet in bagis-pro. I created a google doc with the fields for both this and aoi_v because some of the fields are shared. The ones in yellow are the ones that I am uncertain about.

[jdduh]

Additional remarks added to the highlighted fields in the shared Google sheet.

[lbross]

Thank you. 2 follow-up questions.

1. AOISHPAREA AND AOISHPUNIT. Your comment says 'We can switch to the new attribute fields if the UI can display the area'. What does this mean? Will we be creating a BAGIS-Pro version of AOI Utilities? We can store the area in SQKM (the AOISHPUNIT) and convert it to the other units for display if we need to.
2. awdb_id. Your comments says 'We can remove it for now, until we find its dependency in the rest of BAGIS Pro during the QA/QC stage.' Does this mean we want to eliminate this field from BAGIS-Pro? It shouldn't be too hard to do this because I use the field as a constant I can search for. I just haven't done it yet because it's been easier to leave it in.

[jdduh]

We need to create some types of AOI browser for the BAGIS Pro. See below for some screenshots from BAGIS V3. It will be fine to store the area in the areasqkm (or whatever we are using now) as long as the unit is explicitly documented somewhere (e.g., in the table column name).
If we are sure that BAGIS Pro won't use awdb_id (which is replaced by stationTriplet), then we can remove the awdb_id field.

[lbross]

I have excised the awdb_id field from BAGIS-Pro. I opened two new issues for migrating the screens associated with the screenshots. The units for AOISHPAREA are stored in AOISHPUNIT so it should be easy to convert the area to other units when we need to. For now, I will only support SQKM since we control and populate this field, but the infrastructure is there if we need to add other units later.

[lbross]

For the hillshade layer, in addition to the z factor it also needs an azimuth and an altitude. The default values are 315 and 45 respectively. Is this okay?

[jdduh]

The default values for the hillshade tool are correct.

[jdduh]

The AOISHPAREA value needs to be updated every time BAGIS Pro retrieves its value. In theory, BAGIS Pro can access the shapearea value directly and check the spatial reference metadata (projection) to figure out the unit of the shapearea value. The reason that the original BAGIS maintains the AOISHPAREA and AOISHPUNIT is because the original Weasel AOI uses shapefile to store vector data. A shapefile doesn't have the shapearea field by default. After an AOI is converted to the fileGeodatabase format, all vector data are stored as featureclasses that by default have the shapearea field.

If AOISHPAREA is the source of the AOI area data in BAGIS Pro, then it needs to be updated every time BAGIS Pro retrieves its value. We current have several AOI reports (and summary statistics) that contain incorrect AOI area information due to that the AOISHPAREA values were not updated (due to the modification of the AOI boundary) or were missing (due to unknown errors when the AOI were created in BAGIS V3). It would be ideal to remove the AOISHPAREA dependency that needs to be checked by the user when they modify the AOI geometry. It's not a uncommon scenario to have the AOI geometry modified, particularly in large AOIs that encompass a diverse geographic regions.

[lbross]

Stop creating and populating aoishparea and aoishpunits field. No longer need to calculate AOI area.