MTBS Data Processing - ImageServices

[jdduh]

The USGS MTBS data (rasters) cannot be accessed directly from their WMS for geoprocessing. A local version is acquired and will be published as webservice(s) (e.g., image service) on either PSU ArcGIS Server or NRCS AGOL. BAGIS Pro has existing routines to extract individual raster data directly from ArcGIS Server imageservices. Given the temporal nature of the MTBS data, we could explore the options of publishing the MTBS rasters as 1) a time-aware mosaic dataset as one imageservice on PSU ArcGIS Server, 2) separate imageservices (one service for each year, from 1984 to 2022) on NRCS AGOL (see #15), or 3) a time-aware mosaic dataset as one imageservice on NRCS AGOL. Please recommend a solution. It seems the current method (i.e., separate imageservices on PSU ArcGIS Server) is the least complex in terms of data preparation and publishing. However, BAGIS Pro needs to have the capability of figuring out what MTBS services are available when new services are added to the server. We would also like to reduce the dependency of BAGIS on PSU server infrastructure.

[lbross]

I haven't been able to find a way to publish an image service to AGOL. This is the documentation on publishing image services from Pro. It seems to require ArcGIS Server. I don't know if you have another resource who can answer this question?

[jdduh]

A user needs to have the ArcGIS Image for ArcGIS Online license to publish an image service. I have assigned the license to ebagis. Here is a sample imageservice that I just published: https://iservices2.arcgis.com/6Miy5NqQWjMYTGFY/arcgis/rest/services/mtbs_CONUS_2022_dynamic/ImageServer

There is an AGOL wizard that allows users to publish the imageservice. The publishing is not from ArcGIS Pro.

[lbross]

Thanks for the additional information. We can't use the dynamic imagery layer because according to ESRI, dynamic layers can't be shared with Everyone (public). We would need to do this so they can be accessed by Pro. I was able to clip your sample layer from Pro, but when I look at it through the client there is nothing to see. Also no attribute table. Perhaps there is an error with the symbology? I get the same result when I add the imageservice to a map in Pro.

If we can get it to work, the time-aware mosaic image service would probably be the most efficient. Can you try publishing one of these with 2-3 years of data to AGOL so we can see if it works?

[jdduh]

It seems that AGOL doesn't support time-aware imageservices. I can reconfigure the AGOL dynamic imageservice to regular imageservice and try to publish a time-aware imageservices on the basins GIS server.

[lbross]

It depends on if it's more important to 1)host the data on NRCS AGOL or to 2) have it be a time-aware image service (hosted at PSU). Detecting the new services (years) is a solvable problem. We could come up with a naming convention and use a parameter in the batch tool settings that could be updated with the start and end years of available data. If #1 is more important than we should focus on getting the sample imageservice on AGOL working. If #2 is more important, then yes, please publish a sample time-aware service to basins.

[jdduh]

I prefer serving the data on AGOL. Try this one: https://tiledimageservices2.arcgis.com/6Miy5NqQWjMYTGFY/arcgis/rest/services/mtbs_CONUS_2022/ImageServer
I will remove the dynamic service. This service is currently hosted from PSU AGOL. The NWCC account on NRCS AGOL still doesn't have a license to publish an imageservice. I will move the services to NRCS AGOL once the requested license is assigned. The NRCS AGOL imageservices will have different URLs. Let me know if you need more services to be published on PSU AGOL.

The MTBS data has a separate data explorer (MDE) that is developed using Google Earth Engine. We can check its capability and see if the data can be used directly in ArcGIS/BAGIS Pro. See this page for more information: https://www.mtbs.gov/data-explorer

[jdduh]

The source MTBS data doesn't have an attribute table. The cell values represent:

1. Unburned/unchanged/Low (Dark Green)
2. Low severity (Cyan)
3. Moderate severity (Yellow)
4. High severity (Red)
5. Increased Greenness - increased post-fire response (Lime Green)
6. Non-processing area mask (White)

[lbross]

This layer is still not behaving as I would expect. It doesn't display in the ArcGIS viewer and while it let me created an attribute table on the clipped layer, when I try to look at the table, that option is greyed out.

I will look into the data explorer

[lbross]

FYI I downloaded the 2022 year data from MTBS and tried to add it to both ArcGIS Pro and ArcMap. Both tell me that the .tif is an invalid dataset. I requested the 2020 data and am waiting for their email to download it to see if it is a problem with this particular year.
I can't find a straightforward way to add the GEE data to ArcGIS. We might be able to do it using a Python script inside Pro but it would require installing the google earth libraries into a cloned Python environment. This would be some extra steps for anyone who uses the addIn that we can't script. Connecting to GEE this way also requires us to set up a GEE project. I am leery of adding dependencies to a google product as they have a history of dropping things they no longer want to do. GEE has been around for a long time though.

[jdduh]

There is a gdb with the complete MTBS dataset at D:\projects\gisdata\bagis-static\USGS_mtbs.gdb on basins server. The rasters are in a time-aware mosaic dataset. I haven't heard from NRCS on issuing an image license to the nwcc AGOL account and am inclined to just publishing the images as imageservices on the basins ArcGIS Server. I can build attribute tables for all the images, but just wonder why an attribute table is needed in the analysis.

[lbross]

I continue to struggle to work with this data. Maybe I'm doing something wrong? When I load the .gdb from basins, this is what I see:

I found this post on the ESRI discussion boards about loading the data into a mosaic dataset. Maybe it will help?

I was able to download the 2020 mtbs data this morning and open the layer in Pro. It looks a lot like the layer you posted to AGOL so maybe it's okay after all and I just need to figure out how to use it. I'm not sure if attribute tables are needed yet. But I've not worked with a discrete raster without one.

[jdduh]

Let's go with the regular imageservice with the MTBS data. We need to perform a time query to get a raster from a time-aware mosaic dataset. You can see the time tag (Year) from the attribute table of the Footprint layer.

Please convert the final raster layer for reporting to integer datatype using the INT tool. The output is a raster data with an attribute table containing the VALUE and COUNT fields. We can use the COUNT values to estimate the percentage. If the actual area is needed, then the cell size and linear unit of the raster need to be queried. If raster query is an issue, then here is the raster information for the MTBS dataset that we can put directly in the tool.

cell dimension: 30 meters by 30 meters
cell size: 900 square meters

[jdduh]

I will post the URL prefix for the imageservices when they are available. The raster's naming convention would be something like: mtbs_CONUS_YYYY.

[jdduh]

I will use integer rasters (with attribute tables) to publish the imageservices so that the BAGIS Pro addin doesn't need to deal with the datatype conversion. For future reference, the downloaded MTBS data needs to be converted to INT before being published to PSU ArcGIS Server.

[lbross]

Thanks for working on this. This is probably a silly question, but if we have a different imageservice for each year why does it need to be time aware? Or is the time grain shorter than a year? Or are these instructions for publishing the image services in the future?

[jdduh]

There is no need to be time-aware. I thought that might be a "new" approach we could explore. Given that AGOL doesn't support time-aware imageservices (note: I need to verify this), we can stick to the ArcGIS Server imageservice framework that we are using now. If time-aware imageservices do exist in ArcGIS Online and we can get a license from NRCS, the we can reevaluate the data serving framework in the future.

[lbross]

The key that I was missing was converting the values to integer. I thought I would need an attribute table to get the number of cells in each category. We have used this approach to calculate raster areas in BAGIS before.

It's a good idea to do the image services on basins, for now. If we get the license from NRCS we can publish them there and we 'should' just be able to swap out the server name in the configuration file. And it 'should' still work.

[jdduh]

The imageservices for MTBS are available at: http://webservices.geog.pdx.edu/arcgis/rest/services/usgs_mtbs_conus/mtbs_CONUS_1984/ImageServer

[lbross]

Thanks @jdduh. Given what we talked about yesterday afternoon, can you create these services in descending order from the most recent available year? If we're only going to clip the previous 30 years data and we're using the service name for service discovery, this will speed my progress somewhat.

[jdduh]

I was thinking of not publishing the most recent years so that you can test for future additions of new data. I can publish the data from, say, 2019 and prior. Then, we should have 3 years of services for testing.

[jdduh]

I think I can publish them all and disable/stop the services for testing purpose.

[lbross]

That's a good idea too. Yes 2019 and prior is good. Will let us test when nifc and mtbs have different dates which will often be the case. And stopping the most recent services is fine.

My preference is to add the 'checking for more recent data before running reports' function as an enhancement as I'm not sure I'll be able to get to it by the end of September. The workaround is to always load all the data if it hasn't been recently loaded and you are unsure what is in the catalog. It's not ideal but it's better than not finishing the basic functionality on time.

[lbross]

I'm having trouble clipping the mtbs image services. Getting an error message about size limits. Can you take a look?

[jdduh]

I can increase the max row, col numbers of the imageservices. The is the same error message that I encountered with the nlcd_land_cover imageservice. The 30 meters dem imageservice seems to be fine with the clip size for the same AOI. I wonder if this is related to real number versus integer imageservices. I will change the settings on 2015 - 2019 MTBS imageservices for testing. Let me know if you prefer changing all the imageservices. The new max clip number for rows is set to 15000, same as the max columns number. Please give them a try and let me know if 15000 is large enough.

[lbross]

2015-2019 is a good start. When I was looking through our clip functions, I noticed that we had to add the Alaska NLCD image service as a layer so we could clip it. It just tried that and it still didn't work. It gave me a 9999 error. I don't think the AOI I'm working with is all that big, so I'm not sure how this will work with the larger AOIs. I will test the new size tomorrow.

[jdduh]

The way I was able to clip the nlcd_land_cover is that I also set the Processing Extent to the AOI buffered polygon. I'm not sure if that was the reason because I had already make the max row number to 15000.

[jdduh]

The max clip dimension parameter is updated for mtbs 2004 - 2022 imageservices.

[lbross]

I am setting the extent in the C# code but have not been setting it when testing interactively. I will make sure to try that. And just to confirm, for the fire data we are clipping to the aoi boundary with no buffer? This is different from the original bagis-pro layers so I'm reworking some of the logic.

[lbross]

I was just able to successfully clip all of the mtbs layers with the clip dimension parameter update. Did you change them all or did this just start working? I am looking at 2002 which is in the gap between 2015-2019 and 2004 - 2022.
An item for discussion: It seems that this data may be sparse. ie: Lots of NoData rasters for an AOI. Should we run a clean up process that checks the raster properties and deletes any that are NoData? The downside of this approach is that we won't be able to distinguish if there was no mtbs data for that year. Or just none in a particular AOI. Not sure if that matters?

[jdduh]

2002 imageservice hasn't been updated. Did you update the extent setting in C#? If you did, then maybe that is why. It's ok to clip the mtbs just to the un-buffered AOI. Given that the next step in the process is to find the maximum mtbs value for each pixel for all the rasters in the time period, we might need to reclass NODATA to something else (e.g., -99). This probably also addresses your concern on NODATA versus missing data.

[lbross]

Yes. I added the extent setting in C#. I checked every other place where we clip rasters and we always set the extent which is good. I added comments to all of the implementations so hopefully I won't miss this again. It sounds like we want to keep all of the raw mtbs rasters.

[jdduh]

Yes, please keep all clipped mtbs rasters. We can name the rasters that have only NODATA cells with a distinctive suffix or prefix so that the subsequent processes won't use them. It's still required to recode NODATA cells to, say, -99 for those rasters that have valid burn severity values, otherwise, the final output might only contain NODATA cells (please verify if this is true). See https://pro.arcgis.com/en/pro-app/latest/help/analysis/spatial-analyst/performing-analysis/nodata-and-how-it-affects-analysis.htm

[lbross]

If we use the Cell Statistics tool to find the maximum mtbs value across a series of rasters, it has the 'Ignore NoData' option. I just tested this and it seems to handle the NoData values correctly. Are there any other tools you anticipate that we will need?

[jdduh]

If the ignore NoData option works for the cell statistics tool, then we don't have to recode the NoData cells. I don't think we will use any other tools that might be affected by NoData. Thanks for the verification.

[lbross]

I am working on the request to add a settings table with the value, description, and burn severity level (low, medium, high) or (1, 2,3) to the saved settings file. I can read the attribute table from an mtbs imageservice which allows me to dynamically know that the valid values are 1 .. 6. But this information is recorded for each image service (year). Is it safe to assume that all of the image services have the same values? If not, I guess we would need to define/configure this information for each year :-(

[jdduh]

Please assume that the codes are the same for the complete MTBS dataset. Here is the metadata for the MTBS classes for future reference if USGS changes the coding scheme.

1. Unburned/unchanged/Low (Dark Green) – A discrete burn severity class identified when thresholding dNBR data for Burned Area Emergency Response or Monitoring Trends in Burn Severity assessments or dNDVI for Burned Area Emergency Response assessments. It includes areas that are either unburned, or when visible fire effects occupy a small proportion of the site, on the order of less than 5 percent. The class may also include areas that recover very quickly after fire, such as grasslands or light surface burns under dense, non-impacted forest canopies.

2. Low severity (Cyan) – A discrete burn severity class identified when thresholding dNBR data for Burned Area Emergency Response or Monitoring Trends in Burn Severity assessments, dNDVI for Burned Area Emergency Response assessments or RdNBR data for Rapid Assessment of Vegetation Condition after Wildfire assessments. It includes areas where more than a small proportion of the site burned. All vegetation strata are slightly altered from the pre-fire state, but some may show pronounced burn effects. In forested ecosystems, it typically represents areas affected by fire where:

• Substrates, litter often exhibits fairly high consumption (up to 100 percent).
• Duff, woody debris and newly exposed mineral soil typically exhibit some change.
• Low vegetation (<1 meter) and shrubs or trees (1-5 meters) may show significant aboveground scorch, char or consumption, and vegetation density or cover may be greatly altered.
• Intermediate and large overstory trees may exhibit up to 25 percent mortality evidenced by crown char or scorch.
• Char height from ground flames is typically less than 3 meters.

3. Moderate severity (Yellow) – A discrete burn severity class identified when thresholding dNBR data for Burned Area Emergency Response or Monitoring Trends in Burn Severity (MTBS) assessments, dNDVI for Burned Area Emergency Response assessments or RdNBR data for Rapid Assessment of Vegetation Condition after Wildfire assessments. It includes areas that exhibit conditions that are transitional in magnitude and/or uniformity between characteristics within low and high burn severity classes.

4. High severity (Red) – A discrete burn severity class identified when thresholding dNBR data for Burned Area Emergency Response or Monitoring Trends in Burn Severity assessments, dNDVI for BAER assessments or RdNBR data for Rapid Assessment of Vegetation Condition after Wildfire assessments. In forested ecosystems, it typically represents areas affected by fire where:

• Substrates, litter is totally consumed; duff is typically nearly entirely consumed;
• Medium and heavy woody debris are at least partially consumed and at least deeply charred with mostly ash and charcoal remaining;
• Overstory trees typically exhibit greater than 75 percent mortality;
• Crown char is typically 100 percent from torching fire, and significant branch loss is present at the highest crown levels.
  In grassland and shrubland ecosystems, it typically represents areas affected by fire where:
• Over half of the site exhibits over 50 percent cover of newly exposed mineral soil or rock fragments;
• Herbaceous plants and shrubs are almost completely charred or consumed above ground, often with notable branch loss on taller shrubs;
• Resprouting from perennial plants, except grasses, is strongly reduced.

5. Increased Greenness - increased post-fire response (Lime Green) – A discrete burn severity class identified when thresholding dNBR data for Monitoring Trends in Burn Severity assessments. It typically represents areas that burned but display more vegetation cover, density, and/or productivity, usually within one growing season after fire. This is a fire-caused effect from release of nutrients into soil, and/or reduced competition for nutrients, light and water. These areas are usually herbaceous or low shrub communities that undergo little change in species composition after fire.

6. Non-processing area mask (White) – A discrete class assigned in Burned Area Emergency Response, Rapid Assessment of Vegetation Condition after Wildfire and Monitoring Trends in Burn Severity assessments representing areas of the fire masked out from the analysis. It includes pixels/areas where a reliable burn severity assessment cannot be conducted due to one or more satellite data or atmospheric/terrain interference issues (e.g. data gaps in Landsat 7 Scan Line Corrector-off imagery, clouds, cloud shadows, active fire, smoke, snow, and open water).

[lbross]

The PSU MTBS image services don't cover Alaska. Do we need a separate set of image services for Alaska?

[jdduh]

Yes. I will publish 1984-2022 AK mtbs webservices. The webservices will be in the usgs_mtbs_ak folder on the server with the 'mtbs_AK_YYYY' naming convention. Let me know if you have a preference on the service folder/name.

[lbross]

Those naming conventions sound good. Thank you.

[jdduh]

Some of the AK MTBS image services are available for your testing. All the services from 1984 to 2022 will be available shortly.

[lbross]

I don't think the extent is quite large enough for Alaska. I tried clipping the 15056210_AK_USGS_12072022 AOI to the mtbs image service and it gave me an error about being outside of the extent. This AOI is in southeast Alaska. It is covered by the Alaska DEM but not by the mtbs webservice. Let me know if I can provide additional information.

[jdduh]

It seems that the MTBS extends are bounded by the envelops of the fire perimeter "polygons". As such, the extends vary from year to year. We can assume if the AOI is not covered by the extends of the mtbs layers to be clipped then it means there were no fires for the AOI during these years. Can an error handler catch this and create a dummy mtbs raster for the AOI?

[lbross]

Do you know of a tool that can determine if the extent of one layer is completely included in another? It would be nice to test this before trying to do it rather than having to handle an exception. I think I looked for this before and couldn't find anything.

[jdduh]

Here is a possible solution:

1. Get the extend (xmin, ymin, xmax, ymax) of the raster (https://developers.arcgis.com/rest/services-reference/enterprise/query-boundary/)
2. Construct a rectangle polygon with the raster extend dimensions. https://support.esri.com/en-us/knowledge-base/how-to-create-a-polygon-from-an-xy-data-table-in-arcgis-000029301. Begin with the XY to Line tool to convert the extend csv file to a polyline featureclass. (see screenshot below). I might need to reproject all the AK mtbs layers to the USGS Albers if BAGIS Pro has difficulty clipping these layers.
3. Perform Intersect on the extend polygon and aoi_v. The tool will give a warning if both features don't overlap, also, the output featureclass will contain zero feature.

Here is an example of the csv file.
from_x,from_y,to_x,to_y
633165.0,1464225.0,633165.0,1197315.0
633165.0,1197315.0,-246615.0,1197315.0
-246615.0,1197315.0,-246615.0,1464225.0
-246615.0,1464225.0,633165.0,1464225.0

Here is the coding scheme of the csv file using xmin, ymin, xman, ymax notations.
from_x,from_y,to_x,to_y
xmax,ymax,xmax,ymin
xmax,ymin,xmin,ymin
xmin,ymin,xmin,ymax
xmin,ymax,xmax,ymax

[lbross]

Since the suggested workaround included running a gp tool, I decided to just run the clip tool and check the error message to see if the clip fire is outside the extent. It is working as you suggested by creating a NoData raster if this is the scenario. I have been testing with 15056210_AK_USGS but unfortunately it doesn't seem to have any fires for the 30 years. Do you have any suggested AOIs to test?
I also wonder if the nifc data does cover Alaska? When I zoom into the level I need to to see it, it looks like it stops at the Canadian border, but maybe Alaska is so far away that I'm missing it.
And it appears that the "for more information" link that we are publishing with the nifc data is broken: https://data-nifc.opendata.arcgis.com/datasets/nifc::interagencyfireperimeterhistory-all-years-view/about :-(

Edit: I think I found an Alaska AOI with fires on b-11-1. Stay tuned.

[lbross]

What I have appears to be working for Alaska. Nifc data does cover Alaska The percentages/area between nifc and mtbs appear more variable than in CONUS, if you want to dig into that. I clipped the layers and ran the statistics for 15515500_AK_USGS_Tanana_R_at_Nenana_04272023 on b-11-1.

Would you like an updated AddIn with Alaska support?