BAGIS Pro Basin Report

[jdduh]

Got this error message when I generated the report using the interactive UI. It seems that ArcGIS Pro wasn't able to convert the title_page.html to title_page.pdf. The maps_publish folder of the AOI has both title_page.html and title_page.xml. Their contents look correct.

Is this related to Windows updates? I am still using ArcGIS Pro 3.1.4.

Here are the messages in the log file when I generated the reports in the interactive mode. See the end of the post for the error messages I got when I used the batch mode.

...
2024-10-02 13:45:23.3758 DEBUG GetPortalFile: Renamed C:\BAGIS\BAGIS\annual_runoff_averages.csv so a new copy could be downloaded
2024-10-02 13:45:23.7257 DEBUG GetPortalFile: The requested file was successfully downloaded from the Portal
2024-10-02 13:45:23.7528 DEBUG QueryAnnualRunoffValue: Found Runoff value of -1
2024-10-02 13:45:24.6703 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snotel_sites
2024-10-02 13:45:24.6852 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snolite_sites
2024-10-02 13:45:24.6997 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class coop_pillow_sites
2024-10-02 13:45:24.7130 DEBUG CountFeaturesAsync: Feature class snotel_sites not found. Returning 0 features
2024-10-02 13:45:24.7130 DEBUG CountFeaturesAsync: Feature class snolite_sites not found. Returning 0 features
2024-10-02 13:45:24.7396 DEBUG CountFeaturesAsync: Feature class coop_pillow_sites not found. Returning 0 features
2024-10-02 13:45:24.7543 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snowcourse_sites
2024-10-02 13:45:24.7676 DEBUG CountFeaturesAsync: Feature class snowcourse_sites not found. Returning 0 features
2024-10-02 13:45:25.0808 DEBUG GenerateMapsTitlePageAsync: Title page created!!
2024-10-02 13:45:25.1695 DEBUG GenerateMapsTitlePageAsync: Data sources page created!!
2024-10-02 13:45:25.2868 DEBUG GenerateBlankPage: Blank page E:\NWCC\New_AOIs_2024\PacificNW_Region_NewBasin\351200_MT_USGS_North_Crow_Creek_nr_Ronan_09302024\maps_publish\title_page.pdf created!!

Here are the error messages when I generated the reports in Batch mode.
...
2024-09-30 15:31:25.9066 DEBUG GetPortalFile: Renamed C:\BAGIS\BAGIS\annual_runoff_averages.csv so a new copy could be downloaded
2024-09-30 15:31:26.2721 DEBUG GetPortalFile: The requested file was successfully downloaded from the Portal
2024-09-30 15:31:26.2996 DEBUG QueryAnnualRunoffValue: Found Runoff value of -1
2024-09-30 15:31:28.0047 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snotel_sites
2024-09-30 15:31:28.0338 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snolite_sites
2024-09-30 15:31:28.0600 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class coop_pillow_sites
2024-09-30 15:31:28.0999 DEBUG CountFeaturesAsync: Feature class snotel_sites not found. Returning 0 features
2024-09-30 15:31:28.1275 DEBUG CountFeaturesAsync: Feature class snolite_sites not found. Returning 0 features
2024-09-30 15:31:28.1522 DEBUG CountFeaturesAsync: Feature class coop_pillow_sites not found. Returning 0 features
2024-09-30 15:31:28.1733 ERROR CountPointsWithinInFeatureAsync: Unable to locate point class snowcourse_sites
2024-09-30 15:31:28.1886 DEBUG CountFeaturesAsync: Feature class snowcourse_sites not found. Returning 0 features
2024-09-30 15:31:28.5308 DEBUG GenerateMapsTitlePageAsync: Title page created!!
2024-09-30 15:31:28.6337 DEBUG GenerateMapsTitlePageAsync: Data sources page created!!
2024-09-30 15:31:28.7381 DEBUG GenerateBlankPage: Blank page E:\NWCC\New_AOIs_2024\PacificNW_Region_NewBasin\351200_MT_USGS_North_Crow_Creek_nr_Ronan_09302024\maps_publish\title_page.pdf created!!
2024-09-30 15:31:28.7599 ERROR RunImplAsync: at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access)
at PdfSharp.Pdf.IO.PdfReader.Open(String path, String password, PdfDocumentOpenMode openmode, PdfPasswordProvider provider)
at PdfSharp.Pdf.IO.PdfReader.Open(String path, PdfDocumentOpenMode openmode)
at bagis_pro.GeneralTools.PublishFullPdfDocument(String outputPath, ReportType rType, Int32 sitesAppendixCount) in C:\workspace\bagis-pro\bagis-pro\GeneralTools.cs:line 1993
at bagis_pro.DockAdminToolsViewModel.RunImplAsync(Object param) in C:\workspace\bagis-pro\bagis-pro\DockAdminToolsViewModel.cs:line 1620

[lbross]

I am getting the same error. When we upgraded from Pro 2.x to 3.x the free api that we were using to convert from html no longer worked. I found a solution using the Chrome browser in headless mode. They changed the api a few years ago. Maybe they changed it again? I will do some research. So this problem is due to some change in Chrome rather than our code or ArcGIS Pro.

[jdduh]

FYI. https://www.reddit.com/r/chrome/comments/1f3f3gg/latest_version_html_to_pdf_quit_working/

[lbross]

This solution requires all users to install an additional chromium library locally which I would like to avoid. I am going to try using the --headless=old solution to get you going again. But this is only temporary as it sounds like google is trying to remove this functionality from the Chrome browser.
We need to find a new free third party C# library that works on .NET 6.0 framework for converting html to pdf :-( My research this morning hasn't revealed an obvious candidate. They all have drawbacks :-(

This is a more complete description of the issue and options.

[lbross]

I've been able to fix this by adding the --profile-directory flag and setting it to a temp subdirectory of the maps_publish folder that I create and then delete. So we're not using the deprecated api which is good. I'm trying to run an end-to-end test before I publish a new AddIn and now it's crashing when publishing the multipane maps. I'm working on it.

[lbross]

The convert html to pdf function is temporarily fixed. I implemented the option to set the --user-data-dir parameter. Chrome creates a bunch of temp files in this folder so we create it before we use it and delete when the process is finished. It is very fast. I'm not comfortable with this as a long term solution, so I'll be spending some time exploring other options. But I know we have to deliver some reports to NWCC soon, so this new AddIn should allow you to do that. I uploaded it to basins.
I am still working on a problem where Pro crashes when removing the map tabs for the extra map frames for the multipane maps. It doesn't thrown an exception I can catch. It just crashes. So far it hasn't happened when I start without a template. I wonder if the the template I am using is corrupted? I will try creating a new template and see if that works. It still crashes even when I close all the maps on the template.
You will see the option for creating an AOI from a Shapefile in this AddIn. Only the beginning stages of this function are implemented, but you can review the form/UI. If we need an AddIn to distribute to NWCC, let me know and I will hide this button.

[lbross]

Moving the conversation for this issue back to where it belongs. There is some documentation for this issue on #53. The temporary solution is not working on a computer with the Chinese language pack installed. I am unable to recreate the problem in my development environment so will need @jduh to test the solution. I don't want to dig too far into Chrome development as it seems quite dynamic and we don't have the resources to follow it.

Some options to try:

1. Using the --headless=old flag in the Chrome call. This is a temporary solution because it is deprecated and will eventually be removed, but it could get us through a couple of months, if it works
2. Require that users install the standalone Chrome headless shell in a location of our choosing if they want to generate PDF reports. I don't think this would auto-update so we would have better control but it is a separate step in the installation process. We could look for this component when an AOI is chosen and pop an error if it's not installed. Should be one time for each computer.
3. I can implement the PuppeteerSharp solution that I tried earlier this week. I wasn't able to get the margins to look right, but this may be a better solution than the funny symbols on the output. This is a third party library so maintenance and updates are out of our control. But if we compile against a selected version, it should be relatively stable.

[jdduh]

Is it possible to specify a specific font when generating the html files? It seems that the computer in question might not have the font. I will generate the reports on a lab computer for now and report back.

[lbross]

I might be able to do that in a css stylesheet. If you want to try this, please let me know what font to use. You can also try saving the title_page.html as pdf from Chrome and see if it still looks funny.

[jdduh]

Used Chrome's Print... -> Save As PDF. The output pdf looks correct without any scrambled characters. The fonts properties of the pdf file show that the document has Arial-BoldMT and ArialMT. The Chrome browser doesn't have any 'customized' font settings.

[lbross]

I'm not sure what you want me to try next. Chrome's print function is probably not exactly the same as the headless version that BAGIS uses because it is setting up a new, generic user profile. We are using the font-family 'Arial, Helvetica, sans-serif' for some of the styles. I could try applying it to the entire html tag. Or we could try one of the 3 options I listed above. How did the tests on the lab computer go?

[jdduh]

BAGIS Pro's Batch Tool just crashed ArcGIS Pro 3.3.2 on a lab computer. Attached is the log file. I will need to put the AOIs on a computer that I can remote desktop to first. Don't do anything for now until I exhaust all the possible workarounds.
2024-10-24_logfile.txt

It worked the 2nd time. The crash probably was related to the AGOL log on prompt. I switch the ArcPro logon directly to the NRCS account. As expected, the reports look normal. Is there a way to set the font for the html? It seems that ArialMT and Arial_BoldMT are the ones used by default.

[lbross]

The HTML standard lets you set the font-family and the client fulfills this request the best it can with the fonts that are installed. Per my previous note, our current font-family is 'Arial, Helvetica, sans-serif'. Is there a particular font(s) that you want to try? Do fonts in one part of the report look different from others? As I said previously, I'm setting the font for some of the styles but not at the overall html level.

[jdduh]

The complete pdf files has many different fonts (e.g., those from ESRI). The first several pages that are rendered in Chrome use ArialMT and Arial_BoldMT. The MT font type seems to be associated with PostScript. We can set the font to Arial. BTW, when you suggest that I "can also try saving the title_page.html as pdf from Chrome and see if it still looks funny." Who can I save the html as pdf in Chrome? The other potential issue is that my desktop computer has Acrobat Pro installed. I'm not sure if that makes any difference.

[lbross]

We are already setting the font-family to 'Arial, Helvetica, sans-serif' in the pages that are rendered in Chrome. The client tries each of these in order until it finds a font that it has. Since you say it is using ArialMT and ArialBoldMT, it sounds like Chrome is already finding an Arial font.
We have total control over the fonts that are used in the pages that ESRI generates, much like we used to set the fonts on the layout elements of an .mxd. Do you want me to go through and change all of the fonts on the map documents to Arial? I can probably also change the fonts on the Excel-driven pages to Arial if they aren't already.

To save the html as pdf in Chrome, you open the html page in Chrome and then choose 'Print' and 'Save as PDF'. I don't know that this will exactly replicate what BAGIS is doing because BAGIS doesn't use your Chrome profile. And I'm also not sure if having Acrobat Pro makes a difference. I don't think that it would unless it supplies you some different fonts. But Windows components have a weird way of interacting sometimes.

[jdduh]

Let's do nothing for now since we have a workaround. I will test if the language pack is indeed the culprit. This hasn't been an issue in the past. I believe that I had generated reports without any scrambled texts on this new computer. I will keep monitoring the issue (after future Chrome updates). Thanks for your time investigating this.