Possible issue with group_by or mpi.

[nicholasesposito]

There appears to be an issue with group_by within obsforge or the bufr-query.

I've been using the obsforge to create netcdf and/or ioda files using ADPSFC, SFCSHP, and ACFT_PROFILES prepbufr files as inputs. When I run my ADPSFC and SFCSHP yamls/ scripts/ and associated mapping files, the number of obs in the output are the same regardless of number of processors used. ADPSFC has 149183 data Locations using the bufr_backend, script_backend, bufr2netcdf, and script2netcdf, and regardless of how many processors I use. With the same conditions, SFCSHP runs result in 27144 Locations across the board.

Acft_profiles, however, is all over the place. Notably aircraft_profiles uses "group_by_variable" in the mapping yaml.
There are 3144128 expected data Locations, as in previous versions of bufr_backend and iodaconv. This only shows up when I use bufr2netcdf and script2netcdf with 1 processor. If I increase the mpi, the number of data Locations decreases (when I run 4, the Locations is reduced to 2958349. The decrease is even more apparent when I run with bufr_backend and script_backend. When I run these, regardless of mpi, the number of Locations is 122711, or ~4% of what it should be.

For reference, bufr2netcdf and bufr_backend JUST test the mapping, while the script2netcdf and script_backend test the python scripts. bufr_backend and script_backend produce IODA files while bufr2netcdf and script2netcdf produce regular netcdf files.

There could be multiple issues, but I can't tell. Let me know if you have further questions. Thanks

[nicholasesposito]

@rmclaren just opened this issue. I am unable to assign issues in this repository. Thanks for looking at this.