1.5.0 - Improvements

📦  New Features

• #114 Add support for the same pressure calibration as the Proffast Pylot (passing factors and offsets; see "job settings")

• #108 Ability to specifiy alternate local config location (Credits to @cfleur)

🧵  Refactoring and Updates

• Switching from YAPF to Ruff formatting and linting
• Upgrading to the metadata library to the refactored and simplified version that supports shorter names as well (lid instead of location_id, sid instead of sensor_id, from_dt instead of from_datetime, etc.).

🪲  Bugfixes

• #107 Parsing pressure files fails when it contains null values (this was already fixed in 1.4.1)

• Debug pressure loading and retrieval queue for some edge cases

v1.4.0 - General Usability

📦  New Features

• #97 Fully configurable interface to ground pressure files. Now the pipeline can work with almost any format you store your ground pressure data in. Read about it in the documentation.

• #101 New bundle script that works for all retrieval algorithms. It bundles the data into one file per sensor and retrieval alg./met. profile. Read about it in the documentation.

• #105 Add DC amplitudes to output of Proffast2.4 preprocessor

🧵  Refactoring and Updates

• #60 Removed the calibration factors from the metadata. They have not been used yet and since every team has their own way of calibrating their data, it is hard to set a standard in here.

• #102 Make Interferogram File Regex more flexible (now also supports the placeholders $(YYYY), $(YY), $(MM), and $(DD).

• Polish the profiles downloader logic

🪲  Bugfixes

• Handle directories with no or only corrupt interferograms. Until now, a container failed and did not produce a "failed" output directory.

• #99 Fix failing retrieval start command when pipeline finished in under 0.5 seconds

As always, you can find the exact config options in the API Reference of the Configuration.

v1.3.0 - Support other Python versions

📦  New Features

• Add support for Python 3.10 and above. Previously this project was fixed to 3.11. The reason is that some shared computing environments do not have the newest Python versions available.
• Run CI tests for all supported Python versions (long tests only run for 3.10)
• Add support for internal version Proffast 2.4.1

🧵  Refactoring and Updates

• Atmospheric profiles scope is nullable now (in case you only want to download standard sites)

🪲  Bugfixes

• Automatically create subdirectories for atmospheric profiles (credits to Catherine Fait (FMI) for reporting)
• GGG2020 Profiles from standard sites can now be downloaded independently of the configured scope (manual requests)

1.2.0 - Improved OPUS File Prefiltering, Additional Retrieval Job Options

📦  New Features

• Add option to pass custom ILS values
• Add option to specify output folder suffixes
• Add ETA to jobs watcher

🧵  Refactoring and Updates

• Upgrade to new OPUS file prefiltering based on Proffast 2.4. This OPUS file validator can be used standalone using the tum-esm-utils library

1.1.0 - Proffast 2.4 Support and Other Improvements

This release adds a lot of minor improvements and bugfixes as well as support for Proffast2.4.

📦  New Features

• Store VMR and pT profiles for all Proffast retrievals
• Support storing binary spectra in Proffast 1.0 retrievals too (only 2.X support until now)
• Add support for Proffast 2.4

🧵  Refactoring and Updates

• Update ILS parameter list for all retrieval algorithms
• Improve documentation
• Improve the pipeline watcher (#93)
• Update paths in GGG2020 downloader after the ginput paths have changed

🪲  Bugfixes

• Fix bugs in Proffast 1.0 wrapper
• Fix bugs in GGG2014 profiles downloader
• Don't run CI container tests in parallel (CI nodes were running out of memory)

1.0.0 - Initial Public Release

We retrieve a lot of EM27/SUN data, produced mainly by MUCCnet (Dietrich et al., 2021), and have used this pipeline since end of 2021.

This codebase provides an automated data pipeline for Proffast 1 and 2.X. Under the hood, it uses the Proffast Pylot to interact with Proffast 2 and an in-house connector to interact with Proffast 1. Whenever using this pipeline for Proffast retrievals, please make sure to also cite Proffast and the Proffast Pylot (for Proffast 2.X retrievals).

📚 Read the documentation at em27-retrieval-pipeline.netlify.app.
💾 Get the source code at github.com/tum-esm/em27-retrieval-pipeline.
🐝 Report Issues or discuss enhancements using issues on GitHub.

v1.0.0-rc.1

The next release candidates will still implement some features and a lot of documentation. Everything that is tracked by the milestone v1.0.0 will be realised by v1.0.0. I will just have release candidates now to better keep track of the progress.