Add more traceability from pipeline results back to source code

[bberkeyU]

Can we add some information to the pipeline results to add traceability/information and track which version of the source code was run to create the data? The appropriate data may be captured somewhere; so this may just be education/documentation.

I think it would be helpful to find the have the commit signature saved in a file in the process directory. This might be especially interesting if we are going back and looking at the process.latest directory if anytime has passed since we created that directory.

[mgalloy]

There are two section of the primary FITS header of the level 1 and 2 files that tries to provide all the information about the processing that was done to process the file.

First, the "Level 1 processing info" section:

COMMENT --- Level 1 processing info ---
DOI     = 'https://doi.org/10.26024/g8p7-wy42' / Digital Object Identifier
DATE_DP = '2024-04-16T03:42:44' / [UT] L1 processing date/time
DPSWID  = '1.0.5 [b1417f0]'    / L1 processing software (2024-04-15) [production
LIN_CRCT=                    F / camera linearity corrected
BOPAL   =                14.80 / [B/Bsun] opal radiance
HOTPIXC =                    T / whether hot pixels were corrected
DEMODV  =                    1 / demod coeffs version [2023-05-15T10:07:04Z]
DISTORTC=                    T / whether distortion corrected
CONTSUB =                    T / whether the continuum was subtracted
CAMERAS = 'both    '           / cameras used in processing

In particular, DPSWID can be used to retrieve the exact software used to produce the file. It includes the code version and commit hash in git.

Also, the "HISTORY" section at the end of the primary header gives the exact steps performed in the level 1 processing:

HISTORY
HISTORY Level 1 calibration and processing steps:
HISTORY   - quality check to determine if the file should be processed
HISTORY   - average level 0 data with same onband and wavelength
HISTORY   - apply dark correction
HISTORY   - apply gain correction
HISTORY   - camera corrections such as hot pixel correction
HISTORY   - demodulation
HISTORY   - distortion correction
HISTORY   - find the occulter position and radius
HISTORY   - subtract continuum
HISTORY   - remove hoizontal/vertical bands
HISTORY   - center images using occulter position and rotate to north up
HISTORY   - combine the cameras
HISTORY   - polarimetric correction
HISTORY   - correct for sky transmission
HISTORY   - update FITS keywords

[mgalloy]

Somewhat related, is that YYYYMMDD.ucomp.cfg is created in the root of the process directory, which is the configuration file for the run. So this lets you look up options that were used for the processing.

[bberkeyU]

Thanks Mike.

I think I was picturing adding something like the:
DPSWID = '1.0.5 [b1417f0]' / L1 processing software (2024-04-15) [production
string into the YYYYMMDD.ucomp.cfg file.

[mgalloy]

We could have a VERSION file with the version and git commit hash for the software in the root of the process directory as well. This would be the same information as DPSWID, but easier to read quickly.

[bberkeyU]

A version file would also work if we didnt want to edit the cfg file.

[bberkeyU]

Since we aren't limited to 80 characters could the VERSION information have the whole git commit I think this will make it easier to jump to the commit get hub:
ie:
5d4063f vs 5d4063f
So we could jump to:
5d4063f

[bberkeyU]

Thats annoying github automatically changed the short and long commit id's to links and displayed them as links.

[bberkeyU]

Actually both of these go to the correct place.
https://github.com/NCAR/ucomp-pipeline/commit/ %short id%
https://github.com/NCAR/ucomp-pipeline/commit/ %long id%

So I am ok/happy with the short commit id's as listed in the INSTALLATION and VERSION files.

[mgalloy]

An even more detailed file is the INSTALLATION file from the build. For example, here is the production INSTALLATION file:

ucomp-pipeline$ cat INSTALLATION
Source directory    : /hao/acos/sw/src/ucomp-pipeline
Version             : 1.0.5
Revision            : b1417f0
Branch              : production
Last commit         : Mon Apr 15 19:09:44 2024
Configure date      : Mon Apr 15 19:12:46 2024

We could copy that to the process directory.

[mgalloy]

I am adding the distortion and hot pixel filenames to the headers because that has been an issue for me recently with quickly changing calibration-related files. See #273.