v3.240508

What's Changed

Tag created mainly for new env variable in jobs for frontier monitoring and WMCore version 2.3.3

• define env.var. FRONTIER_ID Fix #8366 by @belforte in #8367
• requirements - wmcore upgraded to 2.3.3 by @mapellidario in #8383

but this tag brings also many other changes

Publisher:

• Publisher support for PyPI image by @novicecpp in #8368
• Deal with failed migrations 8244 by @belforte in #8378 (still buggy)

Corrupted files reporting (not part of CRAB services proper)

• Update ProcessBadFilesList.py require at least 3 failure reports by @belforte in #8379
• do not crash ProcessBadList.py in case of no reports by @belforte in #8365
• no need for a flag in /etc anymore. Use -p(arent) in gfal-copy by @belforte in #8369

TaskWorker

• only user reliable RSEs for tape recall. Fix #8356 by @belforte in #8381

Oracle

• Oracle - many new things cumulated over many months by @mapellidario in #8326

Full Changelog: v3.240426...v3.240508

v3.240426

• small fixes for CI/CI pipeline
• make DBSDataDiscovery test work and save output to JSON file
• add fix by David Lange to speedup parent finding (x100)

List of commits:

• attempt to streamline parent file discovery (#8352) by @davidlange6
• make test of DBSDataDiscovery work again and add printout of output (#8362) by @belforte
• allow to fall back proxy to installed cert in runner machine (#8359) by @novicecpp

List of difference: v3.240423...v3.240426

v3.240423

What's Changed

• Remove T2_DE_DESY hardcode and use generic RucioException for add_replicas() instead by @novicecpp in #8325
• Clean up cmsbot's Rucio storage space by @novicecpp in #8327
• stop using AuthenticatedSubprocess. Fix #8277 by @belforte in #8331
• PoC CI pipeline on Gitlab to verify PyPI image by @novicecpp in #8328
• use classad.quote, not HTCondorUtils. Fix #8332 by @belforte in #8346
• monit - run monit scripts inside docker containers by @mapellidario in #8194
• Fix deployTW.sh script, fix CI job tags to crab3 and crab3-shell by @novicecpp in #8348

Full Changelog: v3.240325...v3.240423

v3.240325

fix previous broken release v3.240318 via #8302

Validation
ran 3 Jenkins tests at same time

• CV + CCV + ST #8304
• CV+ CCV +ST #8305
• CCV + ST #8306

all OK.

v3.240318 - BROKEN

should work on alma9 schedulers

• use classad.parseAds - fix #8285 (#8296)
• avoid empty lines in finished_jobs/ files. For #8285 (#8287)
• make dag_bootstrap.sh support alma9. Fix #8284 (#8286)

This release is broken, see #8299 (comment)

v3.240313

Alma9 support (server), new production-ready PyPI, WMCore version to 2.3.2rc3 to support alma9 job, and small fixes.

What's Changed

• PYPI proof-of-concept by @novicecpp in #8088
• skip FileTransferEvent msgs in cache_status.py by @belforte in #8262
• correct () in logging statement. Fix #8259 by @belforte in #8260
• port JobAutoTuner to py3 and keep only overflow. by @belforte in #8268
• Make task proc wrapper alma9 compatible for 8256 by @belforte in #8271
• make dag_bootstrap_startup work on alma9. For #8256 by @belforte in #8270
• Bring back ./start.sh -g -d by @novicecpp in #8254
• Fix bug from PR#8254 by @novicecpp in #8275
• Bump cmsweb TW to 20230901-stable and WMCore to 2.3.2rc3 by @novicecpp in #8278

Full Changelog: v3.240219...v3.240313

v3.240219

fix bug in latest commit/tag

• fix name of rucio creds in config. Fix #8245

difference from lates tag: v3.240216...v3.240219

Validation: #8249

v3.240216

Small fixes and has reporting of ReadBranches based on secret file available on all Schedulers. Aimed for deployment in production.

List of commit:

• New format for monit secrets fix 8231 (#8234)
• protect against empty/malformed json reports (#8237)
• handle RSE not found in getWritePfn fix #7107 fix #8230 (#8233)
• report ReadBranches to crab-test index in MONIT. Fix #7932 (#8214)

Full differences from last version: v3.240201...v3.240216

Validation: #8239

v3.240201

this captures changes to job wrapper CMSRunAnalysis.py which
introduces a new way of passing arguments when using preparelocal

There is also a minor fix when using Rucio containers in place of DBS datasets as input

Tag as a release to facilitate validation

List of commits:

• only set a default for missing args. Fix #8208 (#8209)
• CMSRunAnalysi.py takes JSON file as arg Fix #8196 (#8198)
• remove scope in case inputDataset is a Rucio container. Fix #7769 (#8193)

Full list of changes from last version:
v3.240122...v3.240201

Deloyed on crab-preprod-tw01

Validation with crab-prod client:

• #8221 CMSSW_13_0_2 all test
• #8220 CMSSW_10_6_26 CCV
• #8221 CMSSW_10_1_0 CCV
• #8222 CMSSW_8_0_36 CCV and StatusTracking
• #8223 CMSSW_7_6_7 CCV

ALL OK

v3.240122

Main goal of this release is to introduce an reporting of ReadBranches

• report numb of read branches as classAd For #7932 (#8190)

There is also a small fix

• allow different scope for dataset. Fix #7739 (#8187)

And a bunch of Jenkins related commits so that they work with all CMSSW releases and Status Tracking test now runs

• Jenkins should not check publication when taskname says so. For #8185 (#8191)
• use getUsername from CRABClient.UserUtilities fix #8179 (#8180)
• Update TaskSubmissionStatusTracking.sh
• Make it easy to run status tracking in singularity (#8175)
• Make it easy to run status tracking in singularity (#8174)
• add x permission (#8173)
• add driving bash script for StatusTracking (#8171)
• adapt StatusTracking test to new Client. Fix #8169 (#8170)
• make sure CV test detects usage of wrong python

Comparison with latest release: v3.240111...v3.240122