v3.241113.p1

a more clean fix. Also better file names. Already tested in crab-prod-tw01 which is now running from GH master

What's Changed

• Fix directory for suspicious file reporting by @belforte in #8787

Full Changelog: v3.241113...v3.241113.p1

v3.241113

small bug fix in sending suspicious file reports to EOS

What's Changed

• add missing / between directory and file name by @belforte in #8786

Full Changelog: v3.241112...v3.241113

v3.241112

Need a new tag to re-run release pipeline.

Also add an improvement to suspicious input file reporting: one directory per task to avoid O(100k++) files in a single EOS directory.

What's Changed

• when reporting suspicious files, use one directory per task by @belforte in #8783

Full Changelog: v3.241108...v3.241112

v3.241108

fixes automatic splitting which was broken in last release.

What's Changed

• fix splittingSummary for automatic splitting. Fix #8774 by @belforte in #8776

Full Changelog: v3.241107...v3.241108

validation

• release pipeline (uses v3.241030 client) https://gitlab.cern.ch/crab3/CRABServer/-/pipelines/8474478
  stuck on Rucio transfers, wait and retry later. Eventually OK
• CV+CCV with prod client v3.241030 #8777 - OK
• CV+CCV with new dev-client (from GH) v3.241107 #8778 - OK
• CV with dev-client from IB (now at v3.241107) #8780 - OK
• CCV with dev-client from IB (now at v3.241107) #8782 - OK

v3.241107

Main goal change is sandbod handling restructuring for implementing dryrun via preparelocal

Also some cleanup

What's Changed

• enable opensearch ssl_verify back and set PYTHONUNBUFFERED for run.py by @novicecpp in #8746
• remove debug_files handling in TW. Do it in scheduler by @belforte in #8747
• Spelling and missing 'not' in exception message by @oljemark in #8751
• remove duplicates sandbox untarring. Fix #8725 by @belforte in #8753
• CI pipeline for test2 to use crab-dev from IB by @belforte in #8756
• Cleanup deployment files by @novicecpp in #8757
• use dm_weight Fix #8724 by @belforte in #8762
• Make dryrun work again by @belforte in #8767

Full Changelog: v3.241018...v3.241107

v3.241018

implements first part of #6544 : restructure code to avoid handling user sandbox in TaskWorker, but download it directly in the scheduler DAG boostrap process. Just minor, changes of code from @novicecpp in #8645 plus rebasing

Also check that object is visible in S3 after uploading

What's Changed

• add checkS3Object and verify S3 upload with it by @belforte in #8740
• Download sandbox from s3 to scheduler by @belforte in #8743
• Stop downloadind user sandbox in tw by @belforte in #8745

BEWARE

last commit above breaks compatibility with current prod client v3.240930
a compatible Client comes with v3.241025
CRABClient MUST be updated before we can deploy this or later versions of TW

Full Changelog: v3.241017...v3.241018

validation pipeline https://gitlab.cern.ch/crab3/CRABServer/-/pipelines/8349112

v3.241017

What's Changed

• Run spark cronjobs on ipynb directly by @novicecpp in #8735
• correct use of rucio.get_local_account_usage. Fix #8738 by @belforte in #8739
• adding dryrun option for task scheduling by @aspiringmind-code in #8734
• Changing the message when no tasks selected by @aspiringmind-code in #8742
• Various fix for process controller scripts (manage.sh/manage.py/etc.) by @novicecpp in #8737
• RucioUtils instead of python.RucioUtils by @aspiringmind-code in #8744

Full Changelog: v3.241009...v3.241017

v3.241009

This release brings in small fixes, cleanup of unnecessary files and some restructuring.

What's Changed

• fix object has no attribute in manage.py by @novicecpp in #8690
• fix the X509_USER_CERT path, and also add X509_USER_PROXY here, by @novicecpp in #8684
• Enable install WMCore from private repository by @novicecpp in #8683
• Fix #8661 changing local account usage to tape recall usage in quota report by @aspiringmind-code in #8697
• Fix #8663 adding getsubmitrefusedtasks api by @aspiringmind-code in #8698
• Fix retag monit image in release pipeline by @novicecpp in #8706
• fix #8703 and minor improv. to CI by @belforte in #8708
• Cleanup job wrapper by @belforte in #8704
• Remove extra files from TW tarballs by @belforte in #8718
• remove use of site.ad file Fix #8720 by @belforte in #8721
• Add Waiting task status to GenerateMonit by @aspiringmind-code in #8729

Full Changelog: v3.240904...v3.241009

v3.240904

What's Changed

• fix pipeline by @aspiringmind-code in #8637
• Fixing registry caching for monit image by @novicecpp in #8634
• make runContainer.sh help works by @novicecpp in #8639
• Add bash-completion to pypi container by @novicecpp in #8641
• add instuctions for TW logstash config by @belforte in #8642
• monit - set CMD to bash by @mapellidario in #8640
• monit - runcontainer improved by @mapellidario in #8643
• jenkins test - add client validation test for crab recover by @mapellidario in #8087
• add comments by @belforte in #8633
• fix .gitlab-ci.yml comment by @novicecpp in #8649
• Change CI variables name for triggering the pipelines by @novicecpp in #8653
• monitoring image - some new additions by @mapellidario in #8656
• stop using username_CRAB for myproxy. Fix #8655 by @belforte in #8657
• No locations found should result in SubmissionRefused by @belforte in #8662
• fix logging format by @belforte in #8668
• add test14 by @novicecpp in #8669
• fix test14 by @novicecpp in #8670
• Fix rucio cleanup.sh and add SUBMITREFUSED to ST by @novicecpp in #8671
• spark monitoring cronjobs - a small refactor by @mapellidario in #8672
• use CRAB_Requested* ads in resubmission. Fix #8654 by @belforte in #8674
• suspend tape recall rule before deleting. Fix #8647 by @belforte in #8652
• Refactor process controller scripts by @novicecpp in #8658
• add status subresource to RESTTask and use it in task_info.js by @belforte in #8680
• fix 8658: manage.py which cause manage.sh always run on debug mode by @novicecpp in #8682
• bump WMCore to 2.3.rc2 by @belforte in #8687
• Task scheduling by @aspiringmind-code in #8676

Full Changelog: v3.240819...v3.240904

v3.240809.p1

simply bump WMCore to 2.3.6rc2 to pick up fix to voms-proxy-info concurrency issue

What's Changed

• bump WMCore to 2.3.6rc2 (#8687) by @belforte in #8688

Full Changelog: v3.240809...v3.240809.p1