Adding Extensions to Support Main Pipeline

[Sohambutala]

These extensions are designed to trigger or support the main pipeline, aiding in data source acquisition, enabling concurrent deployments etc.

1. File Monitoring for Automated Deployments:

Purpose: To monitor file changes and trigger specific deployments.

Current Approach:

• Schedule: watchdog is not used currently as we only need to schedule at regular intervals. This can be changed later, or a separate extension can be added for real-time scenarios.
• Deployment Independence: Each file triggers a new deployment, and a failure in one deployment does not affect others.

Implementation Details:

• Tracking: Utilizes Prefect Blocks to record the last successful run time and the list of processed files.
• Concurrency: Leverages task concurrency to run multiple deployments simultaneously, allowing for concurrent file processing.
• Fail-Safe Option: The fail_safe option toggles between concurrent and sequential processing. In sequential mode, a failure in one file stops the processing of remaining files. In concurrent mode, file processing is independent.

Considerations:
Concurrency must be managed carefully when appending results to a single Zarr store to avoid race conditions and maintain the order of data.

2. File Downloader for Seamless Data Transfer:

Purpose: To facilitate data transfer between two locations.

Implementation Details:

• Tracking: Checks the destination folder for existing files and skips the transfer if the file is already present.
• Concurrency: Employs task concurrency to download multiple files simultaneously. Concurrency can be managed by configuring the global concurrency limit (edf-data-transfer).
• Delete Option: Offers an option to delete files after transfer, with a default setting of false.

Considerations:
Be mindful of rate limitations of both the source and destination. Concurrent requests can trigger rate limits.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 7.19719% with 1586 lines in your changes missing coverage. Please review.

Project coverage is 31.68%. Comparing base (f85b12e) to head (83bb7d1).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...chodataflow/stages/subflows/initialization_flow.py	9.28%	215 Missing ⚠️
echodataflow/extensions/file_monitor.py	0.00%	193 Missing ⚠️
echodataflow/stages/subflows/echoshader_flow.py	0.00%	144 Missing ⚠️
...low/stages/subflows/echoshader_flow_predictions.py	0.00%	141 Missing ⚠️
echodataflow/stages/subflows/mask_prediction.py	0.00%	120 Missing ⚠️
echodataflow/extensions/file_downloader.py	0.00%	99 Missing ⚠️
echodataflow/extensions/panel_app_lasker.py	0.00%	98 Missing ⚠️
echodataflow/extensions/panel_app_shimada.py	0.00%	97 Missing ⚠️
echodataflow/extensions/panel_app.py	0.00%	80 Missing ⚠️
echodataflow/stages/subflows/Sv_pipeline.py	0.00%	80 Missing ⚠️
... and 20 more

❗ There is a different number of reports uploaded between BASE (f85b12e) and HEAD (83bb7d1). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (f85b12e)	HEAD (83bb7d1)
	2	1

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #99       +/-   ##
===========================================
- Coverage   50.50%   31.68%   -18.82%     
===========================================
  Files          34       48       +14     
  Lines        2285     3888     +1603     
===========================================
+ Hits         1154     1232       +78     
- Misses       1131     2656     +1525     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Sohambutala]

Converting this PR to draft since Prefect variables have a limit of 255 characters. An alternative would be to try using the Echodataflow block object.