Initial Draft For Revamped Echoflow Design

[Sohambutala]

• Revamped configuration files
• Separated flows and tasks based on API defined in Echopype
• Organized utility functions based on their corresponding functionalities.

[valentina-s]

Thanks for adding the notebooks with nice explanations @Sohambutala! I tried the LocalEchoflowDemo.ipynb and here are a few notes on the experience:

• first I needed to create some profile to initiate .profile folder. I ran prefect profile create test
• I also was missing the echoflow init step and got the error about the missing .echoflow/echoflow_rules.txt. I ran it through python echoflow/echoflow_cli.py init and it worked, so I assume you had some alias set up. Could add a reminder in the notebook that people should have gone through these steps.
• I got some connection time out errors. In the end I limited to 10 files and the command completed. My laptop had at least 24GB of free RAM at that moment.

* I retried today with 100 files, and it is working, i.e. I obtained the individual objects but I still had some connection refused error:

Anyway, it could be RAM dependence.

• You can also some explanation of what the data object was: It is a list of one entry with 10 elements (the xarray datasets) corresponding to the files.

[Sohambutala]

Thank you @valentina-s! Based on your feedback/comments here is the revision:

1. Added setup and initialization instructions in the offline notebook.
2. Resolved the "connection refused" issue stemming from the outdated method used to fetch previous flows from the Prefect REST API.
3. Introduced a shell script that can be executed on any machine to initiate echoflow as a systemd process.

[valentina-s]

Works great! Thank you so much!