Initial Draft For Revamped Echoflow Design

README.md

@@ -1,75 +1,101 @@
-# echoflow
+## Echoflow: Streamlined Data Pipeline Orchestration
 
-**NOTE: This project is currently under heavy development,
-and has not been tested or deployed on the cloud**
+Welcome to **Echoflow**! Echoflow is a powerful data pipeline orchestration tool designed to simplify and enhance the execution of data processing tasks. Leveraging the capabilities of [Prefect 2.0](https://www.prefect.io/) and YAML configuration files, Echoflow caters to the needs of scientific research and data analysis. It provides an efficient way to define, configure, and execute complex data processing workflows.
 
-Sonar conversion pipeline tool with echopype.
-This tool allows for users to quickly setup sonar data processing pipelines,
-and deploy it locally or on the cloud. It uses [Prefect 2.0](https://www.prefect.io/),
-a data workflow orchestration tool to run these flows on various platforms.
+Echoflow integrates with **echopype**, a renowned package for sonar data analysis, to provide a versatile solution for researchers, analysts, and engineers. With Echoflow, users can seamlessly process and analyze sonar data using a modular and user-friendly approach.
 
-## Development
 
-To develop the code, simply install the package in a python virtual environment in editable mode.
+# Getting Started with Echoflow
+
+This guide will walk you through the initial steps to set up and run your Echoflow pipelines.
+
+## 1. Create a Virtual Environment
+
+To keep your Echoflow environment isolated, it's recommended to create a virtual environment using Conda or Python's built-in `venv` module. Here's an example using Conda:
+
+```bash
+conda create --name echoflow-env
+conda activate echoflow-env
+```
+
+Or, using Python's venv:
+
+```bash
+python -m venv echoflow-env
+source echoflow-env/bin/activate  # On Windows, use `echoflow-env\Scripts\activate`
+```
+
+## 2. Clone the Project
+Now that you have a virtual environment set up, you can clone the Echoflow project repository to your local machine using the following command:
+
+```bash
+git clone <repository_url>
+```
+
+## 3. Install the Package
+Navigate to the project directory you've just cloned and install the Echoflow package. The -e flag is crucial as it enables editable mode, which is especially helpful during development and testing. Now, take a moment and let the echoflow do its thing while you enjoy your coffee.
+
+```bash
+cd <project_directory>
+pip install -e .
+```
+
+## 4. Echoflow and Prefect Initialization
+
+To kickstart your journey with Echoflow and Prefect, follow these simple initialization steps:
+
+### 4.1 Initializing Echoflow
+Begin by initializing Echoflow with the following command:
 
 ```bash
-pip install -e .[all]
+echoflow init
 ```
+This command sets up the groundwork for your Echoflow environment, preparing it for seamless usage.
 
-This will install all of the dependencies that the package need.
-
-**Check out the [Hake Flow Demo](./notebooks/HakeFlowDemo.ipynb) notebook to get started.**
-
-## Package structure
-
-All of the code lives in a directory called [echoflow](./echoflow/).
-
-Under that directory, there are currently 4 main subdirectory:
-
-- [settings](./echoflow/settings/): This is where pipeline configurations object models are found,
-as well as a home for any package configurations.
-  - [models](./echoflow/settings/models/): This sub-directory to `settings` contains [pydantic](https://docs.pydantic.dev/) models to validate the configuration file specified by the user.
-  This can look like below in [YAML](https://yaml.org/) format.
-
-    ```yaml
-    name: Bell_M._Shimada-SH1707-EK60
-    sonar_model: EK60
-    raw_regex: (.*)-?D(?P<date>\w{1,8})-T(?P<time>\w{1,6})
-    args:
-      urlpath: s3://ncei-wcsd-archive/data/raw/{{ ship_name }}/{{ survey_name }}/{{ sonar_model }}/*.raw
-      # Set default parameter values as found in urlpath
-      parameters:
-        ship_name: Bell_M._Shimada
-        survey_name: SH1707
-        sonar_model: EK60
-      storage_options:
-        anon: true
-    transect:
-        # Transect file spec
-        # can be either single or multiple files
-        file: ./hake_transects_2017.zip
-    output:
-      urlpath: ./combined_files
-      overwrite: true
-    ```
-
-    This yaml file turns into a `MainConfig` object that looks like:
-
-    ```python
-    MainConfig(name='Bell_M._Shimada-SH1707-EK60', sonar_model='EK60', raw_regex='(.*)-?D(?P<date>\\w{1,8})-T(?P<time>\\w{1,6})', args=Args(urlpath='s3://ncei-wcsd-archive/data/raw/{{ ship_name }}/{{ survey_name }}/{{ sonar_model }}/*.raw', parameters={'ship_name': 'Bell_M._Shimada', 'survey_name': 'SH1707', 'sonar_model': 'EK60'}, storage_options={'anon': True}, transect=Transect(file='./hake_transects_2017.zip', storage_options={})), output=Output(urlpath='./combined_files', storage_options={}, overwrite=True), echopype=None)
-    ```
-
-- [stages](./echoflow/stages/): Within this directory lives the code for various stages within the sonar data processing pipeline, which is currently sketched out and discussed [here](https://github.com/uw-echospace/data-processing-levels/blob/main/discussion_2022-07-12.md).
-- [subflows](./echoflow/subflows/): Subflows contains flows that support processing level flows.
-Essentially this is the individual smaller flows that need to run within a data processing level.
-  - Currently, each subflow is a directory that contains the following python files:
-    - `flows.py`: Code regarding to flow lives here
-    - `tasks.py`: Code regarding to task lives here
-    - `utils.py`: Code that is used for utility functions lives here
-    - `__init__.py`: File to import flow so the subflow directory can become a module and flow to be easily imported.
-- [tests](./echoflow/tests/): Tests code lives in this directory.
-
-For more details about prefect, go to their extensive [documentation](https://docs.prefect.io/).
+### 4.2 Initializing Prefect
+For Prefect, initialization involves a few extra steps, including secure authentication. Enter the following command to initiate the Prefect authentication process:
+
+- If you have a Prefect Cloud account, provide your Prefect API key to securely link your account. Type your API key when prompted and press Enter.
+
+```bash
+prefect cloud login
+```
+
+- If you don't have a Prefect Cloud account yet, you can use local prefect account. This is especially useful for those who are just starting out and want to explore Prefect without an account.
+
+```bash
+prefect profiles create echoflow-local
+```
+
+The initialization process will ensure that both Echoflow and Prefect are properly set up and ready for you to dive into your cloud-based workflows.
+
+## 5. Configure Blocks
+Echoflow utilizes the concept of [blocks](./docs/configuration/blocks.md) which are secure containers for storing credentials and sensitive data. If you're running the entire flow locally, feel free to bypass this step.To set up your cloud credentials, configure blocks according to your cloud provider. For detailed instructions, refer to the [Blocks Configuration Guide](./docs/configuration/blocks.md#creating-credential-blocks).
+
+## 6. Edit the Pipeline Configuration
+Open the [pipeline.yaml](./docs/configuration/pipeline.md) file. This YAML configuration file defines the processes you want to execute as part of your pipeline. Customize it by adding the necessary stages and functions from echopype that you wish to run.
+
+## 7. Define Data Sources and Destinations
+Customize the [datastore.yaml](./docs/configuration/datastore.md) file to define the source and destination for your pipeline's data. This is where Echoflow will fetch and store data as it executes the pipeline.
+
+## 8. Execute the Pipeline
+You're now ready to execute your Echoflow pipeline! Use the echoflow_start function, which is a central piece of Echoflow, to kick off your pipeline. Import this function from Echoflow and provide the paths or URLs of the configuration files. You can also pass additional options or storage options as needed. Here's an example:
+
+Customize the paths, block name, storage type, and options based on your requirements.
+
+
+```python
+from echoflow import echoflow_start, StorageType, load_block
+
+dataset_config = # url or path of datastore.yaml
+pipeline_config = # url or path of pipeline.yaml
+logfile_config = # url or path of logging.yaml (Optional)
+
+aws = load_block(name="<block_name>", type=<StorageType>)
+
+options = {"storage_options_override": False} # Enabling this assigns the block for universal use, avoiding the need for repetitive configurations when employing a single credential block throughout the application.
+data  = echoflow_start(dataset_config=dataset_config, pipeline_config=pipeline_config, logging_config=logfile_config, storage_options=aws, options=options)
+```
 
 ## License
 

deployment/deploy_echoflow_worker.sh

@@ -0,0 +1,56 @@
+#!/bin/bash
+
+# Step 1: Create a Python Virtual Environment
+python3 -m venv $HOME/env/echoflow-prod
+source $HOME/env/echoflow-prod/bin/activate
+
+# Step 2: Clone the Echoflow Repository
+cd $HOME/
+git clone https://github.com/OSOceanAcoustics/echoflow.git
+cd $HOME/echoflow
+
+# Step 3: Checkout the Dev Branch and Update (Optional) - Skip if using Prod/main branch
+git checkout dev
+git pull origin dev
+
+# Step 4: Install the Echoflow Project in Editable Mode
+pip install -e .
+
+# Step 5: Log in to Prefect Cloud and Set Your API Key - Change to step 5b if using prefect locally
+echo "Enter Prefect API key: "
+read prefectKey
+prefect cloud login -k $prefectKey
+
+# Step 5b: Setup prefect locally
+# prefect profile create echoflow-local
+
+# Step 6: Set Up the Prefect Worker as a Systemd Service
+echo "Enter Work Pool Name: "
+read workPool
+cd /etc/systemd/system
+
+# Create and edit the prefect-worker.service file
+sudo cat <<EOL > prefect-worker.service
+[Unit]
+Description=Prefect-Worker
+
+[Service]
+User=$(whoami)
+WorkingDirectory=$HOME/echoflow
+ExecStart=$(which prefect) agent start --pool $workPool
+Restart=always
+
+[Install]
+WantedBy=multi-user.target
+EOL
+
+# Step 7: Restart to to make systemd aware of the new service
+sudo systemctl daemon-reload
+
+# Optionally, enable the service to start at boot
+sudo systemctl enable prefect-worker.service
+
+# Step 8: Start the Prefect Worker Service
+sudo systemctl start prefect-worker.service
+
+echo "Setup completed. The Echoflow worker is now running. Send tasks to $workPool using Prefect UI or CLI."

docs/configuration/blocks.md

@@ -0,0 +1,105 @@
+# Echoflow Configuration and Credential Blocks
+
+Echoflow leverages the concept of "blocks" from Prefect, which serve as containers for storing various types of data, including credentials and sensitive information. Currently, Echoflow supports two types of blocks: Azure Cosmos DB Credentials Block and AWS Credentials Block. These blocks allow you to securely store sensitive data while benefiting from Prefect's robust integration capabilities.
+
+For a deeper understanding of blocks, you can refer to the [Prefect documentation](https://docs.prefect.io/2.11.5/concepts/blocks/).
+
+## Types of Blocks in Echoflow
+
+In the context of Echoflow, there are two main categories of blocks:
+
+### 1. Echoflow Configuration Blocks
+
+These blocks serve as repositories for references to credential blocks, as well as repositories for the various Prefect profiles that have been established using Echoflow's functions.
+
+### 2. Credential Blocks
+
+Credential blocks store sensitive information, such as authentication keys and tokens, securely. Echoflow integrates with Prefect's capabilities to ensure that sensitive data is protected.
+
+## Creating Credential Blocks
+
+Credential blocks can be conveniently created using an `.ini` file. By leveraging Prefect's integration, Echoflow ensures that the credentials stored in these blocks are handled securely. To create a credential block, you can follow these steps:
+
+1. Open the `credentials.ini` file, which is located under the `.echoflow` directory in your home directory.
+```bash
+# Terminal command
+cd ~/.echoflow
+```
+2. Place the necessary credential information within the `credentials.ini` file.
+```bash
+# Terminal command
+nano credentials.ini # Or use any of your favourite editors
+```
+3. Store the updated `.ini` file in the `.echoflow` directory, which resides in your home directory.
+4. Utilize [echoflow load-credentials](../../echoflow/stages/subflows/echoflow.py#load_credential_configuration) command to generate a new credential block, leveraging the content from the `.ini` file. 
+```bash 
+echoflow load-credentials
+```
+5. Add the name of the block in pipeline or datastore yaml configuration files under `storage_options` section with the appropriate storage type (refer [StorageType](../../echoflow/config/models/datastore.py#StorageType)).
+
+```yaml
+# Example
+storage_options:
+  block_name: echoflow-aws-credentials  # Name of the block containing credentials
+  type: AWS  # Specify the storage type using StorageType enum
+```
+
+By providing the block name and storage type, ensure that the correct block is used for storage operations, and maintain clarity regarding the chosen storage type.
+
+Once a credential block is created, it can be managed through the Prefect Dashboard. Additionally, if needed, you can use the `echoflow load-credentials` command with the `--sync` argument to ensure your blocks stay up-to-date with any changes made in the Prefect UI. This ensures that your configurations remain accurate and aligned across the application. **It is highly recommended to create new blocks whenever possible, as modifying existing blocks can lead to data loss or conflicts.** 
+
+## Considerations When Using `echoflow load-credentials`
+
+When utilizing the `echoflow load-credentials` command, be aware of the following considerations:
+
+- **Overwriting Values**: When using `echoflow load-credentials`, all the values from the `.ini` file will be written to the credential block, potentially overwriting existing values. Exercise caution when using this command to prevent unintentional data loss.
+- **Creating New Blocks**: To maintain data integrity and security, it's advised to create new blocks rather than modifying existing ones. If editing an existing block becomes necessary, it should be done through the Prefect Dashboard.
+- **Sync Argument**: The `--sync` argument is available in the `echoflow load-credentials` command. When set, this option syncs the credential block updates with the Prefect UI. This feature facilitates the seamless management of blocks through the dashboard, enhancing collaboration and control over credentials.
+
+By adhering to these guidelines, you can ensure the secure management of sensitive information while effectively configuring and utilizing Echoflow within your projects.
+
+
+# Configuration File Explanation: credentials.ini
+
+This Markdown file explains the structure and contents of the `credentials.ini` configuration file.
+
+## AWS Section
+
+The `[AWS]` section contains configuration settings related to AWS credentials.
+
+- `aws_access_key_id`: Your AWS access key.
+- `aws_secret_access_key`: Your AWS secret access key.
+- `aws_session_token`: AWS session token (optional).
+- `region_name`: AWS region name.
+- `name`: Name of the AWS credentials configuration.
+- `active`: Indicates if the AWS credentials are active (True/False).
+- `options`: Additional options for AWS configuration.
+
+## AzureCosmos Section
+
+The `[AZCosmos]` section contains configuration settings related to Azure Cosmos DB credentials.
+
+- `name`: Name of the Azure Cosmos DB credentials configuration.
+- `connection_string`: Azure Cosmos DB connection string.
+- `active`: Indicates if the Azure Cosmos DB credentials are active (True/False).
+- `options`: Additional options for Azure Cosmos DB configuration.
+
+Example:
+
+```ini
+[AWS]
+aws_key = my-access-key
+aws_secret = my-secret-key
+token = my-session-token
+region = us-west-1
+name = my-aws-credentials
+active = True
+option_key = option_value
+
+[AZCosmos]
+name = my-az-cosmos-credentials
+connection_string = my-connection-string
+active = True
+option_key = option_value
+```
+

docs/configuration/datastore.md

@@ -0,0 +1,59 @@
+# Echoflow Run Configuration Documentation
+
+This document provides detailed explanations for the keys used in the provided YAML configuration used to define an Echoflow run.
+
+## Run Details
+
+- `name`: This key specifies the name of the Echoflow run. It is used to identify and label the execution of the Echoflow process.
+- `sonar_model`: This key indicates the model of the sonar used for data collection during the run.
+- `raw_regex`: This key indicates the regex to be used while parsing the source directory to match the files to be processed.
+
+## Input Arguments
+
+- `urlpath`: This key defines the source data URL pattern for accessing raw data files. The pattern can contain placeholders that will be dynamically replaced during execution.
+- `parameters`: This section holds parameters used in the source data URL. These parameters dynamically replace placeholders in the URL path.
+- `storage_options`: This section defines storage options for accessing source data. It may include settings to anonymize access to the data.
+- `transect`: This section provides information about the transect data, including the URL of the transect file and storage options.
+- `json_export`: When set to true, this key indicates that raw JSON metadata of files should be exported for processing.
+- `raw_json_path`: This key defines the path where the raw JSON metadata will be stored. It can be used to skip parsing files in the source directory and instead fetch files from this JSON.
+
+## Output Arguments
+
+- `urlpath`: This key defines the destination data URL where processed data will be stored.
+- `overwrite`: When set to true, this key specifies that the data should overwrite any existing data in the output directory.
+- `storage_options`: This section defines storage options for the destination data, which may include details such as the block name and type.
+
+## Notes
+
+- The provided configuration serves as a structured setup for executing an Echoflow run, allowing customization through the specified keys.
+- Dynamic placeholders like `ship_name`, `survey_name`, and `sonar_model` are replaced with actual values based on the context.
+
+Example:
+
+```yaml
+name: Bell_M._Shimada-SH1707-EK60 # Name of the Echoflow Run
+sonar_model: EK60 # Sonar Model
+raw_regex: (.*)-?D(?P<date>\w{1,8})-T(?P<time>\w{1,6}) # Regex to parse the filenames
+args: # Input arguments
+  urlpath: s3://ncei-wcsd-archive/data/raw/{{ ship_name }}/{{ survey_name }}/{{ sonar_model }}/*.raw # Source data URL
+  parameters: # Source data URL parameters
+    ship_name: Bell_M._Shimada
+    survey_name: SH1707
+    sonar_model: EK60
+  storage_options: # Source data storage options
+    anon: true
+  transect: # Source data transect information
+    file: ./x0007_fileset.txt       # Transect file URL. Accepts .zip or .txt file
+    storage_options: # Transect file storage options
+      block_name: echoflow-aws-credentials # Block name. For more information on Blocks refer blocks.md
+      type: AWS # Block type 
+    default_transect_num: 1 # Set when not using a file to pass transect information
+  json_export: true # Export raw json metadata of files to be processed
+  raw_json_path: s3://echoflow-workground/combined_files/raw_json # Path to store the raw json metadata. Can also work to skip the process of parsing the files at source directory and fetch files present in this json instead.
+output: # Output arguments
+  urlpath: s3://echoflow-workground/combined_files_dask # Destination data URL parameters
+  overwrite: true # Flag to overwrite the data if present in the output directory
+  storage_options: # Destination data storage options
+    block_name: echoflow-aws-credentials
+    type: AWS
+```