Dev Update

.github/workflows/main.yaml

@@ -18,7 +18,7 @@ jobs:
       uses: actions/setup-python@v2
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Install echoflow
+    - name: Install echodataflow
       run: pip install .[all]
     - name: Run unit tests
       env:

.github/workflows/python-publish.yml

@@ -6,7 +6,7 @@
 # separate terms of service, privacy policy, and support
 # documentation.
 
-name: echoflow-dev
+name: echodataflow-pypi-publish
 
 on:
   release:
@@ -37,3 +37,4 @@ jobs:
       with:
         user: __token__
         password: ${{ secrets.PYPI_API_TOKEN }}
+

Dockerfile

@@ -4,15 +4,15 @@ FROM prefecthq/prefect:2-python3.10
 # Add our requirements.txt file to the image and install dependencies
 COPY requirements.txt .
 
-# Update setuptools and pip before installing echoflow
+# Update setuptools and pip before installing echodataflow
 RUN pip install --upgrade pip setuptools
 
-RUN pip install --no-cache-dir --trusted-host pypi.python.org echoflow
+RUN pip install --no-cache-dir --trusted-host pypi.python.org echodataflow
 # RUN pip install -r requirements.txt --trusted-host pypi.python.org --no-cache-dir
-RUN echoflow init
+RUN echodataflow init
 
 
 # Run our flow script when the container starts
-CMD ["python", "-m", "echoflow.docker_trigger.py"]
+CMD ["python", "-m", "echodataflow.docker_trigger.py"]
 
 EXPOSE 4200

README.md

@@ -14,15 +14,15 @@ This guide will walk you through the initial steps to set up and run your Echoda
 To keep your Echodataflow environment isolated, it's recommended to create a virtual environment using Conda or Python's built-in `venv` module. Here's an example using Conda:
 
 ```bash
-conda create --name echoflow-env
-conda activate echoflow-env
+conda create --name echodataflow-env
+conda activate echodataflow-env
 ```
 
 Or, using Python's venv:
 
 ```bash
-python -m venv echoflow-env
-source echoflow-env/bin/activate  # On Windows, use `echoflow-env\Scripts\activate`
+python -m venv echodataflow-env
+source echodataflow-env/bin/activate  # On Windows, use `echodataflow-env\Scripts\activate`
 ```
 
 ## 2. Clone the Project
@@ -48,7 +48,7 @@ To kickstart your journey with Echodataflow and Prefect, follow these simple ini
 Begin by initializing Echodataflow with the following command:
 
 ```bash
-echoflow init
+echodataflow init
 ```
 This command sets up the groundwork for your Echodataflow environment, preparing it for seamless usage.
 
@@ -64,7 +64,7 @@ prefect cloud login
 - If you don't have a Prefect Cloud account yet, you can use local prefect account. This is especially useful for those who are just starting out and want to explore Prefect without an account.
 
 ```bash
-prefect profiles create echoflow-local
+prefect profiles create echodataflow-local
 ```
 
 The initialization process will ensure that both Echodataflow and Prefect are properly set up and ready for you to dive into your cloud-based workflows.
@@ -79,13 +79,13 @@ Open the [pipeline.yaml](./docs/configuration/pipeline.md) file. This YAML confi
 Customize the [datastore.yaml](./docs/configuration/datastore.md) file to define the source and destination for your pipeline's data. This is where Echodataflow will fetch and store data as it executes the pipeline.
 
 ## 8. Execute the Pipeline
-You're now ready to execute your Echodataflow pipeline! Use the echoflow_start function, which is a central piece of Echodataflow, to kick off your pipeline. Import this function from Echodataflow and provide the paths or URLs of the configuration files. You can also pass additional options or storage options as needed. Here's an example:
+You're now ready to execute your Echodataflow pipeline! Use the echodataflow_start function, which is a central piece of Echodataflow, to kick off your pipeline. Import this function from Echodataflow and provide the paths or URLs of the configuration files. You can also pass additional options or storage options as needed. Here's an example:
 
 Customize the paths, block name, storage type, and options based on your requirements.
 
 
 ```python
-from echoflow import echoflow_start, StorageType, load_block
+from echodataflow import echodataflow_start, StorageType, load_block
 
 dataset_config = # url or path of datastore.yaml
 pipeline_config = # url or path of pipeline.yaml
@@ -94,7 +94,7 @@ logfile_config = # url or path of logging.yaml (Optional)
 aws = load_block(name="<block_name>", type=<StorageType>)
 
 options = {"storage_options_override": False} # Enabling this assigns the block for universal use, avoiding the need for repetitive configurations when employing a single credential block throughout the application.
-data  = echoflow_start(dataset_config=dataset_config, pipeline_config=pipeline_config, logging_config=logfile_config, storage_options=aws, options=options)
+data  = echodataflow_start(dataset_config=dataset_config, pipeline_config=pipeline_config, logging_config=logfile_config, storage_options=aws, options=options)
 ```
 
 ## License

deployment/deploy_echoflow_worker.sh → deployment/deploy_echodataflow_worker.sh

@@ -1,19 +1,19 @@
 #!/bin/bash
 
 # Step 1: Create a Python Virtual Environment
-python3 -m venv $HOME/env/echoflow-prod
-source $HOME/env/echoflow-prod/bin/activate
+python3 -m venv $HOME/env/echodataflow-prod
+source $HOME/env/echodataflow-prod/bin/activate
 
-# Step 2: Clone the Echoflow Repository
+# Step 2: Clone the Echodataflow Repository
 cd $HOME/
-git clone https://github.com/OSOceanAcoustics/echoflow.git
-cd $HOME/echoflow
+git clone https://github.com/OSOceanAcoustics/echodataflow.git
+cd $HOME/echodataflow
 
 # Step 3: Checkout the Dev Branch and Update (Optional) - Skip if using Prod/main branch
 git checkout dev
 git pull origin dev
 
-# Step 4: Install the Echoflow Project in Editable Mode
+# Step 4: Install the Echodataflow Project in Editable Mode
 pip install -e .
 
 # Step 5: Log in to Prefect Cloud and Set Your API Key - Change to step 5b if using prefect locally
@@ -22,7 +22,7 @@ read prefectKey
 prefect cloud login -k $prefectKey
 
 # Step 5b: Setup prefect locally
-# prefect profile create echoflow-local
+# prefect profile create echodataflow-local
 
 # Step 6: Set Up the Prefect Worker as a Systemd Service
 echo "Enter Work Pool Name: "
@@ -36,7 +36,7 @@ Description=Prefect-Worker
 
 [Service]
 User=$(whoami)
-WorkingDirectory=$HOME/echoflow
+WorkingDirectory=$HOME/echodataflow
 ExecStart=$(which prefect) agent start --pool $workPool
 Restart=always
 
@@ -53,4 +53,4 @@ sudo systemctl enable prefect-worker.service
 # Step 8: Start the Prefect Worker Service
 sudo systemctl start prefect-worker.service
 
-echo "Setup completed. The Echoflow worker is now running. Send tasks to $workPool using Prefect UI or CLI."
+echo "Setup completed. The Echodataflow worker is now running. Send tasks to $workPool using Prefect UI or CLI."

docs/configuration/blocks.md

@@ -1,62 +1,62 @@
-# Echoflow Configuration and Credential Blocks
+# Echodataflow Configuration and Credential Blocks
 
-Echoflow leverages the concept of "blocks" from Prefect, which serve as containers for storing various types of data, including credentials and sensitive information. Currently, Echoflow supports two types of blocks: Azure Cosmos DB Credentials Block and AWS Credentials Block. These blocks allow you to securely store sensitive data while benefiting from Prefect's robust integration capabilities.
+Echodataflow leverages the concept of "blocks" from Prefect, which serve as containers for storing various types of data, including credentials and sensitive information. Currently, Echodataflow supports two types of blocks: Azure Cosmos DB Credentials Block and AWS Credentials Block. These blocks allow you to securely store sensitive data while benefiting from Prefect's robust integration capabilities.
 
 For a deeper understanding of blocks, you can refer to the [Prefect documentation](https://docs.prefect.io/2.11.5/concepts/blocks/).
 
-## Types of Blocks in Echoflow
+## Types of Blocks in Echodataflow
 
-In the context of Echoflow, there are two main categories of blocks:
+In the context of Echodataflow, there are two main categories of blocks:
 
-### 1. Echoflow Configuration Blocks
+### 1. Echodataflow Configuration Blocks
 
-These blocks serve as repositories for references to credential blocks, as well as repositories for the various Prefect profiles that have been established using Echoflow's functions.
+These blocks serve as repositories for references to credential blocks, as well as repositories for the various Prefect profiles that have been established using Echodataflow's functions.
 
 ### 2. Credential Blocks
 
-Credential blocks store sensitive information, such as authentication keys and tokens, securely. Echoflow integrates with Prefect's capabilities to ensure that sensitive data is protected.
+Credential blocks store sensitive information, such as authentication keys and tokens, securely. Echodataflow integrates with Prefect's capabilities to ensure that sensitive data is protected.
 
 ## Creating Credential Blocks
 
-Credential blocks can be conveniently created using an `.ini` file. By leveraging Prefect's integration, Echoflow ensures that the credentials stored in these blocks are handled securely. To create a credential block, you can follow these steps:
+Credential blocks can be conveniently created using an `.ini` file. By leveraging Prefect's integration, Echodataflow ensures that the credentials stored in these blocks are handled securely. To create a credential block, you can follow these steps:
 
-1. Open the `credentials.ini` file, which is located under the `.echoflow` directory in your home directory.
+1. Open the `credentials.ini` file, which is located under the `.echodataflow` directory in your home directory.
 ```bash
 # Terminal command
-cd ~/.echoflow
+cd ~/.echodataflow
 ```
 2. Place the necessary credential information within the `credentials.ini` file.
 ```bash
 # Terminal command
 nano credentials.ini # Or use any of your favourite editors
 ```
-3. Store the updated `.ini` file in the `.echoflow` directory, which resides in your home directory.
-4. Utilize [echoflow load-credentials](../../echoflow/stages/subflows/echoflow.py#load_credential_configuration) command to generate a new credential block, leveraging the content from the `.ini` file. 
+3. Store the updated `.ini` file in the `.echodataflow` directory, which resides in your home directory.
+4. Utilize [echodataflow load-credentials](../../echodataflow/stages/subflows/echodataflow.py#load_credential_configuration) command to generate a new credential block, leveraging the content from the `.ini` file. 
 ```bash 
-echoflow load-credentials
+echodataflow load-credentials
 ```
-5. Add the name of the block in pipeline or datastore yaml configuration files under `storage_options` section with the appropriate storage type (refer [StorageType](../../echoflow/config/models/datastore.py#StorageType)).
+5. Add the name of the block in pipeline or datastore yaml configuration files under `storage_options` section with the appropriate storage type (refer [StorageType](../../echodataflow/config/models/datastore.py#StorageType)).
 
 ```yaml
 # Example
 storage_options:
-  block_name: echoflow-aws-credentials  # Name of the block containing credentials
+  block_name: echodataflow-aws-credentials  # Name of the block containing credentials
   type: AWS  # Specify the storage type using StorageType enum
 ```
 
 By providing the block name and storage type, ensure that the correct block is used for storage operations, and maintain clarity regarding the chosen storage type.
 
-Once a credential block is created, it can be managed through the Prefect Dashboard. Additionally, if needed, you can use the `echoflow load-credentials` command with the `--sync` argument to ensure your blocks stay up-to-date with any changes made in the Prefect UI. This ensures that your configurations remain accurate and aligned across the application. **It is highly recommended to create new blocks whenever possible, as modifying existing blocks can lead to data loss or conflicts.** 
+Once a credential block is created, it can be managed through the Prefect Dashboard. Additionally, if needed, you can use the `echodataflow load-credentials` command with the `--sync` argument to ensure your blocks stay up-to-date with any changes made in the Prefect UI. This ensures that your configurations remain accurate and aligned across the application. **It is highly recommended to create new blocks whenever possible, as modifying existing blocks can lead to data loss or conflicts.** 
 
-## Considerations When Using `echoflow load-credentials`
+## Considerations When Using `echodataflow load-credentials`
 
-When utilizing the `echoflow load-credentials` command, be aware of the following considerations:
+When utilizing the `echodataflow load-credentials` command, be aware of the following considerations:
 
-- **Overwriting Values**: When using `echoflow load-credentials`, all the values from the `.ini` file will be written to the credential block, potentially overwriting existing values. Exercise caution when using this command to prevent unintentional data loss.
+- **Overwriting Values**: When using `echodataflow load-credentials`, all the values from the `.ini` file will be written to the credential block, potentially overwriting existing values. Exercise caution when using this command to prevent unintentional data loss.
 - **Creating New Blocks**: To maintain data integrity and security, it's advised to create new blocks rather than modifying existing ones. If editing an existing block becomes necessary, it should be done through the Prefect Dashboard.
-- **Sync Argument**: The `--sync` argument is available in the `echoflow load-credentials` command. When set, this option syncs the credential block updates with the Prefect UI. This feature facilitates the seamless management of blocks through the dashboard, enhancing collaboration and control over credentials.
+- **Sync Argument**: The `--sync` argument is available in the `echodataflow load-credentials` command. When set, this option syncs the credential block updates with the Prefect UI. This feature facilitates the seamless management of blocks through the dashboard, enhancing collaboration and control over credentials.
 
-By adhering to these guidelines, you can ensure the secure management of sensitive information while effectively configuring and utilizing Echoflow within your projects.
+By adhering to these guidelines, you can ensure the secure management of sensitive information while effectively configuring and utilizing Echodataflow within your projects.
 
 
 # Configuration File Explanation: credentials.ini

docs/configuration/datastore.md

@@ -1,10 +1,10 @@
-# Echoflow Run Configuration Documentation
+# Echodataflow Run Configuration Documentation
 
-This document provides detailed explanations for the keys used in the provided YAML configuration used to define an Echoflow run.
+This document provides detailed explanations for the keys used in the provided YAML configuration used to define an Echodataflow run.
 
 ## Run Details
 
-- `name`: This key specifies the name of the Echoflow run. It is used to identify and label the execution of the Echoflow process.
+- `name`: This key specifies the name of the Echodataflow run. It is used to identify and label the execution of the Echodataflow process.
 - `sonar_model`: This key indicates the model of the sonar used for data collection during the run.
 - `raw_regex`: This key indicates the regex to be used while parsing the source directory to match the files to be processed.
 
@@ -25,13 +25,13 @@ This document provides detailed explanations for the keys used in the provided Y
 
 ## Notes
 
-- The provided configuration serves as a structured setup for executing an Echoflow run, allowing customization through the specified keys.
+- The provided configuration serves as a structured setup for executing an Echodataflow run, allowing customization through the specified keys.
 - Dynamic placeholders like `ship_name`, `survey_name`, and `sonar_model` are replaced with actual values based on the context.
 
 Example:
 
 ```yaml
-name: Bell_M._Shimada-SH1707-EK60 # Name of the Echoflow Run
+name: Bell_M._Shimada-SH1707-EK60 # Name of the Echodataflow Run
 sonar_model: EK60 # Sonar Model
 raw_regex: (.*)-?D(?P<date>\w{1,8})-T(?P<time>\w{1,6}) # Regex to parse the filenames
 args: # Input arguments
@@ -45,15 +45,15 @@ args: # Input arguments
   group: # Source data transect information
     file: ./x0007_fileset.txt       # Transect file URL. Accepts .zip or .txt file
     storage_options: # Transect file storage options
-      block_name: echoflow-aws-credentials # Block name. For more information on Blocks refer blocks.md
+      block_name: echodataflow-aws-credentials # Block name. For more information on Blocks refer blocks.md
       type: AWS # Block type 
     group_name: default_group # Set when not using a file to pass transect information
   json_export: true # Export raw json metadata of files to be processed
-  raw_json_path: s3://echoflow-workground/combined_files/raw_json # Path to store the raw json metadata. Can also work to skip the process of parsing the files at source directory and fetch files present in this json instead.
+  raw_json_path: s3://echodataflow-workground/combined_files/raw_json # Path to store the raw json metadata. Can also work to skip the process of parsing the files at source directory and fetch files present in this json instead.
 output: # Output arguments
-  urlpath: s3://echoflow-workground/combined_files_dask # Destination data URL parameters
+  urlpath: s3://echodataflow-workground/combined_files_dask # Destination data URL parameters
   overwrite: true # Flag to overwrite the data if present in the output directory
   storage_options: # Destination data storage options
-    block_name: echoflow-aws-credentials
+    block_name: echodataflow-aws-credentials
     type: AWS
 ```