Testing AWS Batch and Fargate Executors

[Emmanuel289]

I setup the aws-batch and fargate plugins using pip install to test the batch and fargate executors.

Testing the Fargate Executor

In the case of the fargate executor, I followed these steps:

AWS Configuration setup

Configured aws credentials by copying the secret_access_key, access_key_id, and session_token generated for my profile into the ~/.aws/credentials file. I also set the AWS environment variables to make sure that the right variables were pulled in the event that some variables were not properly set in the ~/.aws/credentials file.

export AWS_REGION="us-east-1"
export AWS_DEFAULT_REGION="us-east-1"
export AWS_DEFAULT_PROFILE=“<profile_name>”
export AWS_ACCESS_KEY_ID=“<given_value>”
export AWS_SECRET_ACCESS_KEY=“<given_value>”

Verifying CLI Access to Resources

I ran some commands to verify cli access to AWS resources, especially those required by the fargate plugin, e.g,
aws ecs describe-clusters —cluster covalent-fargate-cluster to inspect the fargate-cluster

Added entries for the fargate plugin to the covalent configuration file

I updated the covalent.conf file with entries for the fargate plugin

Executing an electron on the fargate executor

I tried to execute an electron in the workflow below, but the task execution failed with exit code - 1. The exception was raised by poll_ecs_task() in fargate.py.

@ct.electron(executor="fargate")
def my_task(x, y):
    return x + y

@ct.lattice
def workflow(x, y, z):
    res =  z + my_task(x, y)
    return res


dispatch_id = ct.dispatch(workflow)(1, 2, 3)

print(dispatch_id)

result_obj = ct.get_result(dispatch_id)

print(result_obj)

Testing the Batch Executor

In the case of the batch executor, I got a similar exception raised by poll_batch_jobs() in awsbatch.py

@ct.electron(executor="awsbatch")
def my_task(x, y):
    return x + y

@ct.lattice
def workflow(x, y, z):
    res =  z + my_task(x, y)
    return res


dispatch_id = ct.dispatch(workflow)(1, 2, 3)

print(dispatch_id)

result_obj = ct.get_result(dispatch_id, wait=True)

print(result_obj)

[AlejandroEsquivel]

Tracking in the following GH issues:

#802
#803