Problem with s3 region solved in k8s and third party block storage (S3), but fails with Fusion

[dariogf]

Hi all,

I would like to know your experience before opening an issue, just in case I am having some mistake.

I am totally new to nextflow, but have made progress, I already have a local, slurm and Google Batch executors working fine, and now I am trying to use it with a managed k8s from OVH, and using their own custom block storage (with S3 interface), not an AWS bucket.

That means that I have to provide a custom endpoint and also a custom region for S3 to work.

In my config file, I have:

profiles {


    ovh {
        workDir = 's3://mybucket/testwf/work'
        
        docker.enabled = true
        wave.enabled = true
        fusion.enabled = true
        fusion.exportStorageCredentials = true

        aws {
        
            accessKey = 'XXXXXXXXXXXXXXXXXXXXXX'
            secretKey = 'YYYYYYYYYYYYYYYYYYYYYY'
            region = 'gra'
            
            client {
                endpoint = 'https://s3.gra.io.cloud.ovh.net'
                s3PathStyleAccess = true
                protocol = 'https'
            }
        
        }

        k8s {
            namespace = 'default'
            computeResourceType = 'Job'
        }

        process {
            executor = 'k8s'
            scratch = false
            ...
        }
    }
}

With this configuration, I get this error, that means that nextflow is not honoring my configuration with 'gra' region:

ERROR ~ The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'gra' (Service: Amazon S3; Status Code: 400; Error Code: AuthorizationHeaderMalformed; Request ID: tx148dbb3399004be684136-006542a3b1; S3 Extended Request ID: null; Proxy: null)

The same error raises when I remove the aws configuration and credentials from the config file and put them on .aws/credentials. Command line aws works fine, so credentials are ok.

I have forked nextflow and looking at the code, I think that I have narrowed the origin of this problem to this line in
S3FileSystemProvider

        final String bucketName = S3Path.bucketName(uri);
        final boolean global = bucketName!=null;
        final AwsClientFactory factory = new AwsClientFactory(awsConfig, Regions.US_EAST_1.getName());
        client = new S3Client(factory.getS3Client(clientConfig, global));

The new AwsClientFactory line pass a fixed region (US_EAST_1) as parameter, and inside the constructor this fixed region is assigned in first place to the s3client region:

        this.region
                = region  // this is the Regions.US_EAST_1.getName() parameter
                ?: config.region
                ?: SysEnv.get('AWS_REGION')
                ?: SysEnv.get('AWS_DEFAULT_REGION')
                ?: fetchRegion()

I would expect that the assignment of the fixed region parameter to be at the end of the assignment block, and the first one to be the one from the config, so I have modified it like this:

diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/AwsClientFactory.groovy b/plugins/nf-amazon/src/main/nextflow/cloud/aws/AwsClientFactory.groovy
index b60c1d794..78a259645 100644
--- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/AwsClientFactory.groovy
+++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/AwsClientFactory.groovy
@@ -109,11 +109,11 @@ class AwsClientFactory {
 
         // -- get the aws default region
         this.region
-                = region
-                ?: config.region
+                = config.region
                 ?: SysEnv.get('AWS_REGION')
                 ?: SysEnv.get('AWS_DEFAULT_REGION')
                 ?: fetchRegion()
+                ?: region
 
         if( !this.region )
             throw new AbortOperationException('Missing AWS region -- Make sure to define in your system environment the variable `AWS_DEFAULT_REGION`')

After make and make pack, the staging of files works as expected with the fixed build, but a new problem arrises with Fusion.

Pods and jobs are correctly created on the k8s cluster, but the Fusion file system fails with an error saying that region is empty and that it is not 'gra'. So I suspect that the fusion binary is not taking into account the region configuration, as it happens with nextflow.

I have also tried to define AWS_REGION, and AWS_DEFAULT_REGION manually on the pod env, and the variables are populated to the pod, but they are not honored by the fusion binary:

10:15AM WRN retrying after error at 'singleStore.ListDirectory error="operation error S3: ListObjectsV2, https response error StatusCode: 400, RequestID: tx1b5aea4b083f484b82d78-0065422534, HostID: tx1b5aea4b083f484b82d78-0065422534, api error AuthorizationHeaderMalformed: The authorization header is malformed; the region '' is wrong; expecting 'gra'" RetryOnError=0

Please, someone with access to the fusion source code could see if region can be set?

Someone know a workaround of how can I tell fusion binary the region to use?

[pditommaso]

Excellent troubleshooting. Regarding the region in the AwsClientFactory, the rationale is to force the us-west-1 region for "global" S3 operations. However, I agree that makes no sense in your case. You want to open an issue for it.

In the case of region propagation for Fusion, I think you are right. Looking at this code it seems the AWS_REGION is not added to the Fusion environment. @jordeu What do you think?

[dariogf]

Thanks, will open an issue.

[jordeu]

AWS_REGION is not use by Fusion when you provide a custom endpoint. Double check that AWS_S3_ENDPOINT is define at your pod.

All the providers that I've tested didn't require the region to be set (the endpoint already fixes the region). I'm going to prepare a snapshot version using the AWS_REGION if present, so you can test if it work on OVH.

[jordeu]

Give it a try to this Fusion snapshot running Nextflow with:

export FUSION_CONTAINER_CONFIG_URL=https://fusion-develop.s3.eu-west-2.amazonaws.com/snapshots/2023/11/02/060449964/fusion-snapshot-amd64.json

Double check that AWS_S3_ENDPOINT and AWS_REGION both are set at the pod.

[dariogf]

Hi there!,

I have launched a complete workflow and it has worked as expected. Nextflow now works with managed Kubernetes clusters and S3 storage from vendors that needs custom regions to be used:

Completed at: 03-Nov-2023 10:36:21
Duration    : 2h 6m 35s
CPU hours   : 3.8
Succeeded   : 28

I suppose that those changes will be incorporated at some point into the main branch... meanwhile I will keep using my modified nextflow version and the fusion snapshot that you have provided.

Thanks for your help.

[jordeu]

At the end of next week we are going to release a new version with this change.

We may use FUSION_AWS_REGION instead of AWS_REGION as variable to set the Fusion region, but this should be transparent because is Nextflow who is going to set these variables.

[dariogf]

That`s great !

[jordeu]

@dariogf Fusion v2.2.8 with this fix has been released and is automatically used by Nextflow unless you fix it using export FUSION_CONTAINER_CONFIG_URL=... to a different version.

Notice that the final variable used is FUSION_AWS_REGION and not AWS_REGION and you need to manually add it until next Nextflow release is available that will automatically add it:
8f8b09f

[dariogf]

Thanks, have just tried it and I can confirm that it is working as expected (with my own modified version of nextflow).