Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESF incorrectly sets region for VPC flow logs from multi-region S3 bucket #741

Open
zmoog opened this issue Jul 11, 2024 · 0 comments
Open
Labels
bug Something isn't working

Comments

@zmoog
Copy link
Contributor

zmoog commented Jul 11, 2024

  • Version: 1.14.0

Context

I am creating an issue for a user who reported it using a private channel.

All VPC flow log data, regardless of originating AWS region, is stored in a single bucket in the us-east-1 region.

In this scenario, the user consolidates VPC logs from multiple regions in a single S3 bucket and uses ESF to ingest all VPC logs from the S3 bucket.

ESF creates documents with the region of the S3 bucket instead of the region of the S3 object.

Current behavior

My understanding of the region problem is that the user:

  • Is sending all the VPC flow logs in a single S3 bucket; for example, a bucket named my-vpcflow-logs hosted on the region eu-north-1.
  • Each VPC Flow log file is stored in an object key like the following:
AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz
  • When ESF processes the AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz object, it creates a document with the following fields:
{
  "cloud": {
    "provider": "aws",
    "region": "eu-north-1",
    "account": {
      "id": "123"
    }
  }
}

Expected behavior

  • However, since the AWSLogs/123/vpcflowlogs/eu-central-11/2024/07/11/123_vpcflowlogs_eu-central-1_fl-0cea9cbf050c152d5_20240711T0000Z_123eabd0.log.gz comes from the eu-central-1 region, the user expects the document to have the following content:
{
  "cloud": {
    "provider": "aws",
    "region": "eu-central-1",
    "account": {
      "id": "123"
    }
  }
}

Notes

In the current version, ESF uses the region from the notification S3 published in the SQS queue.

@zmoog zmoog added the bug Something isn't working label Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant