Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 Batch Operations 2.0 Schema Fields to S3BatchJob/S3BatchJobTask #550

Merged
merged 7 commits into from
Jan 29, 2024

Conversation

lyoung-confluent
Copy link
Contributor

@lyoung-confluent lyoung-confluent commented Jan 25, 2024

This PR adds support for the new S3 Batch Job 2.0 Invocation Schema by adding the userArguments field and the s3Bucket key (instead of s3BucketArn) to the existing S3BatchJob and S3BatchJobTask structs. For more details see this blog post: https://aws.amazon.com/blogs/storage/automate-object-processing-in-amazon-s3-directory-buckets-with-s3-batch-operations-and-aws-lambda/

I'm unsure if this should be a new struct specifically for 2.0 or just a small modification to the existing struct as implemented in this PR. Lambda Powertools for Python has implemented it as the same object/class for both schema versions but also has the benefit of Python's looser typing: aws-powertools/powertools-lambda-python#3572

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@codecov-commenter
Copy link

codecov-commenter commented Jan 25, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (58da2cc) 73.30% compared to head (51690ab) 73.30%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #550   +/-   ##
=======================================
  Coverage   73.30%   73.30%           
=======================================
  Files          26       26           
  Lines        1487     1487           
=======================================
  Hits         1090     1090           
  Misses        324      324           
  Partials       73       73           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@bmoffatt
Copy link
Collaborator

I'm unsure if this should be a new struct specifically for 2.0 or just a small modification to the existing struct as implemented in this PR.

Since the new schema version removes a field, rather than being purely additive, let's do new structs with a V2 suffix.

@bmoffatt bmoffatt added enhancement type/events issue or feature request related to the events package labels Jan 26, 2024
@lyoung-confluent
Copy link
Contributor Author

@bmoffatt I've split it into a new struct S3BatchJobV2, however the response struct has not changed with 2.0, should I duplicate it anyway?

@bmoffatt
Copy link
Collaborator

@bmoffatt I've split it into a new struct S3BatchJobV2, however the response struct has not changed with 2.0, should I duplicate it anyway?

I'd say leave the response struct as-is

@bmoffatt bmoffatt merged commit eae55b8 into aws:main Jan 29, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement type/events issue or feature request related to the events package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants