Skip to content

Commit

Permalink
feat(lambda): support for Provisioned Pollers (#32205)
Browse files Browse the repository at this point in the history
### Reason for this change

Lambda is introducing a new property in Event Sources named `ProvisionedPollerConfig` to set provisioned pollers that read from the event sources. When specified, it allows control over the minimum and maximum number of pollers that can be provisioned to process events from the source.

This feature is currently supported for MSK and Self-managed Kafka event sources.

### Description of changes

This new property can be opted in by setting the `ProvisionedPollerConfig` field while creating event sources. The example of setting `ProvisionedPollerConfig` for is shown below:

```
fn.addEventSource(new sources.ManagedKafkaEventSource(
  {
    clusterArn,
    topic: kafkaTopic,
    startingPosition: lambda.StartingPosition.TRIM_HORIZON,
    provisionedPollerConfig: {
      minimumPollers: 1,
      maximumPollers: 3,
    },
  }))

```
### Description of how you validated changes

Have added unit test and integration test to validate the implementation

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
  • Loading branch information
roger-zhangg authored Nov 21, 2024
1 parent e1195f9 commit 68b969e
Show file tree
Hide file tree
Showing 15 changed files with 781 additions and 11 deletions.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
Expand Up @@ -322,6 +322,138 @@
"my-test-topic2"
]
}
},
"F3ServiceRole2F65FFC0": {
"Type": "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
],
"Version": "2012-10-17"
},
"ManagedPolicyArns": [
{
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
]
]
}
]
}
},
"F3ServiceRoleDefaultPolicy1C0463D1": {
"Type": "AWS::IAM::Policy",
"Properties": {
"PolicyDocument": {
"Statement": [
{
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:GetSecretValue"
],
"Effect": "Allow",
"Resource": [
{
"Ref": "S509448A1"
},
{
"Ref": "SC0855C491"
}
]
}
],
"Version": "2012-10-17"
},
"PolicyName": "F3ServiceRoleDefaultPolicy1C0463D1",
"Roles": [
{
"Ref": "F3ServiceRole2F65FFC0"
}
]
}
},
"F38FF9B13A": {
"Type": "AWS::Lambda::Function",
"Properties": {
"Code": {
"ZipFile": "exports.handler = async function handler(event) {\n console.log('event:', JSON.stringify(event, undefined, 2));\n return { event };\n}"
},
"Handler": "index.handler",
"Role": {
"Fn::GetAtt": [
"F3ServiceRole2F65FFC0",
"Arn"
]
},
"Runtime": "nodejs18.x"
},
"DependsOn": [
"F3ServiceRoleDefaultPolicy1C0463D1",
"F3ServiceRole2F65FFC0"
]
},
"F3KafkaEventSource838c4d5ff3c99c1a617120adfca83e5bmytesttopic3ED015F25": {
"Type": "AWS::Lambda::EventSourceMapping",
"Properties": {
"BatchSize": 100,
"FilterCriteria": {
"Filters": [
{
"Pattern": "{\"numericEquals\":[{\"numeric\":[\"=\",1]}]}"
}
]
},
"FunctionName": {
"Ref": "F38FF9B13A"
},
"ProvisionedPollerConfig": {
"MaximumPollers": 3,
"MinimumPollers": 1
},
"SelfManagedEventSource": {
"Endpoints": {
"KafkaBootstrapServers": [
"my-self-hosted-kafka-broker-1:9092",
"my-self-hosted-kafka-broker-2:9092",
"my-self-hosted-kafka-broker-3:9092"
]
}
},
"SelfManagedKafkaEventSourceConfig": {
"ConsumerGroupId": "myTestConsumerGroup3"
},
"SourceAccessConfigurations": [
{
"Type": "CLIENT_CERTIFICATE_TLS_AUTH",
"URI": {
"Ref": "SC0855C491"
}
},
{
"Type": "SERVER_ROOT_CA_CERTIFICATE",
"URI": {
"Ref": "S509448A1"
}
}
],
"StartingPosition": "TRIM_HORIZON",
"Topics": [
"my-test-topic3"
]
}
}
},
"Parameters": {
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 68b969e

Please sign in to comment.