Sercrets manager secrets caching seems not working

[jacoor]

I am using AWS secrets manager with secretrs.
In the cloudtrail logs for the below query

{
          "match_phrase": {
            "aws.cloudwatch.log_group": "/cloudtrail"
          }
        },
        {
          "match_phrase": {
            "parsed_message.eventName": "GetSecretValue"
          }
        },
        {
          "match_phrase": {
            "parsed_message.userIdentity.sessionContext.sessionIssuer.userName": "serverlessrepo-terraform--ApplicationElasticServerl-SKjNqvlXy88n"
          }
        }

I see 10k events for 20 minutes.

What am I also seeing is that it looks like feedback loop. The forwarder requests secret, this is being logged into Cloudtrail. Cloudtrail is being ingested by forwarder, so this again causes to run another instance. Filtering out "getSecretValue" call on cloudwatch trigger solved the issue for me.

Steps to reproduce

• use SSM in lambda config
• set lambda to ingest cloudtrail w/o filtering on Getsecretvalue
• watch invocation level go mad