sam local start-api invoke lambda unable to access DynamoDB, the error message showed "The security token included in the request is invalid ".

[leo731121]

Description

Before sam local version 0.52.0 everything works fine.Sam local environment can request data from cloud dynamodb. After I upgraded my sam local version to 1.0.0. When I debug my code in sam local environment. I got an error

"com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The security token included in the request is invalid (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: UnrecognizedClientException "

I can request data from dynamoDB with Java main function, so I have enough permission to request data from the dynamodb with ~/.aws/credentials (default)

I had test the same project and the same credentials in another PC with sam local 0.18.0, everything is fine.

Steps to reproduce

sam local start-api --host 127.0.0.1 -p 3000 --profile default

Observed result

sam local api start running after I run the command "sam local start-api --host 127.0.0.1 -p 3000 --profile default". But when lambda need to reqeust data from dynamoDB, There is an error"The security token included in the request is invalid"

Expected result

sam local should get data from cloud dynamoDB.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Ubuntu 18.04 LTS
2. sam --version: 1.0.0

sam local start-api --host 127.0.0.1 -p 3000 --profile default

[unglud]

Same here

[mndeveci]

Hi @leo731121 ,

Thanks for raising this up. Is it possible to test the issue with v0.53.0 of SAM CLI?

[leo731121]

Hi @leo731121 ,

Thanks for raising this up. Is it possible to test the issue with v0.53.0 of SAM CLI?

Hi @mndeveci ,
Yes, before i upgraded my sam local to v1.0.0, i used sam local v0.52.0 and everything is normal. After i reinstalled homebrew in my Ubuntu 18.04LTS, I only can install sam local v1.0.0 now. I had tried to downgrade my sam local to v0.48.0(according to this post, #1987 (comment)) but failed. This issue will occur in Ubuntu 16.04LTS which installed sam local v1.0.0. I have another PC which installed Ubuntu and sam local v0.40.0 is normal. This issue seems to occur only in sam local v1.0.0.

[leo731121]

I have replicated the issue in windows 10 which installed sam local v1.0.0. It seems not to be a problem cause by OS. I got an old version from the link(https://github.com/awslabs/aws-sam-cli/releases) and then installed v0.49.0 (AWS_SAM_CLI_64_PY3.msi)in Windows 10. The issue (security token invalid) will not occur. When using v0.49.0, the sam local will download image lambci/lambda. If I use sam local v1.0.0, the image file will be amazon/aws-sam-cli-emulation-image-java8:rapid-1.0.0. Is there any way to install older sam local with tar.gz file ?

[leo731121]

Hi, @mndeveci
After I installed sam local v0.53.0 in Ubuntu 18.04 LTS. Everything works now!

git clone https://github.com/awslabs/aws-sam-cli
cd aws-sam-cli
git checkout tags/v0.53.0
python3 setup.py install
sam --version

I think the major issue is docker image. v0.53.0 will fetch lambci/lambda:java8 docker image . Instead v1.0.0 will fetch amazon/aws-sam-cli-emulation-image-java8 rapid-1.0.0.
Even i changed the policy or role of lambda function in cloudformation template file.
The issue still occur in v1.0.0.

[ohayak]

I confirm this problem on MacBook using aws sam v1.1.0

[benvogan]

This wasted 3 days of my time debugging (I am on Windows 10 WSL1). I can attach to the running docker image and execute my lambda (Go runtime) directly and code I've put into the main function that accesses DynamoDB will work. However when run with /var/rapid/init that same lambda executable fails. Something in either the init or the bootstrap has changed and is somehow breaking the security context. Printing out my session credentials shows that they are correct independently of how the code is run (directly or through init). This is also not specific to DynamoDB as I originally ran into this trying to communicate with Cognito. I suspect talking to all AWS services is broken although I have only tested these two.

[asyba]

I got same error with SecretManager and AWS CLI 1.1.0 from upgrading from older version

[asyba]

anyone knows how I can downgrade or install another older version of aws sam cli with brew?

[mndeveci]

We tracked down the issue and it is only happening in java8 images. java11 or new java8.al2 images doesn't have this bug.
We are currently working on fixing it and release new version of the image. Sorry for the inconvenience.

About downgrading the SAM CLI, I tried to follow couple of brew documents, but I wasn't able to install specific version.

[asyba]

We tracked down the issue and it is only happening in java8 images. java11 or new java8.al2 images doesn't have this bug.
We are currently working on fixing it and release new version of the image. Sorry for the inconvenience.

About downgrading the SAM CLI, I tried to follow couple of brew documents, but I wasn't able to install specific version.

thanks, with java11 manage to test aws tool kit and debug works now. I can use this until is fixed.

[zhouziyang]

@mndeveci I tried python3.6 image, seems same issue occur. I can get worked by hardcode credentials in boto3 API, but failed if using --profile (already checked env variables are correct). I also checked python 3.7, and seems worked fine.

[mndeveci]

@zhouziyang can you write down the steps to reproduce the issue?
For java8 example, running following command for a lambda function (which lists s3 buckets in your account) fails;
AWS_ACCESS_KEY=<> AWS_ACCESS_KEY_ID=<> AWS_SECRET_KEY=<> AWS_SECRET_ACCESS_KEY=<> sam local invoke

[Ghilteras]

Can we please update the issue description? This is not a specific problem with DynamoDB, but a generic key ingestion problem on the SAM container.

[leo731121]

We tracked down the issue and it is only happening in java8 images. java11 or new java8.al2 images doesn't have this bug.
We are currently working on fixing it and release new version of the image. Sorry for the inconvenience.

About downgrading the SAM CLI, I tried to follow couple of brew documents, but I wasn't able to install specific version.

hi, @mndeveci,

I used git to checkout tags/v0.53.0 and used python to downgrade sam local.
The below is the script

git clone https://github.com/awslabs/aws-sam-cli
cd aws-sam-cli
git checkout tags/v0.53.0
python3 setup.py install
sam --version

I wasn't able to install specific version via brew ,either.

[leo731121]

@zhouziyang can you write down the steps to reproduce the issue?
For java8 example, running following command for a lambda function (which lists s3 buckets in your account) fails;
AWS_ACCESS_KEY=<> AWS_ACCESS_KEY_ID=<> AWS_SECRET_KEY=<> AWS_SECRET_ACCESS_KEY=<> sam local invoke

hi, @mndeveci,
I'm developing the java 8 (openJDK8)lambda functions in Eclipse.
The project is a AWS Cloudformation project and the lambda functions are invoked by API Gateway.
I can run the lambda triggered by API Gateway but it occurs the issue when the sam local use version 1.0.0+.
Even use the cmd sam local start-api with --profile, the issue remains.

[leo731121]

Can we please update the issue description? This is not a specific problem with DynamoDB, but a generic key ingestion problem on the SAM container.

hi, @Ghilteras ,

It would be thankful if you could update the description.
Please give some advice for the description.

[zhouziyang]

@zhouziyang can you write down the steps to reproduce the issue?
For java8 example, running following command for a lambda function (which lists s3 buckets in your account) fails;
AWS_ACCESS_KEY=<> AWS_ACCESS_KEY_ID=<> AWS_SECRET_KEY=<> AWS_SECRET_ACCESS_KEY=<> sam local invoke

@mndeveci Hi, sorry for late reply. I'm running with command sam local invoke <Func> -d <Port> -t <Template> --profile <Profile> --region <Region> --debug -e <Event.json>; I already store AWS credentials in ~/.aws/credentials, and checked ENV variables (AWS_ACCESS_KEY etc.) populated correctly during debugging. the docker image used: amazon/aws-sam-cli-emulation-image-python3.6@sha256:d1c38dbdfd32ad1e908e0180d771b3c8540cb2c114e35e3442a7a2bac5bb97dd

[scottomyers]

I am also experiencing this issue with a Go Lambda that downloads files from S3.

[awood45]

This is now fixed. You do not need to upgrade the AWS SAM CLI to see the changes reflected, but you may need to delete your older images. This impacted:

• Python 2.7
• Python 3.6
• Java 8 (not Java 8 AL2)
• .NET Core 2.1
• .NET Core 3.1
• Go 1.x

If you clean up the associated Docker images and rerun sam local invoke you should immediately see this behavior working properly. Please feel free to reopen if this recurs.

[asyba]

@awood45 how this has been fixed? without doing a new aws sam cli version?

[awood45]

The Docker images themselves have been updated, which all versions from 1.0.0 onwards will pull down.

[TheMassapequaTutor]

I am still getting this issue on SAM CLI, version 1.23.0. I have tried uninstalling and reinstalling the sam cli, but with no success. any suggestions?

[pjoshiii]

Any update on this

[teuber789]

Also seeing this. Running sam cli v1.46.0 with nodejs v14 runtime (image is nodejs14.x-x86_64-f88a901bbe51b578e9260ae49).

[dimasusername]

cli v 1.50.0 with image runtime

getting the same error accessing secrets manager from within the Lambda

[evantobin1]

I got the error when using Java8 as my Handler runtime. When I upgraded to Java11, I was able to access the table.

[saratahmad]

I got the error when using Java8 as my Handler runtime. When I upgraded to Java11, I was able to access the table.

Thank you so much!! This worked for me