OutOfMemoryException thrown after updating .NET 8 application to Amazon.Lambda.RuntimeSupport 1.9.0

[martincostello]

Describe the bug

Spotting that the 1.9.0 release of Amazon.Lambda.RuntimeSupport added a net8.0 TFM, I updated an existing application of mine running .NET 8 RC2 in AWS Lambda to consume the updated library.

After merging the change, my GitHub Actions CI tests started to fail on Linux and Windows with OutOfMemoryException errors.

This is interesting for several reasons:

1. The GitHub Actions CI isn't hosted in AWS Lambda, but does use my test library to simulate running in AWS Lambda.
2. I've not seen these exceptions before in my CI.
3. It didn't happen in the CI on the pull request itself.
4. It doesn't seem to affect macOS.

The test library sets the AWS_LAMBDA_FUNCTION_MEMORY_SIZE environment variable to simulate the Lambda runtime environment, so it may be that this failure is "expected" based on the change in behaviour and my tests need to be configured to use a higher memory limit for the application's workload.

As the net8.0 version explicitly changes the runtime's memory limits, that's the likely root cause culprit for the exceptions.

I'm logging this issue just in case there's maybe some missing runtime guards or something where the library shouldn't adjust the memory limits other than in the explicit cases already commented in the code.

aws-lambda-dotnet/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/LambdaBootstrap.cs

Lines 268 to 279 in 3dea2c0

	int lambdaMemoryInMb;
	if (!int.TryParse(Environment.GetEnvironmentVariable(LambdaEnvironment.EnvVarFunctionMemorySize), out lambdaMemoryInMb))
	return;
	ulong memoryInBytes = (ulong)lambdaMemoryInMb * LambdaEnvironment.OneMegabyte;
	// If the user has already configured the hard heap limit to something lower then is available
	// then make no adjustments to honor the user's setting.
	if ((ulong)GC.GetGCMemoryInfo().TotalAvailableMemoryBytes < memoryInBytes)
	return;
	AppContext.SetData("GCHeapHardLimit", memoryInBytes);

/cc @normj

Expected Behavior

OutOfMemoryException is not thrown.

Current Behavior

OutOfMemoryException is thrown.

Reproduction Steps

Clone martincostello/adventofcode@188be2b and run the build.ps1 script.

Possible Solution

No response

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Amazon.Lambda.RuntimeSupport 1.9.0

Targeted .NET Platform

8.0.100-rc.2.23502.2

Operating System and version

Windows 2022 and Ubuntu 22.04

[martincostello]

4. It doesn't seem to affect macOS.

Looks like this is explained by the Windows and Linux runner having 7GB of RAM and macOS having 14GB when using GitHub-hosted runners.

[ashishdhingra]

Needs review. CC @normj

[normj]

Hey @martincostello

I see in the repo you have have a commit to increase the memory. I assume it was working now. What was the memory size environment variable set to before?

[martincostello]

The default in my test library was 128, and the function runs with 384 when deployed.

Through trial and error I got things to behave stably in CI by setting it to 1GB less than the RAM available on the runner agents. I was getting slightly different weird behaviour if I just set it to int.MaxValue where HTTP connections just seemed to get closed - I wondered if it was trying to allow more memory than the machine had and that was causing a different failure mode.

[normj]

Yeah 128 is pretty low for .NET for anything but the most simplest functions. I haven't had a chance to look through your repo yet other try your repo steps which sadly for me worked without exceptions. Is each function running in these test environment running in its own process and thrown away after the test or within the same process is RuntimeSupport being spun and and torn down? Lambda would be more like the first option but I think you are running a series of tests and I wasn't sure if we are basically swapping out the Lambda code within the process which would also chew up a lot more memory.

It sounds like RuntimeSupport is still doing the right thing but we could be messing up testing solutions that created their own emulations but didn't have to worry about the memory constrains in the past. I could add an environment variable to disable the new memory setting code but I'm not sure what its use would be in the real Lambda environment which makes me think it is probably easier to do what you are doing which is raise the emulated memory setting value. What do you think, would you prefer an environment variable to disable the memory setting code?

[martincostello]

I only used 128 as that was the default for Lambda. Before this change I was only setting it for my emulation because otherwise the Runtime Support library would throw if it wasn't set at all; otherwise it had no real functional purpose/enforcement. You can see this here in this comment past me wrote.

Everything is just running in the normal test process, there's no isolation or anything. It's basically just "pretending" to be Lambda, rather than a full-fidelity simulation. The repo where I hit the issues is probably an outlier as it's just a bunch of Advent of Code solutions, and some of them aren't probably as efficient as they could be and that's never been an issue before now 😄

I think in my case the problem is that Runtime Support is changing the heap size for the entire test process, and that then bleeds into other tests that are unrelated to Lambda where there's quite a hungry memory demand.

While I've found a solution to my issue in the end, it took a of of trial and error to find the sweet spot that got things stable again. That makes me wonder if a "kill switch" is a good just-in-case safety measure for future people who may encounter this as a more straightforward workaround. Initially I set it to 384 as that's what my Lambda is configured to use, but the problem was more the "leftover" heap setting affecting other tests, rather than affecting the Lambda-specific ones (I think).

[normj]

@martincostello I pushed out version 1.9.1 that has the new environment variable we discussed in the PR.

[martincostello]

Thanks @normj, using the new setting is much simpler 😃 martincostello/adventofcode#1295

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.