Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda multiline logging messages - different behaviour between Zip & Image (docker) package type #1662

Closed
TibbsTerry opened this issue Jan 23, 2024 · 6 comments
Assignees
Labels

Comments

@TibbsTerry
Copy link

TibbsTerry commented Jan 23, 2024

Describe the bug

Simple Lambda function with Console.WriteLine statements will behave differently for Zip & image (docker) packages

Console logging of a multi-line statement eg:
Console.WriteLine("Multi line example" + Environment.NewLine + "Multi line example");
will appear as one CloudWatch entry when deployed using a zip file but will appear as two CloudWatch entries in a docker runtime

This is also an issue using logging providers such as Microsoft.Extensions.Logging.Console & Amazon.Lambda.Logging.AspNetCore. (probably because they all use Amazon.Lambda.Core.LambdaLogger?)

Expected Behavior

One logging message should be written to one CloudWatch entry.
Behaviour should be consistent.

Current Behavior

Behaviour is not consistent.
There is one CloudWatch entry when deployed using a zip file but two CloudWatch entries in a docker runtime.

Reproduction Steps

Simple lambda function:

public class Function 
{
    public void FunctionHandler(ILambdaContext context)
    {
        Console.WriteLine($"** Func Start v{ThisAssembly.Info.Version}**");

        Console.WriteLine("Single line console example");

        Console.WriteLine("Multi line console example" + Environment.NewLine + "Multi line console example");
    }
}
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <AWSProjectType>Lambda</AWSProjectType>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
    <PublishReadyToRun>true</PublishReadyToRun>
	  <BuildCounter Condition="'$(BuildCounter)' == ''">$([System.DateTime]::Now.ToString('HHmm'))</BuildCounter>
	  <VersionPrefix>1.0.$(BuildCounter)</VersionPrefix>
  </PropertyGroup>
	<ItemGroup>
		<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.4.0" />
		<PackageReference Include="ThisAssembly" Version="1.4.1">
			<PrivateAssets>all</PrivateAssets>
			<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
		</PackageReference>
	</ItemGroup>
</Project>

Zip deployment:

{
  "profile": "*****",
  "region": "******",
  "configuration": "Release",
  "function-architecture": "x86_64",
  "function-runtime": "dotnet6",
  "function-memory-size": 256,
  "function-timeout": 30,
  "function-handler": "LambdaLoggingZip::LambdaLoggingZip.Function::FunctionHandler"
}

Docker deployment:

FROM public.ecr.aws/lambda/dotnet:6 AS base

FROM mcr.microsoft.com/dotnet/sdk:6.0-bullseye-slim as build
WORKDIR /src
COPY ["LambdaLoggingTest.csproj", "LambdaLoggingTest/"]
RUN dotnet restore "LambdaLoggingTest/LambdaLoggingTest.csproj"

WORKDIR "/src/LambdaLoggingTest"
COPY . .
RUN dotnet build "LambdaLoggingTest.csproj" --configuration Release --output /app/build

FROM build AS publish
RUN dotnet publish "LambdaLoggingTest.csproj" \
            --configuration Release \ 
            --runtime linux-x64 \
            --self-contained false \ 
            --output /app/publish \
            -p:PublishReadyToRun=true  

FROM base AS final
WORKDIR /var/task
COPY --from=publish /app/publish .
CMD ["LambdaLoggingTest::LambdaLoggingTest.Function::FunctionHandler"] 
{
  "profile": "*************",
  "region": "***************",
  "configuration": "Release",
  "package-type": "image",
  "function-memory-size": 256,
  "function-timeout": 30,
  "image-command": "LambdaLoggingTest::LambdaLoggingTest.Function::FunctionHandler"
}

Possible Solution

Issue probably in Amazon.Lambda.Core.LambdaLogger ?

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

Targeted .NET Platform

net6.0 / net7.0

Operating System and version

@TibbsTerry TibbsTerry added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jan 23, 2024
@TibbsTerry
Copy link
Author

Also an issue with the dotnet:7 runtime image

@normj
Copy link
Member

normj commented Jan 24, 2024

This is a limitations with container based Lambda functions. For zip based Lambda functions the Lambda environment is given a special endpoint to send the logs to and we can send the log messages as your code sends them. But with container based images that special endpoint doesn't exist. Logs in containers is basically scrapping stdout and when doing that their is no context when a log message ends so newlines is used in the absence of anything better coming from stdout.

If you are curious from the .NET side here is the code in the .NET Lambda runtime client that chooses between the 2 different systems. https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.RuntimeSupport/Helpers/ConsoleLoggerWriter.cs#L156

@TibbsTerry
Copy link
Author

TibbsTerry commented Jan 25, 2024

I believe Amazon.Lambda.Logging.AspNetCore has the same issue?

So we have to write our own custom ILoggerProvider if we want to use containers?? Seems a bit ridiculous?

@bhoradc bhoradc added module/lambda-client-lib and removed needs-triage This issue or PR still needs to be triaged. labels Jan 30, 2024
@TibbsTerry
Copy link
Author

Any update on this issue?

@normj
Copy link
Member

normj commented Mar 29, 2024

Lambda has no plans to change how containers are run to provide access to the separate telemetry API. Once we get structured logging support in place for .NET that will provide you a work around where the whole structured logging will be compressed into a single line with newlines being escaped.

I'm going to close this issue but if you want to track the structured logging support follow this PR.
#1588

@normj normj closed this as completed Mar 29, 2024
Copy link
Contributor

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants