WriteUnhandledExceptionToLog in LambdaBootstrap seems to break structured logging setup

[LarsAndreasEk]

Description

In 80c440c, the logging behaviour for unhandled exceptions was changed. Exceptions are ToString()-ed and written to Console.Error.

This seems to "break" existing setups where structured logging is in place (ie Serilog). Logs are written twice, once by the application's own structured logging setup, then again to Console.Error using a non-structured approach.

We seem to be left with these options:

1. Live with exceptions being logged twice, in two different ways.
2. Skip the structured logging for unhandled exceptions, and live with the unstructured logging for unhandled exceptions.
3. Downgrade to Amazon.Lambda.RuntimeSupport 1.5

If you have any other suggestions, please let me know!

Reproduction Steps

1. Create a dotnet lambda (I use a custom runtime, but I don't think that changes anything)
2. Setup Serilog or another structured logging library, configured to write to stdout
3. Have the lambda listen to sqs messages, and deliberately throw an exception
4. Make sure to have a try/catch around message processing that logs thrown exceptions using Serilog.
5. Rethrow the exception to let Lambda retry the SQS message.
6. Inspect the logs (you will now see a json string from Serilog AND numerous lines from the WriteUnhandledExceptionToLog method)
7. Ship the logs to Datadog, or similar, that tries to understand the not-so-structured logs :)

Resolution

It would be nice if it was possible to disable the logging that the LambdaBootstrap does when an unhandled exception is thrown. Or perhaps have a way to provide a delegate that does the logging itself, so that we could have the LambdaBootstrap itself use structure logging.

---

Might not be a 🐛 bug-report, but causes problems for us at least :)

[ashishdhingra]

Most likely a feature request rather than a bug.

[martincostello]

Might be related to #1000.

[bruce-lindsay]

@LarsAndreasEk this was an issue for me as well. As a workaround Console.SetError from the handler constructor with a custom implementation of a TextWriter which does nothing when WriteLine(object o) is called suppresses the output. I'm leaving this in case it helps anyone else. This worked for me because I don't use Console.Error from application code.

The new behavior should provide an opt-out for customers who are already using tools like Serilog.

[LarsAndreasEk]

Thanks @bruce-lindsay! I put another workaround in place: Catch exceptions in our own code, log, then throw a custom exception that has a ToString implementation that returns null.

[ashishdhingra]

@LarsAndreasEk Good afternoon. Structured logging support was added as part of PR #1803. Please refer to #1747 for details. Kindly check if it resolves your issue.

Thanks,
Ashish

[LarsAndreasEk]

Glad to hear! I'm currently not working with that particular stack, so I will just take your word for it :)

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.