[Epic] Overhaul Logging

[ThomasLaPiana]

Problem Statement

Current logging for the Fides webserver leaves a lot to be desired. It is exceedingly difficult to debug/track down issues both in production as well as dev environments.

The following issues have been created as a result of internal and external feedback on the current logging solution

Issues

• Adopt structlog as a replacement for loguru #3724
• Enable serializing logs as JSON #3725
• Logging fails on 500s because the process dies, so we get the stack trace but not the proper log message #3729
• Standardize on a single logging format for the entire ecosystem (Privacy Center, Fides webserver, Fides workers) #3726
• Bind Privacy Request IDs to their execution logs to make debugging easier #3727
• Fix exceptions/failures showing up as multiple log lines in Datadog #3728
• Privacy Center has no HTTP logging #3730
• Configure less strict masking for things like connection tests (for debugging) #3731
• Ability to enable SQL-level logging with Backend Database #3750
• Privacy Center - 404s Logged as 200s #4186

[ThomasLaPiana]

cc @daveqnet @RobertKeyser @NevilleS @seanpreston

May the discussion commence! Let me know if I missed anything, thoughts/feeling on the priority, and anything else that you think is worth discussing!

[NevilleS]

Update after our planning chat yesterday - let's identify a couple of the low-hanging fruit items here (e.g. fixing the server swallowing logs, adding TRACE logs with a lot of detail, etc.) and ship a few improvements, but we'll want to break this out into other follow-up issues. I'm not expecting us to solve logging immediately

[ThomasLaPiana]

@daveqnet @seanpreston @RobertKeyser can y'all write wishlists for logging in the following format? I'll then integrate them all into a single list of issues for the Epic

This isn't something we'll solve in a day (or a sprint) so it's worth getting everything into a big Epic and prioritizing as we go 🙂

---

Critical Requirements

• <title> - description

Nice to Haves

• <title> - description

[daveqnet]

Sure thing, @ThomasLaPiana. From my (security) perspective, I've just adapted the L1 (most basic) logging requirements from ASVS 4.0.

Critical Requirements

• Log Content: Secrets - Verify that the application does not log credentials. Session tokens should only be stored in logs in an irreversible, hashed form. (ASVS 4.0 7.1.1)
  • I've excluded payment data from the original req as fides does not include any payment/billing integrations.
  • Not a requirement, but if you want more info on what constitutes a credential and should be excluded, the OWASP cheatsheet is good on this: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html#data-to-exclude . Some examples: database connection strings if they include passwords, bearer tokens in Authorization headers if logging full HTTP headers.
• Log Content: PII - Verify that the application does not by default log other sensitive data as defined under local privacy laws. (ASVS 4.0 7.1.2)
  • You can take "sensitive data as defined under local privacy laws" to mean PII or personal data.
  • The "by default" is my addition. fides should be private by default, but my data protection position here is that the user deploying fides is a data controller and should be empowered (through, say, configuration) to make a decision to collect data in line with local privacy laws. For example, a user deploying fides in the EU could make a decision to collect IP addresses using a legitimate interests lawful basis under the GDPR (e.g. security & abuse detection and prevention) and configure fides to do this, but separately they'd have to include this info on their privacy notice to end users.
• Log Protection: Output Encoding - Verify that all logging components appropriately encode data to prevent log injection.
  • What we're trying to protect against here is an attacker trying to inject fake or malicious entries to fides logs. As a reminder, log4shell was a log injection vulnerability in log4j. 😬 I am hopeful that the logging library used will meet this requirement for us, but it'll need to be verified. If we support logging to locations other than stdout/stderr e.g. logging to file, then this requirement will be more difficult to meet. (ASVS 4.0 7.3.1)

Nice to Haves

• I think the scope of the must-haves above is large enough as it is! If I could include nice-to-haves, basically anything marked as L2 here, excluding the error handling section as it's beyond the scope of this epic.
• It'd be great to have a doc on docs.ethyca.com explaining how logging works.

One piece of a additional info: as a security best practice we recommend to users not to set a lower log level than INFO in production deployments. DEBUG (and TRACE if we add it) should only ever be set temporarily for initial deployment and later troubleshooting.

FATAL
ERROR
WARN
INFO
DEBUG
TRACE

[ThomasLaPiana]

Meeting with @RobertKeyser on June 30, 2023

What're some logging changes you'd like to see?

• Inconsistent logging formats between the Privacy Center, Fides, Workers, etc.
• There is no request ID throughout the Privacy Request task logs, so it's hard to figure out exactly where/why things are failing
• Some exceptions and failures show up as multiple lines in DataDog
• JSON logging would be easier, current format requires some additional parsing especially due to varying log formats
• When an API request returns a 500 we get the stack trace in the logs but not the request itself (We hypothesize this is because we aren't wrapping things in a try/except so when the exception gets raised the logging doesn't get called)
• Privacy Center has no HTTP logging
• Less strict masking for certain logs (such as connection tests), this adds a lot of friction when debugging

What're some changes you'd like to see as far as running and maintaining Fides instances?

• Be able to set up messaging and storage destinations from within the UI
• Inviting users as opposed to setting up their password directly
• When we have an error before graph traversal starts/after it's complete, you can't restart them. This happens when something like building the final package fails.
• No way to delete/clear a failed request
• Admin has no ability to directly resubmit/submit a privacy request. So if a request fails, you can't resubmit on behalf of the user to get it to go through
• No way to trigger a graph rebuild, which means if you update a connector due to a failure the failed privacy request still won't be fixed
• Searching by subject identity would be helpful
• Don't require page refresh to see Privacy Requests change status

[RobertKeyser]

Additional request: Can we enable SQL-level logging in SQLAlchemy when the log level is set to TRACE?