Protect Secrets & Personal Data in App Logs
Application logs are a goldmine for debugging, but they often contain passwords, API keys, tokens, and personal data that can be exposed if logs leak. This guide shows you how to systematically prevent secrets and sensitive data from appearing in your logs, and what to do if they already are. You’ll learn practical techniques for redaction, filtering, and safe logging practices that work across most programming languages and frameworks.
What You’ll Need
- Access to your application’s logging configuration (code or config files)
- Knowledge of where your logs are stored (files, stdout, cloud logging service)
- A list of the sensitive data types you handle (passwords, tokens, PII, etc.)
- Basic familiarity with regular expressions (for pattern-based redaction)
Step-by-Step
- Inventory what you log. Review your code for logging statements. Search for common keywords like
password,token,secret,apiKey,authorization,email,ssn,creditcard. Also check third-party libraries that may log request/response bodies. - Never log raw request bodies or full URLs. If you must log requests, log only the method, path (without query string), and headers that are safe. Query strings often contain tokens or personal data.
- Use structured logging. Instead of string concatenation, use a structured format (JSON, key-value pairs). This makes it easier to apply field-level redaction later. For example, in Python use
logging.info("user %s logged in", user_id)instead oflogging.info("user " + user_id + " logged in"). - Implement a redaction filter. Create a function that scrubs sensitive fields before they are written. For JSON logs, walk the object and replace values for keys named
password,secret,token, etc. with[REDACTED]. - Use regex for pattern-based redaction. For values that may appear anywhere in a message (like API keys or credit card numbers), use regular expressions to replace matches. For example, replace
Bearer [A-Za-z0-9-._~+/]+withBearer [REDACTED]. Test your regexes thoroughly to avoid over-redaction. - Configure your logging library to sanitize. Most logging frameworks support custom filters or formatters. For example, in Logback (Java) you can write a custom
Converter; in Python’s logging, you can add aFilter; in Node.js, you can wrap the console methods or use a library likepinowith redaction built in. - Mask sensitive data at the source. Where possible, don’t even pass secrets to logging functions. For example, when logging a database connection string, strip the password before logging. Use helper functions that return sanitized versions of objects.
- Set log levels appropriately. Debug logs often contain more detail. Ensure that production uses
INFOorWARNby default, and thatDEBUGis only enabled temporarily with extra caution. - Review third-party libraries. Many frameworks log request details automatically. Disable or configure these to avoid leaking sensitive data. For example, in Spring Boot, set
logging.level.org.springframework.web.filter.CommonsRequestLoggingFilter=OFFor configure it to exclude headers. - Test your redaction. Write unit tests that send known secrets through your logging pipeline and assert that the output does not contain them. Also, manually inspect logs after a test run.
- Monitor and alert on potential leaks. Use a log analysis tool to search for patterns like
password=orBearerand alert if they appear. This catches mistakes early. - Rotate and secure log storage. Even with redaction, logs may contain personal data. Encrypt logs at rest, restrict access, and follow your data retention policy.
Common Mistakes
- Relying solely on redaction at the presentation layer—if the log is written before redaction, the secret is already in the file.
- Using blacklists (e.g., replacing known keys) instead of whitelists (only logging safe fields). Blacklists miss unexpected fields.
- Logging exceptions that include the full request body or stack trace with variables. Stack traces can contain local variable values.
- Forgetting that query strings in URLs are often logged by web servers or proxies. Ensure your reverse proxy also strips sensitive query parameters.
- Redacting too aggressively, making logs useless for debugging. For example, masking all numbers can hide order IDs.
Tips
- Create a central logging utility that all code uses, so you only need to enforce redaction in one place.
- Use a dedicated logging library with built-in redaction, such as
pino(Node.js) orstructlog(Python). - For Java, consider using
logstash-logback-encoderwhich supports custom JSON providers and masking. - In development, enable debug logging but use a separate log file that is not shipped to production.
- Conduct a security review of your logging configuration as part of your CI/CD pipeline.
Troubleshooting
- Secrets still appear in logs. Check if the logging happens in a library you don’t control. Use a global filter that processes every log event before output. Also check if the secret is being logged by your web server or container runtime, not your app.
- Redaction breaks legitimate log messages. Your regex may be too broad. Test with sample messages and refine. Consider using a whitelist approach for structured logs.
- Logs are too slow after adding redaction. Redaction can be CPU-intensive. Optimize by only running regex on message strings, not on every field. Use simple string replacement for known keys first.
- You find a secret in historical logs. Immediately rotate the secret (e.g., change passwords, revoke tokens). Then, delete or redact the old logs if possible. Implement a log retention policy to limit exposure.
Frequently Asked Questions
Should I ever log passwords?
No. Never log passwords, even in hashed form. Hashes can still be cracked if weak. If you need to debug authentication, log the username and a success/failure flag only.
What about logging personal data like email addresses?
Personal data should be minimized. If you must log an email for correlation, consider hashing it or logging only the first part (e.g., user@example.com becomes user***). Follow your local data protection regulations.
How can I redact data in logs that are already written?
You can use a log processing tool (like Logstash or a simple script) to scan existing logs and replace patterns with [REDACTED]. However, this does not undo exposure—rotate any secrets that may have leaked.
My framework logs request headers automatically—how do I stop it?
Check your framework’s configuration. For example, in Spring Boot, disable CommonsRequestLoggingFilter or set includeHeaders to false. For Express.js, remove the morgan token that logs headers.
Is it enough to redact the value of the ‘password’ field?
No. Attackers can still see the field name and may infer that a password is being used. Also, secrets can appear in other places, like URLs or stack traces. Use a comprehensive approach that includes pattern matching and whitelisting.
What should I do if a secret leaks in a log?
Immediately rotate the secret (change the password, revoke the token). Then, investigate how it got logged and fix the root cause. Consider notifying affected users if personal data was exposed, as required by law.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.