Logging best practices
Logging levels
Be careful not to log an excessive amount of data. Logs should capture useful and actionable data. Excessive logging can negatively impact performance, and it is can also increase logging storage and processing costs. Excessive logging can also result in issues and security events going undetected.
Logging HTTP response status codes can generate a significant amount of log data, especially 200-level (success) and 300-level (redirection) status codes. We recommend that you consider logging only 400-level (client-side errors) and 500-level (server-side errors) status codes.
Application logging frameworks provide different levels of logging, such asĀ info, debug, or error. For development environments, you might want to use verbose logging, such as including info and debug, to help your developers. However, we recommend that you disable info and debug levels for production environments because these can generate excessive logging data.
Cautions and exclusions
-
Ensure that the data you are logging is legally permitted, particularly in jurisdictions where your organization operates.
-
Don't exclude any events from known users (such as other internal systems), trusted third parties, search engine robots, uptime or process monitors, and other remote monitoring systems. However, you can include a classification flag for each of these in the recorded data. Consider that log files generated by your application might be used by parties, such as third-party log monitoring solutions or external service providers, that are not authorized to view any sensitive data that the application processes.
-
The following attributes should not be recorded directly in the logs. Remove, mask, sanitize, hash, or encrypt the following:
-
Application source code
-
Session identification values (consider replacing this with a hashed value if you need to track session-specific events)
-
Access tokens
-
Sensitive personal data and some forms of personally identifiable information (PII), such as health information or government-issued identifiers
-
Authentication passwords
-
Database connection strings
-
Encryption keys and other primary secrets
-
Bank account or payment card holder data
-
Data of a higher security classification than the logging system is permitted to store
-
Commercially sensitive information
-
Information that is illegal to collect in the relevant jurisdictions
-
Information that a user has either opted out of or hasn't explicitly consented to collection
-
Information that the consent to collect has expired
-
Special data types
Sometimes, the following data can also be recorded in logs. While it can be useful for investigative and troubleshooting purposes, it can reveal sensitive information about the system. You might need to anonymize, hash, or encrypt these data types before the event is recorded:
-
File paths
-
Internal network names and addresses
-
Non-sensitive personal data, such as personal names, telephone numbers, and email addresses
Use data anonymization if the individual's real identity is not required in the log or if the risk is considered too great.
Access and change management
-
Non-administrative users should not be able to disable logging of events, especially those that are necessary to meet compliance requirements.
-
Only administrative users should be able to pause or stop logging services or modify configurations.
-
If your logging service has a log file integrity validation feature, enable it. This helps you detect modification, deletion, or forging of log files. For more information about this feature in AWS services, see Using CloudTrail in this guide.
-
Logging changes must be intrinsic to the application, such as made automatically by the application based on an approved algorithm, or follow an approved change management processes, such as when you change configuration data or modify the source code.