Log Aggregation - Modern Application Development on AWS

Log Aggregation

The more complicated a system is, the more import it is to have good logs. The challenge is that if logs are scattered across different services, it’s difficult to get a unified view of the entire system. It is essential to have a centralized place where logs are uniformly managed and discoverable. Gathering metrics is also important. In a microservice architecture, calls to various services might be required to handle a given request, so it can be more difficult to find the source of poor performance or errors compared to a monolith. This is why it’s critical to have centrally aggregated logs and runtime metrics.

Figure 16 –   Example of an architecture with log aggregation

For aggregated logging in the AWS Cloud, you can use Amazon CloudWatch. If you use AWS Lambda to implement microservices, anything you write to stdout is sent to CloudWatch Logs. Amazon Elastic Container Service (Amazon ECS) and AWS Fargate can also send anything written to stdout to Amazon CloudWatch Logs with the awslogs log driver. If you use Amazon Elastic Kubernetes Service (Amazon EKS), the logs can be sent to Amazon CloudWatch Logs using the sidecar pattern with Fluentd or Fluent Bit. CloudWatch Using Container Insights can also be used to send logs and metrics to CloudWatch for containerized applications running on either Amazon ECS and AWS Fargate or Amazon EKS.

To trace the execution times or errors from calls between services, you can use AWS X-Ray. AWS X-Ray lets you understand how your application and its underlying services are performing so you can identify and troubleshoot the root cause of performance issues and errors. X-Ray provides an end-to-end view of requests as they travel through your application, and shows a map of your application’s underlying components.

Figure 17 – Example of an architecture with log aggregation using Amazon CloudWatch and AWS X-Ray