Menu
Centralized Logging on AWS
Centralized Logging on AWS

Automated Deployment

Before you launch the automated deployment, please review the architecture, configuration, and other information discussed in this guide. Follow the step-by-step instructions in this section to configure and deploy a centralized logging solution into your account.

Time to deploy: Approximately 30 minutes

What We'll Cover

The procedure for deploying this architecture on AWS consists of the following steps. For detailed instructions, follow the links for each step.

Step 1. Launch the Primary Stack

  • Launch the AWS CloudFormation template into your AWS account.

  • Enter values for required parameters: Stack Name, User Name, Password, EC2 Key Pair Name, SSH Access CIDR

  • Review the other template parameters, and adjust if necessary.

Step 2. Launch the Spoke Stack

  • Launch the AWS CloudFormation template into secondary AWS accounts and AWS Regions.

  • Review the template parameters, and adjust if necessary.

Step 3. Configure the Kibana Dashboard

  • Verify the URL for the Nginx proxy server.

  • Add a Kibana index and then import the Kibana dashboard.

Step 1. Launch the Primary Stack

This automated AWS CloudFormation template deploys the centralized logging solution in your primary AWS account.

Note

You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.

  1. Sign in to the AWS Management Console and click the button below to launch the centralized-logging-primary AWS CloudFormation template.

    
                                centralized logging launch button

    You can also download the template as a starting point for your own implementation.

  2. The template is launched in the US East (N. Virginia) Region by default. To launch the centralized logging solution in a different AWS Region, use the region selector in the console navigation bar.

  3. On the Select Template page, verify that you selected the correct template and choose Next.

  4. On the Specify Details page, assign a name to your centralized logging solution stack.

  5. Under Parameters, review the parameters for the template, and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    Domain Name centralized-logging The name of the Amazon ES domain that this template will create.

    Note

    Amazon ES domain names must start with a lowercase letter and must be between 3 and 28 characters. Valid characters are a-z (lowercase only), 0-9, and – (hyphen).

    Cluster Size Small A drop-down box with three Amazon ES cluster sizes: Small, Medium, Large
    Spoke Accounts <Optional Input>

    Comma delimited list of account IDs for log indexing. Enter the secondary account IDs in this parameter before you deploy the spoke template in secondary accounts. To add accounts after you launch the primary template, update the Spoke Accounts parameter in the primary stack with the secondary account IDs. Then, update the primary stack and deploy the spoke template in the secondary accounts.

    Note

    For cross-region log indexing in the primary account, enter the primary account ID. For cross-account indexing, enter secondary (spoke) account IDs. For both, enter primary and secondary account IDs.

    User Name <Requires Input> User name for access to the Nginx proxy server
    Password <Requires Input> Password for access to the Nginx proxy server

    Note

    Must be six characters or longer and must contain one uppercase letter, one lower case letter, and a special character (!@#$%^&+)

    Re-Type Password <Requires Input> Confirm the password for access to the Nginx proxy server
    EC2 Key Pair Name <Requires Input> Public and private key pair, which allows you to connect securely to the Nginx proxy and Apache web servers. When you created an AWS account, this is the key pair you created in your preferred AWS Region.
    SSH Access CIDR <Requires Input> This IP address range will have access to Amazon ES via the proxy, and SSH and HTTP access to both the Nginx proxy servers and the Apache web server.
    VPC CIDR for Proxy Servers 10.249.0.0/16 CIDR block for the solution’s VPC. You can modify the address range to avoid overlapping with existing networks.
    Subnet 1 for Proxy Server 10.249.250.0/24 CIDR block for the VPC subnet created in AZ1
    Subnet 2 for Proxy Server 10.249.249.0/24 CIDR block for the VPC subnet created in AZ2
    Sample Logs No Choose whether to deploy the demo template
    VPC CIDR for Sample Sources 10.250.0.0/16

    CIDR block for the sample logs VPC. You can modify the address range to avoid overlapping with existing networks.

    Note

    Use this parameter only if you choose Yes for Sample Logs.

    Subnet for Sample Web Server 10.250.250.0/24

    CIDR block for the sample web server. You can modify the address range to avoid overlapping with existing networks.

    Note

    Use this parameter only if you choose Yes for Sample Logs.

  6. Choose Next.

  7. On the Options page, choose Next.

  8. On the Review page, review and confirm the settings. Be sure to check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  9. Choose Create to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should see a status of CREATE_COMPLETE in roughly 25 minutes.

  10. To see details for the stack resources, choose the Outputs tab. The following table describes some of these outputs in more detail.

    Key Description
    KibanaURL

    URL for front-end access to the Kibana 4 dashboard via the proxy server

    DomainEndpoint

    URL for the Amazon ES domain endpoint

    MaterRole

    Master IAM role for log indexing on the Amazon ES domain

Note

This solution deploys an AWS Lambda function, solution-helper, which runs only during initial configuration or when resources are updated or deleted. You will see the solution-helper function in the AWS Lambda console, which is necessary to manage associated resources for as long as the solution is running.

Step 2. Launch the Spoke Stack

Use this procedure to launch the components necessary to manage logs in secondary accounts. You must enter the secondary account IDs in the Spoke Accounts parameter of the primary template before you launch this template in secondary accounts.

Note

You are responsible for the cost of the AWS services used while running this solution. See the Cost section for more details. For full details, see the pricing webpage for each AWS service you will be using in this solution.

  1. Sign in to the AWS Management Console and click the button below to launch the centralized-logging-spoke AWS CloudFormation template.

    
                                centralized logging launch button

    You can also download the template as a starting point for your own implementation.

  2. The template is launched in the US East (N. Virginia) Region by default. To launch the centralized logging solution in a different AWS Region, use the region selector in the console navigation bar.

  3. On the Select Template page, verify that you selected the correct template and choose Next.

  4. On the Specify Details page, assign a name to your centralized logging solution stack.

  5. Under Parameters, review the parameters for the template and modify them as necessary. This solution uses the following default values.

    Parameter Default Description
    Elasticsearch Endpoint <Requires Input>

    Amazon Elasticsearch Service (Amazon ES) domain endpoint.

    Note

    You can find the endpoint in the primary AWS CloudFormation stack Outputs tab. The endpoint is the value of the DomainEndpoint key.

    Master Account Role <Requires Input>

    AWS IAM role for cross-account indexing

    Note

    You can find the master role in the primary AWS CloudFormation stack Outputs tab. The role is the value of the MasterRole key.

    Cluster Size Small

    A drop-down box with three Amazon ES cluster sizes: Small, Medium, Large

    Note

    Select the same cluster size you chose for the primary stack. You can find the cluster size in the primary AWS CloudFormation stack Outputs tab. The name of the cluster size is the value of the ClusterSize key.

    Sample Logs No Choose whether to deploy the demo template
    VPC CIDR for Sample Sources 10.250.0.0/16

    CIDR block for the sample logs VPC. You can modify the address range to avoid overlapping with existing networks.

    Note

    Use this parameter only if you choose Yes for Sample Logs.

    Subnet for Sample Web Server 10.250.250.0/24

    CIDR block for the sample web server. You can modify the address range to avoid overlapping with existing networks.

    Note

    Use this parameter only if you choose Yes for Sample Logs.

  6. Choose Next.

  7. On the Options page, choose Next.

  8. On the Review page, review and confirm the settings. Be sure to check the box acknowledging that the template will create AWS Identity and Access Management (IAM) resources.

  9. Choose Create to deploy the stack.

    You can view the status of the stack in the AWS CloudFormation console in the Status column. You should see a status of CREATE_COMPLETE in roughly five minutes.

Step 3. Configure the Kibana Dashboard

A Kibana dashboard displays a group of visualizations that you can modify, save, and share. If you choose to deploy the sample logs, the visualizations for this solution combine data from VPC flow logs, the Apache web server, and AWS CloudTrail to create a centralized view of an application and its supporting resources. Note that you must deploy the demo template before you configure the dashboard.

After the centralized logging solution stack launch completes, you can access the Kibana dashboard and begin importing log data. Use the following steps to log in to Kibana, add an Elasticsearch index, and import the solution’s preconfigured dashboard settings.

  1. Download the dashboard configuration file (basic-dashboard.json) from the centralized logging solution Amazon S3 bucket. You will use this later in the procedure to configure your first dashboard.

  2. Go to the AWS CloudFormation console, and in the Outputs tab, open the KibanaURL link to go to the Kibana dashboard.

  3. When prompted, log in to the dashboard with the user name and password you specified in Step 1. Launch the Primary Stack.

  4. In the left menu bar, choose Management.

  5. Under Configure an index pattern, set the Index name or pattern field to cwl-*.

    You should see the message box underneath change from red to green, confirming that there are matching indices and aliases.

  6. Under Time Filter field name, choose @timestamp.

  7. Choose Create. You will see a list of every field in the index.

  8. On the Saved Objects tab, choose Import and select the basic-dashboard.json file you downloaded in Step 1 of this procedure. If prompted, choose Yes, overwrite all.

    Note

    If this causes an error message, choose Go Back. Delete the cwl-* index you just created. Wait at least 10 minutes for the indices to populate. Then, repeat steps 4-8.

  9. In the Saved Objects tab under Dashboards, you should see a Basic dashboard. Choose the eye icon next to the dashboard to view it.

  10. The solution’s default dashboard will load. In the upper-right corner, you can adjust the data time period (clock icon). You can also adjust interval for the webpage refresh rate (Auto-refresh).


                Sample Kibana dashboard

Figure 2: Sample Kibana dashboard

Explore and experiment with the dashboard settings. You can interact with the Apache server to see the events passed to the dashboard metrics, for example, request a webpage that doesn’t exist to see the 404 error count increase. The VPC visualizations show you information such as the top 10 rejected source IP addresses.

You can create and save additional visualizations based on the data that is relevant to your application. For more information, go to the Kibana User Guide.