View AWS Network Firewall logs and metrics by using Splunk

Created by Ivo Pinto

Environment: PoC or pilot	Technologies: Networking; CloudNative; Content delivery; Operations; Security, identity, compliance	Workload: All other workloads
AWS services: Amazon CloudWatch; Amazon CloudWatch Logs; AWS Network Firewall

Summary

Many organizations use Splunk Enterprise as a centralized aggregation and visualization tool for logs and metrics from different sources. This pattern helps you configure Splunk to fetch AWS Network Firewall logs and metrics from Amazon CloudWatch Logs by using the Splunk Add-On for AWS. 

To achieve this, you create a read-only AWS Identity and Access Management (IAM) role. Splunk Add-On for AWS uses this role to access CloudWatch. You configure the Splunk Add-On for AWS to fetch metrics and logs from CloudWatch. Finally, you create visualizations in Splunk from the retrieved log data and metrics.

Prerequisites and limitations

Prerequisites

• A Splunk account

• A Splunk Enterprise instance, version 8.2.2 or later 

• An active AWS account

• Network Firewall, set up and configured to send logs to CloudWatch Logs

Limitations

• Splunk Enterprise must be deployed as a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances in the AWS Cloud.

• Collecting data by using an automatically discovered IAM role for Amazon EC2 is not supported in the AWS China Regions.

Architecture

The diagram illustrates the following:

1. Network Firewall publishes logs to CloudWatch Logs.

2. Splunk Enterprise retrieves metrics and logs from CloudWatch.

To populate example metrics and logs in this architecture, a workload generates traffic that passes through the Network Firewall endpoint to go to the internet. This is achieved by the use of route tables. Although this pattern uses a single Amazon EC2 instance as the workload, this pattern can apply to any architecture as long as Network Firewall is configured to send logs to CloudWatch Logs.

This architecture also uses a Splunk Enterprise instance in another virtual private cloud (VPC). However, the Splunk instance can be in another location, such as in the same VPC as the workload, as long as it can reach the CloudWatch APIs.

Tools

AWS services

• Amazon CloudWatch Logs helps you centralize the logs from all your systems, applications, and AWS services so you can monitor them and archive them securely.

• Amazon Elastic Compute Cloud (Amazon EC2) provides scalable computing capacity in the AWS Cloud. You can launch as many virtual servers as you need and quickly scale them up or down.

• AWS Network Firewall is a stateful, managed, network firewall and intrusion detection and prevention service for VPCs in the AWS Cloud.

Other tools

• Splunk helps you monitor, visualize, and analyze log data.

Epics

Create an IAM role

Task	Description	Skills required
Create the IAM policy.	Follow the instructions in Creating policies using the JSON editor to create the IAM policy that grants read-only access to the CloudWatch Logs data and CloudWatch metrics. Paste the following policy into the JSON editor. { "Statement": [ { "Action": [ "cloudwatch:List*", "cloudwatch:Get*", "network-firewall:List*", "logs:Describe*", "logs:Get*", "logs:List*", "logs:StartQuery", "logs:StopQuery", "logs:TestMetricFilter", "logs:FilterLogEvents", "network-firewall:Describe*" ], "Effect": "Allow", "Resource": "*" } ], "Version": "2012-10-17" }	AWS administrator
Create a new IAM role.	Follow the instructions in Creating a role to delegate permissions to an AWS service to create the IAM role that the Splunk Add-On for AWS uses to access CloudWatch. For Permissions policies, choose the policy that you created previously.	AWS administrator
Assign the IAM role to the EC2 instances in the Splunk cluster.	Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/. In the navigation pane, choose Instances. Select the EC2 instances in the Splunk cluster. Choose Actions, Security, and then Modify IAM role. Select the IAM role that you created previously, and then choose Save.	AWS administrator

Install the Splunk Add-On for AWS

Task	Description	Skills required
Install the add-on.	In the Splunk dashboard, navigate to Splunk Apps. Search for Splunk Add-on for Amazon Web Services. Choose Install. Provide your Splunk credentials.	Splunk administrator
Configure the AWS credentials.	In the Splunk dashboard, navigate to Splunk Add-on for AWS. Choose Configuration. In the Autodiscovered IAM Role column, select the IAM role that you created previously. For more information, see Find an IAM role within your Splunk platform instance in the Splunk documentation.	Splunk administrator

Configure Splunk access to CloudWatch

Task	Description	Skills required
Configure the retrieval of Network Firewall logs from CloudWatch Logs.	In the Splunk dashboard, navigate to Splunk Add-on for AWS. Choose Input. Choose Create New Input. In the list, choose Custom Data Type, and then choose CloudWatch Logs. Provide the Name, AWS Account, AWS Region, and Log Group for your Network Firewall logs. Choose Save. By default, Splunk fetches the log data every 10 minutes. This is a configurable parameter under Advanced Settings. For more information, see Configure a CloudWatch Logs input using Splunk Web in the Splunk documentation.	Splunk administrator
Configure the retrieval of Network Firewall metrics from CloudWatch.	In the Splunk dashboard, navigate to Splunk Add-on for AWS. Choose Input. Choose Create New Input. In the list, choose CloudWatch. Provide the Name, AWS Account, and AWS Region for your Network Firewall metrics. Next to Metric Configuration, choose Edit in advanced mode. (Optional) Delete all preconfigured namespaces.  Choose Add Namespace, and then name it AWS/NetworkFirewall. In Dimension Value, add the following. [{"AvailabilityZone":[".*"],"Engine":[".*"],"FirewallName":[".*"]}] For Metrics, choose All. For Metric Statistics, choose Sum. Choose OK. Choose Save. By default, Splunk fetches the metric data every 5 minutes. This is a configurable parameter under Advanced Settings. For more information, see Configure a CloudWatch input using Splunk Web in the Splunk documentation.	Splunk administrator

Create Splunk visualizations by using queries

Task	Description	Skills required
View the top source IP addresses.	In the Splunk dashboard, navigate to Search & Reporting. In the enter search here box, enter the following. sourcetype="aws:cloudwatchlogs" | top event.src_ip This query displays a table of the source IP addresses with the most traffic, in descending order. For a graphical representation, choose Visualization.	Splunk administrator
View packet statistics.	In the Splunk dashboard, navigate to Search & Reporting. In the enter search here box, enter the following. sourcetype="aws:cloudwatch"| timechart sum(Sum) by metric_name This query displays a table of the metrics DroppedPackets, PassedPackets, and ReceivedPackets per minute. For a graphical representation, choose Visualization.	Splunk administrator
View the most-used source ports.	In the Splunk dashboard, navigate to Search & Reporting. In the enter search here box, enter the following. sourcetype="aws:cloudwatchlogs" | top event.dest_port This query displays a table of the source ports with the most traffic, in descending order. For a graphical representation, choose Visualization.	Splunk administrator

Related resources

AWS documentation

• Creating a role to delegate permissions to an AWS service (IAM documentation)

• Creating IAM policies (IAM documentation)

• Logging and monitoring in AWS Network Firewall (Network Firewall documentation)

• Route table configurations for AWS Network Firewall (Network Firewall documentation)

AWS blog posts

• AWS Network Firewall deployment models

AWS Marketplace

• Splunk Enterprise Amazon Machine Image (AMI)