Amazon Managed Streaming for Apache Kafka
Developer Guide

Create an Amazon MSK Cluster

Before you can create an Amazon MSK cluster you need to specify a VPC and set up subnets within that VPC. You need two subnets in two different Availability Zones in the following Regions: South America (São Paulo), Canada (Central), and US West (N. California). In all other Regions where Amazon MSK is available, you can specify either two or three subnets. Your subnets must all be in different Availability Zones. When you create a cluster, Amazon MSK distributes the broker nodes evenly over the subnets that you specify. For an example of how to set up a VPC and subnets for an MSK cluster, see Step 1: Create a VPC for Your MSK Cluster and Step 2: Enable High Availability and Fault Tolerance.

Creating a Cluster Using the AWS Management Console

  1. Open the Amazon MSK console at https://console.aws.amazon.com/msk/.

  2. Choose Create cluster.

  3. Specify a name for the cluster.

  4. In the VPC list, choose the VPC you want to use for the cluster. You can also specify which version of Apache Kafka you want Amazon MSK to use to create the cluster.

  5. Specify two subnets if you're using one of the following Regions: South America (São Paulo), Canada (Central), and US West (N. California). In other Regions where Amazon MSK is available, you can specify either two or three subnets. The subnets that you specify must be in different Availability Zones.

  6. Choose the kind of configuration you want. For information about MSK configurations, see Amazon MSK Configuration.

  7. Specify the type and number of brokers you want MSK to create in each Availability Zone. The minimum is one broker per Availability Zone and the maximum is five brokers per Availability Zone.

  8. (Optional) Assign tags to your cluster. Tags are optional. For more information, see Tagging Your Amazon MSK Resources.

  9. You can adjust the storage volume per broker. After you create the cluster, you can increase the storage volume per broker but you can't decrease it.

  10. Choose the settings you want for encrypting data in transit. By default, MSK encrypts data as it transits between brokers within a cluster. If you don't want to encrypt data as it transits between brokers, clear the check box labeled Enable encryption within the cluster.

  11. Choose one of the three settings for encrypting data as it transits between clients and brokers. For more information, see Encryption in Transit.

  12. Choose the kind of CMK that you want to use for encrypting data at rest. For more information, see Encryption at Rest.

  13. If you want to authenticate the identity of clients, choose Enable TLS client authentication by selecting the box next to it. For more information about authentication, see Client Authentication.

  14. Choose the monitoring level you want. This determines the set of metrics you get. For more information, see Monitoring Amazon MSK with Amazon CloudWatch.

  15. (Optional) Choose Advanced settings, and then choose Customize settings. You can specify one or more security groups that you want to give access to your cluster (for example, the security groups of client machines). If you specify security groups that were shared with you, you must ensure that you have permissions to them. Specifically, you need the ec2:DescribeSecurityGroups permission. For an example, see Amazon EC2: Allows Managing EC2 Security Groups Associated With a Specific VPC, Programmatically and in the Console.

  16. Choose Create cluster.

Creating a Cluster Using the AWS CLI

  1. Copy the following JSON and save it to a file. Name the file brokernodegroupinfo.json. Replace the subnet IDs in the JSON with the values that correspond to your subnets. These subnets must be in different Availability Zones. Replace "Security-Group-ID" with the ID of one or more security groups of the client VPC. Clients associated with these security groups get access to the cluster. If you specify security groups that were shared with you, you must ensure that you have permissions to them. Specifically, you need the ec2:DescribeSecurityGroups permission. For an example, see Amazon EC2: Allows Managing EC2 Security Groups Associated With a Specific VPC, Programmatically and in the Console. Finally, save the updated JSON file on the computer where you have the AWS CLI installed.

    { "InstanceType": "kafka.m5.large", "ClientSubnets": [ "Subnet-1-ID", "Subnet-2-ID" ], "SecurityGroups": [ "Security-Group-ID" ] }

    Important

    Specify exactly two subnets if you are using one of the following Regions: South America (São Paulo), Canada (Central), and US West (N. California). For other Regions where Amazon MSK is available, you can specify either two or three subnets. The subnets that you specify must be in distinct Availability Zones. When you create a cluster, Amazon MSK distributes the broker nodes evenly across the subnets that you specify.

  2. Run the following AWS CLI command in the directory where you saved the brokernodegroupinfo.json file, replacing "Your-Cluster-Name" with a name of your choice. For "Monitoring-Level", you can specify one of the following three values: DEFAULT, PER_BROKER, or PER_TOPIC_PER_BROKER. For information about these three different levels of monitoring, see Monitoring Amazon MSK with Amazon CloudWatch. The enhanced-monitoring parameter is optional. If you don't specify it in the create-cluster command, you get the DEFAULT level of monitoring.

    aws kafka create-cluster --cluster-name "Your-Cluster-Name" --broker-node-group-info file://brokernodegroupinfo.json --kafka-version "2.2.1" --number-of-broker-nodes 3 --enhanced-monitoring "Monitoring-Level"

    The output of the command looks like the following JSON:

    { "ClusterArn": "...", "ClusterName": "AWSKafkaTutorialCluster", "State": "CREATING" }

    Note

    The create-cluster command might return an error stating that one or more subnets belong to unsupported Availability Zones. When this happens, the error indicates which Availability Zones are unsupported. Create subnets that don't use the unsupported Availability Zones and try the create-cluster command again.

  3. Save the value of the ClusterArn key because you need it to perform other actions on your cluster.

Creating a Cluster with a Custom MSK Configuration Using the AWS CLI

For information about custom MSK configurations and how to create them, see Amazon MSK Configuration.

  1. Save the following JSON to a file, replacing configuration-arn with the ARN of the configuration that you want to use to create the cluster.

    { "Arn": configuration-arn, "Revision": 1 }
  2. Run the create-cluster command and use the configuration-info option to point to the JSON file you saved in the previous step. The following is an example.

    aws kafka create-cluster --cluster-name ExampleClusterName --broker-node-group-info file://brokernodegroupinfo.json --kafka-version "1.1.1" --number-of-broker-nodes 3 --enhanced-monitoring PER_TOPIC_PER_BROKER --configuration-info file://configuration.json

    The following is an example of a successful response after running this command.

    { "ClusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/CustomConfigExampleCluster/abcd1234-abcd-dcba-4321-a1b2abcd9f9f-2", "ClusterName": "CustomConfigExampleCluster", "State": "CREATING" }

Creating a Cluster Using the API