Clusters
Use this resource to create an Amazon MSK cluster and to get a list of existing clusters.
URI
/v1/clusters
HTTP methods
GET
Operation ID: ListClusters
Returns a list of all the MSK clusters.
Name | Type | Required | Description |
---|---|---|---|
nextToken | String | False | The paginated results marker. When the result of the operation is truncated, the call returns |
clusterNameFilter | String | False | Specify a prefix of the name of the clusters that you want to list. The service lists all the clusters whose names start with this prefix. |
maxResults | String | False | The maximum number of results to return in the response (default maximum 100 results per API call). If there are more results, the response includes a |
Status code | Response model | Description |
---|---|---|
200 |
ListClustersResponse | Successful response. |
400 | Error | The request isn't valid because the input is incorrect. Correct your input and then submit it again. |
401 | Error | The request is not authorized. The provided credentials couldn't be validated. |
403 | Error | Access forbidden. Check your credentials and then retry your request. |
404 | Error | The resource could not be found due to incorrect input. Correct the input, then retry the request. |
429 | Error | 429 response |
500 | Error | There was an unexpected internal server error. Retrying your request might resolve the issue. |
503 | Error | 503 response |
OPTIONS
Enable CORS by returning the correct headers.
Status code | Response model | Description |
---|---|---|
200 | None | The default response for a CORS method. |
POST
Operation ID: CreateCluster
Creates a new MSK cluster. The following Python 3.6 examples shows how you can create a cluster that's distributed over two Availability Zones.
Before you run this Python script, replace the example subnet and security-group IDs with the IDs of your subnets and security group. When you create an MSK cluster, its brokers get evenly distributed over a number of Availability Zones that's equal to the number of subnets that you specify in the BrokerNodeGroupInfo
parameter. In this example, you can add a third subnet to get a cluster that's distributed over three Availability Zones.
import boto3 client = boto3.client('kafka') response = client.create_cluster( BrokerNodeGroupInfo={ 'BrokerAZDistribution': 'DEFAULT', 'ClientSubnets': [ 'subnet-012345678901fedcba', 'subnet-9876543210abcdef01' ], 'InstanceType': 'kafka.m5.large', 'SecurityGroups': [ 'sg-012345abcdef789789' ] }, ClusterName='SalesCluster', EncryptionInfo={ 'EncryptionInTransit': { 'ClientBroker': 'TLS_PLAINTEXT', 'InCluster': True } }, EnhancedMonitoring='PER_TOPIC_PER_BROKER', KafkaVersion='2.2.1', NumberOfBrokerNodes=2 ) print(response)
Status code | Response model | Description |
---|---|---|
200 |
CreateClusterResponse | Successful response. |
400 | Error | The request isn't valid because the input is incorrect. Correct your input and then submit it again. |
401 | Error | The request is not authorized. The provided credentials couldn't be validated. |
403 | Error | Access forbidden. Check your credentials and then retry your request. |
404 | Error | The resource could not be found due to incorrect input. Correct the input, then retry the request. |
409 | Error | This cluster name already exists. Retry your request using another name. |
429 | Error | 429 response |
500 | Error | There was an unexpected internal server error. Retrying your request might resolve the issue. |
503 | Error | 503 response |
Schemas
Request bodies
{ "brokerNodeGroupInfo": { "brokerAZDistribution": enum, "clientSubnets": [ "string" ], "connectivityInfo": { "publicAccess": { "type": "string" }, "vpcConnectivity": { "clientAuthentication": { "sasl": { "iam": { "enabled": boolean }, "scram": { "enabled": boolean } }, "tls": { "enabled": boolean } } } }, "instanceType": "string", "securityGroups": [ "string" ], "storageInfo": { "ebsStorageInfo": { "provisionedThroughput": { "enabled": boolean, "volumeThroughput": integer }, "volumeSize": integer } }, "zoneIds": [ "string" ] }, "clientAuthentication": { "sasl": { "iam": { "enabled": boolean }, "scram": { "enabled": boolean } }, "tls": { "certificateAuthorityArnList": [ "string" ], "enabled": boolean }, "unauthenticated": { "enabled": boolean } }, "clusterName": "string", "configurationInfo": { "arn": "string", "revision": integer }, "encryptionInfo": { "encryptionAtRest": { "dataVolumeKMSKeyId": "string" }, "encryptionInTransit": { "clientBroker": enum, "inCluster": boolean } }, "enhancedMonitoring": enum, "kafkaVersion": "string", "loggingInfo": { "brokerLogs": { "cloudWatchLogs": { "enabled": boolean, "logGroup": "string" }, "firehose": { "deliveryStream": "string", "enabled": boolean }, "s3": { "bucket": "string", "enabled": boolean, "prefix": "string" } } }, "numberOfBrokerNodes": integer, "openMonitoring": { "prometheus": { "jmxExporter": { "enabledInBroker": boolean }, "nodeExporter": { "enabledInBroker": boolean } } }, "storageMode": enum, "tags": { } }
Response bodies
{ "clusterInfoList": [ { "activeOperationArn": "string", "brokerNodeGroupInfo": { "brokerAZDistribution": enum, "clientSubnets": [ "string" ], "connectivityInfo": { "publicAccess": { "type": "string" }, "vpcConnectivity": { "clientAuthentication": { "sasl": { "iam": { "enabled": boolean }, "scram": { "enabled": boolean } }, "tls": { "enabled": boolean } } } }, "instanceType": "string", "securityGroups": [ "string" ], "storageInfo": { "ebsStorageInfo": { "provisionedThroughput": { "enabled": boolean, "volumeThroughput": integer }, "volumeSize": integer } }, "zoneIds": [ "string" ] }, "clientAuthentication": { "sasl": { "iam": { "enabled": boolean }, "scram": { "enabled": boolean } }, "tls": { "certificateAuthorityArnList": [ "string" ], "enabled": boolean }, "unauthenticated": { "enabled": boolean } }, "clusterArn": "string", "clusterName": "string", "creationTime": "string", "currentBrokerSoftwareInfo": { "configurationArn": "string", "configurationRevision": integer, "kafkaVersion": "string" }, "currentVersion": "string", "customerActionStatus": enum, "encryptionInfo": { "encryptionAtRest": { "dataVolumeKMSKeyId": "string" }, "encryptionInTransit": { "clientBroker": enum, "inCluster": boolean } }, "enhancedMonitoring": enum, "loggingInfo": { "brokerLogs": { "cloudWatchLogs": { "enabled": boolean, "logGroup": "string" }, "firehose": { "deliveryStream": "string", "enabled": boolean }, "s3": { "bucket": "string", "enabled": boolean, "prefix": "string" } } }, "numberOfBrokerNodes": integer, "openMonitoring": { "prometheus": { "jmxExporter": { "enabledInBroker": boolean }, "nodeExporter": { "enabledInBroker": boolean } } }, "state": enum, "stateInfo": { "code": "string", "message": "string" }, "storageMode": enum, "tags": { }, "zookeeperConnectString": "string", "zookeeperConnectStringTls": "string" } ], "nextToken": "string" }
{ "clusterArn": "string", "clusterName": "string", "state": enum }
{ "invalidParameter": "string", "message": "string" }
Properties
BrokerAZDistribution
This parameter is currently not in use.
DEFAULT
BrokerLogs
The broker logs configuration for this MSK cluster.
Property | Type | Required | Description |
---|---|---|---|
cloudWatchLogs | False | Details of the CloudWatch Logs destination for broker logs. | |
firehose | False | Details of the Kinesis Data Firehose delivery stream that is the destination for broker logs. | |
s3 | False | Details of the Amazon S3 destination for broker logs. |
BrokerNodeGroupInfo
Describes the setup to be used for the broker nodes in the cluster.
Property | Type | Required | Description |
---|---|---|---|
brokerAZDistribution | False | This parameter is currently not in use. | |
clientSubnets | Array of type string | True | The list of subnets to connect to in the client virtual private cloud (VPC). Amazon creates elastic network interfaces inside these subnets. Client applications use elastic network interfaces to produce and consume data. If you use the US West (N. California) Region, specify exactly two subnets. For other Regions where Amazon MSK is available, you can specify either two or three subnets. The subnets that you specify must be in distinct Availability Zones. When you create a cluster, Amazon MSK distributes the broker nodes evenly across the subnets that you specify. Client subnets can't occupy the Availability Zone with ID |
connectivityInfo | False | Information about the cluster's connectivity setting. | |
instanceType | string MinLength: 5 MaxLength: 32 | True | The type of Amazon EC2 instances to use for brokers. The following instance types are allowed: kafka.m5.large, kafka.m5.xlarge, kafka.m5.2xlarge, kafka.m5.4xlarge, kafka.m5.8xlarge, kafka.m5.12xlarge, kafka.m5.16xlarge, and kafka.m5.24xlarge. |
securityGroups | Array of type string | False | The security groups to associate with the elastic network interfaces in order to specify who can connect to and communicate with the Amazon MSK cluster. If you don't specify a security group, Amazon MSK uses the default security group associated with the VPC. If you specify security groups that were shared with you, you must ensure that you have permissions to them. Specifically, you need the |
storageInfo | False | Contains information about storage volumes attached to Amazon MSK broker nodes. | |
zoneIds | Array of type string | False | The zoneIds for brokers in customer account. |
BrokerSoftwareInfo
Information about the current software installed on the cluster.
Property | Type | Required | Description |
---|---|---|---|
configurationArn | string | False | The Amazon Resource Name (ARN) of the configuration used for the cluster. This field isn't visible in this preview release. |
configurationRevision | integer Format: int64 | False | The revision of the configuration to use. This field isn't visible in this preview release. |
kafkaVersion | string | False | The version of Apache Kafka. You can use Amazon MSK to create clusters that use Apache Kafka versions 1.1.1 and 2.2.1. |
ClientAuthentication
Includes all client authentication information.
Property | Type | Required | Description |
---|---|---|---|
sasl | False | Details for client authentication using SASL. To turn on SASL, you must also turn on | |
tls | False | Details for ClientAuthentication using TLS. To turn on TLS access control, you must also turn on | |
unauthenticated | False | Details for ClientAuthentication using no authentication. |
ClientBroker
Client-broker encryption in transit setting.
TLS
TLS_PLAINTEXT
PLAINTEXT
CloudWatchLogs
Details of the CloudWatch Logs destination for broker logs.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | True | Specifies whether broker logs get sent to the specified CloudWatch Logs destination. |
logGroup | string | False | The CloudWatch log group that is the destination for broker logs. |
ClusterInfo
Returns information about a cluster.
Property | Type | Required | Description |
---|---|---|---|
activeOperationArn | string | False | Arn of active cluster operation. |
brokerNodeGroupInfo | False | Information about the broker nodes. | |
clientAuthentication | False | Includes all client authentication information. | |
clusterArn | string | False | The Amazon Resource Name (ARN) that uniquely identifies the cluster. |
clusterName | string | False | The name of the cluster. |
creationTime | string | False | The time when the cluster was created. |
currentBrokerSoftwareInfo | False | Information about the version of software currently deployed on the brokers in the cluster. | |
currentVersion | string | False | The current version of the MSK cluster. Cluster versions aren't simple integers. You can obtain the current
version by describing the cluster. An example version is
|
customerActionStatus | False | Determines if there is an action required from the customer. | |
encryptionInfo | False | Includes all encryption-related information. | |
enhancedMonitoring | False | Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster. This property has three possible values: | |
loggingInfo | False | You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs. | |
numberOfBrokerNodes | integer | False | The number of broker nodes in the cluster. |
openMonitoring | False | Settings for open monitoring using Prometheus. | |
state | False | The state of the cluster. Amazon MSK automatically renews certificates on clusters every 13 months. It sets the state of the cluster to | |
stateInfo | False | Includes information of the cluster state. | |
storageMode | False | This controls storage mode for supported storage tiers. | |
tags | object | False | Tags attached to the cluster. |
zookeeperConnectString | string | False | The connection string to use to connect to zookeeper cluster on plaintext port. |
zookeeperConnectStringTls | string | False | The connection string to use to connect to the Apache ZooKeeper cluster on a TLS port. |
ClusterState
The state of the cluster. Amazon MSK automatically renews certificates on clusters every 13 months. It sets the state of the cluster to MAINTENANCE
when it starts the certificate-update operation. It sets it back to ACTIVE
when the update is done. While a cluster is in the MAINTENANCE
state, you can continue to produce and consume data, but you can't perform any update operations on it. You can perform update operations on a cluster when it is in the ACTIVE
state.
ACTIVE
CREATING
UPDATING
DELETING
FAILED
MAINTENANCE
REBOOTING_BROKER
HEALING
ConfigurationInfo
Specifies the configuration to use for the brokers.
Property | Type | Required | Description |
---|---|---|---|
arn | string | True | ARN of the configuration to use. |
revision | integer Format: int64 Minimum: 1 | True | The revision of the configuration to use. |
ConnectivityInfo
Broker access controls.
Property | Type | Required | Description |
---|---|---|---|
publicAccess | False | Access control settings for the cluster's brokers. | |
vpcConnectivity | False | VPC connection control settings for brokers |
CreateClusterRequest
Creates a cluster.
Property | Type | Required | Description |
---|---|---|---|
brokerNodeGroupInfo | True | Information about the broker nodes in the cluster. | |
clientAuthentication | False | Includes all client authentication related information. | |
clusterName | string MinLength: 1 MaxLength: 64 | True | The name of the cluster. |
configurationInfo | False | Represents the configuration that you want MSK to use for the cluster. | |
encryptionInfo | False | Includes all encryption-related information. | |
enhancedMonitoring | False | Specifies the level of monitoring for the MSK cluster. The possible values are | |
kafkaVersion | string MinLength: 1 MaxLength: 128 | True | The version of Apache Kafka. You can use Amazon MSK to create clusters that use Apache Kafka versions 1.1.1 and 2.2.1. |
loggingInfo | False | Logging Info details. | |
numberOfBrokerNodes | integer | True | The number of broker nodes in the cluster. |
openMonitoring | False | The settings for open monitoring. | |
storageMode | False | This controls storage mode for supported storage tiers. | |
tags | object | False | Create tags when creating the cluster. |
CreateClusterResponse
Returns information about the created cluster.
Property | Type | Required | Description |
---|---|---|---|
clusterArn | string | False | The Amazon Resource Name (ARN) of the cluster. |
clusterName | string | False | The name of the MSK cluster. |
state | False | The state of the cluster. Amazon MSK automatically renews certificates on clusters every 13 months. It sets the state of the cluster to |
CustomerActionStatus
A type of an action required from the customer.
CRITICAL_ACTION_REQUIRED
ACTION_RECOMMENDED
NONE
EBSStorageInfo
Contains information about the EBS storage volumes attached to the broker nodes.
Property | Type | Required | Description |
---|---|---|---|
provisionedThroughput | False | EBS volume provisioned throughput information. | |
volumeSize | integer Minimum: 1 Maximum: 16384 | False | The size in GiB of the EBS volume for the data drive on each broker node. |
EncryptionAtRest
The data-volume encryption details. You can't update encryption at rest settings for existing clusters.
Property | Type | Required | Description |
---|---|---|---|
dataVolumeKMSKeyId | string | True | The Amazon Resource Name (ARN) of the Amazon KMS key for encrypting data at rest. If you don't specify a KMS key, MSK creates one for you and uses it. |
EncryptionInTransit
The settings for encrypting data in transit.
Property | Type | Required | Description |
---|---|---|---|
clientBroker | False | Indicates the encryption setting for data in transit between clients and brokers. You must set it to one of the following values.
The default value is | |
inCluster | boolean | False | When set to true, it indicates that data communication among the broker nodes of the cluster is encrypted. When set to false, the communication happens in plaintext. The default value is true. |
EncryptionInfo
Includes encryption-related information, such as the Amazon KMS key used for encrypting data at rest and whether you want MSK to encrypt your data in transit.
Property | Type | Required | Description |
---|---|---|---|
encryptionAtRest | False | The data-volume encryption details. | |
encryptionInTransit | False | The details for encryption in transit. |
EnhancedMonitoring
Specifies which Apache Kafka metrics Amazon MSK gathers and sends to Amazon CloudWatch for this cluster. This property has three possible values: DEFAULT
, PER_BROKER
, and PER_TOPIC_PER_BROKER
. For a list of the metrics associated with each of these three levels of monitoring, see Monitoring.
DEFAULT
PER_BROKER
PER_TOPIC_PER_BROKER
PER_TOPIC_PER_PARTITION
Error
Returns information about an error.
Property | Type | Required | Description |
---|---|---|---|
invalidParameter | string | False | The parameter that caused the error. |
message | string | False | The description of the error. |
Firehose
Firehose details for BrokerLogs.
Property | Type | Required | Description |
---|---|---|---|
deliveryStream | string | False | The Kinesis Data Firehose delivery stream that is the destination for broker logs. |
enabled | boolean | True | Specifies whether broker logs get sent to the specified Kinesis Data Firehose delivery stream. |
IAM
Details for SASL/IAM client authentication.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | SASL/IAM authentication is enabled or not. |
JmxExporter
Indicates whether you want to enable or disable the JMX Exporter.
Property | Type | Required | Description |
---|---|---|---|
enabledInBroker | boolean | True | Indicates whether you want to enable or disable the JMX Exporter. |
JmxExporterInfo
JMX Exporter details.
Property | Type | Required | Description |
---|---|---|---|
enabledInBroker | boolean | True | JMX Exporter being enabled in broker. |
ListClustersResponse
The response contains an array containing cluster information and a next token if the response is truncated.
Property | Type | Required | Description |
---|---|---|---|
clusterInfoList | Array of type ClusterInfo | False | Information on each of the MSK clusters in the response. |
nextToken | string | False | The paginated results marker. When the result of a |
LoggingInfo
You can configure your MSK cluster to send broker logs to different destination types. This is a container for the configuration details related to broker logs.
Property | Type | Required | Description |
---|---|---|---|
brokerLogs | True | You can configure your MSK cluster to send broker logs to different destination types. This configuration specifies the details of these destinations. |
NodeExporter
Indicates whether you want to enable or disable the Node Exporter.
Property | Type | Required | Description |
---|---|---|---|
enabledInBroker | boolean | True | Indicates whether you want to enable or disable the Node Exporter. |
NodeExporterInfo
Node Exporter details.
Property | Type | Required | Description |
---|---|---|---|
enabledInBroker | boolean | True | Node Exporter being enabled in broker. |
OpenMonitoring
JMX and Node monitoring for the MSK cluster.
Property | Type | Required | Description |
---|---|---|---|
prometheus | True | Prometheus exporter settings. |
OpenMonitoringInfo
JMX and Node monitoring for cluster.
Property | Type | Required | Description |
---|---|---|---|
prometheus | True | Prometheus details. |
Prometheus
Prometheus settings for open monitoring.
Property | Type | Required | Description |
---|---|---|---|
jmxExporter | False | Indicates whether you want to enable or disable the JMX Exporter. | |
nodeExporter | False | Indicates whether you want to enable or disable the Node Exporter. |
PrometheusInfo
Prometheus details.
Property | Type | Required | Description |
---|---|---|---|
jmxExporter | False | JMX Exporter details. | |
nodeExporter | False | Node Exporter details. |
ProvisionedThroughput
Contains information about provisioned throughput for EBS storage volumes attached to kafka broker nodes.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | Provisioned throughput is enabled or not. |
volumeThroughput | integer | False | Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second. |
PublicAccess
Broker access controls
Property | Type | Required | Description |
---|---|---|---|
type | string | False | DISABLED means that public access is turned off. SERVICE_PROVIDED_EIPS means that public access is turned on. |
S3
The details of the Amazon S3 destination for broker logs.
Property | Type | Required | Description |
---|---|---|---|
bucket | string | False | The name of the S3 bucket that is the destination for broker logs. |
enabled | boolean | True | Specifies whether broker logs get sent to the specified Amazon S3 destination. |
prefix | string | False | The S3 prefix that is the destination for broker logs. |
Sasl
Details for client authentication using SASL. To turn on SASL, you must also turn on EncryptionInTransit
by setting inCluster
to true. You must set clientBroker
to either TLS
or TLS_PLAINTEXT
. If you choose TLS_PLAINTEXT
, then you must also set unauthenticated
to true.
Scram
Details for SASL/SCRAM client authentication.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | SASL/SCRAM authentication is enabled or not. |
StateInfo
Includes information about the state of the cluster.
Property | Type | Required | Description |
---|---|---|---|
code | string | False | If the cluster is in an unusable state, this field contains the code that describes the issue. |
message | string | False | If the cluster is in an unusable state, this field contains a message that describes the issue. |
StorageInfo
Contains information about storage volumes attached to Amazon MSK broker nodes.
Property | Type | Required | Description |
---|---|---|---|
ebsStorageInfo | False | EBS volume information. |
StorageMode
Controls storage mode for various supported storage tiers.
LOCAL
TIERED
Tls
Details for client authentication using TLS.
Property | Type | Required | Description |
---|---|---|---|
certificateAuthorityArnList | Array of type string | False | List of AWS Private CA Amazon Resource Name (ARN)s. |
enabled | boolean | False | TLS authentication is enabled or not. |
Unauthenticated
Details for allowing no client authentication.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | Unauthenticated is enabled or not. |
VpcConnectivity
VPC connection control settings for brokers.
Property | Type | Required | Description |
---|---|---|---|
clientAuthentication | False | VPC connection control settings for brokers. |
VpcConnectivityClientAuthentication
Includes all client authentication information for VpcConnectivity.
Property | Type | Required | Description |
---|---|---|---|
sasl | False | Details for VpcConnectivity ClientAuthentication using SASL. | |
tls | False | Details for VpcConnectivity ClientAuthentication using TLS. |
VpcConnectivityIAM
Details for SASL/IAM client authentication for VpcConnectivity.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | SASL/IAM authentication is enabled or not. |
VpcConnectivitySasl
Details for client authentication using SASL for VpcConnectivity.
Property | Type | Required | Description |
---|---|---|---|
iam | False | Details for ClientAuthentication using IAM for VpcConnectivity. | |
scram | False | Details for SASL/SCRAM client authentication for VpcConnectivity. |
VpcConnectivityScram
Details for SASL/SCRAM client authentication for vpcConnectivity.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | SASL/SCRAM authentication is enabled or not. |
VpcConnectivityTls
Details for client authentication using TLS for vpcConnectivity.
Property | Type | Required | Description |
---|---|---|---|
enabled | boolean | False | TLS authentication is enabled or not. |
See also
For more information about using this API in one of the language-specific AWS SDKs and references, see the following: