# Amazon Managed Streaming for Apache Kafka Developer Guide > Developer guide for Amazon Managed Streaming for Apache Kafka. - [Welcome](https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.md) - [Setting up](https://docs.aws.amazon.com/msk/latest/developerguide/before-you-begin.md) - [Quota](https://docs.aws.amazon.com/msk/latest/developerguide/limits.md) - [Document history](https://docs.aws.amazon.com/msk/latest/developerguide/doc-history.md) ## [MSK Provisioned](https://docs.aws.amazon.com/msk/latest/developerguide/msk-provisioned.md) ### [Get started](https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.md) Get up and running with Amazon MSK. - [Create a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster.md): Create an MSK cluster using the AWS Management Console. - [Create an IAM role](https://docs.aws.amazon.com/msk/latest/developerguide/create-client-iam-role.md): Create an IAM policy that grants access to create topics on the cluster and to send data to those topics. - [Create a client machine](https://docs.aws.amazon.com/msk/latest/developerguide/create-client-machine.md): Create a client machine for creating an Apache Kafka topic and for producing and consuming data in Amazon MSK. - [Create a topic](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.md): Create a topic in your newly created MSK cluster. - [Produce and consume data](https://docs.aws.amazon.com/msk/latest/developerguide/produce-consume.md): Produce and consume data. - [View metrics](https://docs.aws.amazon.com/msk/latest/developerguide/view-metrics.md): View the metrics for your MSK cluster using CloudWatch. - [Delete the tutorial resources](https://docs.aws.amazon.com/msk/latest/developerguide/delete-cluster.md): Delete the resources that you created for this tutorial. ### [How it works](https://docs.aws.amazon.com/msk/latest/developerguide/msk-cluster-management.md) Common Amazon MSK operations. ### [Create a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-cluster.md) Create an MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [Create a cluster using AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster-console.md): The procedures in this topic describe the common task of creating an MSK Provisioned cluster using the Custom create option in the AWS Management Console. - [Create a provisioned Amazon MSK cluster using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster-cli.md): This process describes the common task of creating an MSK Provisioned cluster using the AWS CLI. - [Create an MSK Provisioned cluster with a custom Amazon MSK configuration using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster-cli-custom-config.md): This process describes the common task of creating an MSK Provisioned cluster with a custom Amazon MSK configuration using the AWS CLI. - [Create an MSK Provisioned cluster using the Amazon MSK API](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster-api.md): This process describes the common task of creating and MSK Provisioned cluster using the API. ### [List clusters](https://docs.aws.amazon.com/msk/latest/developerguide/msk-list-clusters.md) List all clusters in your account using the AWS Management Console, the AWS CLI, or the API. - [List clusters using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/list-clusters-console.md): To get a bootstrap broker for an Amazon MSK cluster, you need the cluster Amazon Resource Name (ARN). - [List clusters using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/list-clusters-cli.md): To get a bootstrap broker for an Amazon MSK cluster, you need the cluster Amazon Resource Name (ARN). - [List clusters using the API](https://docs.aws.amazon.com/msk/latest/developerguide/list-clusters-api.md): To get a bootstrap broker for an Amazon MSK cluster, you need the cluster Amazon Resource Name (ARN). ### [Connect to an MSK Provisioned cluster](https://docs.aws.amazon.com/msk/latest/developerguide/client-access.md) How to connect to your MSK Provisioned cluster from client machines. - [Turn on public access](https://docs.aws.amazon.com/msk/latest/developerguide/public-access.md): Amazon MSK gives you the option to turn on public access to the brokers of MSK Provisioned clusters running Apache Kafka 2.6.0 or later versions. ### [Access from within AWS](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access.md) To connect to an MSK cluster from inside AWS but outside the cluster's Amazon VPC, the following options exist. ### [Multi-VPC private connectivity in a single Region](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.md) Multi-VPC private connectivity for cross-account clients in the same Region. ### [Multi-VPC get started tutorial](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-getting-started.md) Step-by-step use case for setting up multi-VPC private connectivity - [Cluster owner turns on multi-VPC](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-turn-on.md): The MSK cluster owner needs to make configuration settings on the MSK cluster after the cluster is created and in an ACTIVE state. - [Cluster owner attaches cluster policy](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cluster-owner-action-policy.md): The cluster owner can attach a cluster policy (also known as a resource-based policy) to the MSK cluster where you will turn on multi-VPC private connectivity. - [Cross-account user configures VPC connections](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-user-action.md): To set up multi-VPC private connectivity between a client in a different account from the MSK cluster, the cross-account user creates a managed VPC connection for the client. - [Update auth schemes on a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-update-authschemes.md): Multi-VPC private connectivity supports several authorization schemes: SASL/SCRAM, IAM, and TLS. - [Reject managed VPC connection](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-reject-connection.md): From the Amazon MSK console on the cluster admin account, you can reject a client VPC connection. - [Delete managed VPC connection](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-delete-connection.md): The cross-account user can delete a managed VPC connection for an MSK cluster from the client account console. - [Multi-VPC private connectivity permissions](https://docs.aws.amazon.com/msk/latest/developerguide/mvpc-cross-account-permissions.md): This section summarizes the permissions needed for clients and clusters using the multi-VPC private connectivity feature. - [Port information](https://docs.aws.amazon.com/msk/latest/developerguide/port-info.md): Use the following port numbers so that Amazon MSK can communicate with client machines: ### [Get the bootstrap brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-get-bootstrap-brokers.md) Get the bootstrap brokers for your cluster using the AWS Management Console, the AWS CLI, or the API. - [Get the bootstrap brokers using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/get-bootstrap-console.md): This process describes how to get bootstrap brokers for a cluster using the AWS Management Console. - [Get the bootstrap brokers using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/get-bootstrap-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) that you obtained when you created your cluster. - [Get the bootstrap brokers using the API](https://docs.aws.amazon.com/msk/latest/developerguide/get-bootstrap-api.md): To get the bootstrap brokers using the API, see GetBootstrapBrokers. ### [Monitor a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/monitoring.md) Learn how to monitor your Amazon MSK Provisioned cluster. - [View metrics using CloudWatch](https://docs.aws.amazon.com/msk/latest/developerguide/cloudwatch-metrics.md): You can monitor metrics for Amazon MSK using the CloudWatch console, the command line, or the CloudWatch API. ### [CloudWatch metrics for Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details.md) Amazon MSK integrates with Amazon CloudWatch so that you can collect, view, and analyze CloudWatch metrics for your MSK Standard brokers. - [MSK Provisioned cluster states](https://docs.aws.amazon.com/msk/latest/developerguide/msk-cluster-states.md): The following table shows the possible states of a MSK Provisioned cluster and describes what they mean. - [CloudWatch metrics for Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/metrics-details-express.md): Amazon MSK integrates with CloudWatch so that you can collect, view, and analyze CloudWatch metrics for your MSK Express brokers. ### [Monitor with Prometheus](https://docs.aws.amazon.com/msk/latest/developerguide/open-monitoring.md) You can monitor your MSK Provisioned cluster with Prometheus, an open-source monitoring system for time-series metric data. - [Enable open monitoring on new clusters](https://docs.aws.amazon.com/msk/latest/developerguide/enable-open-monitoring-at-creation.md): This procedure describes how to enable open monitoring on a new MSK cluster using the AWS Management Console, the AWS CLI, or the Amazon MSK API. - [Enable open monitoring on existing Provisioned clusters](https://docs.aws.amazon.com/msk/latest/developerguide/enable-open-monitoring-after-creation.md): To enable open monitoring, make sure that the MSK Provisioned cluster is in the ACTIVE state. - [Set up a Prometheus host](https://docs.aws.amazon.com/msk/latest/developerguide/set-up-prometheus-host.md): This procedure describes how to set up a Prometheus host using a prometheus.yml file. - [Use Prometheus metrics](https://docs.aws.amazon.com/msk/latest/developerguide/prometheus-metrics.md): All metrics emitted by Apache Kafka to JMX are accessible using open monitoring with Prometheus. - [Store Prometheus metrics](https://docs.aws.amazon.com/msk/latest/developerguide/managed-service-prometheus.md): Amazon Managed Service for Prometheus is a Prometheus-compatible monitoring and alerting service that you can use to monitor Amazon MSK clusters. - [Monitor consumer lags](https://docs.aws.amazon.com/msk/latest/developerguide/consumer-lag.md): Monitoring consumer lag allows you to identify slow or stuck consumers that aren't keeping up with the latest data available in a topic. - [Use storage capacity alerts](https://docs.aws.amazon.com/msk/latest/developerguide/cluster-alerts.md): On Amazon MSK provisioned clusters, you choose the cluster's primary storage capacity. ### [Update cluster security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-security.md) Update the authentication and encryption settings for an existing Amazon MSK cluster using the AWS Management Console, the AWS CLI, or the API. - [Update cluster security settings using console](https://docs.aws.amazon.com/msk/latest/developerguide/update-security-console.md): Describes how to updated Amazon MSK cluster security settings using the AWS Management Console. - [Update cluster security settings using CLI](https://docs.aws.amazon.com/msk/latest/developerguide/update-security-cli.md): Describes how to updated Amazon MSK cluster security settings using the AWS CLI. - [Update cluster security settings using API](https://docs.aws.amazon.com/msk/latest/developerguide/update-security-api.md): Describes how to updated Amazon MSK cluster security settings using the API. - [Expand a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-broker-count.md): Increase the number of brokers in an Amazon MSK cluster using the AWS Management Console, the AWS CLI, or the API. - [Remove a broker](https://docs.aws.amazon.com/msk/latest/developerguide/msk-remove-broker.md): Increase the number of brokers in an Amazon MSK cluster using the AWS Management Console, the AWS CLI, or the API. - [Update cluster broker size](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-broker-type.md): Update the Amazon MSK cluster broker size for all the brokers in the cluster using the AWS Management Console, the AWS CLI, or the API. - [Use Cruise Control](https://docs.aws.amazon.com/msk/latest/developerguide/cruise-control.md): How to use Cruise Control to rebalance and monitor your Amazon MSK cluster. - [Update cluster configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-cluster-config.md): Update the configuration of a Amazon MSK clustercluster. - [Configure dual-stack network type](https://docs.aws.amazon.com/msk/latest/developerguide/mskp-choose-cluster-network-type.md): Configure IPv4 or dual-stack network type for your MSK Provisioned clusters. - [Reboot a broker for an Amazon MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-reboot-broker.md): Reboot a broker for an Amazon MSK cluster using the AWS Management Console, the AWS CLI, or the API. ### [Tag a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tagging.md) Assign your own metadata tags to your Amazon MSK resources (such as clusters) to help you manage them. - [Tag basics for clusters](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tagging-basics.md): Assign your own metadata tags to your Amazon MSK resources (such as clusters) to help you manage them. - [Track costs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tagging-billing.md): Assign your own metadata tags to your Amazon MSK resources, such as clusters, to help you manage them. - [Tag restrictions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tagging-restrictions.md): Assign your own metadata tags to your Amazon MSK resources, such as clusters, to help you manage them. - [Tag resources using API](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tagging-howto.md): Describes API operations to tag or untag an Amazon MSK resource or list the current set of tags. - [Migrate to MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/migration.md): Explore cluster migration options using Amazon MSK Replicator and Apache MirrorMaker 2.0. ### [Delete a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-delete-cluster.md) Delete an Amazon MSK cluster using the AWS Management Console, the AWS CLI, or the API. - [Delete an Amazon MSK Provisioned cluster using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/delete-cluster-console.md): This process describes how to delete an Amazon MSK Provisioned cluster using the AWS Management Console. - [Delete an Amazon MSK Provisioned cluster using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/delete-cluster-cli.md): This process describes how to delete an MSK Provisioned cluster using the AWS CLI. - [Delete an Amazon MSK Provisioned cluster using the API](https://docs.aws.amazon.com/msk/latest/developerguide/delete-cluster-api.md): The Amazon MSK API allows you to programmatically create and manage your MSK Provisioned cluster as part of automated infrastructure provisioning or deployment scripts. ### [Key features and concepts](https://docs.aws.amazon.com/msk/latest/developerguide/operations.md) Amazon MSK key features and concepts explanations and reference material. ### [Broker types](https://docs.aws.amazon.com/msk/latest/developerguide/broker-instance-types.md) MSK Provisioned offers two broker types - Standard and Express. - [Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-standard.md): Standard brokers for MSK Provisioned offer the most flexibility to configure your cluster's performance. - [Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-broker-types-express.md): Express brokers for MSK Provisioned make Apache Kafka simpler to manage, more cost-effective to run at scale, and more elastic with the low latency you expect. - [Broker sizes](https://docs.aws.amazon.com/msk/latest/developerguide/broker-instance-sizes.md): When you create an Amazon MSK Provisioned cluster you specify the size of brokers that you want it to have. ### [Storage management](https://docs.aws.amazon.com/msk/latest/developerguide/msk-storage-management.md) Configuring tiered storage, provisioning storage throughput, and scaling up broker storage. ### [Tiered storage](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tiered-storage.md) Enable and disable tiered storage mode on Amazon MSK clusters and topics. - [Tiered storage scenario](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tiered-storage-retention-rules.md): When you enable tiered storage for a new or existing topic, Apache Kafka copies closed log segments from primary storage to tiered storage. - [Create tiered storage cluster with console](https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-cluster-tiered-storage-console.md): Create a tiered storage-enabled Amazon MSK cluster with the AWS Management Console. - [Create tiered storage cluster with AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-cluster-tiered-storage-cli.md): Create a tiered storage-enabled Amazon MSK cluster with the AWS CLI. - [Enable and disable tiered storage on a topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-enable-disable-topic-tiered-storage-cli.md): Enable and disable tiered storage on a topic - [Enable tiered storage using CLI](https://docs.aws.amazon.com/msk/latest/developerguide/msk-enable-cluster-tiered-storage-cli.md): Enable tiered storage on an existing Amazon MSK cluster using AWS CLI - [Update tiered storage using console](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-tiered-storage-console.md): Make sure the current Apache Kafka version of your MSK cluster is 2.8.2.tiered. ### [Scale up broker storage](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-storage.md) Update the volume of storage associated with Amazon MSK brokers. ### [Automatic scaling for clusters](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand.md) Use automatic scaling for Amazon MSK to increase storage capacity for a Amazon MSK cluster. - [Auto-scaling policies](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand-details.md): An auto-scaling policy defines the following parameters for your cluster: ### [Set up automatic scaling](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand-setup.md) You can use the Amazon MSK console, the Amazon MSK API, or CloudFormation to implement automatic scaling for storage. - [Set up automatic scaling](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand-setup-console.md): This process describes how to use the Amazon MSK console to implement automatic scaling for storage. - [Set up auto scaling using CLI](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand-setup-cli.md): This process describes how to use the Amazon MSK CLI to implement automatic scaling for storage. - [Set up auto scaling using API](https://docs.aws.amazon.com/msk/latest/developerguide/msk-autoexpand-setup-api.md): This process describes how to use the Amazon MSK API to implement automatic scaling for storage. - [Manual scaling](https://docs.aws.amazon.com/msk/latest/developerguide/manually-expand-storage.md): To increase storage, wait for the cluster to be in the ACTIVE state. ### [Manage broker storage throughput](https://docs.aws.amazon.com/msk/latest/developerguide/msk-provision-throughput-management.md) manage provisioned storage throughput for brokers in a Amazon MSK cluster. - [Provision storage throughput for brokers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-provision-throughput.md): Configure provisioned storage throughput for brokers in a Amazon MSK cluster. ### [Broker configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration.md) Configure properties and values for Amazon MSK cluster, brokers, topics, and Apache ZooKeeper nodes. ### [Standard broker configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-standard.md) Setting Standard broker configuration properties for Amazon MSK cluster. - [Custom Amazon MSK configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-properties.md): Setting custom configuration properties for Amazon MSK cluster. - [Default Amazon MSK configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-default-configuration.md): The values of the properties in the default Amazon MSK configuration. - [Guidelines for tiered storage topic-level configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-guidelines-tiered-storage-topic-level-config.md): Guidelines for Amazon MSK tiered storage topic-level configuration. ### [Express broker configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-express.md) Setting Express broker configuration properties for Amazon MSK cluster. - [Custom read/write access configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-express-read-write.md): Create, describe, update, delete, and list custom Amazon MSK configurations, and use them to create Amazon MSK cluster. - [Read-only configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-express-read-only.md): Express brokers read-only configurations ### [Broker configuration operations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations.md) Create, describe, update, delete, and list custom Amazon MSK Express broker configurations, and use them to create Amazon MSK cluster. - [Create configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations-create.md): Create a custom Amazon MSK configuration, and use it to create Amazon MSK cluster. - [Update configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations-update.md): Update a custom Amazon MSK configuration. - [Delete configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations-delete.md): Delete a custom Amazon MSK configuration. - [Get configuration metadata](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations-describe.md): Get metadata about a custom Amazon MSK configuration. - [Get configuration revision details](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations-describe-revision.md): Get a detailed description of the Amazon MSK configuration revision. - [List configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-operations-list.md): Describes how to list all Amazon MSK configurations in your account for the current AWS Region - [Amazon MSK configuration states](https://docs.aws.amazon.com/msk/latest/developerguide/msk-configuration-states.md): An Amazon MSK configuration can be in one of the following states. ### [Configure custom domain names](https://docs.aws.amazon.com/msk/latest/developerguide/custom-domain-names.md) Advertise custom domain names instead of the default Amazon MSK-generated broker addresses. - [Set up a custom domain name end to end](https://docs.aws.amazon.com/msk/latest/developerguide/custom-domain-setup.md): Add, apply, track, and verify a custom domain name on your Amazon MSK cluster. ### [Intelligent rebalancing](https://docs.aws.amazon.com/msk/latest/developerguide/intelligent-rebalancing.md) Amazon MSK provides intelligent rebalancing for all new MSK Provisioned clusters with Express brokers. - [Scaling Amazon MSK clusters](https://docs.aws.amazon.com/msk/latest/developerguide/intelligent-rebalancing-scaling-clusters.md): With intelligent rebalancing, you can scale your clusters up or down by editing the broker count in your clusters in a single action. - [Steady state rebalancing](https://docs.aws.amazon.com/msk/latest/developerguide/intelligent-rebalancing-self-balancing-paritions.md) - [Patching Provisioned clusters](https://docs.aws.amazon.com/msk/latest/developerguide/patching-impact.md): The impact of software updates on Kafka clients and applications. - [Broker offline and client failover](https://docs.aws.amazon.com/msk/latest/developerguide/troubleshooting-offlinebroker-clientfailover.md): Kafka allows for an offline broker; a single offline broker in a healthy and balanced cluster following best practices will not see impact or cause failure to produce or consume. ### [Security](https://docs.aws.amazon.com/msk/latest/developerguide/security.md) Configure Amazon Managed Streaming for Apache Kafka to meet your security and compliance objectives, and learn how to use other AWS services that help you to secure your Amazon MSK resources. ### [Data protection](https://docs.aws.amazon.com/msk/latest/developerguide/data-protection.md) Learn how the AWS shared responsibility model applies to data protection in Amazon Managed Streaming for Apache Kafka. - [Amazon MSK encryption](https://docs.aws.amazon.com/msk/latest/developerguide/msk-encryption.md): How to encrypt your data when using Amazon MSK. - [TLS security policies](https://docs.aws.amazon.com/msk/latest/developerguide/msk-tls-security-policies.md): Select a TLS security policy to control the TLS protocol versions and cipher suites that your Amazon MSK cluster accepts, including TLS 1.3. ### [Get started with Amazon MSK encryption](https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption.md) Describes how to encrypt your data when using Amazon MSK. - [Specify encryption settings when creating a Amazon MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption-cluster-create.md): Describes how to specify encryption settings when creating a Amazon MSK cluster. - [Test Amazon MSK TLS encryption](https://docs.aws.amazon.com/msk/latest/developerguide/msk-working-with-encryption-test-tls.md): Describes how to test TLS encryption with Amazon MSK. - [APIs with Interface VPC Endpoints](https://docs.aws.amazon.com/msk/latest/developerguide/privatelink-vpc-endpoints.md): Use an Interface VPC endpoint, powered by AWS PrivateLink, to prevent traffic between your Amazon VPC and Amazon MSK APIs from leaving the Amazon network. ### [Authentication and authorization for Amazon MSK APIs](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam.md) How to authenticate clients and manage access to Amazon MSK APIs. ### [How Amazon MSK works with IAM](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam.md) Before you use IAM to manage access to Amazon MSK, you should understand what IAM features are available to use with Amazon MSK. - [Amazon MSK identity-based policies](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam-id-based-policies.md): With IAM identity-based policies, you can specify allowed or denied actions and resources as well as the conditions under which actions are allowed or denied. - [Amazon MSK resource-based policies](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam-resource-based-policies.md): Amazon MSK supports a cluster policy (also known as a resource-based policy) for use with Amazon MSK clusters. - [Authorization based on Amazon MSK tags](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam-tags.md): You can attach tags to Amazon MSK clusters. - [Amazon MSK IAM roles](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam-roles.md): An IAM role is an entity within your Amazon Web Services account that has specific permissions. ### [Identity-based policy examples](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_id-based-policy-examples.md) By default, IAM users and roles don't have permission to execute Amazon MSK API actions. - [Policy best practices](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_service-with-iam-policy-best-practices.md): Identity-based policies determine whether someone can create, access, or delete Amazon MSK resources in your account. - [Allow users to view their own permissions](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_id-based-policy-examples-view-own-permissions.md): This example shows how you might create a policy that allows IAM users to view the inline and managed policies that are attached to their user identity. - [Accessing one Amazon MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_id-based-policy-examples-access-one-cluster.md): In this example, you want to grant an IAM user in your Amazon Web Services account access to one of your clusters, purchaseQueriesCluster. - [Accessing Amazon MSK clusters based on tags](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_id-based-policy-examples-view-widget-tags.md): You can use conditions in your identity-based policy to control access to Amazon MSK resources based on tags. ### [Service-linked roles](https://docs.aws.amazon.com/msk/latest/developerguide/using-service-linked-roles.md) Describes service-linked roles to give Amazon MSK access to resources in your AWS account. - [Service-linked role permissions](https://docs.aws.amazon.com/msk/latest/developerguide/slr-permissions.md): Amazon MSK uses the service-linked role named AWSServiceRoleForKafka. - [Create a service-linked role](https://docs.aws.amazon.com/msk/latest/developerguide/create-slr.md): You don't need to create a service-linked role manually. - [Edit a service-linked role](https://docs.aws.amazon.com/msk/latest/developerguide/edit-slr.md): Amazon MSK does not allow you to edit the AWSServiceRoleForKafka service-linked role. - [Supported Regions for service-linked roles](https://docs.aws.amazon.com/msk/latest/developerguide/slr-regions.md): Amazon MSK supports using service-linked roles in all of the Regions where the service is available. ### [AWS managed policies](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol.md) Learn about AWS managed policies for Amazon MSK and recent changes to those policies. - [Managed policy AmazonMSKFullAccess](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol-AmazonMSKFullAccess.md): This policy grants administrative permissions that allow a principal full access to all Amazon MSK actions. - [Managed policy AmazonMSKReadOnlyAccess](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol-AmazonMSKReadOnlyAccess.md): This policy grants read-only permissions that allow users to view information in Amazon MSK. - [Managed policy KafkaServiceRolePolicy](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol-KafkaServiceRolePolicy.md): You can't attach KafkaServiceRolePolicy to your IAM entities. - [Managed policy AWSMSKReplicatorExecutionRole](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol-AWSMSKReplicatorExecutionRole.md): The AWSMSKReplicatorExecutionRole policy grants permissions to the Amazon MSK replicator to replicate data between MSK clusters. - [Managed policy updates](https://docs.aws.amazon.com/msk/latest/developerguide/security-iam-awsmanpol-updates.md): View details about updates to AWS managed policies for Amazon MSK since this service began tracking these changes. - [Troubleshoot Amazon MSK identity and access](https://docs.aws.amazon.com/msk/latest/developerguide/security_iam_troubleshoot.md): Use the following information to help you diagnose and fix common issues that you might encounter when working with Amazon MSK and IAM. ### [Authentication and authorization for Apache Kafka APIs](https://docs.aws.amazon.com/msk/latest/developerguide/kafka_apis_iam.md) You can use IAM to authenticate clients and to allow or deny Apache Kafka actions. ### [IAM access control](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.md) How to use IAM for authentication and authorization. ### [How IAM access control for Amazon MSK works](https://docs.aws.amazon.com/msk/latest/developerguide/how-to-use-iam-access-control.md) To use IAM access control for Amazon MSK, perform the following steps, which are described in detail in these topics: - [Create a Amazon MSK cluster that uses IAM access control](https://docs.aws.amazon.com/msk/latest/developerguide/create-iam-access-control-cluster-in-console.md): This section explains how you can use the AWS Management Console, the API, or the AWS CLI to create a Amazon MSK cluster that uses IAM access control. - [Configure clients for IAM access control](https://docs.aws.amazon.com/msk/latest/developerguide/configure-clients-for-iam-access-control.md): To enable clients to communicate with an MSK cluster that uses IAM access control, you can use either of these mechanisms: - [Create authorization policies for the IAM role](https://docs.aws.amazon.com/msk/latest/developerguide/create-iam-access-control-policies.md): Attach an authorization policy to the IAM role that corresponds to the client. - [Get the bootstrap brokers for IAM access control](https://docs.aws.amazon.com/msk/latest/developerguide/get-bootstrap-brokers-for-iam.md): See Get the bootstrap brokers for an Amazon MSK cluster. - [Semantics of IAM authorization policy actions and resources](https://docs.aws.amazon.com/msk/latest/developerguide/kafka-actions.md) - [Common use cases for client authorization policy](https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control-use-cases.md): The first column in the following table shows some common use cases. ### [Mutual TLS authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.md) How to use TLS-based client authentication with Amazon MSK. - [Create a Amazon MSK cluster that supports client authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication-cluster.md) - [Set up a client to use authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication-client.md): This process describes how to set up an Amazon EC2 instance to use as a client to use authentication. - [Produce and consume messages using authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication-messages.md): This process describes how to produce and consume messages using authentication. ### [SASL/SCRAM authentication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.md) How to use sign-in credentials-based client authentication with AWS Secrets Manager and Amazon MSK. - [How sign-in credentials authentication works](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-howitworks.md): Sign-in credentials authentication for Amazon MSK uses SASL/SCRAM (Simple Authentication and Security Layer/ Salted Challenge Response Mechanism) authentication. ### [Set up SASL/SCRAM authentication for an Amazon MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial.md) To set up a secret in AWS Secrets Manager, follow the Creating and Retrieving a Secret tutorial in the AWS Secrets Manager User Guide. - [Connecting to your cluster with sign-in credentials](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-tutorial-connect.md): After you create a secret and associate it with your cluster, you can connect your client to the cluster. - [Working with users](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-users.md): Creating users: You create users in your secret as key-value pairs. - [Rotating sign-in credentials](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-rotation.md): How to rotate SASL/SCRAM credentials and replace a username for an Amazon MSK cluster. - [Limitations when using SCRAM secrets](https://docs.aws.amazon.com/msk/latest/developerguide/msk-password-limitations.md): Note the following limitations when using SCRAM secrets: - [Apache Kafka ACLs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-acls.md): How to use Apache Kafka ACLs with Amazon MSK. - [Changing security groups](https://docs.aws.amazon.com/msk/latest/developerguide/change-security-group.md): How to change the security group that's associated with an existing Amazon MSK cluster. ### [Controlling access to Apache ZooKeeper](https://docs.aws.amazon.com/msk/latest/developerguide/zookeeper-security.md) How to limit access to the Apache ZooKeeper nodes in your Amazon MSK cluster. - [Disable or enable ZooKeeper access](https://docs.aws.amazon.com/msk/latest/developerguide/zookeeper-disable-access.md): How to disable or re-enable direct Apache ZooKeeper client access on your Amazon MSK cluster. - [To place your Apache ZooKeeper nodes in a separate security group](https://docs.aws.amazon.com/msk/latest/developerguide/zookeeper-security-group.md): To limit access to Apache ZooKeeper nodes, you can assign a separate security group to them. - [Using TLS security with Apache ZooKeeper](https://docs.aws.amazon.com/msk/latest/developerguide/zookeeper-security-tls.md): You can use TLS security for encryption in transit between your clients and your Apache ZooKeeper nodes. - [Compliance validation](https://docs.aws.amazon.com/msk/latest/developerguide/MSK-compliance.md): Learn what AWS services are in scope of a specific compliance program. - [Resilience](https://docs.aws.amazon.com/msk/latest/developerguide/disaster-recovery-resiliency.md): Learn how AWS architecture supports data redundancy, and learn about specific Amazon Managed Streaming for Apache Kafka features for data resiliency. - [Infrastructure security](https://docs.aws.amazon.com/msk/latest/developerguide/infrastructure-security.md): Learn how Amazon Managed Streaming for Apache Kafka isolates service traffic. ### [Amazon MSK logging](https://docs.aws.amazon.com/msk/latest/developerguide/msk-logging.md) Amazon MSK supports broker and authorizer logs on Standard and Express brokers. - [CloudTrail events](https://docs.aws.amazon.com/msk/latest/developerguide/logging-API-using-cloudtrail.md) ### [Metadata management](https://docs.aws.amazon.com/msk/latest/developerguide/metadata-management.md) Amazon MSK supports Apache ZooKeeper and KRaft metadata management modes. - [Kafka cluster ID for KRaft clusters](https://docs.aws.amazon.com/msk/latest/developerguide/kraft-cluster-id.md): Each Apache Kafka cluster has a cluster ID that identifies it to Kafka clients and tools. - [Migrate from ZooKeeper to KRaft mode](https://docs.aws.amazon.com/msk/latest/developerguide/zk-to-kraft-migration.md): You can migrate an existing ZooKeeper-based MSK cluster to KRaft mode using the UpdateClusterKafkaVersion API. ### [Topic Operations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-topic-operations-information.md) Manage topics in your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. ### [List topics](https://docs.aws.amazon.com/msk/latest/developerguide/msk-list-topics.md) List all topics in your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [List topics using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/list-topics-console.md) - [List topics using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/list-topics-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) of your cluster. - [List topics using the API](https://docs.aws.amazon.com/msk/latest/developerguide/list-topics-api.md): To list topics using the API, see ListTopics. ### [Describe a topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-describe-topic.md) Get detailed information about a specific topic in your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [Describe a topic using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/describe-topic-console.md) - [Describe a topic using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/describe-topic-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) of your cluster and TopicName with the name of the topic you want to describe. - [Describe a topic using the API](https://docs.aws.amazon.com/msk/latest/developerguide/describe-topic-api.md): To describe a topic using the API, see DescribeTopic. ### [Describe topic partitions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-describe-topic-partitions.md) View detailed partition information for a specific topic in your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [View partition information using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/describe-topic-partitions-console.md) - [View partition information using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/describe-topic-partitions-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) of your cluster and TopicName with the name of the topic. - [View partition information using the API](https://docs.aws.amazon.com/msk/latest/developerguide/describe-topic-partitions-api.md): To view partition information using the API, see DescribeTopicPartitions. ### [Create topics](https://docs.aws.amazon.com/msk/latest/developerguide/msk-create-topic.md) Create topics in your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [Create topics using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic-console.md) - [Create a topic using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) of your cluster. - [Create a topic using the API](https://docs.aws.amazon.com/msk/latest/developerguide/create-topic-api.md): To create a topic using the API, see CreateTopic. ### [Update a topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-update-topic.md) Update a topic in your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [Update a topic using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/update-topic-console.md) - [Update a topic using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/update-topic-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) of your cluster and TopicName with the name of the topic you want to update. - [Update a topic using the API](https://docs.aws.amazon.com/msk/latest/developerguide/update-topic-api.md): To update a topic using the API, see UpdateTopic. ### [Delete a topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-delete-topic.md) Delete a topic from your MSK Provisioned cluster using the AWS Management Console, the AWS CLI, or the API. - [Delete a topic using the AWS Management Console](https://docs.aws.amazon.com/msk/latest/developerguide/delete-topic-console.md) - [Delete a topic using the AWS CLI](https://docs.aws.amazon.com/msk/latest/developerguide/delete-topic-cli.md): Run the following command, replacing ClusterArn with the Amazon Resource Name (ARN) of your cluster and TopicName with the name of the topic you want to delete. - [Delete a topic using the API](https://docs.aws.amazon.com/msk/latest/developerguide/delete-topic-api.md): To delete a topic using the API, see DeleteTopic. - [Resources](https://docs.aws.amazon.com/msk/latest/developerguide/resources.md): Amazon MSK resources. ### [Apache Kafka versions](https://docs.aws.amazon.com/msk/latest/developerguide/kafka-versions.md) Information about the Apache Kafka versions that Amazon Managed Streaming for Apache Kafka (Amazon MSK) supports and related cluster operations. - [Supported Apache Kafka versions](https://docs.aws.amazon.com/msk/latest/developerguide/supported-kafka-versions.md): Amazon Managed Streaming for Apache Kafka (Amazon MSK) supports the following Apache Kafka and Amazon MSK versions. ### [Amazon MSK version support](https://docs.aws.amazon.com/msk/latest/developerguide/version-support.md) This topic describes the Amazon MSK version support policy and the procedure for Upgrade the Apache Kafka version. - [Upgrade the Apache Kafka version](https://docs.aws.amazon.com/msk/latest/developerguide/version-upgrades.md): You can upgrade an existing MSK cluster to a newer version of Apache Kafka. - [Best practices for version upgrades](https://docs.aws.amazon.com/msk/latest/developerguide/version-upgrades-best-practices.md): To ensure client continuity during the rolling update that is performed as part of the Kafka version upgrade process, review the configuration of your clients and your Apache Kafka topics as follows: - [Troubleshoot Amazon MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/troubleshooting.md): Troubleshoot issues with your Amazon MSK cluster. ### [Best practices](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-intro.md) This section describes best practices to follow for Standard brokers and Express brokers. - [Best practices for Standard brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices.md): Best practices for managing your MSK Provisioned cluster when using Standard brokers. - [Recommended CloudWatch alarms](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-cw-alarms.md): Recommended CloudWatch alarms to monitor the health of your Amazon MSK Provisioned clusters. - [Best practices for Express brokers](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-express.md): Best practices for managing your Amazon MSK cluster when using Express brokers. - [Best practices for Apache Kafka clients](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-kafka-client.md): Best practices for managing Apache Kafka clients. - [Analyze best practices with the managing-amazon-msk skill](https://docs.aws.amazon.com/msk/latest/developerguide/bestpractices-msk-skill.md): Use the managing-amazon-msk Agent Skill to assess your Amazon MSK cluster and clients against best practices, diagnose operational issues, and implement recommended configurations. ## [MSK Serverless](https://docs.aws.amazon.com/msk/latest/developerguide/serverless.md) ### [Use MSK Serverless clusters](https://docs.aws.amazon.com/msk/latest/developerguide/serverless-getting-started.md) Get up and running with Amazon MSK serverless clusters. - [Create a cluster](https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster.md): Create an Amazon MSK Serverless cluster. - [Create an IAM role](https://docs.aws.amazon.com/msk/latest/developerguide/create-iam-role.md): Create an IAM role that can create topics on the cluster and write to them. - [Create a client machine](https://docs.aws.amazon.com/msk/latest/developerguide/create-serverless-cluster-client.md): Create a client that can access the serverless cluster. - [Create a topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-serverless-create-topic.md): Create an Apache Kafka topic on an MSK Serverless cluster. - [Produce and consume data](https://docs.aws.amazon.com/msk/latest/developerguide/msk-serverless-produce-consume.md): In this step, you produce and consume data using the topic that you created in the previous step. - [Delete resources](https://docs.aws.amazon.com/msk/latest/developerguide/delete-resources.md): Delete the resources that you created in this getting started tutorial for MSK Serverless. - [Configuration properties](https://docs.aws.amazon.com/msk/latest/developerguide/serverless-config.md): Amazon MSK sets broker configuration properties for serverless clusters. - [Configure dual-stack network type](https://docs.aws.amazon.com/msk/latest/developerguide/serverless-config-dual-stack.md): Amazon MSK supports dual-stack network type for existing MSK Serverless clusters that use Kafka version 3.6.0 or later at no additional cost. - [Monitoring](https://docs.aws.amazon.com/msk/latest/developerguide/serverless-monitoring.md): Amazon MSK integrates with Amazon CloudWatch so that you can collect, view, and analyze metrics for your MSK Serverless cluster. ## [MSK Connect](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.md) ### [Getting started](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-getting-started.md) This is a step-by-step tutorial that uses the AWS Management Console to create an MSK cluster and a sink connector that sends data from the cluster to an S3 bucket. - [Set up resources required for MSK Connect](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-tutorial-setup.md): In this step you create the following resources that you need for this getting-started scenario: - [Create custom plugin](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-plugin.md): A plugin contains the code that defines the logic of the connector. - [Create client machine and Apache Kafka topic](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-topic.md): In this step you create an Amazon EC2 instance to use as an Apache Kafka client instance. - [Create connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-connector.md): This procedure describes how to create a connector using the AWS Management Console. - [Send data to the MSK cluster](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-send-data.md): In this step you send data to the Apache Kafka topic that you created earlier, and then look for that same data in the destination S3 bucket. ### [Understand connectors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-connectors.md) A connector integrates external systems and Amazon services with Apache Kafka by continuously copying streaming data from a data source into your Apache Kafka cluster, or continuously copying data from your cluster into a data sink. - [Understand connector capacity](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-capacity.md): The total capacity of a connector depends on the number of workers that the connector has, as well as on the number of MSK Connect Units (MCUs) per worker. - [Understand maximum autoscaling task count](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-max-autoscaling-task-count.md): The maxAutoscalingTaskCount parameter is an optional capacity field available for autoscaling connectors in Amazon MSK Connect. - [Configure dual-stack network type](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-dual-stack.md): Amazon MSK Connect supports dual-stack network type for new connectors. - [Create a connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-create-connector-intro.md): This procedure describes how to create a connector using the AWS Management Console. - [Update a connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-update-connector.md) - [Restart a connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-restart-connector.md): You can restart a newly created connector and all of its tasks, or selectively restart only the failed tasks, while preserving configuration and committed offsets. - [Pause and resume a connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-pause-resume-connector.md): You can pause a running connector to stop it from processing records, and resume it later. - [Connecting from connectors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-from-connectors.md): The following best practices can improve the performance of your connectivity to Amazon MSK Connect. - [Create custom plugins](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-plugins.md): A plugin is an AWS resource that contains the code that defines your connector logic. ### [Understand MSK Connect workers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-workers.md) Learn about workers for Amazon MSK Connect. - [Supported worker configuration properties](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-supported-worker-config-properties.md): MSK Connect provides a default worker configuration. - [Create a custom configuration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-create-custom-worker-config.md): This procedure describes how to create a custom worker configuration using the AWS Management Console. ### [Manage connector offsets](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-manage-connector-offsets.md) This section provides information to help you manage source connector offsets using the offset storage topic. - [Use the default offset storage topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-default-offset-storage-topic.md): By default, Amazon MSK Connect generates a new offset storage topic on your Kafka cluster for each connector that you create. - [Use custom offset storage topic](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-set-offset-storage-topic.md): To provide offset continuity between source connectors, you can use an offset storage topic of your choice instead of the default topic. - [Configuration providers](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-config-provider.md): This page provides information about externalizing secrets for Amazon MSK Connect with an open source configuration provider. ### [IAM roles and policies](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-iam.md) This section helps you set up the appropriate IAM policies and roles to securely deploy and manage Amazon MSK Connect within your AWS environment. - [Understand service execution role](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-service-execution-role.md) - [Example policy](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-iam-policy-examples.md): To give a non-admin user full access to all MSK Connect functionality, attach a policy like the following one to the user's IAM role. - [Prevent cross-service confused deputy problem](https://docs.aws.amazon.com/msk/latest/developerguide/cross-service-confused-deputy-prevention.md): The confused deputy problem is a security issue where an entity that doesn't have permission to perform an action can coerce a more-privileged entity to perform the action. - [AWS managed policies](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-security-iam-awsmanpol.md): Learn about AWS managed policies for MSK Connect and recent changes to those policies. - [Use service-linked roles](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-using-service-linked-roles.md): How to use service-linked roles to give MSK Connect access to resources in your AWS account. ### [Enable internet access](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-internet-access.md) This page includes information to help you configure internet access for Amazon MSK Connect. - [Set up a NAT gateway](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-internet-access-private-subnets-example.md): The following steps show you how to set up a NAT gateway to enable internet access for a connector. ### [Understand private DNS hostnames](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-dns.md) Describes Private DNS hostname resolution feature in MSK Connect. - [Configure a VPC DHCP option](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-dns-config-dhcp.md): Describes the configuration requirements for publicly and privately resolvable DNS hostnames. - [Configure DNS attributes](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-dns-attributes.md): Describes recommended settings for your connector VPC. - [Handle connector creation failures](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-dns-failure-handling.md): Possible connector creation failures associated with DNS resolution and suggested actions to resolve the issues. - [Security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-security.md): Security and data protection in Amazon MSK Connect using Amazon MSK APIs with Interface VPC Endpoints. - [Logging](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-logging.md): MSK Connect can write log events that you can use to debug your connector. - [Monitoring MSK Connect](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-monitoring-overview.md): Monitor MSK Connect to maintain reliability, availability, and performance. ### [Examples](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-examples.md) Examples for Amazon MSK Connect that demonstrate how to set up common connectors and configuration providers. - [Set up Amazon S3 sink connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-S3sink-connector-example.md): This example shows how to set up the Confluent Amazon S3 sink connector for Amazon MSK connect. - [Set up EventBridge Kafka sink connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-eventbridge-kafka-connector.md): Learn how to set up MSK Connect with EventBridge to stream data from Apache Kafka topics to EventBridge for building event-driven architectures. ### [Use Debezium source connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-debeziumsource-connector-example.md) This example shows how to use the Debezium MySQL connector with a MySQL-compatible Amazon Aurora database as the source. - [Complete prerequisites to use Debezium source connector](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-debeziumsource-connector-example-prereqs.md): Your connector must be able to access the internet so that it can interact with services such as AWS Secrets Manager that are outside of your Amazon Virtual Private Cloud. - [Create a Debezium source connector](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-debeziumsource-connector-example-steps.md): This procedure describes how to create a Debezium source connector. - [Update a Debezium connector configuration](https://docs.aws.amazon.com/msk/latest/developerguide/mkc-debeziumsource-connector-update.md): To update the configuration of the Debezium connector, follow these steps: ### [Migrate to Amazon MSK Connect](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-migrating.md) This section describes how to migrate your Apache Kafka connector application to Amazon Managed Streaming for Apache Kafka Connect (Amazon MSK Connect). - [Understand internal topics used by Kafka Connect](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-kafka-connect-topics.md): An Apache Kafka Connect application that’s running in distributed mode stores its state by using internal topics in the Kafka cluster and group membership. - [State management](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-state-management.md): By default, Amazon MSK Connect creates three separate topics in the Kafka cluster for each Amazon MSK Connector to store the connector’s configuration, offset, and status. - [Migrate source connectors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-migrate-source-connectors.md): Source connectors are Apache Kafka Connect applications that import records from external systems into Kafka. - [Migrate sink connectors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-migrate-sink-connectors.md): Sink connectors are Apache Kafka Connect applications that export data from Kafka to external systems. - [Troubleshooting](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-troubleshooting.md): The following information can help you troubleshoot problems that you might have while using MSK Connect. ## [MSK Replicator](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator.md) ### [Get started](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-get-started.md) Review supported configurations and prerequisites, and create your first replicator. - [Supported configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-supported-configs.md): The following are requirements for cluster types, Kafka versions, instance types, and network configurations when using MSK Replicator. - [Prerequisites](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-prerequisites.md): Before you create an MSK Replicator, ensure that you have the following prerequisites in place. - [Prepare source and target clusters](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-prepare-clusters.md): Before creating an MSK Replicator, you need to prepare both a source cluster and a target cluster. - [Create your first replicator](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-first-replicator.md): This walkthrough guides you through creating your first MSK Replicator between two MSK clusters using the AWS console. ### [Replicator operations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-manage.md) Create, edit, delete, and monitor MSK Replicators using the AWS console, CLI, or infrastructure as code. ### [Create a replicator](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-create-op.md) Before you create an MSK Replicator, make sure you have the IAM permissions required to create an MSK Replicator and have completed Prepare source and target clusters. - [Using the console](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-create-console.md) - [Using the CLI](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-create-cli.md): Use the create-replicator command to create an MSK Replicator. - [Using CloudFormation/Terraform](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-create-cfn.md) - [Edit a replicator](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-edit.md): You cannot change the source cluster, target cluster, Replicator starting position, or topic name replication configuration once the MSK Replicator has been created. - [Delete a replicator](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-delete.md): You may need to delete an MSK Replicator if it fails to create (FAILED status). - [Monitor replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-monitor.md): You can use Amazon CloudWatch in the target cluster Region to view metrics for ReplicationLatency, MessageLag, and ReplicatorThroughput at a topic and aggregate level for each MSK Replicator. ### [Replication patterns](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-patterns.md) Set up same-region, cross-region, active-passive, and active-active replication patterns, and perform failover and failback operations. - [Same-region replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-same-region.md): In same-region replication (SRR), both the source and target MSK clusters are in the same AWS Region. - [Cross-region replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-cross-region.md): In cross-region replication (CRR), the source and target MSK clusters are in different AWS Regions. - [Active-passive replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-active-passive.md): In an active-passive setup, only one MSK cluster at a time is actively serving streaming data, while the other cluster is on standby. - [Active-active replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-active-active.md): In an active-active setup, both MSK clusters are actively serving reads and writes. - [Failover to the secondary Region](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-when-failover.md): We recommend that you monitor replication latency in the secondary AWS Region using Amazon CloudWatch. - [Planned failover](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-planned-failover.md): You can conduct a planned failover to test the resiliency of your application against an unexpected event in your primary AWS Region. - [Unplanned failover](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-unplanned-failover.md): You can conduct an unplanned failover when there is a service event in the primary AWS Region and you want to temporarily redirect your traffic to the secondary Region. - [Failback](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-failback.md): You can failback to the primary AWS Region after the service event in that Region has ended. ### [Migration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-migration.md) Migrate between Amazon MSK clusters, from self-managed Kafka to Amazon MSK, and from MirrorMaker 2 to Amazon MSK Replicator. - [Migrate between Amazon MSK clusters](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-migrate-cluster.md): Amazon MSK Replicator supports migrations between Amazon MSK clusters in the same AWS account. ### [Migrate from self-managed Kafka](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-migrate-external.md) You can use MSK Replicator to migrate Apache Kafka workloads from self-managed environments to Amazon MSK Provisioned clusters. - [Self-managed Kafka prerequisites](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-external-prereqs.md) - [API examples for self-managed Kafka](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-external-api-examples.md) - [Migrate from MirrorMaker 2](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-migrate-mm2.md): To migrate from MirrorMaker (MM2) to MSK Replicator, follow these steps: - [Assess readiness with AI](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-migrate-ai-assist.md): Before you begin a migration, you can use AI-assisted tooling to evaluate your source Apache Kafka cluster and plan your target Amazon MSK deployment. - [Cross-account migration](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-cross-account.md): For migrations across different AWS accounts, you must use Apache MirrorMaker 2.0. ### [Key concepts](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-concepts.md) Understand how Amazon MSK Replicator works, including data replication, topic naming, metadata replication, consumer group offset syncing, and network requirements. - [How replication works](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-how-replication-works.md): When you create a Replicator, MSK Replicator deploys all required resources in the target cluster's AWS Region to optimize for data replication latency. - [Topic naming](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-topic-naming.md): MSK Replicator has two topic name configuration modes: Prefixed (default) or Identical topic name replication. - [Metadata and ACL replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-metadata-acl.md): MSK Replicator supports copying metadata from the source cluster to the target cluster. - [Consumer group offset synchronization](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-bidirectional-offset-sync.md): MSK Replicator can synchronize consumer group offsets from a source cluster to a target cluster, allowing consumers to switch clusters and resume processing without skipping records. - [Network requirements](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-network-reqs.md): MSK Replicator uses IAM access control to connect to your clusters, which uses port 9098. ### [Security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-security.md) Understand the security model for Amazon MSK Replicator, including the service execution role, encryption, and IAM permissions. ### [Service execution role (SER)](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-ser.md) MSK Replicator uses a service execution role (SER) to read from your source cluster and write to your target cluster. - [Additional SER permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-ser-additional-perms.md): The AWSMSKReplicatorExecutionRole managed policy covers cluster, topic, and consumer group permissions for IAM auth. ### [Service execution role permissions reference](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-permissions-reference.md) Understand the minimal IAM permissions and Kafka access control lists (ACLs) required by the Amazon MSK Replicator service execution role for each replicator feature. - [Topic replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-perms-topic-replication.md): Use Amazon MSK Replicator to replicate topics between clusters. - [Topic configuration replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-perms-topic-config-replication.md): When copyTopicConfigurations is enabled (the default), the replicator reads topic-level configurations from the source cluster. - [ACL replication](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-perms-acl-replication.md): When copyAccessControlListsForTopics is enabled (the default), the replicator copies access control lists from the source cluster to the target cluster. - [Consumer group offset sync](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-perms-consumer-group-sync.md): When synchroniseConsumerGroupOffsets is enabled (the default), the replicator translates consumer group offsets from the source cluster and writes them to the target cluster. - [Log delivery](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-perms-log-delivery.md): Grant log delivery permissions to the caller role (the IAM principal that calls CreateReplicator), not to the service execution role. - [Combined policy example](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-perms-combined-example.md): The following example shows a complete service execution role policy for a replicator with all features enabled, including topic configuration replication and LEGACY offset sync mode. - [Encryption](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-encryption.md): All communication and data between MSK Replicator and your clusters is always encrypted in-transit. ### [Required IAM permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-create-iam-perms.md) The IAM principal (user or role) that calls CreateReplicator needs the permissions described in this section. - [Log delivery permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-create-iam-perms-logs.md): If you configure log delivery on the Replicator, append the appropriate statements below to the base policy. - [IAM permissions reference](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-iam-ref.md): The following table summarizes the IAM actions that the API caller (the IAM principal that calls CreateReplicator and other replicator APIs) needs. - [Pricing](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-pricing.md): Understand the pricing model for Amazon MSK Replicator. ### [Observability](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-observability.md) Monitor Amazon MSK Replicator using Amazon CloudWatch metrics and troubleshoot common issues. - [Metrics reference](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-metrics-ref.md): The following metrics describes performance or connection metrics for the MSK Replicator. - [Monitor replication from self-managed Kafka clusters](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-monitor-external.md): After you create an MSK Replicator to replicate data from a self-managed Apache Kafka cluster to an Amazon MSK Provisioned cluster, monitor the following Amazon CloudWatch metrics to track replication progress and determine when your applications are ready to migrate. - [MSK Replicator logs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-logs.md): MSK Replicator supports forwarding operational logs to help you monitor and troubleshoot replication. ### [Best practices](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-best-practices.md) Best practices for sizing, throughput management, and retention when using Amazon MSK Replicator. - [Sizing and partitions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-bp-sizing.md): The number of partitions on your source and target MSK clusters directly impacts replication performance. - [Managing throughput with Kafka quotas](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-bp-quotas.md): Since MSK Replicator acts as a consumer for your source cluster, replication can cause other consumers to be throttled on your source cluster. - [Retention period](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-bp-retention.md): You can set the log retention period for MSK Provisioned and Serverless clusters. - [Cluster configuration changes after Replicator creation](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-post-creation-config.md) - [Considerations for multi-Region applications](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-bp-multi-region.md): When building multi-Region Apache Kafka applications with MSK Replicator, keep the following in mind: - [Troubleshooting](https://docs.aws.amazon.com/msk/latest/developerguide/msk-replicator-troubleshooting.md): The following information can help you troubleshoot problems with MSK Replicator. ## [MSK streaming tables to Apache Iceberg (New)](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg.md) - [Key concepts](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-concepts.md): This section describes the key concepts behind Amazon MSK Data Delivery. ### [Get started](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-getting-started.md) - [Supported configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-supported-configs.md) - [Prerequisites](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-prerequisites.md) - [Create your first Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-first-channel.md): To create your first Channel, complete the following steps: ### [IAM permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-iam.md) There are two distinct sets of permissions: the Channel lifecycle management permissions that you need to create and manage Channels, and the service execution role that the Channel assumes to deliver data. - [Channel lifecycle management permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-iam-lifecycle.md): The following permissions are for the IAM principal (the user or role) you use to create, update, describe, delete, and list Channels. - [Service execution role](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-iam-service-role.md): A Channel assumes a service execution role to deliver data. ### [Manage Channels](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-manage.md) Resource states: CREATING, ACTIVE, UPDATING, DELETING, FAILED, SUSPENDING, SUSPENDED. - [Create a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-create.md) - [Describe a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-describe.md) - [Update a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-update.md): You can modify only the data freshness interval (DataFreshnessInSeconds, 300–900). - [Delete a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-delete.md): Deleting a Channel stops delivery from the Kafka topic to the destination. - [List Channels](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-list.md) ### [Iceberg table behaviors](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-behaviors.md) This section describes how a Channel maps schemas and manages the Apache Iceberg table it creates in Amazon S3 Tables. - [Schema type mapping](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-schema-mapping.md): When you register a schema in the AWS Glue Schema Registry, the Channel maps the JSON Schema types to Iceberg column types as follows. - [Table type](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-table-type.md): The Iceberg table that a Channel creates in S3 Tables is managed by the Amazon MSK service. - [Table maintenance](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-maintenance.md): S3 Tables provides optional automated table maintenance jobs — compaction, snapshot expiration, and unreferenced file cleanup. - [Table attributes](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-attributes.md): The Iceberg table a Channel creates has the following attributes (designed for governance and compliance): - [Core library version](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-version.md) ### [Security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-security.md) - [Encryption in transit](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-security-transit.md): A Channel encrypts all data in transit between Amazon MSK Express brokers and the Channel service using TLS 1.2 or higher. - [Encryption at rest](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-security-rest.md): Data delivered to S3 Table buckets is encrypted at rest using the destination bucket's default encryption settings: - [IAM and access control](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-security-iam.md): A Channel uses IAM roles for authorization. ### [Monitoring](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-monitoring.md) A Channel publishes data-delivery metrics to Amazon CloudWatch under the AWS/Kafka namespace. - [Amazon CloudWatch metrics](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-metrics.md) - [Metric dimensions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-metric-dimensions.md) - [Recommended alarms](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-alarms.md) - [Metrics in the console](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-viewing-metrics.md) ### [Logging](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-logging.md) - [CloudTrail logging](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-cloudtrail.md): All Channel API calls are logged in AWS CloudTrail: CreateChannel, DescribeChannel, UpdateChannel, DeleteChannel, ListChannels. - [Amazon CloudWatch Logs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-cw-logs.md): A Channel can publish operational logs to a Amazon CloudWatch Logs log group, including schema resolution events, delivery attempts and outcomes, and error details for failed deliveries. ### [Best practices](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-bestpractices.md) - [Throughput and data freshness](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-bp-throughput.md): Follow these recommendations for throughput and data freshness: - [Schema management](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-bp-schema.md) - [Security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-bp-security.md) ### [Troubleshooting](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-troubleshooting.md) - [Channel stuck in CREATING state](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-creating.md) - [Checking operation status](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-operation-status.md) - [Data freshness higher than configured](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-freshness.md) - [Throughput exceeds the supported limit](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-throughput.md) - [Failed records (FailedRowCount greater than 0)](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-failed-records.md) - [No data appearing in the table](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-no-data.md) - [Channel is suspended](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-suspended.md) - [Delivery stops after a schema change](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-schema-change.md) - [Permission denied errors in Amazon CloudWatch Logs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-permission-denied.md) - [Channel not available on cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-iceberg-ts-not-available.md) ## [MSK data delivery to Amazon S3 (New)](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3.md) - [Key concepts](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-concepts.md): This section describes the key concepts behind Amazon MSK Data Delivery. ### [Get started](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-getting-started.md) - [Supported configurations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-supported-configs.md) - [Prerequisites](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-prerequisites.md) - [Create your first Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-first-channel.md): To create your first Channel, complete the following steps: ### [IAM permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-iam.md) There are two distinct sets of permissions: the Channel lifecycle management permissions that you need to create and manage Channels, and the service execution role that the Channel assumes to deliver data. - [Channel lifecycle management permissions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-iam-lifecycle.md): The following permissions are for the IAM principal (the user or role) you use to create, update, describe, delete, and list Channels. - [Service execution role](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-iam-service-role.md): A Channel assumes a service execution role to deliver data. ### [Manage Channels](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-manage.md) Resource states: CREATING, ACTIVE, UPDATING, DELETING, FAILED, SUSPENDING, SUSPENDED. - [Create a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-create.md) - [Describe a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-describe.md) - [Update a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-update.md): You can modify only the data freshness interval (DataFreshnessInSeconds, 300–900). - [Delete a Channel](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-delete.md): Deleting a Channel stops delivery from the Kafka topic to the destination. - [List Channels](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-list.md) ### [Output key template](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-output-key-template.md) For Amazon S3 general purpose buckets, the output key template controls the S3 object key (path and file name) of each delivered object. - [Variables](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-template-variables.md): Variables use the !{name} syntax. - [Partitioning](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-template-partitioning.md) - [Rules](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-template-rules.md): If you provide a template, it must satisfy all of the following. - [Examples](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-template-examples.md) ### [Security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-security.md) - [Encryption in transit](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-security-transit.md): A Channel encrypts all data in transit between Amazon MSK Express brokers and the Channel service using TLS 1.2 or higher. - [Encryption at rest](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-security-rest.md): Data delivered to general-purpose S3 buckets is encrypted at rest using the destination bucket's default encryption settings: - [IAM and access control](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-security-iam.md): A Channel uses IAM roles for authorization. ### [Monitoring](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-monitoring.md) A Channel publishes data-delivery metrics to Amazon CloudWatch under the AWS/Kafka namespace. - [Amazon CloudWatch metrics](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-metrics.md) - [Metric dimensions](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-metric-dimensions.md) - [Recommended alarms](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-alarms.md) - [Metrics in the console](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-viewing-metrics.md) ### [Logging](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-logging.md) - [CloudTrail logging](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-cloudtrail.md): All Channel API calls are logged in AWS CloudTrail: CreateChannel, DescribeChannel, UpdateChannel, DeleteChannel, ListChannels. - [Amazon CloudWatch Logs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-cw-logs.md): A Channel can publish operational logs to a Amazon CloudWatch Logs log group, including schema resolution events, delivery attempts and outcomes, and error details for failed deliveries. ### [Best practices](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-bestpractices.md) - [Throughput and data freshness](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-bp-throughput.md) - [Object layout](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-bp-layout.md) - [Security](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-bp-security.md) ### [Troubleshooting](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-troubleshooting.md) - [Channel stuck in CREATING state](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-creating.md) - [Checking operation status](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-operation-status.md) - [Data freshness higher than configured](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-freshness.md) - [Throughput exceeds the supported limit](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-throughput.md) - [Failed records (FailedRecordCount greater than 0)](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-failed-records.md) - [No objects appearing in the bucket](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-no-data.md) - [Channel is suspended](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-suspended.md) - [Permission denied errors in Amazon CloudWatch Logs](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-permission-denied.md) - [Channel not available on cluster](https://docs.aws.amazon.com/msk/latest/developerguide/msk-data-delivery-s3-ts-not-available.md) ## [MSK integrations](https://docs.aws.amazon.com/msk/latest/developerguide/msk-integrations.md) - [Athena connector for Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/integrations-athena.md): Reference for Amazon Athena connector for Amazon MSK. - [Redshift integration for Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/integrations-redshift.md): Reference for Amazon MSK support for Amazon Redshift streaming data ingestion feature. - [Firehose integration for Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/integrations-kinesis-data-firehose.md): Reference for Amazon MSK support for Amazon Data Firehose streaming data ingestion feature. - [Lambda integration with Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/integrations-lambda.md): Reference for Amazon MSK support for data processing with AWS Lambda. - [Access EventBridge pipes](https://docs.aws.amazon.com/msk/latest/developerguide/eb-pipes-integration.md): Create pipes with a Amazon MSK cluster as the source from within the Amazon MSK console. - [Kafka Streams with MSK Serverless](https://docs.aws.amazon.com/msk/latest/developerguide/use-kafka-streams-express-brokers-msk-serverless.md): Configure and run Kafka Streams applications with MSK Serverless. ### [Real-time vector embedding blueprints](https://docs.aws.amazon.com/msk/latest/developerguide/ai-vector-embedding-integration-learn-more.md) Create real-time AI apps with a Amazon MSK cluster as the source from within the Amazon MSK console. - [Logging and observability](https://docs.aws.amazon.com/msk/latest/developerguide/ai-vector-embedding-integration-logging-observability.md): This section lists the CloudWatch metrics for real-time vector embedding blueprints. - [Notes before enabling real-time vector embedding blueprints](https://docs.aws.amazon.com/msk/latest/developerguide/ai-vector-embedding-integration-notes.md): This section lists some notes to be aware of before enabling real-time vector embedding blueprints. - [Deploy streaming data vectorization blueprint](https://docs.aws.amazon.com/msk/latest/developerguide/ai-vector-embedding-integration-deploy.md): This section is a procedure for deploying a streaming data vectorization blueprint. ## Full content The entire guide is also available as a single markdown file, useful for feeding the full guide to an LLM in one pass: - [Complete guide (single file)](https://docs.aws.amazon.com/msk/latest/developerguide/full.md)