Set up resources required for MSK Connect

In this step you create the following resources that you need for this getting-started scenario:

• An S3 bucket to serve as the destination that receives data from the connector.

• An MSK cluster to which you will send data. The connector will then read the data from this cluster and send it to the destination S3 bucket.

• An IAM role that allows the connector to write to the destination S3 bucket.

• An Amazon VPC endpoint to make it possible to send data from the Amazon VPC that has the cluster and the connector to Amazon S3.

To create the S3 bucket

1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/.

2. Choose Create bucket.

3. For the name of the bucket, enter a descriptive name such as mkc-tutorial-destination-bucket.

4. Scroll down and choose Create bucket.

5. In the list of buckets, choose the newly created bucket.

6. Choose Create folder.

7. Enter tutorial for the name of the folder, then scroll down and choose Create folder.

To create the cluster

1. Open the Amazon MSK console at https://console.aws.amazon.com/msk/home?region=us-east-1#/home/.

2. In the left pane, under MSK Clusters, choose Clusters.

3. Choose Create cluster.

4. Choose Custom create.

5. For the cluster name enter mkc-tutorial-cluster.

6. Under General cluster properties, choose Provisioned for the cluster type.

7. Under Networking, choose an Amazon VPC. Then select the Availability Zones and subnets that you want to use. Remember the IDs of the Amazon VPC and subnets that you selected because you need them later in this tutorial.

8. Under Access control methods ensure that only Unauthenticated access is selected.

9. Under Encryption ensure that only Plaintext is selected.

10. Continue through the wizard and then choose Create cluster. This takes you to the details page for the cluster. On that page, under Security groups applied, find the security group ID. Remember that ID because you need it later in this tutorial.

To create the IAM role that can write to the destination bucket

1. Open the IAM console at https://console.aws.amazon.com/iam/.

2. In the left pane, under Access management, choose Roles.

3. Choose Create role.

4. Under Or select a service to view its use cases, choose S3.

5. Scroll down and under Select your use case, again choose S3.

6. Choose Next: Permissions.

7. Choose Create policy. This opens a new tab in your browser where you will create the policy. Leave the original role-creation tab open because we'll get back to it later.

8. Choose the JSON tab, and then replace the text in the window with the following policy.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListAllMyBuckets"
         ],
         "Resource": "arn:aws:s3:::*"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:ListBucket",
           "s3:GetBucketLocation",
           "s3:DeleteObject"
         ],
         "Resource": "arn:aws:s3:::<my-tutorial-destination-bucket>"
       },
       {
         "Effect": "Allow",
         "Action": [
           "s3:PutObject",
           "s3:GetObject",
           "s3:AbortMultipartUpload",
           "s3:ListMultipartUploadParts",
           "s3:ListBucketMultipartUploads"
         ],
         "Resource": "*"
       }
     ]
   }

9. Choose Next: Tags.

10. Choose Next: Review.

11. Enter mkc-tutorial-policy for the policy name, then scroll down and choose Create policy.

12. Back in the browser tab where you were creating the role, choose the refresh button.

13. Find the mkc-tutorial-policy and select it by choosing the button to its left.

14. Choose Next: Tags.

15. Choose Next: Review.

16. Enter mkc-tutorial-role for the role name, and delete the text in the description box.

17. Choose Create role.

To allow MSK Connect to assume the role

1. In the IAM console, in the left pane, under Access management, choose Roles.

2. Find the mkc-tutorial-role and choose it.

3. Under the role's Summary, choose the Trust relationships tab.

4. Choose Edit trust relationship.

5. Replace the existing trust policy with the following JSON.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": "kafkaconnect.amazonaws.com"
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

6. Choose Update Trust Policy.

To create an Amazon VPC endpoint from the cluster's VPC to Amazon S3

1. Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.

2. In the left pane, choose Endpoints.

3. Choose Create endpoint.

4. Under Service Name choose the com.amazonaws.us-east-1.s3 service and the Gateway type.

5. Choose the cluster's VPC and then select the box to the left of the route table that is associated with the cluster's subnets.

6. Choose Create endpoint.

Next Step

Create custom plugin