Authentication with mTLS for Redshift streaming ingestion from Amazon MSK

Mutual transport-layer security (mTLS) provides the means for a server to authenticate a client it's sending information to, and for the client to authenticate the server. The benefit of using mTLS is to provide trusted authentication for a variety of use cases in several industry-vertical applications. These include use cases in financial, retail, government and healthcare industries. In the case of streaming ingestion to Redshift, authentication occurs between a server, in this case Amazon MSK, and an Amazon Redshift provisioned cluster or an Amazon Redshift Serverless workgroup.

This topic provides procedures and SQL-command examples that show ways to create an external schema that uses mTLS to authenticate between the Redshift client and Amazon MSK server. The steps in this topic complement the full set of steps for setting up streaming ingestion from Amazon MSK. These are detailed at Getting started with streaming ingestion from Amazon Managed Streaming for Apache Kafka (Amazon MSK).

Prerequisites for using mTLS for streaming ingestion

This section provides prerequisite steps for using mTLS for streaming ingestion with either AWS Certificate Manager or Amazon SageMaker.

1. As a preliminary step, you must have or create a private certificate authority (PCA), which you can use to issue certificates that, among other functions, enable secure communication through secure communication channels. AWS Private Certificate Authority (Private CA) is an available service that performs this function. For more information, see Creating a private CA in the AWS Private Certificate Authority User Guide. In a later step, you issue a certificate and attach it to your Amazon MSK cluster to enable encrypted communication to Redshift.

2. Create an Amazon MSK cluster that supports mtls client authentication. For more information about configuring an Amazon MSK cluster, see To create a cluster that supports client authentication in the Amazon Managed Streaming for Apache Kafka Developer Guide.

3. Edit the security settings for the Amazon MSK cluster, turning on TLS client authentication using AWS Certificate Manager (ACM) and selecting the AWS Private CA (PCA) you created previously. For more information, see Updating security settings of a cluster in the Amazon Managed Streaming for Apache Kafka Developer Guide.

Using mTLS for streaming ingestion with AWS Certificate Manager

The following procedure shows how to configure mTLS for Redshift streaming ingestion by leveraging AWS Certificate Manager (ACM) for certificate storage and management:

1. Request a private certificate through ACM. When you do this, select the PCA you created in the Prerequisites section as the certificate authority. ACM stores the signed certificate and attached private key for secure communication. For more information about managing certificates with ACM, see Issuing and managing certificates in the AWS Certificate Manager User Guide.

2. For the IAM role that you use to manage your Redshift cluster or Amazon Redshift Serverless workgroup, attach the permission to export the certificate, which is acm:ExportCertificate. For more information about setting up the necessary IAM resources for streaming ingestion with Amazon MSK, see Setting up IAM permissions and performing streaming ingestion from Kafka. Specify the same IAM role in the next step to create the external schema.

3. Get the bootstrap broker URI for the Amazon MSK cluster. Getting the bootstrap broker URI is detailed in Getting the bootstrap brokers for an Amazon MSK cluster in the Amazon Managed Streaming for Apache Kafka Developer Guide.

4. Run a SQL command like the following sample to create an external schema that maps a stream from an Amazon MSK cluster to a Redshift external schema, using mtls.

   CREATE EXTERNAL SCHEMA my_schema
   FROM MSK
   IAM_ROLE 'arn:aws:iam::012345678901:role/my_role'
   AUTHENTICATION mtls
   URI 'b-1.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9094,b-2.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9094'
   AUTHENTICATION_ARN 'arn:aws:acm:Region:444455556666:certificate/certificate_ID';

   Important parameters:

   • IAM_ROLE – The IAM role associated with the cluster, for streaming ingestion.

   • URI – The bootstrap broker URI for the Amazon MSK cluster. Note that port 9094 is specified for communicating with brokers for TLS encryption.

   • AUTHENTICATION_ARN – The ARN of the ACM certificate. The ARN is available in the ACM console when you choose the issued certificate.

After performing these configuration steps, you can create a Redshift materialized view that references the schema defined in the sample and then use REFRESH MATERIALIZED VIEW to stream data. This is detailed in the getting-started steps for streaming from Amazon MSK at Getting started with streaming ingestion from Amazon Managed Streaming for Apache Kafka (MSK) .

Using mTLS for streaming ingestion with AWS Secrets Manager

You can configure mTLS for Redshift streaming ingestion by leveraging AWS Secrets Manager for certificate management, if you don't want to reference the certificate in ACM. The steps that follow describe how to configure it.

1. Create a certificate signing request and private key with your tool of choice. Then you can use the signing request to generate a signed certificate, using the same AWS Private CA (PCA) that you used to generate the certificate for the Amazon MSK cluster. For more information about issuing a certificate, see IssueCertificate in the AWS Private Certificate Authority API Reference.

2. Extract the certificate using AWS Private Certificate Authority. For more information, see Retrieve a private certificate> in the AWS Private Certificate Authority User Guide.

3. Store the certificate and private key generated in the previous step in AWS Secrets Manager. Choose Other type of secret and use the plaintext format. The key-value pairs should be in the format {"certificate":"<cert value>","privateKey":"<pkey value>"}, as in the following example. For more information about creating and managing secrets in AWS Secrets Manager, see Create and manage secrets with AWS Secrets Manager in the AWS Secrets Manager User Guide.

   {"certificate":"-----BEGIN CERTIFICATE-----
   klhdslkfjahksgdfkgioeuyihbflahabhbdslv6akybeoiwv1hoaiusdhbahsbdi 
   H4hAX8/eE96qCcjkpfT84EdvHzp6fC+/WwM0oXlwUEWlvfMCXNaG5D8SqRq3qA==
   -----END CERTIFICATE-----
   -----BEGIN CERTIFICATE-----
   klhdslkfjahksgdfkgioeuyihbflahabhbdslv6akybeoiwv1hoaiusdhbahsbdi 
   wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
   -----END CERTIFICATE-----",
   "privateKey":"-----BEGIN PRIVATE KEY-----
   klhdslkfjahksgdfkgioeuyihbflahabhbdslv6akybeoiwv1hoaiusdhbahsbdi
   7OD4m1dBEs3Fj++hDMH9rYRp99RqtCOf0EWOUe139KOilOsW+cyhAoc9Ci2+Jo/k
   17u2N1iGILMQEZuCRtnJOkFYkw==
   -----END PRIVATE KEY-----"}

4. In Redshift, run the SQL command to create the external schema. You use the AUTHENTICATION type mtls. You also specify the URI of the Amazon MSK cluster and the secret ARN in AWS Secrets Manager.

   CREATE EXTERNAL SCHEMA my_schema
   FROM MSK
   IAM_ROLE 'arn:aws:iam::012345678901:role/my_role'
   AUTHENTICATION mtls
   URI 'b-1.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9094,b-2.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9094'
   SECRET_ARN 'arn:aws:secretsmanager:us-east-1:012345678910:secret:myMTLSSecret';

Important parameters:

• IAM_ROLE – The IAM role associated with the cluster, for streaming ingestion.

• URI – The bootstrap broker URI for the Amazon MSK cluster. Note that port 9094 is specified for communicating with brokers for TLS encryption.

• SECRET_ARN – The ARN of the secret from Secrets Manager, containing the certificate to use for mTLS.

Enabling mTLS authentication for an existing external schema

If you have an existing external schema that you use for streaming ingestion and you want to implement mutual TLS for authentication, you can run a command such as the following, which specifies mTLS authentication and the ACM certificate ARN in ACM.

ALTER EXTERNAL SCHEMA schema_name 
AUTHENTICATION mtls
AUTHENTICATION_ARN 'arn:aws:acm:Region:444455556666:certificate/certificate_ID';

Or you can specify mTLS authentication, with reference to the secret ARN in AWS Secrets Manager.

ALTER EXTERNAL SCHEMA schema_name 
AUTHENTICATION mtls
SECRET_ARN 'arn:aws:secretsmanager:us-east-1:012345678910:secret:myMTLSSecret';