MLSEC-09: Secure inter-node cluster communications

For frameworks such as TensorFlow, it’s common to share information like coefficients as part of the inter-node cluster communications. The algorithms require that exchanged information stay synchronized across nodes. Secure this information through encryption in transit.

Implementation plan

• Enable inter-node encryption in Amazon SageMaker- In distributed computing environments, data transmitted between nodes can traverse wide networks, or even the internet. Enable inter-node encryption through the appropriate controls for the technology choices made. You can instruct SageMaker to automatically encrypt inter-container communication for your training job to ensure that data is passed over an encrypted tunnel.

• Enable encryption in transit in Amazon EMR - There are many applications and execution engines in the Hadoop ecosystem, providing a variety of tools to match the needs of your ML and analytics workloads. Amazon EMR has distributed cluster capabilities and is also an option for running training jobs on the data that is either stored locally on the cluster or in Amazon S3. Amazon EMR makes it easy to create and manage fully configured, elastic clusters of Amazon EC2 instances running Hadoop and other applications in the Hadoop ecosystem. Amazon EMR provides security configurations to set up data encryption at rest while stored on Amazon S3 and local Amazon EBS volumes. It also allows the set-up of Transport Layer Security (TLS) certificates for the encryption of data in transit.

Documents

• Protect Communications Between ML Compute Instances in a Distributed Training Job

• Amazon EMR Management guide - Security Data Protection & Encryption Options

• Apache Hadoop on Amazon EMR

Blogs

• Encrypt data in transit using a TLS custom certificate provider with Amazon EMR

• Secure Amazon EMR with Encryption

Examples

• Protect Communications Between ML Compute Instances in a Distributed Training Job

• TF Encrypted