Give SageMaker Processing Jobs Access to Resources in Your Amazon VPC
To control access to your data and processing jobs, we recommend that you create a private Amazon VPC and configure it so that your jobs aren't accessible over the public internet. For information about creating and configuring a VPC, see Getting Started With Amazon VPC in the Amazon VPC User Guide. Using a VPC helps to protect your processing containers and data because you can configure your VPC so that it is not connected to the internet. Using a VPC also allows you to monitor all network traffic in and out of your processing containers by using VPC flow logs. For more information, see VPC Flow Logs in the Amazon VPC User Guide.
This document explains how to add Amazon VPC configurations for processing jobs.
Configure a Processing Job for Amazon VPC Access
To specify subnets and security groups in your private VPC, use the
NetworkConfig.VpcConfig
request parameter of the CreateProcessingJob
API, or provide this information when you create a
processing job in the SageMaker console. SageMaker uses this information to create network interfaces and attach
them to your processing containers. The network interfaces provide your processing containers with a
network connection within your VPC that is not connected to the internet. They also enable your
processing job to connect to resources in your private VPC.
The following is an example of the VpcConfig
parameter that you
include in your call to CreateProcessingJob
:
VpcConfig: { "Subnets": [ "subnet-0123456789abcdef0", "subnet-0123456789abcdef1", "subnet-0123456789abcdef2" ], "SecurityGroupIds": [ "sg-0123456789abcdef0" ] }
Configure Your Private VPC for SageMaker Processing
When configuring the private VPC for your SageMaker processing jobs, use the following guidelines. For information about setting up a VPC, see Working with VPCs and Subnets in the Amazon VPC User Guide.
Topics
Ensure That Subnets Have Enough IP Addresses
Your VPC subnets should have at least two private IP addresses for each instance in a processing job. For more information, see VPC and Subnet Sizing for IPv4 in the Amazon VPC User Guide.
Create an Amazon S3 VPC Endpoint
If you configure your VPC so that processing containers don't have access to the internet, they can't connect to the Amazon S3 buckets that contain your data unless you create a VPC endpoint that allows access. By creating a VPC endpoint, you allow your processing containers to access the buckets where you store your data. We recommend that you also create a custom policy that allows only requests from your private VPC to access to your S3 buckets. For more information, see Endpoints for Amazon S3.
To create an S3 VPC endpoint:
-
Open the Amazon VPC console at https://console.aws.amazon.com/vpc/
. -
In the navigation pane, choose Endpoints, then choose Create Endpoint
-
For Service Name, choose com.amazonaws.
region
.s3, whereregion
is the name of the region where your VPC resides. -
For VPC, choose the VPC you want to use for this endpoint.
-
For Configure route tables, select the route tables to be used by the endpoint. The VPC service automatically adds a route to each route table you select that points any S3 traffic to the new endpoint.
-
For Policy, choose Full Access to allow full access to the S3 service by any user or service within the VPC. Choose Custom to restrict access further. For information, see Use a Custom Endpoint Policy to Restrict Access to S3.
Use a Custom Endpoint Policy to Restrict Access to S3
The default endpoint policy allows full access to S3 for any user or service in your VPC. To further restrict access to S3, create a custom endpoint policy. For more information, see Using Endpoint Policies for Amazon S3. You can also use a bucket policy to restrict access to your S3 buckets to only traffic that comes from your Amazon VPC. For information, see Using Amazon S3 Bucket Policies.
Restrict Package Installation on the Processing Container
The default endpoint policy allows users to install packages from the Amazon Linux and Amazon Linux 2 repositories on the processing container. If you don't want users to install packages from that repository, create a custom endpoint policy that explicitly denies access to the Amazon Linux and Amazon Linux 2 repositories. The following is an example of a policy that denies access to these repositories:
{ "Statement": [ { "Sid": "AmazonLinuxAMIRepositoryAccess", "Principal": "*", "Action": [ "s3:GetObject" ], "Effect": "Deny", "Resource": [ "arn:aws:s3:::packages.*.amazonaws.com/*", "arn:aws:s3:::repo.*.amazonaws.com/*" ] } ] } { "Statement": [ { "Sid": "AmazonLinux2AMIRepositoryAccess", "Principal": "*", "Action": [ "s3:GetObject" ], "Effect": "Deny", "Resource": [ "arn:aws:s3:::amazonlinux.*.amazonaws.com/*" ] } ] }
Configure Route Tables
Use default DNS settings for your endpoint route table, so that standard Amazon S3
URLs (for example, http://s3-aws-region.amazonaws.com/MyBucket
)
resolve. If you don't use default DNS settings, ensure that the URLs that you
use to specify the locations of the data in your processing jobs resolve by
configuring the endpoint route tables. For information about VPC endpoint route
tables, see Routing for Gateway Endpoints in the Amazon VPC User
Guide.
Configure the VPC Security Group
In distributed processing, you must allow communication between the different containers in the same processing job. To do that, configure a rule for your security group that allows inbound connections between members of the same security group. For more information, see Security Group Rules.
Connect to Resources Outside Your VPC
If you configure your VPC so that it doesn't have internet access, processing jobs that use that VPC do not have access to resources outside your VPC. If your processing job needs access to resources outside your VPC, provide access with one of the following options:
-
If your processing job needs access to an AWS service that supports interface VPC endpoints, create an endpoint to connect to that service. For a list of services that support interface endpoints, see VPC Endpoints in the Amazon VPC User Guide. For information about creating an interface VPC endpoint, see Interface VPC Endpoints (AWS PrivateLink) in the Amazon VPC User Guide.
-
If your processing job needs access to an AWS service that doesn't support interface VPC endpoints or to a resource outside of AWS, create a NAT gateway and configure your security groups to allow outbound connections. For information about setting up a NAT gateway for your VPC, see Scenario 2: VPC with Public and Private Subnets (NAT) in the Amazon Virtual Private Cloud User Guide.