Step 4. Create the pipeline

After you define the pipeline logically, it's time to create the infrastructure to support the pipeline. This step requires the following capabilities, at a minimum:

• Storage, to host and manage pipeline inputs and outputs, including code, model artifacts, and data used in training and inference runs.

• Compute (GPU or CPU), for modeling and inference as well as data preprocessing and postprocessing.

• Orchestration, to manage the resources being used and to schedule any regular runs. For example, the model might be retrained on a periodic basis as new data becomes available.

• Logging and alerting, to monitor the pipeline model accuracy, for resource utilization, and for troubleshooting.

Implementation with AWS CloudFormation

To create the pipeline we used AWS CloudFormation, which is an AWS service for deploying and managing infrastructure as code. The AWS CloudFormation templates include the Step Functions definition that was created in the previous step with the Step Functions SDK. This step includes the creation of the AWS-managed Step Functions instance, which is called the Step Functions state machine. No resources for training and inference are created at this stage, because training and and inference jobs run on demand, only when they’re needed, as SageMaker AI jobs. This step also includes creating AWS Identity and Access Management (IAM) roles to run the Step Functions, run SageMaker AI, and read and write from Amazon S3.

Modifying the output from the Step Functions SDK

We had to make some minor modifications to the AWS CloudFormation output from the previous section. We used simple Python string matching to do the following:

• We added logic for creating the Parameters section of the AWS CloudFormation template. This is because we want to create two roles and define the pipeline name as a parameter along with the deployment environment. This step also covers any additional resources and roles that you might want to create, as discussed in step 6.

• We reformatted three fields to have the required !Sub prefix and quotation marks so that they can be updated dynamically as a part of the deployment process:

  • The StateMachineName property, which names the state machine.

  • The DefinitionString property, which defines the state machine.

  • The RoleArn property, which is returned by the state machine.