SageMaker Workflows - Amazon SageMaker

SageMaker Workflows

As you scale your machine learning (ML) operations, you can use Amazon SageMaker fully managed workflow services to implement continuous integration and deployment (CI/CD) practices for your ML lifecycle. With the SageMaker Pipelines SDK, you choose and integrate pipeline steps into a unified solution that automates the model-building process from data preparation to model deployment. For Kubernetes based architectures, you can install SageMaker Operators on your Kubernetes cluster to create SageMaker jobs natively using the Kubernetes API and command-line Kubernetes tools such as kubectl. With SageMaker components for Kubeflow pipelines, you can create and monitor native SageMaker jobs from your Kubeflow Pipelines. The job parameters, status, and outputs from SageMaker are accessible from the Kubeflow Pipelines UI. Lastly, if you want to schedule non-interactive batch runs of your Jupyter notebook, use the notebook-based workflows service to initiate standalone or regular runs on a schedule you define.

In summary, SageMaker offers the following workflow technologies:

You can also leverage other services that integrate with SageMaker to build your workflow. Options include the following services:

  • Airflow Workflows: SageMaker APIs to export configurations for creating and managing Airflow workflows.

  • AWS Step Functions: Multi-step ML workflows in Python that orchestrate SageMaker infrastructure without having to provision your resources separately.

For more information on managing SageMaker training and inference, see Amazon SageMaker Python SDK Workflows.