Use Amazon SageMaker Elastic Inference (EI)

By using Amazon Elastic Inference (EI), you can speed up the throughput and decrease the latency of getting real-time inferences from your deep learning models that are deployed as Amazon SageMaker hosted models, but at a fraction of the cost of using a GPU instance for your endpoint. EI allows you to add inference acceleration to a hosted endpoint for a fraction of the cost of using a full GPU instance. Add an EI accelerator in one of the available sizes to a deployable model in addition to a CPU instance type, and then add that model as a production variant to an endpoint configuration that you use to deploy a hosted endpoint. You can also add an EI accelerator to a SageMaker notebook instance so that you can test and evaluate inference performance when you are building your models.

Elastic Inference is supported in EI-enabled versions of TensorFlow, Apache MXNet, and PyTorch. To use any other deep learning framework, export your model by using ONNX, and then import your model into MXNet. You can then use your model with EI as an MXNet model. For information about importing an ONNX model into MXNet, see Importing an ONNX model into MXNet.

Topics

• How EI Works
• Choose an EI Accelerator Type
• Use EI in a SageMaker Notebook Instance
• Use EI on a Hosted Endpoint
• Frameworks that Support EI
• Use EI with SageMaker Built-in Algorithms
• EI Sample Notebooks
• Set Up to Use EI
• Attach EI to a Notebook Instance
• Use EI on Amazon SageMaker Hosted Endpoints

How EI Works

Amazon Elastic Inference accelerators are network attached devices that work along with SageMaker instances in your endpoint to accelerate your inference calls. Elastic Inference accelerates inference by allowing you to attach fractional GPUs to any SageMaker instance. You can select the client instance to run your application and attach an Elastic Inference accelerator to use the right amount of GPU acceleration for your inference needs. Elastic Inference helps you lower your cost when not fully utilizing your GPU instance for inference. We recommend trying Elastic Inference with your model using different CPU instances and accelerator sizes.

The following EI accelerator types are available. You can configure your endpoints or notebook instances with any EI accelerator type.

In the table, the throughput in teraflops (TFLOPS) is listed for both single-precision floating-point (F32) and half-precision floating-point (F16) operations. The memory in GB is also listed.

Accelerator Type	F32 Throughput in TFLOPS	F16 Throughput in TFLOPS	Memory in GB
ml.eia2.medium	1	8	2
ml.eia2.large	2	16	4
ml.eia2.xlarge	4	32	8
ml.eia1.medium	1	8	1
ml.eia1.large	2	16	2
ml.eia1.xlarge	4	32	4

Choose an EI Accelerator Type

Consider the following factors when choosing an accelerator type for a hosted model:

• Models, input tensors and batch sizes influence the amount of accelerator memory you need. Start with an accelerator type that provides at least as much memory as the file size of your trained model. Factor in that a model might use significantly more memory than the file size at runtime.

• Demands on CPU compute resources, main system memory, and GPU-based acceleration and accelerator memory vary significantly between different kinds of deep learning models. The latency and throughput requirements of the application also determine the amount of compute and acceleration you need. Thoroughly test different configurations of instance types and EI accelerator sizes to make sure you choose the configuration that best fits the performance needs of your application.

For more information on selecting an EI accelerator, see:

• Amazon Elastic Inference Overview

• Choosing an Instance and Accelerator Type for Your Model

• Optimizing costs in Amazon Elastic Inference with TensorFlow

Use EI in a SageMaker Notebook Instance

Typically, you build and test machine learning models in a SageMaker notebook before you deploy them for production. You can attach EI to your notebook instance when you create the notebook instance. You can set up an endpoint that is hosted locally on the notebook instance by using the local mode supported by TensorFlow, MXNet, and PyTorch estimators and models in the Amazon SageMaker Python SDK to test inference performance. Elastic Inference enabled PyTorch is not currently supported on notebook instances. For instructions on how to attach EI to a notebook instance and set up a local endpoint for inference, see Attach EI to a Notebook Instance. There are also Elastic Inference-enabled SageMaker Notebook Jupyter kernels for Elastic Inference-enabled versions of TensorFlow and Apache MXNet. For information about using SageMaker notebook instances, see Use Amazon SageMaker Notebook Instances

Use EI on a Hosted Endpoint

When you are ready to deploy your model for production to provide inferences, you create a SageMaker hosted endpoint. You can attach EI to the instance where your endpoint is hosted to increase its performance at providing inferences. For instructions on how to attach EI to a hosted endpoint instance, see Use EI on Amazon SageMaker Hosted Endpoints.

Frameworks that Support EI

Amazon Elastic Inference is designed to be used with AWS enhanced versions of TensorFlow, Apache MXNet, or PyTorch machine learning frameworks. These enhanced versions of the frameworks are automatically built into containers when you use the Amazon SageMaker Python SDK, or you can download them as binary files and import them in your own Docker containers.

You can download the EI-enabled TensorFlow binary files from the public amazonei-tensorflow Amazon S3 bucket to the TensorFlow serving containers. For more information about building a container that uses the EI-enabled version of TensorFlow, see Amazon Elastic Inference with TensorFlow in SageMaker.

You can download the EI-enabled MXNet binary files from the public amazonei-apachemxnet Amazon S3 bucket to the MXNet serving containers. For more information about building a container that uses the EI-enabled version of MXNet, see Amazon Elastic Inference with MXNet in SageMaker.

You can download the EI-enabled PyTorch binary files from the public amazonei-pytorch Amazon S3 bucket to the PyTorch serving containers. For more information about building a container that uses the EI-enabled version of PyTorch, see Amazon Elastic Inference with PyTorch in SageMaker.

To use Elastic Inference in a hosted endpoint, you can choose any of the following frameworks depending on your needs.

• SageMaker Python SDK - Deploy TensorFlow models

• SageMaker Python SDK - Deploy MXNet models

• SageMaker Python SDK - Deploy PyTorch models

If you need to create a custom container for deploying your model that is complex and requires extensions to a framework that the SageMaker pre-built containers do not support, use the low-level AWS SDK for Python (Boto 3) .

Use EI with SageMaker Built-in Algorithms

Currently, the Image Classification - MXNet and Object Detection - MXNet built-in algorithms support EI. For an example that uses the Image Classification algorithm with EI, see End-to-End Multiclass Image Classification Example.

EI Sample Notebooks

The following Sample notebooks provide examples of using EI in SageMaker:

• Using Amazon Elastic Inference with MXNet on Amazon SageMaker

• Using Amazon Elastic Inference with MXNet on an Amazon SageMaker Notebook Instance

• Using Amazon Elastic Inference with Neo-compiled TensorFlow model on SageMaker

• Using Amazon Elastic Inference with a pre-trained TensorFlow Serving model on SageMaker