Amazon Elastic Inference
Developer Guide

Using TensorFlow Models with Elastic Inference

Amazon Elastic Inference(EI) is available only on instances that were launched with an Elastic Inference Accelerator.

The Elastic Inference enabled version of TensorFlow allows you to use Elastic Inference accelerators with minimal changes to your TensorFlow code.

Elastic Inference Enabled TensorFlow

Preinstalled EI Enabled TensorFlow

The Elastic Inference enabled packages are available in the AWS Deep Learning AMI. You also have Docker container options.

Installing EI Enabled TensorFlow

If you're not using a DLAMI instance, you can download the packages from the Amazon S3 bucket to build it in to your own Amazon Linux or Ubuntu AMIs.

Additional Requirements and Considerations

Model Formats Supported

Elastic Inference supports the TensorFlow saved_model format via TensorFlow Serving.

OpenSSL Requirement

Elastic Inference TensorFlow Serving requires OpenSSL for IAM authentication. OpenSSL is pre-installed in the AWS Deep Learning AMI. If you are building your own AMI or Docker container, you must install OpenSSL .

  • Command to install OpenSSL for Ubuntu:

    sudo apt-get install libssl-dev
  • Command to install OpenSSL for Amazon Linux:

    sudo yum install openssl-devel


Elastic Inference TensorFlow Serving provides a warmup feature to preload models and reduce the delay that is typical of the first inference request. Amazon Elastic Inference TensorFlow Serving only supports warming up the "fault-finders" signature definition.

Signature Definitions

Using multiple signature definitions can have a multiplicative effect on the amount of accelerator memory consumed. If you plan to exercise more than one signature definition for your inference calls, you should test these scenarios as you determine the accelerator type for your application.

For large models, EI tends to have larger memory overhead. This may lead to an out-of-memory error. If you receive this error, try switching to a higher EI Accelerator type.