Using TensorFlow Models with Elastic Inference - Amazon Elastic Inference
Elastic Inference Enabled TensorFlowAdditional Requirements and Considerations

Using TensorFlow Models with Elastic Inference

Amazon Elastic Inference (Elastic Inference) is available only on instances that were launched with an Elastic Inference accelerator.

The Elastic Inference enabled version of TensorFlow allows you to use Elastic Inference accelerators with minimal changes to your TensorFlow code.

Topics

Elastic Inference Enabled TensorFlow

Preinstalled EI Enabled TensorFlow

The Elastic Inference enabled packages are available in the AWS Deep Learning AMI. AWS Deep Learning AMIs come with supported TensorFlow version and ei_for_tf pre-installed. Elastic Inference enabled TensorFlow 2 requires AWS Deep Learning AMI v28 or higher. You also have Docker container options with Using Amazon Deep Learning Containers With Elastic Inference.

Installing EI Enabled TensorFlow

If you're not using a AWS Deep Learning AMI instance, you can download the packages from the Amazon S3 bucket to build it in to your own Amazon Linux or Ubuntu AMIs.

Install ei_for_tf.

pip install -U ei_for_tf*.whl

If the TensorFlow version is lower than the required version, pip upgrades TensorFlow to the appropriate version. If the TensorFlow version is higher than the required version, there will be a warning about the incompatibility. Your program fails at run-time if the TensorFlow version incompatibility isn’t fixed.

Additional Requirements and Considerations

TensorFlow 2.0 Differences

Starting with TensorFlow 2.0, the Elastic Inference package is a separate pip wheel, instead of an enhanced TensorFlow pip wheel. The prefix for import statements for the Elastic Inference specific API have changed from tensorflow.contrib.ei to ei_for_tf.

To see the compatible TensorFlow version for a specific ei_for_tf version, see the ei_for_tf_compatibility.txt file in the Amazon S3 bucket.

Model Formats Supported

Elastic Inference supports the TensorFlow saved_model format via TensorFlow Serving.

Warmup

Elastic Inference TensorFlow Serving provides a warmup feature to preload models and reduce the delay that is typical of the first inference request. Amazon Elastic Inference TensorFlow Serving only supports warming up the "fault-finders" signature definition.

Amazon Elastic Inference supports SageMaker Neo compiled TensorFlow models

Amazon Elastic Inference supports TensorFlow 2 models optimized by SageMaker Neo. A pre-trained TensorFlow model can be compiled in SageMaker Neo with EIA as the target device. The resulting model artifacts can be used for inference in Elastic Inference Accelerators. This functionality only works for ei_for_tf version 1.6 and greater. For more information, see Use Elastic Inference with SageMaker Neo compiled models.