Amazon Elastic Inference
Developer Guide

The AWS Documentation website is getting a new look!
Try it now and let us know what you think. Switch to the new look >>

You can return to the original look by selecting English in the language selector above.

TensorFlow Elastic Inference with Python

With Elastic Inference TensorFlow Serving, the standard TensorFlow Serving interface remains unchanged. The only difference is that the entry point is a different binary named amazonei_tensorflow_model_server.

TensorFlow Serving and Predictor are the only inference modes that EI supports. If you haven't tried TensorFlow Serving before, we recommend that you try the TensorFlow Serving tutorial first.

This release of Elastic Inference TensorFlow Serving has been tested to perform well and provide cost-saving benefits with the following deep learning use cases and network architectures (and similar variants):

Use Case Example Network Topology

Image Recognition

Inception, ResNet, MVCNN

Object Detection

SSD, RCNN

Neural Machine Translation

GNMT

Activate the Tensorflow Amazon EI Environment

  1. Note

    This tutorial assumes usage of a DLAMI with Tensorflow Amazon EI.

    • (Option for Python 3) - Activate the Python 3 TensorFlow EI environment:

      $ source activate amazonei_tensorflow_p36
    • (Option for Python 2) - Activate the Python 2.7 TensorFlow EI environment:

      $ source activate amazonei_tensorflow_p27
  2. The remaining parts of this guide assume you are using the amazonei_tensorflow_p27 environment.

If you are switching between MXNet or TensorFlow Elastic Inference environments, you must Stop and then Start your instance to reattach the Elastic Inference Accelerator. Rebooting is not sufficient since the process requires a complete shutdown.

Use Elastic Inference with TensorFlow Serving

The following is an example of serving a Single Shot Detector (SSD) with a Resnet backbone. As a general rule, you need a servable model and client scripts downloaded to your DLAMI.

Serve and Test Inference with an Inception Model

  1. Download the model.

    curl -O https://s3-us-west-2.amazonaws.com/aws-tf-serving-ei-example/ssd_resnet.zip
  2. Unzip the model.

    unzip ssd_resnet.zip -d /tmp
  3. Download a picture of three dogs to your home directory.

    curl -O https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/3dogs.jpg
  4. Navigate to the folder where AmazonEI_TensorFlow_Serving is installed and run the following command to launch the server. Note, "model_base_path" must be an absolute path.

    amazonei_tensorflow_model_server --model_name=ssdresnet --model_base_path=/tmp/ssd_resnet50_v1_coco --port=9000
  5. While the server is running in the foreground, launch another terminal session. Open a new terminal and activate the TensorFlow environment.

    source activate amazonei_tensorflow_p27
  6. Use your preferred text editor to create a script that has the following content. Name it ssd_resnet_client.py. This script will take an image filename as a parameter and get a prediction result from the pre-trained model.

    from __future__ import print_function import grpc import tensorflow as tf from PIL import Image import numpy as np import time import os from tensorflow_serving.apis import predict_pb2 from tensorflow_serving.apis import prediction_service_pb2_grpc tf.app.flags.DEFINE_string('server', 'localhost:9000', 'PredictionService host:port') tf.app.flags.DEFINE_string('image', '', 'path to image in JPEG format') FLAGS = tf.app.flags.FLAGS coco_classes_txt = "https://raw.githubusercontent.com/amikelive/coco-labels/master/coco-labels-paper.txt" local_coco_classes_txt = "/tmp/coco-labels-paper.txt" # it's a file like object and works just like a file os.system("curl -o %s -O %s"%(local_coco_classes_txt, coco_classes_txt)) NUM_PREDICTIONS = 5 with open(local_coco_classes_txt) as f: classes = ["No Class"] + [line.strip() for line in f.readlines()] def main(_): channel = grpc.insecure_channel(FLAGS.server) stub = prediction_service_pb2_grpc.PredictionServiceStub(channel) # Send request with Image.open(FLAGS.image) as f: f.load() # See prediction_service.proto for gRPC request/response details. data = np.asarray(f) data = np.expand_dims(data, axis=0) request = predict_pb2.PredictRequest() request.model_spec.name = 'ssdresnet' request.inputs['inputs'].CopyFrom( tf.contrib.util.make_tensor_proto(data, shape=data.shape)) result = stub.Predict(request, 60.0) # 10 secs timeout outputs = result.outputs detection_classes = outputs["detection_classes"] detection_classes = tf.make_ndarray(detection_classes) num_detections = int(tf.make_ndarray(outputs["num_detections"])[0]) print("%d detection[s]" % (num_detections)) class_label = [classes[int(x)] for x in detection_classes[0][:num_detections]] print("SSD Prediction is ", class_label) if __name__ == '__main__': tf.app.run()
  7. Now run the script passing the server location, port, and the dog photo's filename as the parameters.

    python ssd_resnet_client.py --server=localhost:9000 --image 3dogs.jpg

Use Elastic Inference with the TensorFlow EIPredictor API

Elastic Inference TensorFlow packages for Python 2 and 3 provide an EIPredictor API. This API function provides you with a flexible way to run models on EI as an alternative to using TensorFlow Serving. The EIPredictor API provides a simple interface to perform repeated inference on a pre-trained model. The following code sample shows the available parameters.

ei_predictor = EIPredictor(model_dir, signature_def_key=None, signature_def=None, input_names=None, output_names=None, tags=None, graph=None, config=None, use_ei=True) output_dict = ei_predictor(feed_dict)

Thus use of EIPredictor is similar to TensorFlow Predictor for a saved model . EIPredictor can be used in the following ways:

//EIPredictor class picks inputs and outputs from default serving signature def with tag “serve”. (similar to TF predictor) ei_predictor = EIPredictor(model_dir) //EI Predictor class picks inputs and outputs from the signature def picked using the signtaure_def_key (similar to TF predictor) ei_predictor = EIPredictor(model_dir, signature_def_key='predict') // Signature_def can be provided directly (similar to TF predictor) ei_predictor = EIPredictor(model_dir, signature_def= sig_def) // You provide the input_names and output_names dict. // similar to TF predictor ei_predictor = EIPredictor(model_dir, input_names, output_names) // tag is used to get the correct signature def. (similar to TF predictor) ei_predictor = EIPredictor(model_dir, tags='serve')

Additional EI Predictor functionality includes:

  • Support for frozen models.

    // For Frozen graphs, model_dir takes a file name , input_names and output_names // input_names and output_names should be provided in this case. ei_predictor = EIPredictor(model_dir, input_names=None, output_names=None )
  • Ability to disable use of EI by using the use_ei flag, which is defaulted to True. This is useful for testing EIPredictor against TensorFlow Predictor.

  • EIPredictor can also be created from a TensorFlow Estimator. Given a trained Estimator, you can first export a SavedModel. See the SavedModel documentation for more details. The following shows example usage:

    saved_model_dir = estimator.export_savedmodel(my_export_dir, serving_input_fn) ei_predictor = EIPredictor(export_dir=saved_model_dir) // Once the EIPredictor is created, inference is done using the following: output_dict = ei_predictor(feed_dict)

Use Elastic Inference with TensorFlow Predictor

Installing Elastic Inference TensorFlow

EI enabled TensorFlow comes bundled in the Deep Learning AMIs. You can also download pip wheels for Python 2 and 3 from the Elastic Inference S3 bucket. Follow these instructions to download and install the pip package:

Choose the tar file for the Python version and operating system of your choice from the S3 bucket. Copy the path to the tar file and run the following command:

curl -O [URL of the tar file of your choice]

To untar the file:

tar -xvzf [name of tar file]

Try the following example to serve different models, such as ResNet, using a Single Shot Detector (SSD). As a general rule, you need a servable model and client scripts downloaded to your Deep Learning AMI (DLAMI) before proceeding.

Serve and Test Inference with an SSD Model

  1. Download the model. If you already downloaded the model in the Serving example, skip this step.

    curl -O https://s3-us-west-2.amazonaws.com/aws-tf-serving-ei-example/ssd_resnet.zip
  2. Unzip the model. Again, you may skip this step if you already have the model.

    unzip ssd_resnet.zip -d /tmp
  3. Download a picture of three dogs to your current directory.

    curl -O https://raw.githubusercontent.com/awslabs/mxnet-model-server/master/docs/images/3dogs.jpg
  4. Open a text editor, such as vim, and paste the following inference script. Save the file as ssd_resnet_predictor.py.

    from __future__ import absolute_import from __future__ import division from __future__ import print_function import os import sys import numpy as np import tensorflow as tf import matplotlib.image as mpimg from tensorflow.contrib.ei.python.predictor.ei_predictor import EIPredictor tf.app.flags.DEFINE_string('image', '', 'path to image in JPEG format') FLAGS = tf.app.flags.FLAGS coco_classes_txt = "https://raw.githubusercontent.com/amikelive/coco-labels/master/coco-labels-paper.txt" local_coco_classes_txt = "/tmp/coco-labels-paper.txt" # it's a file like object and works just like a file os.system("curl -o %s -O %s"%(local_coco_classes_txt, coco_classes_txt)) NUM_PREDICTIONS = 5 with open(local_coco_classes_txt) as f: classes = ["No Class"] + [line.strip() for line in f.readlines()] def get_output(eia_predictor, test_input): pred = None for curpred in range(NUM_PREDICTIONS): pred = eia_predictor(test_input) num_detections = int(pred["num_detections"]) print("%d detection[s]" % (num_detections)) detection_classes = pred["detection_classes"][0][:num_detections] print([classes[int(i)] for i in detection_classes]) def main(_): img = mpimg.imread(FLAGS.image) img = np.expand_dims(img, axis=0) ssd_resnet_input = {'inputs': img} print('Running SSD Resnet on EIPredictor using specified input and outputs') eia_predictor = EIPredictor( model_dir='/tmp/ssd_resnet50_v1_coco/1/', input_names={"inputs": "image_tensor:0"}, output_names={"detection_classes": "detection_classes:0", "num_detections": "num_detections:0", "detection_boxes": "detection_boxes:0"}, ) get_output(eia_predictor, ssd_resnet_input) print('Running SSD Resnet on EIPredictor using default Signature Def') eia_predictor = EIPredictor( model_dir='/tmp/ssd_resnet50_v1_coco/1/', ) get_output(eia_predictor, ssd_resnet_input) if __name__ == "__main__": tf.app.run()
  5. Run the inference script.

    python ssd_resnet_predictor.py --image 3dogs.jpg

For more tutorials and examples, see the TensorFlow Python API.