Inference - AWS Deep Learning Containers

Inference

This section shows how to run inference on AWS Deep Learning Containers for Amazon Elastic Compute Cloud using MXNet, PyTorch, TensorFlow, and TensorFlow 2. You can also use Elastic Inference to run inference with AWS Deep Learning Containers. For tutorials and more information on Elastic Inference, see Using AWS Deep Learning Containers with Elastic Inference on Amazon EC2.

For a complete list of Deep Learning Containers, refer to Deep Learning Containers Images.

Note

MKL users: read the AWS Deep Learning Containers Intel Math Kernel Library (MKL) Recommendations to get the best training or inference performance.

TensorFlow Inference

To demonstrate how to use Deep Learning Containers for inference, this example uses a simple half plus two model with TensorFlow Serving. We recommend using the Deep Learning Base AMI for TensorFlow. After you log into your instance, run the following:

$ git clone -b r1.15 https://github.com/tensorflow/serving.git $ cd serving $ git checkout r1.15

Use the commands here to start TensorFlow Serving with the Deep Learning Containers for this model. Unlike the Deep Learning Containers for training, model serving starts immediately upon running the container and runs as a background process.

  • For CPU instances:

    $ docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_cpu,target=/models/saved_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d <cpu inference container>

    For example:

    $ docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_cpu,target=/models/saved_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:1.15.0-cpu-py36-ubuntu18.04
  • For GPU instances:

    $ nvidia-docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,target=/models/saved_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d <gpu inference container>

    For example:

    $ nvidia-docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,target=/models/sad_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:1.15.0-gpu-py36-cu100-ubuntu18.04

Next, run inference with the Deep Learning Containers.

$ curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://127.0.0.1:8501/v1/models/saved_model_half_plus_two:predict

The output is similar to the following:

{ "predictions": [2.5, 3.0, 4.5 ] }
Note

If you want to debug the container's output, you can attach to it using the container name, as in the following command:

$ docker attach <your docker container name>

In this example you used tensorflow-inference.

TensorFlow 2 Inference

To demonstrate how to use Deep Learning Containers for inference, this example uses a simple half plus two model with TensorFlow 2 Serving. We recommend using the Deep Learning Base AMI for TensorFlow 2. After you log into your instance run the following.

$ git clone -b r2.0 https://github.com/tensorflow/serving.git $ cd serving

Use the commands here to start TensorFlow Serving with the Deep Learning Containers for this model. Unlike the Deep Learning Containers for training, model serving starts immediately upon running the container and runs as a background process.

  • For CPU instances:

    $ docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_cpu,target=/models/saved_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d <cpu inference container>

    For example:

    $ docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_cpu,target=/models/saved_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.0.0-cpu-py36-ubuntu18.04
  • For GPU instances:

    $ nvidia-docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,target=/models/saved_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d <gpu inference container>

    For example:

    $ nvidia-docker run -p 8500:8500 -p 8501:8501 --name tensorflow-inference --mount type=bind,source=$(pwd)/tensorflow_serving/servables/tensorflow/testdata/saved_model_half_plus_two_gpu,target=/models/sad_model_half_plus_two -e MODEL_NAME=saved_model_half_plus_two -d 763104351884.dkr.ecr.us-east-1.amazonaws.com/tensorflow-inference:2.0.0-gpu-py36-cu100-ubuntu18.04
    Note

    Loading the GPU model server may take some time.

Next, run inference with the Deep Learning Containers.

$ curl -d '{"instances": [1.0, 2.0, 5.0]}' -X POST http://127.0.0.1:8501/v1/models/saved_model_half_plus_two:predict

The output is similar to the following.

{ "predictions": [2.5, 3.0, 4.5 ] }
Note

To debug the container's output, you can use the name to attach to it as shown in the following command:

$ docker attach <your docker container name>

This example used tensorflow-inference.

MXNet Inference

To begin inference with MXNet, this example uses a pretrained model from a public S3 bucket.

For CPU instances, run the following command.

$ docker run -it --name mms -p 80:8080 -p 8081:8081 <your container image id> \ mxnet-model-server --start --mms-config /home/model-server/config.properties \ --models squeezenet=https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model

For GPU instances, run the following command:

$ nvidia-docker run -it --name mms -p 80:8080 -p 8081:8081 <your container image id> \ mxnet-model-server --start --mms-config /home/model-server/config.properties \ --models squeezenet=https://s3.amazonaws.com/model-server/models/squeezenet_v1.1/squeezenet_v1.1.model

The configuration file is included in the container.

With your server started, you can now run inference from a different window by using the following command.

$ curl -O https://s3.amazonaws.com/model-server/inputs/kitten.jpg curl -X POST http://127.0.0.1/predictions/squeezenet -T kitten.jpg

After you are done using your container, you can remove it using the following command:

$ docker rm -f mms

MXNet Inference with GluonCV

To begin inference using GluonCV, this example uses a pretrained model from a public S3 bucket.

For CPU instances, run the following command.

$ docker run -it --name mms -p 80:8080 -p 8081:8081 <your container image id> \ mxnet-model-server --start --mms-config /home/model-server/config.properties \ --models gluoncv_yolo3=https://dlc-samples.s3.amazonaws.com/mxnet/gluon/gluoncv_yolo3.mar

For GPU instances, run the following command.

$ nvidia-docker run -it --name mms -p 80:8080 -p 8081:8081 <your container image id> \ mxnet-model-server --start --mms-config /home/model-server/config.properties \ --models gluoncv_yolo3=https://dlc-samples.s3.amazonaws.com/mxnet/gluon/gluoncv_yolo3.mar

The configuration file is included in the container.

With your server started, you can now run inference from a different window by using the following command.

$ curl -O https://dlc-samples.s3.amazonaws.com/mxnet/gluon/dog.jpg curl -X POST http://127.0.0.1/predictions/gluoncv_yolo3/predict -T dog.jpg

Your output should look like the following:

{ "bicycle": [ "[ 79.674225 87.403786 409.43515 323.12167 ]", "[ 98.69891 107.480446 200.0086 155.13412 ]" ], "car": [ "[336.61322 56.533463 499.30566 125.0233 ]" ], "dog": [ "[100.50538 156.50375 223.014 384.60873]" ] }

After you are done using your container, you can remove it using this command.

$ docker rm -f mms

PyTorch Inference

To begin inference with PyTorch, this example uses a model pretrained on Imagenet from a public S3 bucket. Similar to MXNet containers, inference is served using mxnet-model-server, which can support any framework as the backend. For more information, see Model Server for Apache MXNet and this blog on Deploying PyTorch inference with MXNet Model Server.

For CPU instances:

$ docker run -itd --name mms -p 80:8080 -p 8081:8081 <your container image id> \ mxnet-model-server --start --mms-config /home/model-server/config.properties \ --models densenet=https://dlc-samples.s3.amazonaws.com/pytorch/multi-model-server/densenet/densenet.mar

For GPU instances

$ nvidia-docker run -itd --name mms -p 80:8080 -p 8081:8081 <your container image id> \ mxnet-model-server --start --mms-config /home/model-server/config.properties \ --models densenet=https://dlc-samples.s3.amazonaws.com/pytorch/multi-model-server/densenet/densenet.mar

If you have docker-ce version 19.03 or later, you can use the --gpus flag when you start Docker.

The configuration file is included in the container.

With your server started, you can now run inference from a different window by using the following.

$ curl -O https://s3.amazonaws.com/model-server/inputs/flower.jpg curl -X POST http://127.0.0.1/predictions/densenet -T flower.jpg

After you are done using your container, you can remove it using the following.

$ docker rm -f mms

Next steps

To learn about using custom entrypoints with Deep Learning Containers on Amazon ECS, see Custom entrypoints.