MXNet Elastic Inference with Scala - Amazon Elastic Inference

MXNet Elastic Inference with Scala

Starting from Apache MXNet version 1.4, the Scala API can now integrate with Amazon Elastic Inference. You can use Elastic Inference with the following MXNet Scala API operations:

  • MXNet Scala Symbol API

  • MXNet Scala Module API

  • MXNet Scala Infer API

Install Elastic Inference Enabled Apache MXNet

Elastic Inference enabled Apache MXNet is available in the AWS Deep Learning AMI. A maven repository is also available on Amazon S3. You can build it in to your own Amazon Linux or Ubuntu AMIs, or Docker containers.

For Maven projects, Elastic Inference with Scala can be included by adding the following to your project's pom.xml:

<repositories>     <repository>       <id>Amazon Elastic Inference</id>       <url>https://s3.amazonaws.com/amazonei-apachemxnet/scala</url>     </repository> </repositories>

In addition, add the Elastic Inference flavor of MXNet as a dependency using:

        <dependency>             <groupId>com.amazonaws.ml.mxnet</groupId>             <artifactId>mxnet-full_2.11-linux-x86_64-eia</artifactId>             <version>[1.4.0,)</version>         </dependency>

Check MXNet for Scala Version

You can use the commit hash number to determine which release of the Scala-specific version of MXNet is installed using the following code:

// Imports import org.apache.mxnet.util.Version // Line to run println(Version.getCommitHash)

You can then compare the commit hash with the Release Notes to find the specific info about the version you have.  ​

Use Amazon Elastic Inference with the MXNet Symbol API

To use Elastic Inference with the MXNet Symbol API, pass Context.eia() as the context in a call to either the Symbol.bind or Symbol.simpleBind methods. See the MXNet Symbol Reference for more information.

The following is an example using Context.eia() in a call to simpleBind:

import org.apache.mxnet._ object Example {   def main(args: Array[String]): Unit = {     val data = Symbol.Variable("data", shape=Shape(1))     val sym = Symbol.api.exp(Some(data))     // Pass mx.eia() as context during simple bind operation     val executor = sym.simpleBind(Context.eia(), gradReq = "null", shapeDict = Map("data" -> Shape(1)))     for( i  <- 1 to 10) {       executor.forward(false, ("data", NDArray.ones(1)))       println(s"Inference ${i}, output = ${executor.outputs.head}")     }   } }

Note, the GPU context is not supported. All values and computations that are not Elastic Inference should use the CPU context. Use the Elastic Inference context only with the bind call.

The following is an example using bind. Note, you cannot use the Elastic Inference context to allocate memory or it will throw an error.

import org.apache.mxnet._ object Example {   def main(args: Array[String]): Unit = {     val a = Symbol.Variable("a")     val b = Symbol.Variable("b")     val c = a + b     // Even for EIA workloads, declare NDArrays on the CPU     val aData = NDArray.array(Array(1f,2f), Shape(2), Context.cpu())     val bData = NDArray.array(Array(2f,3f), Shape(2), Context.cpu())     // Then in the bind call, use Context.eia()     val executor = c.bind(Context.eia(), Map("a" -> aData, "b" -> bData))     // The forward call is performed on the remote accelerator     executor.forward()     println(s"1st Inference, output = ${executor.outputs.head}")     // Subsequent calls can pass new data in a forward call     executor.forward(false, ("a", NDArray.ones((2))), ("b", NDArray.ones((2))))     println(s"2nd Inference, output = ${executor.outputs.head}")   } }

Use Amazon Elastic Inference with the MXNet Module API

To use Elastic Inference with the MXNet Module API, pass Context.eia() as the context when creating the Module object. See the MXNet Module Reference for more information. 

The following is an example using Elastic Inference with the Module API on a pre-trained real model (Resnet-152).

import java.io.File import java.net.URL import org.apache.commons.io.FileUtils import org.apache.mxnet._ import org.apache.mxnet.infer.ImageClassifier import org.apache.mxnet.module.Module import scala.io.Source object Example {   def main(args: Array[String]): Unit = {     val urlPath = "http://data.mxnet.io/models/imagenet"     val filePath = System.getProperty("java.io.tmpdir")     // Download Model and Image     FileUtils.copyURLToFile(new URL(s"${urlPath}/resnet/152-layers/resnet-152-0000.params"),       new File(s"${filePath}resnet-152/resnet-152-0000.params"))     FileUtils.copyURLToFile(new URL(s"${urlPath}/resnet/152-layers/resnet-152-symbol.json"),       new File(s"${filePath}resnet-152/resnet-152-symbol.json"))     FileUtils.copyURLToFile(new URL(s"${urlPath}/synset.txt"),       new File(s"${filePath}resnet-152/synset.txt"))     FileUtils.copyURLToFile(new URL("https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg?raw=true"),       new File(s"${filePath}cat.jpg"))     // Load model     val (symbol, argParams, auxParams) = Model.loadCheckpoint(s"${filePath}resnet-152/resnet-152", 0)     val mod = new Module(symbol, contexts = Context.eia(), labelNames = IndexedSeq())     mod.bind(dataShapes=IndexedSeq(DataDesc("data", Shape(1, 3, 224, 224))), forTraining = false)     mod.setParams(argParams, auxParams, allowMissing = true)     val labels = Source.fromFile(s"${filePath}resnet-152/synset.txt").getLines().map(_.trim).toIndexedSeq     // Load image     val originalImg = ImageClassifier.loadImageFromFile(s"${filePath}cat.jpg")     val resizedImg = ImageClassifier.reshapeImage(originalImg, 224, 224)     val img = ImageClassifier.bufferedImageToPixels(resizedImg, Shape(1, 3, 224, 224))     mod.forward(new DataBatch(IndexedSeq(img), IndexedSeq(), IndexedSeq(), 0))          val probabilities = mod.getOutputs().head.head.toArray     val best = probabilities.zipWithIndex.sortBy(-_._1).take(5)     best.zipWithIndex.foreach {       case ((prob, nameIndex), i) => println(s"Option ${i}: ${labels(nameIndex)} - ${prob}")     }   } }

Use Amazon Elastic Inference with the MXNet Infer API

To use Elastic Inference with the MXNet Infer API, pass Context.eia() as the context when creating the Infer Predictor object. See the MXNet Infer Reference for more information. The following example also uses the pre-trained real model (Resnet-152).

import java.io.File import java.net.URL import org.apache.commons.io.FileUtils import org.apache.mxnet._ import org.apache.mxnet.infer.ImageClassifier object Example {   def main(args: Array[String]): Unit = {     val urlPath = "http://data.mxnet.io/models/imagenet"     val filePath = System.getProperty("java.io.tmpdir")     // Download Model and Image     FileUtils.copyURLToFile(new URL(s"${urlPath}/resnet/152-layers/resnet-152-0000.params"),       new File(s"${filePath}resnet-152/resnet-152-0000.params"))     FileUtils.copyURLToFile(new URL(s"${urlPath}/resnet/152-layers/resnet-152-symbol.json"),       new File(s"${filePath}resnet-152/resnet-152-symbol.json"))     FileUtils.copyURLToFile(new URL(s"${urlPath}/synset.txt"),       new File(s"${filePath}resnet-152/synset.txt"))     FileUtils.copyURLToFile(new URL("https://github.com/dmlc/web-data/blob/master/mxnet/doc/tutorials/python/predict_image/cat.jpg?raw=true"),       new File(s"${filePath}cat.jpg"))     val inputShape = Shape(1, 3, 224, 224)     val inputDesc = IndexedSeq(DataDesc("data", inputShape, DType.Float32, "NCHW"))     val imgClassifier = new ImageClassifier(s"${filePath}resnet-152/resnet-152", inputDesc, Context.eia())     val img = ImageClassifier.loadImageFromFile(s"${filePath}cat.jpg")     val topK = 5     val output = imgClassifier.classifyImage(img, Some(topK)).head     output.zipWithIndex.foreach{       case ((name, prob), i) => println(s"Option ${i}: ${name} - ${prob}")     }   } }

More Models and Resources

For more tutorials and examples, see:

Troubleshooting

  • MXNet Elastic Inference is built with MKL-DNN, so all operations using Context.cpu() are supported and runs with the same performance as the standard release. MXNet Elastic Inference does not support Context.gpu(), so all operations using that context will throw an error.

  • You cannot allocate memory for NDArray on the remote accelerator by writing something like the following.

    x = NDArray.array(Array(1,2,3), ctx=Context.eia())

    This throws an error. Instead you should use Context.cpu(). Look at the previous bind() example to see how MXNet automatically transfers your data to the accelerator as necessary. Sample error message:

  • Amazon Elastic Inference is only for production inference use cases and does not support any model training. When you use either the Symbol API or the Module API, do not call the backward() method or call bind() with forTraining=True. This throws an error. Because the default value of forTraining is True, make sure you set for_training=False manually in cases such as the example in Use Elastic Inference with the MXNet Module API. Sample error using test.py:

  • Because training is not allowed, there is no point tp initializing an optimizer for inference.

  • A model trained on an earlier version of MXNet will work on a later version of MXNet EI because it is backwards compatible. For example, you can train a model on MXNet 1.3 and run it on MXNet EI 1.4. However, you may run into undefined behavior if you train on a later version of MXNet. For example training a model on MXNet Master and running it on MXNet Elastic Inference 1.4.

  • Different sizes of Elastic Inference accelerators have different amounts of GPU memory. If your model requires more GPU memory than is available in your accelerator, you get a message that looks like the log below. If you run into this message, you should use a larger accelerator size with more memory. Stop and restart your instance with a larger accelerator.

  • Calling reshape explicitly by using either the Module or the Symbol API can lead to OOM errors. Implicitly using different shapes for input NDArrays in different forward passes can also lead to OOM errors. Before being reshaped, the model is not cleaned up on the accelerator until the session is destroyed.