Perform machine learning inference - AWS IoT Greengrass

Perform machine learning inference

With AWS IoT Greengrass, you can perform machine learning (ML) inference on your edge devices on locally generated data using cloud-trained models. You benefit from the low latency and cost savings of running local inference, yet still take advantage of cloud computing power for training models and complex processing.

AWS IoT Greengrass makes the steps required to perform inference more efficient. You can train your inference models anywhere and deploy them locally as machine learning components. For example, you can build and train deep-learning models in Amazon SageMaker or computer vision models in Amazon Lookout for Vision. Then, you can store these models in an Amazon S3 bucket, so you can use these models as artifacts in your components to perform inference on your core devices.

How AWS IoT Greengrass ML inference works

AWS provides machine learning components that you can use to create one-step deployments to perform machine learning inference on your device. You can also use these components as templates to create custom components to meet your specific requirements.

AWS provides the following categories of machine learning components:

  • Model component—Contains machine learning models as Greengrass artifacts.

  • Runtime component—Contains the script that installs the machine learning framework and its dependencies on the Greengrass core device.

  • Inference component—Contains the inference code and includes component dependencies to install the machine learning framework and download pre-trained machine learning models.

Each deployment that you create to perform machine learning inference consists of at least one component that runs your inference application, installs the machine learning framework, and downloads your machine learning models. To perform sample inference with AWS-provided components, you deploy an inference component to your core device, which automatically includes the corresponding model and runtime components as dependencies. To customize your deployments, you can plug in or swap out the sample model components with custom model components, or you can use the component recipes for the AWS-provided components as templates to create your own custom inference, model, and runtime components.

To perform machine learning inference by using custom components:

  1. Create a model component. This component contains the machine learning models that you want to use to perform inference. AWS provides sample pre-trained DLR and TensorFlow Lite models. To use a custom model, create your own model component.

  2. Create a runtime component. This component contains the scripts required to install the machine learning runtime for your models. AWS provides sample runtime components for Deep Learning Runtime (DLR) and TensorFlow Lite. To use other runtimes with your custom models and inference code, create your own runtime components.

  3. Create an inference component. This component contains your inference code, and includes your model and runtime components as dependencies. AWS provides sample inference components for image classification and object detection using DLR and TensorFlow Lite. To perform other types of inference, or to use custom models and runtimes, create your own inference component.

  4. Deploy the inference component. When you deploy this component, AWS IoT Greengrass also automatically deploys the model and runtime component dependencies.

To get started with AWS-provided components, see Tutorial: Perform sample image classification inference using TensorFlow Lite.

For information about creating custom machine learning components, see Customize your machine learning components.

What's different in AWS IoT Greengrass Version 2?

AWS IoT Greengrass consolidates functional units for machine learning—such as models, runtimes, and inference code— into components that enable you to use a one-step process to install the machine learning runtime, download your trained models, and perform inference on your device.

By using the AWS-provided machine learning components, you have the flexibility to start performing machine learning inference with sample inference code and pre-trained models. You can plug in custom model components to use your own custom-trained models with the inference and runtime components that AWS provides. For a completely customized machine learning solution, you can use the public components as templates to create custom components and use any runtime, model, or inference type that you want.

Requirements

To create and use machine learning components, you must have the following:

Supported model sources

AWS IoT Greengrass supports using custom-trained machine learning models that are stored in Amazon S3. You can also use Amazon SageMaker edge packaging jobs to directly create model components for your SageMaker Neo-compiled models. For information about using SageMaker Edge Manager with AWS IoT Greengrass, see Use Amazon SageMaker Edge Manager on Greengrass core devices. You can also use Amazon Lookout for Vision model packaging jobs to create model components for your Lookout for Vision models. For more information about using Lookout for Vision with AWS IoT Greengrass, see Use Amazon Lookout for Vision on Greengrass core devices.

The S3 buckets that contain your models must meet the following requirements:

  • They must not be encrypted using SSE-C. For buckets that use server-side encryption, AWS IoT Greengrass machine learning inference currently supports the SSE-S3 or SSE-KMS encryption options only. For more information about server-side encryption options, see Protecting data using server-side encryption in the Amazon Simple Storage Service User Guide.

  • Their names must not include periods (.). For more information, see the rule about using virtual hosted-style buckets with SSL in Rules for bucket naming in the Amazon Simple Storage Service User Guide.

  • The S3 buckets that store your model sources must be in the same AWS account and AWS Region as your machine learning components.

  • AWS IoT Greengrass must have read permission to the model source. To enable AWS IoT Greengrass to access the S3 buckets, the Greengrass device role must allow the s3:GetObject action. For more information about the device role, see Authorize core devices to interact with AWS services.

Supported machine learning runtimes

AWS IoT Greengrass enables you to create custom components to use any machine learning runtime of your choice to perform machine learning inference with your custom-trained models. For information about creating custom machine learning components, see Customize your machine learning components.

To make the process of getting started with machine learning more efficient, AWS IoT Greengrass provides sample inference, model, and runtime components that use the following machine learning runtimes:

AWS-provided machine learning components

The following table lists the AWS-provided components used for machine learning.

Note

Several AWS-provided components depend on specific minor versions of the Greengrass nucleus. Because of this dependency, you need to update these components when you update the Greengrass nucleus to a new minor version. For information about the specific versions of the nucleus that each component depends on, see the corresponding component topic. For more information about updating the nucleus, see Update the AWS IoT Greengrass Core software (OTA).

Component Description Component type Supported OS Open source
Lookout for Vision Edge Agent Deploys the Amazon Lookout for Vision runtime on the Greengrass core device, so you can use computer vision to find defects in industrial products. Generic Linux No
SageMaker Edge Manager Deploys the Amazon SageMaker Edge Manager agent on the Greengrass core device. Generic Linux, Windows No
DLR image classification Inference component that uses the DLR image classification model store and the DLR runtime component as dependencies to install DLR, download sample image classification models, and perform image classification inference on supported devices. Generic Linux, Windows No
DLR object detection Inference component that uses the DLR object detection model store and the DLR runtime component as dependencies to install DLR, download sample object detection models, and perform object detection inference on supported devices. Generic Linux, Windows No
DLR image classification model store Model component that contains sample ResNet-50 image classification models as Greengrass artifacts. Generic Linux, Windows No
DLR object detection model store Model component that contains sample YOLOv3 object detection models as Greengrass artifacts. Generic Linux, Windows No
DLR runtime Runtime component that contains an installation script that is used to install DLR and its dependencies on the Greengrass core device. Generic Linux, Windows No
TensorFlow Lite image classification Inference component that uses the TensorFlow Lite image classification model store and the TensorFlow Lite runtime component as dependencies to install TensorFlow Lite, download sample image classification models, and perform image classification inference on supported devices. Generic Linux, Windows No
TensorFlow Lite object detection Inference component that uses the TensorFlow Lite object detection model store and the TensorFlow Lite runtime component as dependencies to install TensorFlow Lite, download sample object detection models, and perform object detection inference on supported devices. Generic Linux, Windows No
TensorFlow Lite image classification model store Model component that contains a sample MobileNet v1 model as a Greengrass artifact. Generic Linux, Windows No
TensorFlow Lite object detection model store Model component that contains a sample Single Shot Detection (SSD) MobileNet model as a Greengrass artifact. Generic Linux, Windows No
TensorFlow Lite runtime Runtime component that contains an installation script that is used to install TensorFlow Lite and its dependencies on the Greengrass core device. Generic Linux, Windows No