Solution components - Machine to Cloud Connectivity Framework

Solution components

User interface

User interface and messaging architecture detail

Figure 2. User interface and messaging architecture detail

This solution creates a user interface to help you get data from on-premises equipment. To begin, you must define a connection through this user interface. The connection contains information about the equipment, the tags associated with the equipment, the protocol used, and the read frequency for the equipment data.

When you create the CloudFormation stack, you must provide an email address. The email address is used to generate the initial credentials to access the user interface. After the stack creates the Amazon Cognito user and the user interface, the user receives an email containing the web URL and the access credentials, including a temporary password. This web URL is also shown in the CloudFormation console, in the Output tab, under the value M2C2UIDomainName.

Note

When you initially log in to the interface, you will be prompted to change the auto-generated password.

AWS Lambda functions

Connection builder

The connection builder Lambda function validates and processes API requests from Amazon API Gateway. For a list of APIs, refer to API specification in this guide.

This Lambda function interacts with the connection DynamoDB table and the logs DynamoDB table. It also invokes the Greengrass deployer Lambda function to manage the AWS IoT Greengrass group and the AWS IoT SiteWise gateway configuration.

Greengrass deployer

The Greengrass deployer Lambda function manages the AWS IoT Greengrass group. For OPC Data Access (OPC DA) connections, this function creates a machine collector Lambda function to read data from the OPC DA machine. For OPC Unified Architecture (OPC UA) connections, this function adds the machine details as a source for the AWS IoT SiteWise gateway.

This function also creates a publisher Lambda function and AWS IoT Greengrass group subscriptions with the machine collector and publisher Lambda functions as the source and a target topic.

Once the machine collector and publisher Lambda functions and supporting configurations are created, the Greengrass deployer Lambda function activates a deployment of the AWS IoT Greengrass group to the industrial gateway running the AWS IoT Greengrass software.

For OPC DA connections, after the updates are deployed to the industrial gateway, a message to the m2c2/job/<connectionName> MQTT topic activates the OPC DA machine collector Lambda function to start reading data.

For OPC UA connections, the machine collector Lambda function does not need a message delivered over the AWS IoT topic to start collecting data.

Messages consumer

The messages consumer Lambda function stores the messages in an Amazon Simple Queue Service (Amazon SQS) queue. The Amazon SQS queue gets data from the m2c2/info/<connectionName> and m2c2/error/<connectionName> IoT topics. The messages consumer Lambda function then subscribes to the SQS queue, processes the queue messages, and stores them in the messages DynamoDB table. These messages contain a time-to-live (TTL) attribute of one week, which identifies the length of time the message is stored.

OPC Data Access (OPC DA) collector

The OPC DA collector Lambda function is provisioned by the connection builder Lambda function and is deployed to the edge gateway when the AWS IoT Greengrass group is deployed. This machine connector Lambda function writes the connection configuration to the local gateway, establishes connectivity with the OPC DA server configuration specified in the connection configuration, and reads the telemetry data from the source machine. When it receives the data, it writes the data to the m2c2_<connectionName>_stream Greengrass Stream Manager stream.

OPC Unified Architecture (OPC UA) and IoT SiteWise connector

This solution uses the AWS IoT SiteWise managed connector. This connector contains two Lambda functions: swCollector and swPublisher. The swCollector Lambda function connects to the OPC UA server, which is defined as a source in the AWS IoT SiteWise gateway. This function reads the telemetry data from the server and writes the data to the connection specific Greengrass stream.

The swPublisher Lambda function is configured to read from the SiteWise_Stream Greengrass stream. If the job defined AWS IoT SiteWise as a destination, the publisher Lambda function publishes the data to the SiteWise_Stream stream and the swPublisher Lambda function forwards the data to AWS IoT SiteWise.

Publisher

The publisher Lambda function manages the connectivity with the AWS resources to send data to the AWS Cloud, and reads the telemetry data from the AWS IoT Greengrass stream and sends it back to the AWS Cloud.

By default, the connection definition and configuration provided in the CloudFormation template, the publisher Lambda function sends the data to Amazon Kinesis Data Streams. Amazon Kinesis Data Firehose retrieves the data from the stream, bundles batches of the data into GZIP files, and stores them in an Amazon S3 bucket. Optionally, you can connect to an existing Kinesis data stream in your AWS account and the publisher Lambda function will publish the telemetry data to the Kinesis data stream.

When a job is defined to send data to an AWS IoT topic, the publisher Lambda function sends the data to the m2c2/data/<connectionName>/<machineName>/<tagName> MQTT topic in AWS IoT Core. Error messages are sent to the m2c2/error/<connectionName> MQTT topic and informational messages are sent to the m2c2/info/<connectionName> MQTT topic.

You can use an AWS IoT rules engine to subscribe to these MQTT topics to either store their data in your own data lake, in an Amazon S3 bucket, or you can initiate notifications using Amazon Simple Notification Service (Amazon SNS).

When a job is defined to send the data to AWS IoT SiteWise, you must create an asset model and an asset to be able to consume, read, and analyze the data. Once you have an asset model and have created an asset, you must assign an alias to the asset measurements that matches the data that is coming in from your device. Data from OPC DA uses the following alias naming convention: <siteName>/<area>/<process>/<machineName>/<tag>. If the data is from OPC UA, the alias is the full tag structure from the root. To learn more about modeling industrial assets in AWS IoT SiteWise, refer to Modeling industrial assets in the AWS IoT SiteWise User Guide.

Solution workflow

This solution uses AWS IoT Greengrass to communicate with on-premises equipment using either the OPC Data Access (OPC DA) protocol or the OPC Unified Architecture (OPC UA). After launching the CloudFormation template, the following workflow is set up to collect your telemetry data.

Workflow for the Machine to Cloud Connectivity Framework solution

Figure 3: Workflow for the Machine to Cloud Connectivity Framework solution

  1. Through the user interface, you create a connection defining the telemetry data you want to collect from your on-premises hardware.

    • The hardware has to use either OPC DA or OPC UA protocols.

    • A connection defines the details around the telemetry data being collected, including the name of the machine whose telemetry you are collecting, the protocol the source machine is using, the data collection-specific tags, and the location information of the machine.

  2. The user interface activates the connection builder Lambda function to create a connection.

  3. The connection builder Lambda function validates the connection configuration and invokes the Greengrass deployer Lambda function.

    • If the source uses an OPC DA protocol, the Greengrass deployer Lambda function creates a collector Lambda function and uses the connection configuration to configure the collector Lambda function.

    • If the source uses an OPC UA protocol, the Greengrass deployer Lambda function creates an OPC UA source in the SiteWise Gateway configuration.

    • The Greengrass deployer Lambda function creates the data publisher Lambda function, including the machine location information as well as the data destination definition.

    • The Greengrass deployer Lambda function configures the Greengrass group, including adding the new collector and publisher Lambda functions, subscriptions, and other supporting configurations.

    • The Greengrass deployer Lambda function activates a deployment of the Greengrass group updates and Lambda functions to the industrial gateway.

  4. The collectors Lambda function connects to the configured host and collects the data as defined in the connection configuration.

    • If the source uses an OPC DA protocol, the collectors Lambda function adds an alias and name to the telemetry information.

    • The alias and name follow the same format: <siteName>/<area>/<process>/<machineName>/<tag>

  5. The collected telemetry is written to a Greengrass stream manager stream, one stream per connection.

    • Information messages or error messages from the OPC DA collector is published in IoT topics: m2c2/info/<connectionName> and m2c2/error/<connectionName>.

    • The information and error messages are accessible through the user interface.

  6. The data publisher Lambda function deployed with the connection reads the data from the connection stream.

  7. Depending on the configured destination, the data publisher Lambda function updates the data and forwards to the destination.

    • The default destination is Amazon Kinesis Data Streams. When sending data to a Kinesis data stream and the IoT topic, the publisher adds metadata to the telemetry with the connection information.

    • Optionally, you can provide your own Kinesis data stream instead of using the one created by the solution. Provide the name of the data stream that is in your AWS account when launching the CloudFormation stack (using the Name of the Existing Data Stream in Kinesis Data Streams parameter), and create your own consumer to read from the stream and send the data to your preferred destination.

    • Optionally, you can also send data to an AWS IoT topic or to AWS IoT SiteWise.

      You can send your data to an AWS IoT topic and use IoT rules to interact with AWS services. For more information, refer to Rules for AWS IoT in the AWS IoT Developer Guide.

      You can send your data to AWS IoT SiteWise to monitor operations across facilities, quickly compute common industrial performance metrics, and create applications that analyze industrial equipment data to prevent costly equipment issues and reduce gaps in production.

    • For OPC DA to AWS IoT SiteWise, the timestamp is transformed to milliseconds.

  8. In the default configuration, when the data is sent to the Kinesis data stream, a Kinesis Data Firehose delivery stream takes that data, batches the data, and stores it in an Amazon S3 bucket. You can also choose to send your data to AWS IoT SiteWise or to AWS IoT Core.

    • When sending the data to AWS IoT Core, the data is transmitted via the m2c2/data/<connectionName/<machineName>/<tagName> MQTT topic. You can specify where to send the data once it gets to AWS IoT Core by configuring your own rules engine in AWS IoT Core.

    • When sending your data to AWS IoT SiteWise, you must create an asset model and an asset to be able to consume, read, and analyze the data. Once you have an asset model and have created an asset, you must assign an alias to the asset measurements that matches the data transmitted from your device. If the data is from OPC DA, the alias is <siteName>/<area>/<process>/<machineName>/<tag>. If the data is from OPC UA, the alias is the full tag structure from the root. To learn more about modeling industrial assets in AWS IoT SiteWise, refer to Modeling industrial assets in the AWS IoT SiteWise User Guide.

Communicating with your AWS IoT Greengrass industrial gateway

The solution’s user interface (UI) communicates with the on-premises industrial gateway that is running AWS IoT Greengrass. The UI issues command and control signals on an IoT topic, for example, m2c2/job/<connectionName>.

The OPC DA collector Lambda function reads the message sent to the topic and takes the appropriate action. The OPC UA collector Lambda function receives its configuration directly from AWS IoT SiteWise.

Getting data from on-premises equipment to AWS

You can define your data destination via the user interface. As shown in Figure 4, you can choose to send telemetry data to one or more of the following destinations: AWS IoT SiteWise, an IoT Topic, or Amazon Kinesis Data Streams.

User interface - select the data destination

Figure 4: User interface - select the data destination

AWS CloudFormation parameters

This solution provides the following AWS CloudFormation parameters: ID of the Existing AWS IoT Greengrass Group, Name of the Existing Data Stream in Kinesis Data Streams, Log level, and Initial User Email.

The ID of the Existing AWS IoT Greengrass Group and Name of the Existing Data Stream in Kinesis Data Streams parameters are used to determine whether the necessary AWS resources are created on your behalf or if you are providing the necessary AWS resources. By default, these parameters are left empty, which means the solution automatically creates new AWS resources on your behalf.

The Log level parameter determines the level of logging detail you want when using the solution. The default logging level is ERROR. Additional logging levels include VERBOSE, DEBUG, INFO, and WARN.

The Initial User Email parameter creates the initial user that is authorized to log in to the solution’s user interface. This solution uses Amazon Cognito to create the user and a temporary password. When the initial user logs in to the UI for the first time, they must change the password in order to continue to the home page.

ID of the existing AWS IoT Greengrass group

When you provide a Greengrass group ID value for this parameter, the solution uses this Greengrass group to make configuration changes and as destination to deploy the machine connector Lambda function. The solution does not create Greengrass or AWS IoT resources used by the solution, including a Greengrass group, an AWS IoT thing to represent your industrial gateway, or an X.509 certificate or RSA keypair.

Name of the existing data stream in Kinesis Data Streams

When you provide a data stream name value for this parameter, the solution uses the specified stream to push telemetry data. The solution does not create a new Kinesis data stream, a Kinesis Data Firehose delivery stream, or an S3 bucket. You are responsible for creating a consumer to pull messages from the data stream and manage the storage of that data.