AWS IoT Analytics
AWS IoT Analytics User Guide

Getting Started with AWS IoT Analytics

Follow these steps to collect, store, process and query your device data using AWS IoT Analytics.

Note

Be aware as you enter the names of AWS IoT Analytics entities (channel, data set, data store, and pipeline) in the steps that follow, that any upper-case letters you use will be automatically changed to lower-case by the system. The names of entities must start with a lower-case letter and contain only lower-case letters, underscores and digits.

Create a Data Store

  1. Start by creating a data store to receive and store your messages. You can create multiple data stores to store data according to your needs, or you can start with a single data store to receive all of your AWS IoT messages:

    aws iotanalytics create-datastore --datastore-name mydatastore
  2. To list the data stores you have already created:

    aws iotanalytics list-datastores
  3. To get more information about a data store:

    aws iotanalytics describe-datastore --datastore-name mydatastore

Create a Channel

  1. Incoming messages are sent to a channel, so the next step is to create a channel for your data. You will point a rule engine rule at this channel in a later step:

    aws iotanalytics create-channel --channel-name mychannel
  2. To list the channels you have already created:

    aws iotanalytics list-channels
  3. To get more information about a channel:

    aws iotanalytics describe-channel --channel-name mychannel

Create a Pipeline

To connect a channel to a data store, you need to create a pipeline. The simplest possible pipeline contains no activities other than specifying the channel that collects the data and identifying the data store to which the messages are sent. For information about more complicated pipelines, see Pipeline Activities.

We recommend that you start with a pipeline that does nothing other than connect a channel to a data store. You can see how raw data flows to the data store before you introduce pipeline activities to process this data.

  1. Create a pipeline:

    aws iotanalytics create-pipeline --cli-input-json file://mypipeline.json

    where the file mypipeline.json contains:

    { "pipelineName": "mypipeline", "pipelineActivities": [ { "channel": { "name": "mychannelactivity", "channelName": "mychannel", "next": "mystoreactivity" } }, { "datastore": { "name": "mystoreactivity", "datastoreName": "mydatastore" } } ] }
  2. List your existing pipelines:

    aws iotanalytics list-pipelines
  3. View the configuration of an individual pipeline:

    aws iotanalytics describe-pipeline --pipeline-name mypipeline

You now have a channel that routes data to a pipeline that stores data in a data store where it can be queried. To get device messages flowing into a channel, you must create a special rule in the AWS IoT platform to route messages from a topic into AWS IoT Analytics. (You can also use the "BatchPutMessage" command to send messages directly to a channel as shown in a later step.)

Create an IAM Role

You must create an IAM role that grants permission to send message data to an AWS IoT Analytics channel. First, create the role:

aws iam create-role --role-name myAnalyticsRole --assume-role-policy-document file://arpd.json

where the contents of the file arpd.json should look like this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "iot.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }

Then, attach a policy document to the role:

aws iam put-role-policy --role-name myAnalyticsRole --policy-name myAnalyticsPolicy --policy-document file://pd.json

where the contents of the file pd.json looks like this:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iotanalytics:BatchPutMessage", "Resource": [ "arn:aws:iotanalytics:us-west-2:<your-account-number>:channel/mychannel" ] } ] }

Create an IoT Rule

To connect a channel to a source of AWS IoT messages, create an AWS IoT rule that sends data to an AWS IoT Analytics channel:

aws iot create-topic-rule --rule-name analyticsTestRule --topic-rule-payload file://rule.json

The contents of the rule.json file should look like this:

{ "sql": "SELECT * FROM 'iot/test'", "ruleDisabled": false, "awsIotSqlVersion": "2016-03-23", "actions": [ { "iotAnalytics": { "channelName": "mychannel", "roleArn": "arn:aws:iam::<your-account-number>:role/myAnalyticsRole" } } ] }

where the channel name and the role are the ones you created in the previous steps.

Now you have joined a rule to a channel, a channel to a pipeline, and a pipeline to a data store. Any data matching the rule will now flow through AWS IoT Analytics to the data store ready to be queried.

Send an IoT Message

You can test the connections between the channel, pipeline, and data store by sending a message using either the AWS IoT console or the "BatchPutMessage" command (using the CLI).

Note

The field names of message payloads (data) that you send to AWS IoT Analytics:

  • Must contain only alphanumeric characters and undescores (_); no other special characters are allowed.

  • Must begin with an alphabetic character or single underscore (_).

  • Cannot contain hyphens (-).

  • In regular expression terms: "^[A-Za-z_]([A-Za-z0-9]*|[A-Za-z0-9][A-Za-z0-9_]*)$".

  • Cannot be greater than 255 characters.

  • Are case-insensitive. (Fields named "foo" and "FOO" in the same payload will be considered duplicates.)

For example, {"temp_01": 29} or {"_temp_01": 29} are valid, but {"temp-01": 29}, {"01_temp": 29} or {"__temp_01": 29} are invalid in message payloads.

Option 1- Use the AWS IoT console to send an AWS IoT message:

  1. In the AWS IoT console, in the left navigation pane, choose Test.

  2. On the MQTT client page, in the Publish section, in Specify a topic, type iot/test. In the message payload section, verify the following JSON contents are present, or type them if not:

    { "message": "Hello from AWS IoT console" }
  3. Choose Publish to topic.

    This publishes a message that is routed to the data store you created earlier.

Option 2- Use the "BatchPutMessage" command to send an IoT message to the channel:

Note

This method does not require that you set up an IoT rule to route messages with a specific topic to your channel. But it does require that the device which sends its data/messages to the channel is capable of running software created with the AWS SDK or is capable of using the AWS CLI to call "BatchPutMessage".

  1. Create a file "messages.json" which contains the messages to be sent (in this example only one message is sent):

    [ { "messageId": "message01", "payload": "{ \"message\": \"Hello from the CLI\" }" } ]
  2. Call batch-put-message:

    aws iotanalytics batch-put-message --channel-name mychannel --messages file://messages.json
  3. If there are no errors, you see the following output:

    { "batchPutMessageErrorEntries": [] }

Query Your Data

Now that you have data in a data store, you can query it to answer analytical questions. Although a data store is not a database, you use SQL expressions to query the data and produce results that are stored in a data set.

To query the data, you create a data set. A data set contains the SQL that you use to query the data store along with an optional schedule that repeats the query at a day and time you choose. You create the optional schedules using expressions similar to Amazon CloudWatch schedule expressions.

  1. Create a data set:

    aws iotanalytics create-dataset --cli-input-json file://mydataset.json

    where the file mydataset.json contains:

    { "datasetName": "mydataset", "actions": [ { "actionName":"myaction", "queryAction": { "sqlQuery": "select * from mydatastore" } } ] }
  2. Create the data set content by executing the query:

    aws iotanalytics create-dataset-content --dataset-name mydataset

    Wait a few minutes for the data set content to be created before you continue.

Access the Query Results

Data set contents are your results stored as a file, in CSV format. The file is made available to you through Amazon S3. The following example shows how you can check that your results are ready and download the file.

  1. Call get-dataset-content:

    aws iotanalytics get-dataset-content --dataset-name mydataset
  2. If your data set contains any data, then the output from get-dataset-content will have "state": "SUCCEEDED" in the status field, like this:

    { "timestamp": 1508189965.746, "entries": [ { "entryName": "someEntry", "dataURI": "https://aws-iot-analytics-datasets-f7253800-859a-472c-aa33- e23998b31261.s3.amazonaws.com/results/f881f855-c873-49ce-abd9-b50e9611b71f.csv?X-Amz- Security-Token=<TOKEN>&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20171016T214541Z&X-Amz- SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=<CREDENTIAL>&X-Amz- Signature=<SIGNATURE>" } ], "status": { "state": "SUCCEEDED", "reason": "A useful comment." } }

    dataURI is a signed URL to the output results. It is valid for a short period of time (a few hours). Depending on your workflow, you might want to always call get-dataset-content before you access the content because calling this command generates a new signed URL.

Explore Your Data

AWS IoT Analytics provides direct integration with Amazon QuickSight. Amazon QuickSight is a fast business analytics service you can use to build visualizations, perform ad-hoc analysis, and quickly get business insights from your data. Amazon QuickSight enables organizations to scale to hundreds of thousands of users, and delivers responsive performance by using a robust in-memory engine (SPICE).

AWS IoT Analytics data sets can also be directly consumed by Jupyter Notebooks in order to perform advanced analytics and data exploration. Jupyter Notebooks is an open source solution. You can install and download from http://jupyter.org/install.html. Additional integration with SageMaker, an Amazon hosted notebook solution, is also available.