AWS IoT Analytics
AWS IoT Analytics User Guide

AWS IoT Analytics Console Quickstart Guide

This section shows you how to use the AWS IoT Analytics console to collect, store, process, and query your device data.

Note

Be aware as you enter the names of AWS IoT Analytics entities (channel, data set, data store, and pipeline) in the steps that follow, that any upper-case letters you use will be automatically changed to lower-case by the system. The names of entities must start with a lower-case letter and contain only lower-case letters, underscores and digits.

Sign in to the AWS IoT Analytics Console

If you do not have an AWS account, create one.

  1. To create an AWS account, open the AWS home page and choose Create an AWS Account.

  2. Follow the online instructions. Part of the sign-up procedure involves receiving a phone call and entering a PIN using your phone's keypad.

  3. Sign in to the AWS Management Console and open the AWS IoT Analytics console.

  4. On the AWS IoT Analytics console home page, in the left navigation pane, choose Prepare.

Create a Channel

Incoming messages are sent to a channel.

  1. In the left navigation pane, choose Channels, and then choose Create a channel:

  2. Enter the channel ID, then scroll down and choose Next.

  3. Enter an IoT Core (MQTT) topic filter. Make a note of the topic filter you entered here, because you need it in a later step in order to create a message that will get picked up by your channel.

  4. In the IAM role name area choose Create new. This will automatically create a new role with an appropriate policy attached to it. Then choose Create channel.

Create a Data store

A data store receives and stores your messages. You can create multiple data stores to store data according to your needs. For this example, you create a single data store to receive your AWS IoT messages:

  1. In the left navigation pane, choose Data Stores. On the Create a data store page, choose Create a data store:

  2. Enter the ID for your data store, and then choose Create data store:

Create a Pipeline

To connect a channel to a data store, you need to create a pipeline. The simplest possible pipeline contains no activities other than specifying the channel that collects the data and identifying the data store to which the messages are sent. We explore more complicated pipelines in Pipeline Activities.

For this example, you create a pipeline that does nothing other than connect a channel to a data store. You can see how raw data flows to the data store. Later, you can introduce pipeline activities to process this data.

  1. In the left navigation pane, choose Pipelines. On the Process messages with pipelines page, choose Create a pipeline:

  2. Enter a pipeline ID, and then in Pipeline source, choose Edit:

  3. Select the channel you created before, and then choose Next:

  4. On the Set attributes of your messages page, enter an attribute name, select a type from the pull-down list, and enter an example value, then choose Add new. Repeat this for as many attributes as you want. When done, choose Next.

  5. You won't be adding any pipeline activities right now, so on the Enrich, transform, and filter messages page, just choose Next.

  6. On the Save your processed messages in a data store page, choose Edit, select the data store you created earlier, and then choose Create pipeline.

Create a Data Set

You now have a channel that routes data to a pipeline that stores data in a data store where it can be queried. To query the data, you create a data set. A data set contains SQL expressions that you use to query the data store along with an optional schedule that repeats the query at a day and time you choose. You can create the optional schedules by using expressions similar to Amazon CloudWatch schedule expressions.

  1. On the left navigation pane, choose Analyze, and then choose Data Sets. On the Explore your data with a data set page, choose Create a data set:

  2. On the Set ID and source page, enter an ID. In Select data store source choose Edit:

  3. Select the data store you created earlier, and then choose Next:

  4. On the Author SQL Query page, in the Query area, enter a SQL expression that selects your attributes, and then choose Next. Here is an example SQL expression that uses the attributes we specified previously in this tutorial:

    SELECT thingid, temperature, humidity, datetime FROM my_datastore
  5. You won't schedule a recurring run of the query at this point, so on the Set query schedule page just choose Create data set:

Send an AWS IoT Message

To generate some sample data, use the AWS IoT console to send an AWS IoT message.

Note

The field names of message payloads (data) that you send to AWS IoT Analytics:

  • Must contain only alphanumeric characters and undescores (_); no other special characters are allowed.

  • Must begin with an alphabetic character or single underscore (_).

  • Cannot contain hyphens (-).

  • In regular expression terms: "^[A-Za-z_]([A-Za-z0-9]*|[A-Za-z0-9][A-Za-z0-9_]*)$".

  • Cannot be greater than 255 characters.

  • Are case-insensitive. (Fields named "foo" and "FOO" in the same payload will be considered duplicates.)

For example, {"temp_01": 29} or {"_temp_01": 29} are valid, but {"temp-01": 29}, {"01_temp": 29} or {"__temp_01": 29} are invalid in message payloads.

  1. In the AWS IoT console, in the left navigation pane, choose Test.

  2. On the MQTT client page, in the Publish section, in Specify a topic, type update/environment/dht1. In the message payload section, enter the following JSON contents:

    { "thingid": "dht1", "temperature": 26, "humidity": 29, "datetime": "2018-01-26T07:06:01" }
  3. Choose Publish to topic.

    If you have followed the example to this point, then this publishes a message that is captured by your channel, and then routed by your pipeline to your data store.

Access the Query Results

Data set contents are your results in a file, in CSV format.

  1. In the AWS IoT Analytics console, in the left navigation pane, choose Analyze and then choose Data sets:

  2. On the Data sets page, choose the data set you just created:

  3. On the data set information page, in the upper-right corner, choose Actions, and then choose Run query now:

  4. To check if the data set is ready, look for SUCCEEDED under the name of the data set in the upper left-hand corner. The Details section contains the query results:

  5. In the left navigation pane, click CSV. Choose Download to view or save the CSV file that contains the query results.

    It looks like this:

    "thingid","temperature","humidity","datetime" "dht1","26","29","2018-01-26T07:06:01"
  6. Choose the left arrow in the upper-left corner to return to the main page of the AWS IoT Analytics console.

Explore Your Data

Jupyter Notebooks is an open source solution for advanced analyses and ad-hoc data exploration. Notebooks enable you to use templates and a scripting language, typically Python, to apply different transformations or normalizations to the data, aggregate metrics, and analyze and visualize data using data science libraries. You can even apply more complex analytics and machine learning, such as k-means clustering, to your data using these notebooks.

AWS IoT Analytics uses Amazon SageMaker notebook instances to host its Jupyter notebooks. Before you create a notebook instance, you must create a relationship between AWS IoT Analytics and Amazon SageMaker:

  1. Go to the Amazon SageMaker console and create a notebook instance:

    1. Fill in the details, and then choose Create a new role. Make a note the role ARN.

    2. Create a notebook instance.

    This creates a role that Amazon SageMaker can use and a new notebook instance

  2. Go to the IAM console and modify the Amazon SageMmaker role:

    1. Open the role. It should have one managed policy.

    2. Choose add inline policy, and then for Service, choose IotAnalytics. Choose Select actions, and then type "GetDatasetContent" in the search box and select it. Choose Review policy.

    3. Review the policy for accuracy, give it a name, and then choose Create policy.

    This gives the newly created role permission to read a data set from AWS IoT Analytics.

  3. Return to the AWS IoT Analytics console, and in the left navigation pane, choose Analyze, and then Notebooks:

  4. On the Notebooks page, choose Create:

  5. On the Select method page, choose Blank Notebook:

  6. On the Set up notebook page, enter a name for the notebook. In Select data set sources, choose Select, and then select the data set you created earlier. In Select a Notebook Instance, choose Select, and then choose the notebook instance you created in Amazon SageMaker. Choose Create Notebook:

  7. On the Notebooks page, use the triangles to open your notebook instance and the IoTAnalytics folder. Use the links to explore your data in Jupyter Notebooks:

You can download and install Jupyter Notebooks on your computer. Additional integration with an Amazon hosted notebook solution is also available.

Notebook Templates

The AWS IoT Analytics notebook templates contain AWS-authored machine learning models and visualizations to help you get started with IoT analytics use cases. These notebook templates can be explored as-is for educational purposes, or re-purposed to fit your data and deliver immediate value.

AWS IoT Analytics provides the following notebook templates:

  1. Detecting Contextual Anomalies: Application of contextual anomaly detection in measured wind speed with a PEWMA model for time series data.

  2. Solar Panel Output Forecasting: Application of piecewise, seasonal, linear time series models with trend to predicting the output of solar panels.

  3. Predictive Maintenance on Jet Engines: Application of multivariate LSTM neural networks and logistic regression to predict remaining useful life of jet engines.

  4. Smart Home Customer Segmentation: Application of k-means and PCA analysis to detect different customer segments in smart home usage data.

  5. Smart City Congestion Forecasting: Application of LSTM to predict the utilization rates for city highways.

  6. Smart City Air Quality Forecasting: Application of LSTM to predict particulate pollution in city centers.

You can find more information about notebook templates in the AWS IoT Analytics console under Analyze/Notebooks.