AWS IoT Analytics is no longer available to new customers. Existing customers of AWS IoT Analytics can continue to use the service as normal.
Learn more
Getting late data notifications through Amazon CloudWatch Events
When you create dataset contents using data from a specified time frame, some data might not
arrive in time for processing. To allow for a delay, you can specify a deltaTime
offset for the QueryFilter
when you create a dataset by applying a
queryAction
(a SQL query). AWS IoT Analytics still processes the data that arrives within the
delta time, and your dataset contents have a time lag. The late data notification feature enables
AWS IoT Analytics to send notifications through Amazon CloudWatch Events when data arrives after the delta time.
You can use the AWS IoT Analytics console, API, AWS Command Line Interface (AWS CLI), or AWS SDK to specify late data rules for a dataset.
In the AWS IoT Analytics API, the LateDataRuleConfiguration
object represents the late data
rule settings of a dataset. This object is part of the Dataset
object associated with
the CreateDataset
and UpdateDataset
API operations.
Parameters
When you create a late data rule for a dataset with AWS IoT Analytics, you must specify the following information:
ruleConfiguration
(LateDataRuleConfiguration
)-
A structure that contains the configuration information of a late data rule.
deltaTimeSessionWindowConfiguration
-
A structure that contains the configuration information of a delta time session window.
DeltaTime specifies a time interval. You can use
DeltaTime
to create dataset contents with data that has arrived in the data store since the last execution. For an example ofDeltaTime
, see Creating a SQL dataset with a delta window (CLI).timeoutInMinutes
-
A time interval. You can use
timeoutInMinutes
so that AWS IoT Analytics can batch up late data notifications that have been generated since the last execution. AWS IoT Analytics sends one batch of notifications to CloudWatch Events at one time.Type: Integer
Valid range: 1-60
ruleName
-
The name of the late data rule.
Type: String
Important
To specify lateDataRules
, the dataset must use a DeltaTime
filter.
Configure late data rules (console)
The following procedure shows you how to configure the late data rule of a dataset in the AWS IoT Analytics console.
To configure late data rules
-
Sign in to the AWS IoT Analytics console
. -
In the navigation pane, choose Data sets.
-
Under Data sets, choose the target data set.
-
In the navigation pane, choose Details.
-
In the Delta window section, choose Edit.
-
Under Configure data selection filter, do the following:
-
For Data selection window, choose Delta time.
-
For Offset, enter a time period, and then choose a unit.
-
For Timestamp expression, enter an expression. This can be the name of a timestamp field or a SQL expression that can derive the time, such as
from_unixtime(time)
.For more information about how to write a timestamp expression, see Date and Time Functions and Operators
in the Presto 0.172 Documentation. -
For Late data notification, choose Active.
-
For Delta time, enter an integer. The valid range is 1-60.
-
Choose Save.
-
Configure late data rules (CLI)
In the AWS IoT Analytics API, the LateDataRuleConfiguration
object represents the late data
rule settings of a dataset. This object is part of the Dataset
object associated
with CreateDataset
and UpdateDataset
. You can use the API, AWS CLI, or AWS SDK to specify late
data rules for a dataset. The following example uses the AWS CLI.
To create your dataset with specified late data rules, run the following command. The
command assumes that the dataset.json
file is in the current
directory.
Note
You can use the UpdateDataset API to update an existing dataset.
aws iotanalytics create-dataset --cli-input-json file://dataset.json
The dataset.json
file should contain the following:
-
Replace
demo_dataset
with the target dataset name. -
Replace
demo_datastore
with the target data store name. -
Replace
from_unixtime(time)
with the name of a timestamp field or a SQL expression that can derive the time.For more information about how to write a timestamp expression, see Date and Time Functions and Operators
in the Presto 0.172 Documentation. -
Replace
timeout
with an integer between 1–60. -
Replace
demo_rule
with any name.
{ "datasetName": "
demo_dataset
", "actions": [ { "actionName": "myDatasetAction
", "queryAction": { "filters": [ { "deltaTime": { "offsetSeconds":-180
, "timeExpression": "from_unixtime(time)
" } } ], "sqlQuery": "SELECT * FROMdemo_datastore
" } } ], "retentionPeriod": { "unlimited": false, "numberOfDays": 90 }, "lateDataRules": [ { "ruleConfiguration": { "deltaTimeSessionWindowConfiguration": { "timeoutInMinutes":timeout
} }, "ruleName": "demo_rule
" } ] }
Subscribing to receive late data notifications
You can create rules in CloudWatch Events that defines how to process late data notifications sent from AWS IoT Analytics. When CloudWatch Events receives the notifications, it invokes specified the target actions defined in your rules.
Prerequisites for creating CloudWatch Events rules
Before you create a CloudWatch Events rule for AWS IoT Analytics, you should do the following:
-
Familiarize yourself with events, rules, and targets in CloudWatch Events.
-
Create and configure the targets invoked by your CloudWatch Events rules. Rules can invoke many types of targets, such as the following:
-
Amazon Kinesis streams
-
AWS Lambda functions
-
Amazon Simple Notification Service (Amazon SNS) topics
-
Amazon Simple Queue Service (Amazon SQS) queues
Your CloudWatch Events rule, and the associated targets must be in the AWS Region where you created your AWS IoT Analytics resources. For more information, see Service endpoints and quotas in the AWS General Reference.
-
For more information, see What is CloudWatch Events? and Getting started with Amazon CloudWatch Events in the Amazon CloudWatch Events User Guide.
Late data notification event
The event for late data notifications uses the following format.
{ "version": "0", "id": "7f51dfa7-ffef-97a5-c625-abddbac5eadd", "detail-type": "IoT Analytics Dataset Lifecycle Notification", "source": "aws.iotanalytics", "account": "123456789012", "time": "2020-05-14T02:38:46Z", "region": "us-east-2", "resources": ["arn:aws:iotanalytics:us-east-2:123456789012:dataset/demo_dataset"], "detail": { "event-detail-version": "1.0", "dataset-name": "demo_dataset", "late-data-rule-name": "demo_rule", "version-ids": ["78244852-8737-4650-aa4d-3071a01338fa"], "message": null } }
Create a CloudWatch Events rule to receive late data notifications
The following procedure shows you how to create a rule that sends AWS IoT Analytics late data notifications to an Amazon SQS queue.
To create a CloudWatch Events rule
-
Sign in to the Amazon CloudWatch console
. -
In the navigation pane, under Events, choose Rules.
-
On the Rules page, choose Create rule.
-
Under Event Source, choose Event Pattern.
-
In the Build event pattern to match events by service section, do the following:
-
For Service Name, choose IoT Analytics
-
For Event Type, choose IoT Analytics Dataset Lifecycle Notification.
-
Choose Specific dataset name(s), and then enter the name of the target dataset.
-
-
Under Targets, choose Add target*.
-
Choose SQS queue, and then do the following:
-
For Queue*, choose the target queue.
-
-
Choose Configure details.
-
On the Step 2: Configure rule details page, enter a name and a description.
-
Choose Create rule.