AWS Documentation » AWS IoT » Developer Guide » Fleet Indexing Service

Fleet Indexing Service

Fleet Indexing is a managed service that allows you to index and search your registry and shadow data in the cloud. After your fleet index is set up, the service manages the indexing of all your registry and shadow updates. You can use a simple query language based on the popular open source search engine, Apache Lucene, to search across this data.

To get started, enable indexing and AWS IoT creates the index for your things. After it is active, you can run queries on your index. AWS IoT keeps it continuously updated with your latest data.

You can use the AWS IoT console to manage your indexing configuration and run your search queries. If you prefer programmatic access, you can use the AWS SDKs or the AWS CLI.

Please note that there are additional costs for using this service, beyond the standard charges for AWS IoT services, which are outlined in AWS IoT Device Management Pricing.

Managing Indexing

AWS_Things is the index created for all of your things. You can control whether you want to index only registry data or both registry and shadow data.

Enabling Indexing

You can create the AWS_Things index and control its configuration by using the thing-indexing-configuration setting in the UpdateIndexingConfiguration API. You can retrieve the current indexing configuration by using the GetIndexingConfiguration API.

The following command shows how to use the get-indexing-configuration CLI command to retrieve the current thing-indexing configuration:

aws iot get-indexing-configuration
{
    "thingIndexingConfiguration": {
        "thingIndexingMode": "OFF"
    }
}

You can use the AWS IoT update-indexing-configuration CLI command to update the thing-indexing configuration:

aws iot update-indexing-configuration --thing-indexing-configuration thingIndexingMode=REGISTRY_AND_SHADOW

Valid values for thing-indexing-configuration are:

OFF

No indexing/delete index.

REGISTRY

Create or configure the AWS_Things index to index registry data only.

REGISTRY_AND_SHADOW

Create or configure the AWS_Things index to index registry and shadow data.

Note

Normally, shadows are encrypted. However, if you decide to include shadows in the AWS_Things index, the data will be decrypted in order to index it.

Describing Indexes

The following command shows you how to use the describe-index CLI command to retrieve the current status of the index:

aws iot describe-index --index-name "AWS_Things"
{
    "indexName": "AWS_Things", 
    "indexStatus": "BUILDING", 
    "schema": "REGISTRY_AND_SHADOW"
}

The first time you enable indexing, AWS IoT builds your index. You cannot query the index if the indexStatus is BUILDING. The schema indicates which type of data, REGISTRY or REGISTRY_AND_SHADOW, is indexed.

Changing the configuration for your index causes the index to be rebuilt. The indexStatus during this process is REBUILDING. You can execute queries on the existing data while a rebuild is in progress. For example, if you change the index configuration from REGISTRY to REGISTRY_AND_SHADOW, while the index is being rebuilt, you can query registry data, including the latest updates, but you cannot query the shadow data until the rebuild is complete. The amount of time it takes to build or rebuild the index depends on the amount of data.

What is Indexed

Note the following restrictions and limitations:

Shadow fields with complex types:

A shadow field is indexed only if the value of the field is a simple type or an array consisting entirely of simple types. (By "simple type" we mean a string, a number, or one of the literals true or false). If a field's value is itself a JSON object, or an array containing an object, indexing will not be done on that field. For example, given shadow state:

{
    "state": {
        "reported": {
            "switched": "ON",
            "colors": [ "RED", "GREEN", "BLUE" ],
            "palette": [
                {
                    "name": "RED", 
                    "intensity": 124
                },
                {
                    "name": "GREEN", 
                    "intensity": 68
                },
                {
                    "name": "BLUE", 
                    "intensity": 201
                }
            ]
        }
    }
}

the value of field "palette" will NOT be indexed since it is an array whose items are "objects". The value of field "colors" WILL be indexed since each value in the array is a string.

Shadow metadata:

A field in a shadow's metadata section is indexed, but only if the corresponding field in the the shadow's "state" section is indexed. (In the above example, the "palette" field in the shadow's metadata section will not be indexed either.)

Unregistered shadows:

If you create a shadow using a thing name which has not been registered in your AWS IoT account (using CreateThing), fields in this shadow will not be indexed.

Numeric values:

If any registry or shadow data which is being indexed is recognized by the service as a numeric value, it is indexed as such. You may form queries involving ranges and comparison operators on numeric values, for example "attribute.foo<5" or "shadow.reported.foo:[75 TO 80]". To be recognized as numeric, the value of the data must be a valid JSON "number" type literal (an integer in the range -2^53...2^53-1, or a double-precision floating point with optional exponential notation) or part of an array containing only such values.

Null values:

Null values are not indexed.

Querying an Index

The following command shows how to use the search-index CLI command to query data in the index:

aws iot search-index --index-name "AWS_Things" --query-string "thingName:mything*"
{  
    "things":[{  
         "thingName":"mything1",
         "thingGroupNames":[  
            "mygroup1"
         ],
         "thingId":"a4b9f759-b0f2-4857-8a4b-967745ed9f4e",
         "attributes":{  
            "attribute1":"abc"
         }
    },
    {  
        "thingName":"mything2",
        "thingTypeName":"MyThingType",
        "thingGroupNames":[  
            "mygroup1",
            "mygroup2"
        ],
        "thingId":"01014ef9-e97e-44c6-985a-d0b06924f2af",
        "attributes":{  
            "model":"1.2",
            "country":"usa"
        },
        "shadow":{  
            "desired":{  
                "location":"new york",
                "myvalues":[3, 4, 5]
            },
            "reported":{  
                "location":"new york",
                "myvalues":[1, 2, 3],
                "stats":{  
                    "battery":78
                }
            },
            "metadata":{  
                 "desired":{  
                       "location":{  
                            "timestamp":123456789
                        },
                       "myvalues":{  
                             "timestamp":123456789
                       }
                  },
                  "reported":{  
                        "location":{  
                             "timestamp":34535454
                         },
                        "myvalues":{  
                             "timestamp":34535454
                        },
                        "stats":{  
                             "battery":{  
                                   "timestamp":34535454
                             }
                        }
                 }
            },
            "version":10,
            "timestamp":34535454
        }
    }],
    "nextToken":"AQFCuvk7zZ3D9pOYMbFCeHbdZ+h=G"
}

Query Syntax

Queries are specified using a Lucene-like query syntax. For more information, see Lucene query syntax overview on the Apache website.

The following features of Lucene query syntax are supported:

• Terms and phrases

• Searching fields

• Prefix search

• Range search

• Boolean operators AND, OR, NOT and –

• Grouping

• Field grouping

• Escaping special characters

The following features of Lucene query syntax are NOT supported:

• Leading wildcard search (such as "*xyz"), but searching for "*" will match all things

• Regular expressions

• Boosting

• Ranking

• Fuzzy searches

• Proximity search

• Sorting

• Aggregation

A few things to note about the query language:

• The default operator is AND. A query for “thingName:abc thingType:xyz” is equivalent to “thingName:abc AND thingType:xyz”.

• If a field is not specified, AWS IoT searches for the term in all fields.

• All field names are case-sensitive.

• Search is case-insensitive. Words are separated by whitespace characters as defined by Java's Character.isWhitespace(int).

• Indexing of shadow data includes reported, desired, delta, and metadata sections.

• Shadow and registry versions are not searchable, but are present in the response.

• The maximum number of terms in a query is 5.

Example Queries

Queries are specified in a query string using a Lucene-like query syntax and passed to the SearchIndex API. The following table lists some example query strings:

Query String	Result
"abc"	Queries for "abc" in any registry or shadow field.
"thingName:myThingName"	Queries for a thing with name "myThingName".
"thingName:my*"	Queries for things with names that begin with "my".
"thingName:ab?"	Queries for things with names that have "ab" plus one additional character, for example: "aba", "abb", "abc" etc.
"attributes.myAttribute:75"	Queries for things with an attribute called "MyAttribute" that has the value 75.
"attributes.myAttribute:[75 TO 80]"	Queries for things with an attribute called "MyAttribute" whose value falls within a numeric range (75 - 80, inclusive).
"attributes.myAttribute:{75 TO 80]"	Queries for things with an attribute called "MyAttribute" whose value falls within the numeric range (>75 and <=80).
'attributes.serialNumber["abcd" TO "abcf"]'	Queries for things with an attribute called "serialNumber" whose value is within an alphanumeric string range. This query will return things with a "serialNumber" attribute with values "abcd", "abce", or "abcf".
"attributes.myAttribute:i*t"	Queries for things with an attribute called "MyAttribute" whose value is 'i', followed by any number of characters, followed by 't'.
"attributes.attr1:abc AND attributes.attr2<5 NOT attributes.attr3>10	Queries for things that combine terms using Boolean expressions. This query will return things that have an attribute named "attr1" with a value "abc", an attribute named "attr2" that is less than 5, and an attribute named "attr3" that is not greater than 10.
"shadow.hasDelta:true"	Queries for things whose shadow has a delta element.
"-attributes.model:legacy"	Queries for things where the attribute model is not "legacy".
"shadow.reported.stats.battery:(>70 AND <100) (v2 | v3) -attributes.model:legacy"	Queries for things with the following: The thing's shadow stats.battery attribute has a value between 70 and 100. The text "v2" or "v3" occurs in a thing's name, type name, or attribute values. The thing's model attribute is not set to "legacy".
"shadow.reported.myvalues:2"	Queries for things where the myvalues array in the shadow's reported section contains a value of 2.
"shadow.reported.location:* NOT shadow.desired.stats.battery:*"	Queries for things with the following: The location attribute exists in the shadow's reported section. The stats.battery attribute does not exist in the shadow's desired section.

Authorization

You can specify the things index as a resource ARN in an AWS IoT policy action:

Action	Resource
iot:SearchIndex	An index ARN (for example, arn:aws:iot:<your-aws-region>:index/AWS_Things).
iot:DescribeIndex	An index ARN (for example, arn:aws:iot:<your-aws-region>:index/AWS_Things).