Processing and visualizing vehicle data
The Edge Agent for AWS IoT FleetWise software transfers selected vehicle data to Amazon Timestream or Amazon Simple Storage Service (Amazon S3). After your data arrives in the data destination, you can use other AWS services to visualize and share it.
Processing vehicle data in Timestream
Timestream is a fully managed time series database that can store and analyze trillions of time series data points per day. Your data is stored in a customer managed Timestream table. You can use Timestream to query vehicle data so that you can gain insights about your vehicles. For more information, see What is Amazon Timestream?
The default schema of data that is transferred to Timestream contains the following fields.
| Field name | Data type | Description |
|---|---|---|
|
|
varchar |
The ID of the data collection event. |
|
|
varchar |
The ID of the vehicle from which the data was collected. |
|
|
varchar |
The name of the campaign that the Edge Agent software uses to collect data. |
|
|
timestamp |
The timestamp of the data point. |
|
|
varchar |
The name of the signal. |
|
|
bigint |
Signal values of type Integer. |
|
|
double |
Signal values of type Double. |
|
|
boolean |
Signal values of type Boolean. |
Visualizing vehicle data stored in Timestream
After your vehicle data is transferred to Timestream, you can use the following AWS services to visualize, monitor, analyze, and share your data.
-
Visualize and monitor data in dashboards by using Grafana or Amazon Managed Grafana. You can visualize data from multiple AWS sources (such as Amazon CloudWatch and Timestream) and other data sources with a single Grafana dashboard.
-
Analyze and visualize data in dashboards by using Amazon QuickSight.
Processing vehicle data in S3
Amazon S3 is an object storage service that stores and protects any amount of data. You can use S3 for a variety of use cases, such as data lakes, backup and restore, archive, enterprise applications, AWS IoT devices, and big data analytics. Your data is stored in S3 as objects in buckets. For more information, see What is Amazon S3?
The default schema of data that is transferred to Amazon S3 contains the following fields.
| Field name | Data type | Description |
|---|---|---|
|
|
varchar |
The ID of the data collection event. |
|
|
varchar |
The ID of the vehicle from which the data was collected. |
|
|
varchar |
The name of the campaign that the Edge Agent software uses to collect data. |
|
|
timestamp |
The timestamp of the data point. |
|
|
varchar |
The name of the signal. |
|
|
bigint |
Signal values of type Integer. |
|
|
double |
Signal values of type Double. |
|
|
boolean |
Signal values of type Boolean. |
|
|
struct |
Signal values of type Struct. |
S3 object format
AWS IoT FleetWise transfers vehicle data to S3 where it's saved as an object. You can use the object URI that uniquely identifies the data to find data from the campaign. The S3 object URI format depends on if the collected data is unstructured or processed data.
Unstructured data is stored in S3 in a not pre-defined manner. It can be in various formats, such as images or videos.
Vehicle messages passed to AWS IoT FleetWise with signal data from Amazon Ion files are decoded and transferred to S3 as objects. The S3 objects represent each signal and are binary encoded.
The unstructured data S3 object URI uses the following format:
s3://bucket-name/prefix/unstructured-data/random-ID-yyyy-MM-dd-HH-mm-ss-SSS-vehicleName-signalName-fieldName
Processed data is stored in S3 and undergoes processing steps that validate, enrich, and transform messages. Object lists and velocity are examples of processed data.
Data transferred to S3 are stored as objects that represent records that were buffered
for a period of about 10 minutes. By default, AWS IoT FleetWise adds a UTC time prefix in the
format year=YYYY/month=MM/date=DD/hour=HH before writing objects to S3.
This prefix creates a logical hierarchy in the bucket where each forward slash
(/) creates a level in the hierarchy. The processed data also contains the S3 object URI to unstructured data.
The processed data S3 object URI uses the following format:
s3://bucket-name/prefix/processed-data/year=YYYY/month=MM/day=DD/hour=HH/part-0000-random-ID.gz.parquet
Raw data, also known as primary data, are data collected from Amazon Ion files. You can use raw data to troubleshoot any issues or to root cause errors.
The raw data S3 object URI uses the following format:
s3://bucket-name/prefix/raw-data/vehicle-name/eventID-timestamp.10n
Analyzing vehicle data stored in S3
After your vehicle data is transferred to S3, you can use the following AWS services to monitor, analyze, and share your data.
Extract and analyze data using Amazon SageMaker for downstream labeling and machine learning (ML) workflows.
For more information, see the following topics in the Amazon SageMaker Developer Guide:
Catalog your data using AWS Glue crawler and analyze it in Amazon Athena. By default, objects written to S3 have Apache Hive style time partitions, with data paths that contain key-value pairs connected by equal signs.
For more information, see the following topics in the Amazon Athena User Guide:
Visualize data using Amazon QuickSight by either reading your Athena table or S3 bucket directly.
Tip
If you're reading from S3 directly, confirm that your vehicle data is in JSON format because Amazon QuickSight doesn't support Apache Parquet format.
For more information, see the following topics in the Amazon QuickSight User Guide:
