Best Practice 19.1 – Identify the ingestion mechanisms that best fit your use case - IoT Lens Checklist

Best Practice 19.1 – Identify the ingestion mechanisms that best fit your use case

Identify which data ingestion method best fits with your use case to obtain the best performance/operational complexity tradeoff. Multiple mechanisms might be needed. It provides the optimal ingestion path for the data generated by your devices to obtain the best performance and costs trade-offs.

Recommendation 19.1.1 – Evaluate ingestion mechanism for telemetry data

  • Determine if the communication pattern is uni-directional (device to backend) or bi-directional. For example:

    • HTTPS should be considered if your device is acting as an aggregator and needs to send more than 100 messages per second instead of opening multiple MQTT connections. Use multiple threads and multiple HTTP connections to maximize the throughput for high delay networks as HTTP calls are synchronous.

  • Consider the APIs provided by the destination for your data and adopt them if you can securely access them. For example:

    • AWS IoT Analytics provides an HTTP API that is capable of batching several messages and is suitable for high rate data ingestion when the data is consumed in near-real-time fashion and a service-integrated data storage, data processing and data retention and replay are desired.

    • AWS IoT SiteWise provides an HTTP API to ingest operational data from industrial applications which needs to be stored for a limited period of time and processed as a time series with hierarchical aggregation capabilities.

    • Real-time video (for example, video surveillance cameras) has specific characteristics that makes it more suitable to ingest in a dedicated service, such as Amazon Kinesis Video Streams.

  • Consider the need for data to be buffered locally while the device is disconnected and the transmission resumed as soon as the connection is re-established. For example:

    • AWS IoT Greengrass stream manager provides a managed stream service with local persistence, local processing pipelines and out-of-the-box exporters to Amazon Kinesis Data Streams and AWS IoT Analytics (for example, industrial gateways).

  • Consider the latency, throughput and ordering characteristics of the data you want to ingest. For example:

    • For applications with a high ingestion rate (high-frequency sensor data) and where message ordering is important, Amazon Kinesis Data Streams provides stream-oriented processing capabilities and the ability to act as temporary storage.

    • For applications that do not have any real time requirements (such as logging, large images) and when the devices have the possibility to store data locally, uploading data directly to Amazon S3 can be both performant and cost efficient.

  • For more: