Choosing between the WriteRecords API operation and batch load - Amazon Timestream

Choosing between the WriteRecords API operation and batch load

With the WriteRecords API operation, you can write your streaming time series data into Timestream for LiveAnalytics as it's generated by your system. By using WriteRecords, you can continuously ingest a single data point or smaller batches of data in real time. Timestream for LiveAnalytics offers you a flexible schema that auto detects the column names and data types for your Timestream for LiveAnalytics tables, based on the dimension names and data types of the data points you specify when invoking writes into the database.

In contrast, batch load enables the robust ingestion of batched time-series data from source files (CSV files) into Timestream for LiveAnalytics, using a data model that you define. A few examples for when to use batch load with a source file are importing time series data in bulk for the evaluation of Timestream for LiveAnalytics through a proof of concept, importing time series data in bulk from an IoT device that was offline for some time, and migrating historical time series data from Amazon S3 to Timestream for LiveAnalytics. For information about batch load, see Using batch load in Timestream for LiveAnalytics.

Both solutions are secure, reliable, and performant.

Use WriteRecords when:

  • Streaming smaller amounts (less than 10 MB) of data per request.

  • Populating existing tables.

  • Ingesting data from a log stream.

  • Performing real-time analytics.

  • Requiring lower latency.

Use batch load when:

  • Ingesting larger loads of data that originate in Amazon S3 in CSV files. For more information about limits, see Quotas.

  • Populating new tables, such as in the case of a data migration.

  • Enriching databases with historical data (ingestion into new tables).

  • You have source data that changes slowly or not at all.

  • You have flexible wait times because a batch load task might be in a pending state until resources are available, especially if you load a very large amount of data. Batch load is suitable for data that doesn't need to be readily available for querying or analysis to add more clarity.