Configuring Timestream - Amazon Timestream

Configuring Timestream

  • Configure the data retention period for the memory store and the magnetic store to match the data processing, storage, query performance, and cost requirements.

    • Set the data retention of the memory store to match your application's requirements for processing late arrival data. Late arrival data is incoming data with a timestamp earlier than the current time. It is emitted from resources that batch events for a time period before sending the data to Timestream, and also from resources with intermittent connectivity e.g. an IoT sensor that is online intermittently.

    • Consider the characteristics of queries you plan to run on Timestream such as the types of queries, frequency, time range, and performance requirements. This is because the memory store and magnetic store are optimized for different scenarios. The memory store is optimized for fast point-in-time queries that process small amounts of recent data sent to Timestream. The magnetic store is optimized for fast analytical queries that process medium to large volumes of data sent to Timestream.

    • Your data retention period should also be influenced by the cost requirements of your system.

      For example, consider a scenario where the late arrival data threshold for your application is 2 hours and your applications send many queries that process a day's-worth, week's-worth, or month's-worth of data. In that case, you may want to configure a smaller retention period for the memory store (2-3 hours) and allow more data to flow to the magnetic store given the magnetic store is optimized for fast analytical queries

  • Understand the impact of increasing or decreasing the data retention period of the memory store and the magnetic store of an existing table.

    • When you decrease the retention period of the memory store, the data is moved from the memory store to the magnetic store, and this data transfer is permanent. Timestream does not retrieve data from the magnetic store to populate the memory store. When you decrease the retention period of the magnetic store, the data is deleted from the system and the data deletion is permanent

    • When you increase the retention period of the memory store or the magnetic store, the change takes effect for data being sent to Timestream from that point onwards. Timestream does not retrieve data from the magnetic store to populate the memory store. For example, if the retention period of the memory store was initially set to 2 hours and then increased to 24 hours, it will take 22 hours for the memory store to contain 24 hours worth of data.