Write modes with global tables - Amazon DynamoDB

Write modes with global tables

Global tables are always active-active at the table level. However, you might want to treat them as active-passive by controlling how you route write requests. For example, you might decide to route write requests to a single Region to avoid potential write conflicts.

There are three main categorizations of managed write patterns:

  • Write to any Region mode (no primary)

  • Write to one Region mode (single primary)

  • Write to your Region mode (mixed primary)

You should consider which write pattern fits your use case. This choice affects how you route requests, evacuate a Region, and handle disaster recovery. Overall best practices can differ depending on your application’s write mode.

Write to any Region mode (no primary)

The write to any Region mode is fully active-active and doesn’t impose restrictions on where a write may occur. Any Region may accept a write at any time. This is the simplest mode. It can be used only with some type of applications. It’s suitable when all writers are idempotent, and therefore safely repeatable so that concurrent or repeated write operations across Regions are not in conflict. For example, when a user updates their contact data. This mode also works well for a special case of being idempotent, an append-only dataset where all writes are unique inserts under a deterministic primary key. Lastly, this mode is suitable where the risk of conflicting writes would be acceptable.

Diagram of how client writes to any region works.

The write to any Region mode is the most straightforward architecture to implement. Routing is easier because any Region can be the write target at any time. Failover is easier, because any recent writes can be replayed any number of times to any secondary Region. Where possible, you should design for this write mode.

For example, video streaming services often use global tables for tracking bookmarks, reviews, watch status flags, and so on. These deployments can use the write to any Region mode as long as they ensure that every write is idempotent and the next correct value for an item doesn’t depend on its current value. This will be the case for user updates which assign the user’s new state directly, such as setting a new latest time code, assigning a new review, or setting a new watch status. If the user’s write requests are routed to different Regions, the last write operation will persist and the global state will settle according to the last assignment. Read operations in this mode will eventually become consistent, after being delayed by the latest ReplicationLatency value.

In another example, a financial services firm uses global tables as part of a system to maintain a running tally of debit card purchases for each customer, to calculate that customer’s cash-back rewards. New transactions stream in from around the world and go to multiple Regions. For their current design that doesn't take advantage of global tables, they use a single RunningBalance item per customer. Customer actions update the balance with an ADD expression, which is not idempotent because the new correct value depends on the current value. This means the balance got out of sync if there were two write operations to the same balance at around the same time in different Regions.

This same firm could achieve a write to any Region mode through a careful redesign with DynamoDB’s global tables. The new design could follow an “event streaming” model - essentially a ledger with an append-only workflow. Each customer action appends a new item to the item collection maintained for that customer. The item collection is the set of items that share a primary key, having different sort keys. Each write action that appends the cusomter action is an idempotent insert, using the customer ID as the partition key and transaction ID as the sort key. This design makes the calculation of the balance more involved, because it requires a Query to pull the items followed by some client-side math. But the advantage is that it makes all writes idempotent, which provides significant routing and failover simplifications. For more information see Request routing with global tables.

For a third example, let’s say there’s a customer doing online ad placement. They’ve decided a low risk of data loss would be acceptable to achieve the design simplifications of the write to any Region mode. When they serve ads, they have just a few milliseconds to retrieve enough metadata to determine what ad to show, and then to record the ad impression so the same ad won't repeated to that user. With global tables they can get both low-latency reads for end users across the world and low-latency writes. They can record all ad impressions for a user within a single item, and represent that as a growing List. They can use one item instead of appending to an item collection, because this way they can remove older ad impressions as part of each write without paying for a delete. This write operation is NOT idempotent, so if the same end user sees ads served out of multiple Regions at approximately the same time there’s a chance one ad impression write could overwrite the other. For online ad placement, the risk that a user might occasionally see a repeated ad is worth having this simpler and more efficient design.

Single primary (“Write to one Region”)

The write to one Region mode is active-passive and routes all table writes to a single active region. Note that DynamoDB doesn’t have a notion of a single active region; the application routing outside DynamoDB manages this. The write to one Region mode avoids write conflicts by ensuring writes only flow to one Region at a time. This write mode is helpful when you want to use conditional expressions or transactions, because they won't work unless you know you’re acting against the latest data. So using conditional expressions and transactions requires sending all write requests to the one Region with the latest data.

Eventually consistent reads can go to any replica Regions to achieve lower latencies. Strongly consistent reads must go to the single primary region.

Diagram of how writing to one Region works.

It’s sometimes necessary to change the active Region in response to a Regional failure, to help with data. Evacuating a Region with global tables is one example ofthis use case. Some customers will change the currently active Region on a regular schedule, such as a "follow- the-sun" deployment. This places the active Region near the geography with the most activity, giving it the lowest latency reads and writes. It also has the side benefit of calling the Region- changing code path on a daily basis, making sure it’s well tested before any disaster recovery.

The passive Region(s) may keep a downscaled set of infrastructure surrounding DynamoDB that gets built up only should it become the active region. For a more in-depth discussion of pilot light and warm standby designs see Disaster Recovery (DR) Architecture on AWS, Part III: Pilot Light and Warm Standby.

Using the write to one Region mode works well when leveraging global tables for low-latency globally distributed reads. For example, a large social media company has millions of users and billions of posts. Each user gets assigned to a Region at time of account creation, placed geographically near to their location. Into that non-global table goes all their data. The company uses a separate global table to hold the mapping of users to their home regions, using a write to one Region mode. It keeps read-only copies around the world to help locate each user’s data directly with minimum added latency. Updates are rare (only when moving a user’s home Region from one to another) and always go through one Region for writing, to avoid any chance of write conflicts.

As another example, consider a financial services customer who implemented a daily cash back calculation. They use write to any Region mode for calculating the balance but use write to one Region mode for tracking the actual cash back payments. If they want to reward a customer 1 penny for every $10 spent a day, they will need to Query for all transactions from the previous day, calculate the total spent, write the cash back decision to a new table, delete the queried set of items to mark them as consumed, and replace them with a singular item storing any remainder amount that should go into the next day’s calculations. This work requires transactions, and so will work better with the write to one Region mode. An application may mix write modes, even on the same table, as long as the workloads have no chance of overlapping.

Mixed primary (“Write to your Region”)

The write to your Region mode assigns different data subsets to different Regions and allows write operations only to items through its home region. this mode is active-passive but assigns the active Region based on the item. Every Region is primary for its own non-overlapping data set, and writes must be guarded to ensure proper locality.

This mode is similar to write to one Region except that it enables lower latency writes, because the data associated with each end user can be placed in closer network proximity to that user. It also spreads the surrounding infrastructure more evenly between Regions and requires less work to build out infrastructure during a failover scenario, because all regions will have a portion of their infrastructure already active.

Diagram of how client writes to each item in a single Region works.

Determining the home region for items can be done in a variety of ways:

  • Intrinsic: Some aspect of the data makes clear to what Region it’s homed, like its partition key. For example, a customer and all data about that customer would be marked within the customer data as homed to a certain region. This technique is described in Use Region pinning to set a home Region for items in an Amazon DynamoDB global table

  • Negotiated: The home region of each data set is negotiated in some external manner, such as with a separate global service that maintains assignments. The assignment may have a finite duration after which it’s subject to renegotiation.

  • Table-oriented: Instead of a single replicating global table, have as many global tables as there are replicating Regions. Each table’s name indicates its home Region. In standard operations, all data is written to the home Region while other Regions keep a read-only copy. During a failover, another Region will temporarily adopt write duties for that table.

For example, imagine you’re working for a gaming company. You need low latency reads and writes for all gamers around the world. You can home each gamer to the Region closest them. That region takes all their reads and writes, ensuring there’s always strong read-after-write consistency. However, if that gamer travels or their home Region suffers an outage, a complete copy of their data will be available in alternative Regions. So the gamer can be assigned to different home Region as is useful.

As another example, imagine you’re working at a video conferencing company. Each conference call's metadata gets assigned to a particular Region. Callers can use the Region that's closest to them for lowest latency. If there’s a Region outage, using global tables allows quick recovery because the system can move the processing of the call to a different Region where there’s already a replicated copy of the data.