Map¶
Apply a function to each item in a collection¶
Map executes a function for each item in a collection concurrently. It manages concurrency, collects results as items complete, and checkpoints the outcome.
Each item runs in its own child context. The default nested mode checkpoints that context and its result. Flat mode omits the per-item context checkpoint to reduce operation overhead.
Use map to apply the same operation to every item in a collection. Use parallel instead to execute different operations concurrently.
import {
BatchResult,
DurableContext,
withDurableExecution,
} from "@aws/durable-execution-sdk-js";
export const handler = withDurableExecution(
async (event: any, context: DurableContext): Promise<number[]> => {
const result: BatchResult<number> = await context.map(
"square-numbers",
[1, 2, 3, 4, 5],
async (ctx, item, index) =>
ctx.step(`square-${index}`, async () => item * item),
);
return result.getResults();
},
);
from aws_durable_execution_sdk_python import (
BatchResult,
DurableContext,
durable_execution,
)
def square(ctx: DurableContext, item: int, index: int, items: list[int]) -> int:
return ctx.step(lambda _: item * item, name=f"square-{index}")
@durable_execution
def handler(event: dict, context: DurableContext) -> list[int]:
result: BatchResult[int] = context.map(
[1, 2, 3, 4, 5],
square,
name="square-numbers",
)
return result.get_results()
import java.util.List;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.model.MapResult;
public class SimpleMap extends DurableHandler<Void, List<Integer>> {
@Override
public List<Integer> handleRequest(Void input, DurableContext context) {
MapResult<Integer> result = context.map(
"square-numbers",
List.of(1, 2, 3, 4, 5),
Integer.class,
(item, index, ctx) -> ctx.step(
"square-" + index, Integer.class, s -> item * item));
return result.results();
}
}
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class SimpleMapExample
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<object, IReadOnlyList<int>>(Workflow, input, context);
private async Task<IReadOnlyList<int>> Workflow(object input, IDurableContext ctx)
{
IBatchResult<int> result = await ctx.MapAsync(
new[] { 1, 2, 3, 4, 5 },
async (itemCtx, item, index, items, ct) =>
await itemCtx.StepAsync(
async (_, _) => item * item,
name: $"square-{index}"),
name: "square-numbers");
return result.GetResults();
}
}
Method signature¶
context.map¶
// Named overload
map<TInput, TOutput>(
name: string | undefined,
items: TInput[],
mapFunc: MapFunc<TInput, TOutput>,
config?: MapConfig<TInput, TOutput>,
): DurablePromise<BatchResult<TOutput>>
// Unnamed overload
map<TInput, TOutput>(
items: TInput[],
mapFunc: MapFunc<TInput, TOutput>,
config?: MapConfig<TInput, TOutput>,
): DurablePromise<BatchResult<TOutput>>
Parameters:
name(optional) A name for the map operation. Passundefinedto omit.itemsAn array of items to process.mapFuncAMapFunccalled for each item. See Map Function.config(optional) AMapConfig<TInput, TOutput>object.
Returns: DurablePromise<BatchResult<TOutput>>. Use await to get the result.
Throws: Item exceptions are captured in the BatchResult. Call throwIfError() to
re-throw the first failure.
def map(
inputs: Sequence[U],
func: Callable[[DurableContext, U, int, Sequence[U]], T],
name: str | None = None,
config: MapConfig | None = None,
) -> BatchResult[T]: ...
Parameters:
inputsA sequence of items to process.funcA callable called for each item. See Map Function.name(optional) A name for the map operation.config(optional) AMapConfigobject.
Returns: BatchResult[T].
Raises: Item exceptions are captured in the BatchResult. Call throw_if_error()
to re-raise the first failure.
// sync — blocks until all items complete
<I, O> MapResult<O> map(String name, Collection<I> items, Class<O> resultType,
MapFunction<I, O> function)
<I, O> MapResult<O> map(String name, Collection<I> items, Class<O> resultType,
MapFunction<I, O> function, MapConfig config)
<I, O> MapResult<O> map(String name, Collection<I> items, TypeToken<O> resultType,
MapFunction<I, O> function)
<I, O> MapResult<O> map(String name, Collection<I> items, TypeToken<O> resultType,
MapFunction<I, O> function, MapConfig config)
// async — returns immediately
<I, O> DurableFuture<MapResult<O>> mapAsync(String name, Collection<I> items,
Class<O> resultType, MapFunction<I, O> function)
<I, O> DurableFuture<MapResult<O>> mapAsync(String name, Collection<I> items,
Class<O> resultType, MapFunction<I, O> function,
MapConfig config)
<I, O> DurableFuture<MapResult<O>> mapAsync(String name, Collection<I> items,
TypeToken<O> resultType, MapFunction<I, O> function)
<I, O> DurableFuture<MapResult<O>> mapAsync(String name, Collection<I> items,
TypeToken<O> resultType, MapFunction<I, O> function,
MapConfig config)
Parameters:
name(required) A name for the map operation.itemsACollection<I>of items to process. Its iteration order must remain stable during replay so item indexes match their checkpoints. Use an ordered collection such asList,LinkedHashSet, orTreeSet. The SDK rejectsHashSetand thekeySet(),values(), andentrySet()views of known unordered maps. Copy or sort unordered inputs into aListbefore callingmap()ormapAsync().resultTypeClass<O>orTypeToken<O>for deserialization.functionAMapFunction<I, O>called for each item. See Map Function.config(optional) AMapConfigobject.
Returns: MapResult<O> from map(), or DurableFuture<MapResult<O>> from
mapAsync().
Throws: IllegalArgumentException if items is null or has a known
non-deterministic iteration order. Item exceptions are captured in MapResult.
Inspect failed() to detect failures. If the SDK cannot reconstruct the original
exception, it throws MapIterationFailedException.
Task<IBatchResult<TResult>> MapAsync<TItem, TResult>(
IReadOnlyList<TItem> items,
Func<IDurableContext, TItem, int, IReadOnlyList<TItem>, CancellationToken, Task<TResult>> func,
string? name = null,
MapConfig<TItem>? config = null,
CancellationToken cancellationToken = default);
Parameters:
itemsAnIReadOnlyList<TItem>of items to process.funcA function called for each item. See Map Function.name(optional) A name for the map operation. Omit it to infer one from the call site.config(optional) AMapConfig<TItem>object.cancellationToken(optional) A token linked with the SDK's workflow-shutdown signal, forwarded tofunc.
Returns: Task<IBatchResult<TResult>>. Use await to get the result.
Throws: Item exceptions are captured in the IBatchResult. Inspect Failed to
detect failures, or call ThrowIfError() to re-throw the first failure. The map
throws MapException only when the CompletionConfig criteria are violated.
Map Function¶
type MapFunc<TInput, TOutput> = (
context: DurableContext,
item: TInput,
index: number,
array: TInput[],
) => Promise<TOutput>
Parameters:
contextThe childDurableContextfor this item's execution.itemThe current item being processed.indexThe zero-based index of the item in the input array.arrayThe full input array.
Returns: Promise<TOutput>.
Parameters:
ctxThe childDurableContextfor this item's execution.itemThe current item being processed.indexThe zero-based index of the item in the input sequence.itemsThe full input sequence.
Returns: R.
@FunctionalInterface
interface MapFunction<I, O> {
O apply(I item, int index, DurableContext context);
}
Parameters:
itemThe current item being processed.indexThe zero-based index of the item in the input collection.contextThe childDurableContextfor this item's execution.
Returns: O.
Parameters:
contextThe childIDurableContextfor this item's execution.itemThe current item being processed.indexThe zero-based index of the item in the input list.itemsThe full input list.cancellationTokenA token linked with the SDK's workflow-shutdown signal. It is also tripped when a sibling item satisfies theCompletionConfigand the map short-circuits.
Returns: Task<TResult>.
MapConfig¶
interface MapConfig<TItem, TResult> {
maxConcurrency?: number;
itemNamer?: (item: TItem, index: number) => string;
completionConfig?: CompletionConfig;
serdes?: Serdes<BatchResult<TResult>>;
itemSerdes?: Serdes<TResult>;
summaryGenerator?: (result: BatchResult<TResult>) => string;
nesting?: NestingType;
}
Parameters:
maxConcurrency(optional) Maximum items running at once. Default: unlimited.itemNamer(optional) A function that returns a custom name for each item, used in logs and tests.completionConfig(optional) When to stop. Default: wait for all items.serdes(optional) CustomSerdesfor theBatchResult.itemSerdes(optional) CustomSerdesfor individual item results.summaryGenerator(optional) A function invoked when the serializedBatchResultexceeds 256KB. See Checkpointing.nesting(optional)NestingType.NESTED(default) orNestingType.FLAT. See Nesting.
@dataclass(frozen=True)
class MapConfig(Generic[T]):
max_concurrency: int | None = None
completion_config: CompletionConfig = CompletionConfig()
serdes: SerDes | None = None
item_serdes: SerDes | None = None
summary_generator: SummaryGenerator | None = None
nesting_type: NestingType = NestingType.NESTED
item_namer: Callable[[T, int], str] | None = None
Parameters:
max_concurrency(optional) Maximum items running at once. Default: unlimited.completion_config(optional) When to stop. Default:CompletionConfig()(lenient, all items run regardless of failures).serdes(optional) CustomSerDesfor theBatchResult.item_serdes(optional) CustomSerDesfor individual item results.summary_generator(optional) A callable invoked when the serializedBatchResultexceeds 256KB. See Checkpointing.nesting_type(optional)NestingType.NESTED(default) orNestingType.FLAT. See Nesting.item_namer(optional) A deterministic callable that returns a custom name for each item from the item and its zero-based index.
MapConfig.builder()
.maxConcurrency(Integer) // optional
.completionConfig(CompletionConfig) // optional
.serDes(SerDes) // optional
.nestingType(NestingType) // optional
.itemNamer(BiFunction<Object, Integer, String>) // optional
.itemNamer(Class<I>, BiFunction<? super I, Integer, String>) // optional
.build()
Parameters:
maxConcurrency(optional) Maximum items running at once. Default: unlimited.completionConfig(optional) When to stop. Default:CompletionConfig.allCompleted().serDes(optional) CustomSerDesfor item results and the overall result.nestingType(optional)NestingType.NESTED(default) orNestingType.FLAT. See Nesting.itemNamer(optional) A function that returns a custom name for each item from the item and its zero-based index. Pass the item type as the first argument to receive the item strongly typed instead of asObject. Java does not supportitemNamerwithNestingType.FLAT.
public sealed class MapConfig<TItem>
{
public int? MaxConcurrency { get; set; } // null = unlimited
public CompletionConfig CompletionConfig { get; set; } // default AllSuccessful()
public NestingType NestingType { get; set; } // default Nested
public Func<TItem, int, string>? ItemNamer { get; set; }
}
Parameters:
MaxConcurrency(optional) Maximum items running at once.null(default) is unlimited; must be at least 1 when set.CompletionConfig(optional) When to stop. Default:CompletionConfig.AllSuccessful(). Any item failure completes the map withFailureToleranceExceeded. SetCompletionConfig.AllCompleted()to run every item regardless of failures.NestingType(optional)NestingType.Nested(default) orNestingType.Flat. See Nesting.ItemNamer(optional) A function that returns a custom name for each item, given the item and its zero-based index. Used in logs and traces. Whennull(default), items are named by index.
The BatchResult and per-item results are serialized with the ILambdaSerializer
registered on ILambdaContext.Serializer; there is no per-item serializer slot. See
Serialization.
CompletionConfig¶
See Completion strategies for how CompletionConfig affects
execution and the completion status of the result.
CompletionConfig.allCompleted()
CompletionConfig.allSuccessful()
CompletionConfig.firstSuccessful()
CompletionConfig.minSuccessful(int count)
CompletionConfig.toleratedFailureCount(int count)
CompletionConfig.toleratedFailurePercentage(double percentage)
CompletionConfig.shouldComplete(
Function<CompletionStatus, CompletionDecision> decision)
Use the static factories or set the properties directly:
CompletionConfig.AllSuccessful() // default for map: ToleratedFailureCount = 0
CompletionConfig.AllCompleted() // every item runs regardless of failures
CompletionConfig.FirstSuccessful() // MinSuccessful = 1
new CompletionConfig { MinSuccessful = count }
new CompletionConfig { ToleratedFailureCount = count }
new CompletionConfig { ToleratedFailurePercentage = ratio } // ratio in [0.0, 1.0]
Result types¶
Map returns the same BatchResult<TResult> type as parallel.
interface BatchResult<TResult> {
all: BatchItem<TResult>[];
status: BatchItemStatus.SUCCEEDED | BatchItemStatus.FAILED;
completionReason: "ALL_COMPLETED" | "MIN_SUCCESSFUL_REACHED" | "FAILURE_TOLERANCE_EXCEEDED";
hasFailure: boolean;
successCount: number;
failureCount: number;
startedCount: number;
totalCount: number;
getResults(): TResult[];
getErrors(): ChildContextError[];
succeeded(): BatchItem<TResult>[];
failed(): BatchItem<TResult>[];
started(): BatchItem<TResult>[];
throwIfError(): void;
}
allallBatchItementries, one per item, in input ordergetResults()results of succeeded items, preserving input ordergetErrors()ChildContextError[]for failed itemssucceeded()/failed()/started()BatchItem[]filtered by statussuccessCount/failureCount/startedCount/totalCountitem countsstatusSUCCEEDEDif no failures,FAILEDotherwisecompletionReasonwhy the operation completed. See Completion strategies.hasFailuretrueif any item failedthrowIfError()throws the first item error, if any
Map returns the same BatchResult[R] type as parallel.
@dataclass(frozen=True)
class BatchResult(Generic[R]):
all: list[BatchItem[R]]
completion_reason: CompletionReason
def get_results(self) -> list[R]: ...
def get_errors(self) -> list[ErrorObject]: ...
def succeeded(self) -> list[BatchItem[R]]: ...
def failed(self) -> list[BatchItem[R]]: ...
def started(self) -> list[BatchItem[R]]: ...
def throw_if_error(self) -> None: ...
def to_dict(self) -> dict: ...
@property
def status(self) -> BatchItemStatus: ...
@property
def has_failure(self) -> bool: ...
@property
def success_count(self) -> int: ...
@property
def failure_count(self) -> int: ...
@property
def started_count(self) -> int: ...
@property
def total_count(self) -> int: ...
allallBatchItementries, one per item, in input orderget_results()results of succeeded items, preserving input orderget_errors()list[ErrorObject]for failed itemssucceeded()/failed()/started()BatchItemlists filtered by statussuccess_count/failure_count/started_count/total_countitem countsstatusBatchItemStatus.SUCCEEDEDif no failures,FAILEDotherwisecompletion_reasonwhy the operation completed. See Completion strategies.has_failureTrueif any item failedthrow_if_error()raises the first item error as aCallableRuntimeErrorto_dict()serializes to a plain dict. Serializability depends onR.
Map returns MapResult<O>, which differs from ParallelResult. It holds per-item
results with individual status, result, and error fields.
record MapResult<T>(
List<MapResultItem<T>> items,
ConcurrencyCompletionStatus completionReason
) {
MapResultItem<T> getItem(int index)
T getResult(int index)
MapError getError(int index)
boolean allSucceeded()
int size()
List<T> results() // all results, nulls for failed/skipped items
List<T> succeeded() // results of succeeded items only
List<MapError> failed() // errors of failed items only
}
record MapResultItem<T>(Status status, T result, MapError error) {
enum Status { SUCCEEDED, FAILED, SKIPPED }
}
record MapError(String errorType, String errorMessage, List<String> stackTrace) {}
enum ConcurrencyCompletionStatus {
ALL_COMPLETED,
MIN_SUCCESSFUL_REACHED,
FAILURE_TOLERANCE_EXCEEDED,
CUSTOM_COMPLETION_SUCCEEDED,
CUSTOM_COMPLETION_FAILED
}
itemsordered list ofMapResultItem, one per input itemgetItem(index)theMapResultItemat the given indexgetResult(index)the result at the given index, ornullif failed or skippedgetError(index)theMapErrorat the given index, ornullif succeeded or skippedallSucceeded()trueif every item has statusSUCCEEDEDsize()total number of itemsresults()all results as a list, withnullfor failed or skipped itemssucceeded()results of items with statusSUCCEEDEDfailed()MapErrorobjects for items with statusFAILEDcompletionReasonwhy the operation completed. See Completion strategies.
Items that did not start before the operation reached its completion criteria have
status SKIPPED (not STARTED as in TypeScript and Python).
Map returns the same IBatchResult<TResult> type as parallel. It holds per-item
results with individual status, result, and error.
public interface IBatchResult<T> : IBatchResult
{
IReadOnlyList<IBatchItem<T>> All { get; } // one per item, index order
IReadOnlyList<IBatchItem<T>> Succeeded { get; }
IReadOnlyList<IBatchItem<T>> Failed { get; }
IReadOnlyList<IBatchItem<T>> Started { get; }
IReadOnlyList<T> GetResults(); // succeeded results, index order
IReadOnlyList<DurableExecutionException> GetErrors();
void ThrowIfError(); // throws first item error, if any
}
public interface IBatchResult
{
CompletionReason CompletionReason { get; }
bool HasFailure { get; }
int SuccessCount { get; }
int FailureCount { get; }
int StartedCount { get; }
int TotalCount { get; }
}
AllallIBatchItementries, one per item, in original index orderGetResults()results of succeeded items, preserving index orderGetErrors()DurableExecutionExceptionfor failed items, in index orderSucceeded/Failed/StartedIBatchItemlists filtered by statusSuccessCount/FailureCount/StartedCount/TotalCountitem countsCompletionReasonwhy the operation completed. See Completion strategies.HasFailuretrueif any item failedThrowIfError()throws the first item error, if any
public interface IBatchItem<T>
{
int Index { get; }
string? Name { get; }
BatchItemStatus Status { get; }
T? Result { get; } // set when Status == Succeeded
DurableExecutionException? Error { get; } // set when Status == Failed
}
public enum BatchItemStatus
{
Succeeded,
Failed,
Started
}
Items that did not start before the operation reached its completion criteria have
status Started.
The map function¶
The map function can use any durable operation such as steps, waits, or nested map and parallel operations. Each item runs in its own child context, so items do not share state with each other or with the parent context.
import { DurableContext } from "@aws/durable-execution-sdk-js";
type Order = { id: string; amount: number };
type Receipt = { orderId: string; charged: number };
async function processOrder(
ctx: DurableContext,
order: Order,
index: number,
orders: Order[],
): Promise<Receipt> {
const validated = await ctx.step("validate", async () => {
if (order.amount <= 0) throw new Error("Invalid amount");
return order;
});
const charged = await ctx.step("charge", async () => validated.amount);
return { orderId: validated.id, charged };
}
from aws_durable_execution_sdk_python import DurableContext
def process_order(
ctx: DurableContext,
order: dict,
index: int,
orders: list[dict],
) -> dict:
def validate(_):
if order["amount"] <= 0:
raise ValueError("Invalid amount")
return order
validated = ctx.step(validate, name="validate")
charged = ctx.step(lambda _: validated["amount"], name="charge")
return {"orderId": validated["id"], "charged": charged}
import software.amazon.lambda.durable.DurableContext;
record Order(String id, double amount) {}
record Receipt(String orderId, double charged) {}
// MapFunction<Order, Receipt> implementation
Receipt processOrder(Order order, int index, DurableContext ctx) {
var validated = ctx.step("validate", Order.class, s -> {
if (order.amount() <= 0) throw new IllegalArgumentException("Invalid amount");
return order;
});
var charged = ctx.step("charge", Double.class, s -> validated.amount());
return new Receipt(validated.id(), charged);
}
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class MapFunctionExample
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<IReadOnlyList<Order>, IReadOnlyList<Receipt>>(
Workflow, input, context);
private async Task<IReadOnlyList<Receipt>> Workflow(
IReadOnlyList<Order> orders, IDurableContext ctx)
{
IBatchResult<Receipt> result = await ctx.MapAsync(
orders, ProcessOrder, name: "process-orders");
return result.GetResults();
}
// The map function: receives (ctx, item, index, allItems, cancellationToken)
private static async Task<Receipt> ProcessOrder(
IDurableContext ctx, Order order, int index,
IReadOnlyList<Order> orders, CancellationToken ct)
{
Order validated = await ctx.StepAsync(async (_, _) =>
{
if (order.Amount <= 0) throw new ArgumentException("Invalid amount");
return order;
}, name: "validate");
decimal charged = await ctx.StepAsync(
async (_, _) => validated.Amount, name: "charge");
return new Receipt(validated.Id, charged);
}
}
public record Order(string Id, decimal Amount);
public record Receipt(string OrderId, decimal Charged);
Naming map operations¶
Name your map operations to make them easier to identify in logs and tests.
import {
BatchResult,
DurableContext,
withDurableExecution,
} from "@aws/durable-execution-sdk-js";
export const handler = withDurableExecution(
async (event: { userIds: string[] }, context: DurableContext): Promise<string[]> => {
// Named: pass name as first argument, undefined to omit
const result: BatchResult<string> = await context.map(
"process-users",
event.userIds,
async (ctx, userId, index) =>
ctx.step(`process-${index}`, async () => `processed-${userId}`),
);
return result.getResults();
},
);
The name is the first argument. Pass undefined to omit it.
Use itemNamer in MapConfig to give each item a custom name:
from aws_durable_execution_sdk_python import (
BatchResult,
DurableContext,
durable_execution,
)
def process_user(
ctx: DurableContext, user_id: str, index: int, user_ids: list[str]
) -> str:
return ctx.step(lambda _: f"processed-{user_id}", name=f"process-{index}")
@durable_execution
def handler(event: dict, context: DurableContext) -> list[str]:
# Pass name as keyword argument; omit or pass None to leave unnamed
result: BatchResult[str] = context.map(
event["userIds"],
process_user,
name="process-users",
)
return result.get_results()
Pass name as a keyword argument. Omit it or pass None to leave it unnamed.
Use item_namer in MapConfig to give each item a custom name:
import java.util.List;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.model.MapResult;
public class NamedMap extends DurableHandler<List<String>, List<String>> {
@Override
public List<String> handleRequest(List<String> userIds, DurableContext context) {
// The name is always required in Java
MapResult<String> result = context.map(
"process-users",
userIds,
String.class,
(userId, index, ctx) -> ctx.step(
"process-" + index, String.class, s -> "processed-" + userId));
return result.succeeded();
}
}
The name is always required in Java. The SDK derives each item's name from the operation
name: {name}-iteration-{index}.
Use itemNamer in MapConfig to give each item a custom name:
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class NamedMapExample
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<UserEvent, IReadOnlyList<string>>(
Workflow, input, context);
private async Task<IReadOnlyList<string>> Workflow(
UserEvent input, IDurableContext ctx)
{
// The name is the optional trailing argument; omit it to leave the map unnamed
IBatchResult<string> result = await ctx.MapAsync(
input.UserIds,
async (itemCtx, userId, index, userIds, ct) =>
await itemCtx.StepAsync(
async (_, _) => $"processed-{userId}",
name: $"process-{index}"),
name: "process-users");
return result.GetResults();
}
}
public record UserEvent(IReadOnlyList<string> UserIds);
The name is the optional trailing argument. Omit it to infer one from the call site.
Use ItemNamer in MapConfig to give each item a custom name:
Configuration¶
Configure map behavior using MapConfig:
import {
BatchResult,
DurableContext,
NestingType,
withDurableExecution,
} from "@aws/durable-execution-sdk-js";
export const handler = withDurableExecution(
async (event: { urls: string[] }, context: DurableContext): Promise<string[]> => {
const result: BatchResult<string> = await context.map(
"fetch-urls",
event.urls,
async (ctx, url, index) =>
ctx.step(`fetch-${index}`, async () => {
const response = await fetch(url);
return response.text();
}),
{
maxConcurrency: 5,
completionConfig: { toleratedFailureCount: 2 },
nesting: NestingType.FLAT,
},
);
return result.getResults();
},
);
import urllib.request
from aws_durable_execution_sdk_python import (
BatchResult,
DurableContext,
durable_execution,
)
from aws_durable_execution_sdk_python.config import (
CompletionConfig,
MapConfig,
NestingType,
)
def fetch_url(
ctx: DurableContext, url: str, index: int, urls: list[str]
) -> str:
def do_fetch(_):
with urllib.request.urlopen(url) as response:
return response.read().decode()
return ctx.step(do_fetch, name=f"fetch-{index}")
@durable_execution
def handler(event: dict, context: DurableContext) -> list[str]:
config = MapConfig(
max_concurrency=5,
completion_config=CompletionConfig(tolerated_failure_count=2),
nesting_type=NestingType.FLAT,
)
result: BatchResult[str] = context.map(
event["urls"],
fetch_url,
name="fetch-urls",
config=config,
)
return result.get_results()
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.util.List;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.config.CompletionConfig;
import software.amazon.lambda.durable.config.MapConfig;
import software.amazon.lambda.durable.config.NestingType;
import software.amazon.lambda.durable.model.MapResult;
public class MapConfigExample extends DurableHandler<List<String>, List<String>> {
private static final HttpClient HTTP = HttpClient.newHttpClient();
@Override
public List<String> handleRequest(List<String> urls, DurableContext context) {
var config = MapConfig.builder()
.maxConcurrency(5)
.completionConfig(CompletionConfig.toleratedFailureCount(2))
.nestingType(NestingType.FLAT)
.build();
MapResult<String> result = context.map(
"fetch-urls",
urls,
String.class,
(url, index, ctx) -> ctx.step("fetch-" + index, String.class, s -> {
var request = HttpRequest.newBuilder(URI.create(url)).build();
return HTTP.sendAsync(request, HttpResponse.BodyHandlers.ofString())
.join()
.body();
}),
config);
return result.succeeded();
}
}
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class MapConfigExample
{
private static readonly HttpClient Http = new();
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<UrlEvent, IReadOnlyList<string>>(
Workflow, input, context);
private async Task<IReadOnlyList<string>> Workflow(
UrlEvent input, IDurableContext ctx)
{
var config = new MapConfig<string>
{
MaxConcurrency = 5,
CompletionConfig = new CompletionConfig { ToleratedFailureCount = 2 },
NestingType = NestingType.Flat,
};
IBatchResult<string> result = await ctx.MapAsync(
input.Urls,
async (itemCtx, url, index, urls, ct) =>
await itemCtx.StepAsync(
async (_, stepCt) => await Http.GetStringAsync(url, stepCt),
name: $"fetch-{index}"),
name: "fetch-urls",
config: config);
return result.GetResults();
}
}
public record UrlEvent(IReadOnlyList<string> Urls);
Nesting¶
Nested mode is the default. The SDK records each item context as a separate CONTEXT
operation and checkpoints the item result there. Each item appears separately in the
execution history.
In flat mode, the SDK uses a virtual context for each item and omits the per-item
CONTEXT operation. Durable operations inside the map function still checkpoint and
appear as children of the map operation. The SDK records the item outcome with the parent
map operation.
Use flat mode for maps with many items when each item performs few durable operations and you do not need each item represented separately in the execution history. Flat mode removes one checkpointed operation per item while preserving checkpoints for durable operations inside each item.
Completion strategies¶
CompletionConfig controls when the map operation completes. When the operation reaches
the completion criteria, it abandons items that have not completed yet. The abandoned
items will keep running in the background but cannot checkpoint their results after the
parent completes. The SDK makes a best-effort attempt to cancel ongoing work in
abandoned items, but cancellation is not guaranteed.
The BatchResult's completionReason indicates the stop condition. Items that had not
started yet do not appear in result.all. Items that had started but not completed
appear with status STARTED.
completionConfig |
Early exit completionReason |
Full completion completionReason |
|---|---|---|
{} or omitted |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
toleratedFailureCount=N |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
toleratedFailurePercentage=N |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
minSuccessful=N |
MIN_SUCCESSFUL_REACHED |
ALL_COMPLETED |
The BatchResult's completion_reason indicates the stop condition. Items that were
never started appear in result.all with status STARTED.
completion_config |
Early exit completion_reason |
Full completion completion_reason |
|---|---|---|
CompletionConfig() (default) |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
first_successful() |
MIN_SUCCESSFUL_REACHED |
ALL_COMPLETED |
tolerated_failure_count=N |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
tolerated_failure_percentage=N |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
min_successful=N |
MIN_SUCCESSFUL_REACHED |
ALL_COMPLETED |
Warning
CompletionConfig.all_completed() is deprecated. Use the default CompletionConfig()
instead.
The MapResult's completionReason indicates the stop condition. Items that did not
start before the operation completed have status SKIPPED.
completionConfig |
Early exit completionReason |
Full completion completionReason |
|---|---|---|
allCompleted() (default) |
n/a | ALL_COMPLETED |
allSuccessful() |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
firstSuccessful() |
MIN_SUCCESSFUL_REACHED |
ALL_COMPLETED |
minSuccessful(N) |
MIN_SUCCESSFUL_REACHED |
ALL_COMPLETED |
toleratedFailureCount(N) |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
toleratedFailurePercentage(p) |
FAILURE_TOLERANCE_EXCEEDED |
ALL_COMPLETED |
Use CompletionConfig.shouldComplete(...) when the predefined thresholds cannot
express the completion rule. The SDK evaluates the function as completion state
changes. It receives a CompletionStatus with successCount, failureCount,
completedCount, totalCount, and allItemsRegistered. Map registers all items
before processing begins.
Return CompletionDecision.continueExecution() to keep processing. Return
CompletionDecision.complete(...) with CUSTOM_COMPLETION_SUCCEEDED or
CUSTOM_COMPLETION_FAILED to stop and classify the result.
var completion = CompletionConfig.shouldComplete(status -> {
if (status.successCount() >= requiredSuccesses) {
return CompletionConfig.CompletionDecision.complete(
ConcurrencyCompletionStatus.CUSTOM_COMPLETION_SUCCEEDED);
}
if (status.failureCount() >= failureLimit) {
return CompletionConfig.CompletionDecision.complete(
ConcurrencyCompletionStatus.CUSTOM_COMPLETION_FAILED);
}
return CompletionConfig.CompletionDecision.continueExecution();
});
A custom completion function is mutually exclusive with minSuccessful,
toleratedFailureCount, and toleratedFailurePercentage. It must return a
non-null decision. Keep it deterministic and free of side effects. Items that have
not started when it completes have status SKIPPED.
CUSTOM_COMPLETION_FAILED does not throw automatically. Inspect
result.completionReason().isSucceeded() to distinguish the custom outcomes.
The IBatchResult's CompletionReason indicates the stop condition. Items that were
not dispatched before the operation completed have status Started.
CompletionConfig |
Early exit CompletionReason |
Full completion CompletionReason |
|---|---|---|
AllSuccessful() (default) |
FailureToleranceExceeded |
AllCompleted |
AllCompleted() |
n/a | AllCompleted |
FirstSuccessful() |
MinSuccessfulReached |
AllCompleted |
MinSuccessful = N |
MinSuccessfulReached |
AllCompleted |
ToleratedFailureCount = N |
FailureToleranceExceeded |
AllCompleted |
ToleratedFailurePercentage = ratio |
FailureToleranceExceeded |
AllCompleted |
Note
When using a minSuccessful strategy, failures do not trigger early exit. If all items
fail before the success threshold is reached, the operation completes with
ALL_COMPLETED.
import {
BatchResult,
DurableContext,
withDurableExecution,
} from "@aws/durable-execution-sdk-js";
export const handler = withDurableExecution(
async (event: { items: string[] }, context: DurableContext): Promise<string[]> => {
const result: BatchResult<string> = await context.map(
"process-items",
event.items,
async (ctx, item, index) =>
ctx.step(`process-${index}`, async () => item.toUpperCase()),
{
completionConfig: { minSuccessful: 3 },
},
);
return result.getResults();
},
);
from aws_durable_execution_sdk_python import (
BatchResult,
DurableContext,
durable_execution,
)
from aws_durable_execution_sdk_python.config import CompletionConfig, MapConfig
def process_item(
ctx: DurableContext, item: str, index: int, items: list[str]
) -> str:
return ctx.step(lambda _: item.upper(), name=f"process-{index}")
@durable_execution
def handler(event: dict, context: DurableContext) -> list[str]:
config = MapConfig(
completion_config=CompletionConfig(min_successful=3),
)
result: BatchResult[str] = context.map(
event["items"],
process_item,
name="process-items",
config=config,
)
return result.get_results()
import java.util.List;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.config.CompletionConfig;
import software.amazon.lambda.durable.config.MapConfig;
import software.amazon.lambda.durable.model.MapResult;
public class MapCompletionConfig extends DurableHandler<List<String>, List<String>> {
@Override
public List<String> handleRequest(List<String> items, DurableContext context) {
var config = MapConfig.builder()
.completionConfig(CompletionConfig.minSuccessful(3))
.build();
MapResult<String> result = context.map(
"process-items",
items,
String.class,
(item, index, ctx) -> ctx.step(
"process-" + index, String.class, s -> item.toUpperCase()),
config);
return result.succeeded();
}
}
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class CompletionConfigExample
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<ItemEvent, IReadOnlyList<string>>(
Workflow, input, context);
private async Task<IReadOnlyList<string>> Workflow(
ItemEvent input, IDurableContext ctx)
{
var config = new MapConfig<string>
{
CompletionConfig = new CompletionConfig { MinSuccessful = 3 },
};
IBatchResult<string> result = await ctx.MapAsync(
input.Items,
async (itemCtx, item, index, items, ct) =>
await itemCtx.StepAsync(
async (_, _) => item.ToUpperInvariant(),
name: $"process-{index}"),
name: "process-items",
config: config);
return result.GetResults();
}
}
public record ItemEvent(IReadOnlyList<string> Items);
Error handling¶
When an item throws an error, map captures the error in the result rather than propagating it immediately. Other items continue running.
BatchResult.status is FAILED if any item failed. Call throwIfError() to propagate
the first item error as an exception, or inspect getErrors() to handle errors
individually.
import {
BatchResult,
DurableContext,
withDurableExecution,
} from "@aws/durable-execution-sdk-js";
export const handler = withDurableExecution(
async (event: { items: string[] }, context: DurableContext): Promise<void> => {
const result: BatchResult<string> = await context.map(
"process-items",
event.items,
async (ctx, item, index) =>
ctx.step(`process-${index}`, async () => {
if (item === "bad") throw new Error("bad item");
return item.toUpperCase();
}),
);
if (result.hasFailure) {
const errors = result.getErrors();
console.log(`${result.failureCount} items failed:`, errors);
}
const successes = result.getResults();
console.log(`${result.successCount} items succeeded:`, successes);
},
);
BatchResult.status is FAILED if any item failed. Call throw_if_error() to
propagate the first item error as an exception, or inspect get_errors() to handle
errors individually.
from aws_durable_execution_sdk_python import (
BatchResult,
DurableContext,
durable_execution,
)
def process_item(
ctx: DurableContext, item: str, index: int, items: list[str]
) -> str:
def do_process(_):
if item == "bad":
raise ValueError("bad item")
return item.upper()
return ctx.step(do_process, name=f"process-{index}")
@durable_execution
def handler(event: dict, context: DurableContext) -> None:
result: BatchResult[str] = context.map(
event["items"],
process_item,
name="process-items",
)
if result.has_failure:
errors = result.get_errors()
print(f"{result.failure_count} items failed:", errors)
successes = result.get_results()
print(f"{result.success_count} items succeeded:", successes)
Check result.failed() to detect item failures. Each MapError contains errorType,
errorMessage, and stackTrace as plain strings. If the SDK cannot reconstruct the
original exception, it throws MapIterationFailedException.
import java.util.List;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.model.MapResult;
public class MapErrorHandling extends DurableHandler<List<String>, Void> {
@Override
public Void handleRequest(List<String> items, DurableContext context) {
MapResult<String> result = context.map(
"process-items",
items,
String.class,
(item, index, ctx) -> ctx.step("process-" + index, String.class, s -> {
if ("bad".equals(item)) throw new IllegalArgumentException("bad item");
return item.toUpperCase();
}));
var failures = result.failed();
if (!failures.isEmpty()) {
System.out.println(failures.size() + " items failed");
failures.forEach(e -> System.out.println(e.errorType() + ": " + e.errorMessage()));
}
var successes = result.succeeded();
System.out.println(successes.size() + " items succeeded: " + successes);
return null;
}
}
IBatchResult.HasFailure is true if any item failed. Call ThrowIfError() to
propagate the first item error as an exception, or inspect GetErrors() (which returns
DurableExecutionException objects) to handle errors individually.
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class ErrorHandlingExample
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<ItemEvent, Summary>(Workflow, input, context);
private async Task<Summary> Workflow(ItemEvent input, IDurableContext ctx)
{
IBatchResult<string> result = await ctx.MapAsync(
input.Items,
async (itemCtx, item, index, items, ct) =>
await itemCtx.StepAsync(async (_, _) =>
{
if (item == "bad") throw new InvalidOperationException("bad item");
return item.ToUpperInvariant();
}, name: $"process-{index}"),
name: "process-items");
// BatchResult captures per-item failures rather than throwing. Inspect
// HasFailure/GetErrors to handle them, or call ThrowIfError to propagate
// the first failure.
IReadOnlyList<DurableExecutionException> errors =
result.HasFailure ? result.GetErrors() : Array.Empty<DurableExecutionException>();
IReadOnlyList<string> successes = result.GetResults();
return new Summary(result.SuccessCount, result.FailureCount, successes, errors);
}
}
public record ItemEvent(IReadOnlyList<string> Items);
public record Summary(
int Succeeded,
int Failed,
IReadOnlyList<string> Results,
IReadOnlyList<DurableExecutionException> Errors);
Checkpointing¶
Checkpoint behavior depends on the nesting type. In nested mode, each item checkpoints
its result in a per-item CONTEXT operation. In flat mode, the SDK omits that context
checkpoint and records the item outcome with the parent map operation. Durable operations
inside an item still checkpoint in both modes.
Items that have not completed when the map operation reaches its completion criteria receive no further checkpoint updates. Unless noted otherwise, the language-specific details below describe nested mode.
The parent map operation also checkpoints the serialized BatchResult for
observability. On replay, the SDK deserializes the BatchResult directly from that
checkpoint.
For results over 256KB, the SDK cannot store the full BatchResult in the checkpoint.
Instead, the SDK reconstructs the BatchResult from the checkpointed results of the
individual items. In that case, the checkpoint stores a compact JSON summary, which is
for observability only.
The default summary generator produces:
The parent map operation also checkpoints the serialized BatchResult for
observability. On replay, the SDK deserializes the BatchResult directly from that
checkpoint.
For results over 256KB, the SDK cannot store the full BatchResult in the checkpoint,
so it re-executes the items to reconstruct it instead. In that case, the checkpoint
stores the output of summary_generator, which is for observability only.
The default summary generator produces:
{
"type": "MapResult",
"totalCount": 5,
"successCount": 4,
"failureCount": 1,
"completionReason": "ALL_COMPLETED",
"status": "FAILED"
}
When you pass a custom MapConfig without setting summary_generator, the SDK
checkpoints an empty string for large payloads.
SummaryGenerator is a callable protocol you can pass by setting summary_generator on
MapConfig:
For results under 256KB, the SDK checkpoints the serialized MapResult payload. On
replay, the SDK deserializes the MapResult directly from that checkpoint without
re-executing items.
For results over 256KB, the SDK checkpoints with an empty payload and a replayChildren
flag. On replay, the SDK re-executes the items to reconstruct the MapResult from their
individual checkpoints.
In nested mode, the SDK reconstructs IBatchResult from the per-item child-context
checkpoints without re-executing completed items. In flat mode, the SDK records item
results and errors inline on the parent map operation instead.
The SDK serializes results with the ILambdaSerializer registered on
ILambdaContext.Serializer; there is no per-item summary generator to configure.
Nesting map operations¶
A map function can call context.map() or context.parallel() to create nested
operations. Each nested map creates its own set of child contexts.
import {
BatchResult,
DurableContext,
withDurableExecution,
} from "@aws/durable-execution-sdk-js";
type Region = { name: string; items: string[] };
export const handler = withDurableExecution(
async (event: { regions: Region[] }, context: DurableContext): Promise<string[][]> => {
const result: BatchResult<string[]> = await context.map(
"process-regions",
event.regions,
async (ctx, region, index) => {
const inner: BatchResult<string> = await ctx.map(
`process-${region.name}`,
region.items,
async (innerCtx, item, i) =>
innerCtx.step(`item-${i}`, async () => item.toUpperCase()),
);
return inner.getResults();
},
);
return result.getResults();
},
);
from aws_durable_execution_sdk_python import (
BatchResult,
DurableContext,
durable_execution,
)
def process_item(
ctx: DurableContext, item: str, index: int, items: list[str]
) -> str:
return ctx.step(lambda _: item.upper(), name=f"item-{index}")
def process_region(
ctx: DurableContext, region: dict, index: int, regions: list[dict]
) -> list[str]:
inner: BatchResult[str] = ctx.map(
region["items"],
process_item,
name=f"process-{region['name']}",
)
return inner.get_results()
@durable_execution
def handler(event: dict, context: DurableContext) -> list[list[str]]:
result: BatchResult[list[str]] = context.map(
event["regions"],
process_region,
name="process-regions",
)
return result.get_results()
import java.util.List;
import software.amazon.lambda.durable.DurableContext;
import software.amazon.lambda.durable.DurableHandler;
import software.amazon.lambda.durable.TypeToken;
import software.amazon.lambda.durable.model.MapResult;
public class NestedMap extends DurableHandler<List<Region>, List<List<String>>> {
record Region(String name, List<String> items) {}
@Override
public List<List<String>> handleRequest(List<Region> regions, DurableContext context) {
MapResult<List<String>> result = context.map(
"process-regions",
regions,
new TypeToken<List<String>>() {},
(region, index, ctx) -> {
MapResult<String> inner = ctx.map(
"process-" + region.name(),
region.items(),
String.class,
(item, i, innerCtx) -> innerCtx.step(
"item-" + i, String.class, s -> item.toUpperCase()));
return inner.succeeded();
});
return result.succeeded();
}
}
using Amazon.Lambda.Core;
using Amazon.Lambda.DurableExecution;
public class NestedMapExample
{
public Task<DurableExecutionInvocationOutput> Handler(
DurableExecutionInvocationInput input, ILambdaContext context)
=> DurableFunction.WrapAsync<RegionEvent, IReadOnlyList<IReadOnlyList<string>>>(
Workflow, input, context);
private async Task<IReadOnlyList<IReadOnlyList<string>>> Workflow(
RegionEvent input, IDurableContext ctx)
{
IBatchResult<IReadOnlyList<string>> result = await ctx.MapAsync(
input.Regions,
async (regionCtx, region, index, regions, ct) =>
{
IBatchResult<string> inner = await regionCtx.MapAsync(
region.Items,
async (itemCtx, item, i, items, innerCt) =>
await itemCtx.StepAsync(
async (_, _) => item.ToUpperInvariant(),
name: $"item-{i}"),
name: $"process-{region.Name}");
return inner.GetResults();
},
name: "process-regions");
return result.GetResults();
}
}
public record Region(string Name, IReadOnlyList<string> Items);
public record RegionEvent(IReadOnlyList<Region> Regions);
See also¶
- Parallel operations execute different functions concurrently
- Child contexts understand child context isolation
- Steps use steps within map functions
- Error handling in durable functions
Checkpoint consumption
Durable operations consume checkpoints. To understand how this operation affects your checkpoint usage, see Checkpoint consumption.