Find the maximum concurrency an endpoint can serve under an SLA

When you benchmark a generative artificial intelligence (generative AI) endpoint, you often want to find its maximum concurrency. Concurrency is the number of simultaneous requests the endpoint handles while still meeting a performance target, such as a latency ceiling. You can explore concurrency in two ways in a single benchmark job:

• Concurrency list – Benchmark a fixed set of concurrency levels (for example, 1, 10, and 100) in one job. Use this when you already know the levels you want to compare.

• Find-max-concurrency search – Use this option to find the largest concurrency that still satisfies one or more service-level agreement (SLA) thresholds, without manually specifying the levels to test.

SLA parameters

Specify one or more SLA thresholds in the workload configuration parameters. The winning concurrency must satisfy all of the thresholds that you set.

Parameter	Meaning	Requires streaming
ttft_sla_ms	Maximum time to first token, in milliseconds	Yes
tpot_sla_ms	Maximum time per output token (inter-token latency), in milliseconds	Yes
e2e_sla_ms	Maximum end-to-end request latency, in milliseconds	No
error_rate_sla	Maximum fraction of failed requests (for example, 0.01 for 1 percent)	No

Note

Set search_stat to choose the statistic that the latency SLA is evaluated on: avg (default), p50, p90, p95, or p99. Bound the search with the optional concurrency_min and concurrency_max parameters, and cap cost with search_max_iterations.

Benchmark a list of concurrency levels

Pass a list to the concurrency parameter of the workload configuration. The benchmark runs each level and writes per-level results under concurrency_N/ directories in the output.

Python (Boto3)

client.create_ai_workload_config(
    AIWorkloadConfigName="concurrency-list-config",
    AIWorkloadConfigs={"WorkloadSpec": {"Inline": json.dumps({
        "benchmark": {"type": "aiperf"},
        "parameters": {
            "tokenizer": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
            "streaming": True,
            "prompt_input_tokens_mean": 128,
            "output_tokens_mean": 256,
            "concurrency": [1, 10, 100],   # list -> sweep each level in one job
            "request_count": 100,
        },
    })}},
)

Search for the maximum concurrency under an SLA

Set search_recipe to max-concurrency-under-sla and provide at least one SLA threshold. Do not also set a fixed concurrency value; the search selects it.

Python (Boto3)

client.create_ai_workload_config(
    AIWorkloadConfigName="find-max-config",
    AIWorkloadConfigs={"WorkloadSpec": {"Inline": json.dumps({
        "benchmark": {"type": "aiperf"},
        "parameters": {
            "tokenizer": "TinyLlama/TinyLlama-1.1B-Chat-v1.0",
            "streaming": True,
            "prompt_input_tokens_mean": 128,
            "output_tokens_mean": 256,
            "search_recipe": "max-concurrency-under-sla",
            "e2e_sla_ms": 5000,        # end-to-end latency ceiling (ms)
            "ttft_sla_ms": 1000,       # time-to-first-token ceiling (ms)
            "search_stat": "p99",      # evaluate the SLA on p99
            "concurrency_min": 1,
            "concurrency_max": 256,
            "search_max_iterations": 15,
        },
    })}},
)

Create a benchmark job that references the workload configuration and targets your endpoint, and then poll it to a terminal state. For the full create_ai_benchmark_job and describe_ai_benchmark_job sequence, see Benchmark generative AI inference endpoints.

Important

The error_rate_sla threshold requires a benchmarking tooling version that reports the request error-rate metric on runs that have no errors. On tooling versions that do not report this metric, the service treats every candidate concurrency as failing the SLA and finds no feasible level. Until your account's tooling is updated, gate the search on latency SLAs (ttft_sla_ms, tpot_sla_ms, or e2e_sla_ms). Then review the winning level's error rate in the results.

Read the search result

A find-max-concurrency search writes a search_history.json file at the root of the job's output. The winning concurrency is at boundary_summary.feasible_max.value, which is the largest level that satisfied every SLA. If no level met the SLA, feasible_max is null and boundary_summary.infeasible_min records the smallest level that breached it.

Python (Boto3)

import io, json, tarfile

resp = client.describe_ai_benchmark_job(AIBenchmarkJobName="find-max-job")
location = resp["OutputConfig"]["S3OutputLocation"]     # s3://bucket/prefix/
bucket, _, prefix = location.replace("s3://", "").partition("/")

obj = s3.list_objects_v2(Bucket=bucket, Prefix=prefix)["Contents"]
tar_key = next(o["Key"] for o in obj if o["Key"].endswith("output.tar.gz"))
body = s3.get_object(Bucket=bucket, Key=tar_key)["Body"].read()

with tarfile.open(fileobj=io.BytesIO(body), mode="r:gz") as tar:
    member = next(m for m in tar.getnames() if m.endswith("search_history.json"))
    history = json.loads(tar.extractfile(member).read())

boundary = history["boundary_summary"]
feasible_max = boundary.get("feasible_max")
if feasible_max and feasible_max.get("value") is not None:
    print("Max concurrency under SLA:", feasible_max["value"])
else:
    print("No level met the SLA. Smallest breaching level:",
          boundary.get("infeasible_min", {}).get("value"))