View a markdown version of this page

Best practices for GPU-accelerated functions on Lambda Managed Instances - AWS Lambda

Best practices for GPU-accelerated functions on Lambda Managed Instances

This page describes best practices for GPU-accelerated functions. GPU-accelerated functions are Event Functions on Lambda Managed Instances, so the practices in Best practices for Lambda Managed Instances apply to them as well. For an overview, see GPU support for Lambda Managed Instances.

Capacity provider configuration

  • Specify GPU instance types from several generations if your workload supports them. – Lambda attempts to launch the latest generation in your AllowedInstanceTypes list with the smallest GPU that meets your accelerator memory, and falls back to older generations when that instance type is unavailable. A list with a single instance type has nothing to fall back to and stops scaling out when that type is short in your Availability Zones. Specify g6.4xlarge, g5.4xlarge, and g4dn.4xlarge together rather than g6.4xlarge alone when your code runs on all of their GPUs. Execution speed varies across instance types. For more information about instance type selection, see Instance Type Selection.

  • Include instance types with at least as much CPU and memory as your function configuration. – An instance type has to satisfy your function's MemorySize and vCPU count, not only its AcceleratorMemorySize. The vCPUs an execution environment needs are MemorySize divided by ExecutionEnvironmentMemoryGiBPerVCpu (2 GiB per vCPU by default), and Lambda reserves part of the instance's system memory for management, so the full instance memory is not available to your function. A size that is too small on either dimension does not qualify even when its GPU is large enough. For example, a 16,384 MB function needs 8 vCPUs at the default ratio, which is every vCPU on a g6.2xlarge, so list the 4xlarge size of each generation rather than the 2xlarge, or raise ExecutionEnvironmentMemoryGiBPerVCpu so the same memory needs fewer vCPUs. For more information about incompatible accelerator instance types, see the IncompatibleAcceleratorInstanceTypes troubleshooting entry.

  • Choose instances with sufficient accelerator memory. – Lambda rejects a function configuration whose AcceleratorMemorySize no instance type in the capacity provider can satisfy, so when functions with different accelerator memory share a capacity provider, make sure the AllowedInstanceTypes list contains an instance type for each of them.

  • Reserve GPU capacity for production workloads. – GPU capacity varies by Region and Availability Zone more than CPU capacity does. If you purchase an Amazon EC2 On-Demand Capacity Reservation (ODCR) with open instance eligibility for an instance type and Availability Zone that match your capacity provider, Lambda uses the reserved capacity automatically. Lambda launches instances without targeting a reservation, so a targeted reservation is not used. For more information, see Reserve compute capacity with Amazon EC2 On-Demand Capacity Reservations.

Instance type selection

  • Request the smallest accelerator memory that fits one copy of your workload. – AcceleratorMemorySize is a minimum, and Lambda places the execution environment on the GPU with the least memory that meets it, so a larger value than you need moves your function to a larger and more expensive GPU. Measure the GPU memory that one copy of your model or working set occupies, leave room for the CUDA context and for what concurrent invocations allocate on top of it, and choose the next supported value (3, 6, 12, 16, 24, or 48 GB). For more information about accelerator memory, see Understanding accelerator memory.

  • Do not use fractional GPU instance types for encoder, decoder, and graphics workloads. – On G6f and Gr6f instances the GPU is a fraction of a physical L4 whose encoder, decoder, and graphics engines are shared with other EC2 instances, so throughput varies with their load. A transcoding or rendering function may need less GPU memory than a whole GPU provides, but leaving G6f and Gr6f out of AllowedInstanceTypes places the execution environment on a whole GPU and gives the function those engines to itself.

  • Do not size your workload from the GPU memory the driver reports. – During instance type fallback, Lambda can place the execution environment on a GPU with more memory than you requested, so treat AcceleratorMemorySize as the GPU memory budget your function has to stay within and it behaves the same on every instance type in the capacity provider.

Deployments

  • Prefer $LATEST.PUBLISHED for most workloads. – Republishing $LATEST.PUBLISHED is the simplest way to deploy as Lambda rolls out the new code and reclaims the old execution environments as it goes, so you don't manage or clean up numbered versions. This is the right default when your function's invocation timeout is 15 minutes or less.

  • Keep spare GPU capacity for deployments. – During $LATEST.PUBLISHED deployment, Lambda tries to start execution environments for the new version before it removes the old ones. If you publish a new version and Lambda cannot launch the new execution environments, it does an in-place deployment. With an in-place deployment, Lambda replaces the existing execution environments in place, in batches. The function runs at reduced capacity until the deployment finishes and throttles the invocations it cannot serve. Lambda also pauses scaling of the capacity provider during the deployment. If you publish a numbered version instead, it requires additional capacity to activate, because in-place mode is not available when publishing numbered versions. For production workloads, use the EC2 On-Demand Capacity Reservations (ODCRs) and size the reservation for the function's capacity plus one batch of execution environments. For more information, see Deploying and updating Lambda Managed Instances functions.

  • Use numbered versions and a weighted alias for long-running invocations. – If your function's invocation timeout is longer than 15 minutes, use numbered versions with a weighted alias instead of $LATEST.PUBLISHED. A rolling $LATEST.PUBLISHED deployment reclaims capacity by replacing execution environments, but it does not terminate an in-progress invocation. For long-running functions, the deployment can fail to complete before it reaches the 2-hour deployment limit, at which point the update fails and Lambda rolls back to the last successful version. If the deployment runs in place, long invocations also extend the time the function runs at reduced capacity. For these workloads, publish numbered versions and shift traffic with a weighted alias. AWS CodeDeploy supports alias-shift deployments of up to 50 hours, or you can shift the weights yourself incrementally. For more information, see Deploying and updating Lambda Managed Instances functions.

Concurrency and GPU state

  • Start with the default concurrency and tune it with GPU metrics. – The default PerExecutionEnvironmentMaxConcurrency for GPU-accelerated functions is 4. Publish the version, send it representative traffic, and adjust the setting based on ExecutionEnvironmentGPUMemoryUsed:

    • Lower the concurrency if GPU memory use approaches the accelerator memory you requested, or invocations fail while allocating GPU memory

    • Raise the concurrency if GPU memory use stays well below the accelerator memory and ExecutionEnvironmentGPUUtilization is low under load

    When you raise the concurrency, raise MemorySize with it. Every worker process initializes your handler module before the version becomes Active, so memory use during initialization grows with the concurrency, and a function that activates at a low concurrency can fail with FunctionError.InitResourceExhausted at a higher one even when each invocation is small. For more information about concurrency, see Concurrency.

  • Initialize the model once per worker. – Each worker imports your handler and initializes independently, so load your weights (from the image or Amazon S3) and move the model to the GPU in initialization code; each worker reuses it across its invocations

  • Serve one copy of a large model from a separate process. – If one copy of your model occupies most of the accelerator memory, do not let every worker load its own copy. Either set PerExecutionEnvironmentMaxConcurrency to 1, or start an inference server so that all workers send requests to the same copy over local HTTP, which is how the vLLM and SGLang base images work. For more information about serving one copy of a large model, see Serve one copy of a large model.

  • Choose the concurrency for media workloads by batch time. – Each concurrent transcoding or rendering invocation runs its own encoder, decoder, or graphics session, and the hardware engines process sessions from queues, so concurrency overlaps the download, upload, and CPU work of some invocations with the GPU work of others rather than multiplying encoding throughput. Measure the total time for a batch of files rather than the duration of a single invocation when you choose the concurrency.

Container images and startup time

  • Keep model weights and other large artifacts out of the image. – Package your code, dependencies, and framework in the image, and load model weights, rendering assets, and reference data from Amazon S3 in the same Region in initialization code. The PyTorch image includes the Amazon S3 Connector for PyTorch and the vLLM and SGLang images include the Run:ai Model Streamer, both of which stream weights into memory without writing to /tmp, so the image pull stays short and you can change artifacts without rebuilding the image. The capacity provider's VPC needs a route to Amazon S3; for more information, see Networking for Lambda Managed Instances.

  • Do not download models from public external repositories in production. – Setting MODEL_ID to a model ID on a public model hub such as the Hugging Face Hub is the quickest way to try an image, but the download crosses the internet, depends on the repository's speed and your NAT gateway, and repeats on every new execution environment, which slows scale-out. Copy the model to an S3 bucket in the function's Region once and load it from there.

  • Spread image content across layers and keep single files under 10 GB. – Lambda downloads layers concurrently and unpacks each layer as soon as it arrives, so an image with one very large layer starts more slowly than the same content split across several layers, and you should not flatten the image into a single layer. Order the layers by how often they change: put weights and other large, stable content in earlier layers and your handler code in the last layer, so that a rebuild reuses the cached layers and a deployment that changes only your code pulls only that layer. A single file in the image can be up to 10 GB, and an image with a larger file fails during activation, so split large checkpoints into shards or load them from Amazon S3.

  • Use SOCI for large images, and test with and without the index. – With a Seekable OCI (SOCI) index, Lambda starts the execution environment before the whole image has been downloaded and fetches files when they are first read. This shortens the time to first invocation for images of several gigabytes, but invocations that read image content your initialization code did not touch take longer than with a fully downloaded image, so import the libraries your handler uses in initialization code and compare startup time and invocation duration with and without the index. For more information, see Lazy-load container images with Seekable OCI (SOCI).

  • Pin the base image tag. – The GPU base images are published at public.ecr.aws/lambda/deep-learning-containers/:, and the cuda tag moves with every release. Pin cuda-v1 to follow a major version, or an immutable tag such as cuda-v1.0.0 for reproducible builds, so that a base image release does not change your function's behavior until you choose to rebuild. For the available images and tags, see GPU base images.

  • Match your libraries to the GPU driver when you build from your own base image. – Lambda installs and maintains NVIDIA driver 580 or later on the instances, and the libraries in the Lambda GPU base images are compatible with it. If you build from your own base image, do not install a driver. Instead, use a CUDA 13.x or earlier runtime and framework builds that match it. To use a newer CUDA version, bundle the matching cuda-compat package. The image must also include a runtime interface client that supports multi-concurrent mode. For more information about building your own base image, see Build your own base image.

Scaling configuration

  • Set a GPU utilization target that matches your workload. – By default, Lambda maintains a GPU utilization target that provides headroom for your traffic to double within five minutes without throttles. Override it with a TargetTrackingScalingPolicy on the capacity provider based on your workload characteristics:

    • For bursty, latency-sensitive workloads, choose a low target to maintain additional headroom

    • For steady workloads that tolerate throttles, choose a high target to achieve higher utilization and lower costs

    For more information about how GPU scaling works, see How GPU scaling works.

  • Plan for slower scale-out than CPU functions. – A new GPU execution environment on a new instance waits for the instance launch, the image pull, and your initialization code, which for a model of a few billion parameters includes loading the weights. Shorten initialization as described in Optimizing startup time for GPU-accelerated functions, and for planned traffic increases pre-warm capacity as described in Scaling Lambda Managed Instances.

  • Monitor GPU capacity metrics to understand headroom. – GPUAllocated shows how many GPUs in the capacity provider are serving execution environments, and GPUAvailable shows how many are idle and can take a new execution environment without launching an instance. When GPUAvailable stays at zero under load, new execution environments wait for instance launches.

Cost optimization

  • Clean up numbered versions you no longer use. – Every function version in the Active state keeps execution environments running on GPU instances and bills whether or not it receives traffic. If you deploy by republishing $LATEST.PUBLISHED (the default rolling-update path), Lambda reclaims the old version's execution environments as it rolls out the new one, so versions don't pile up and there's nothing to clean up. If you use numbered versions with a weighted alias or AWS CodeDeploy for explicit control over traffic shifting or per-version metrics, delete superseded versions once you cut over, or deactivate one you want to keep for rollback by setting MinExecutionEnvironments and MaxExecutionEnvironments to 0 with put-function-scaling-config.

  • Use the smallest GPU that fits. – A lower AcceleratorMemorySize lets Lambda place the function on smaller and cheaper instance types, including the G6f and Gr6f fractions, when your workload fits in 3, 6, or 12 GB. For instance pricing in your Region, see AWS Lambda pricing.

Monitoring and observability

  • Use the Duration metric for encoder, decoder, and graphics workloads. – GPUUtilization and ExecutionEnvironmentGPUUtilization measure the GPU's compute cores, and the video encoder and decoder and the graphics pipeline are separate engines, so a transcoding or rendering function keeps these metrics low while fully using the GPU. The function's Duration metric is the one that reflects throughput for these workloads.

  • Set up CloudWatch alarms for GPU metrics. – In addition to the alarms recommended for all Event Functions on Lambda Managed Instances, create Amazon CloudWatch alarms so that you learn about GPU memory pressure and GPU capacity shortfalls before they cause failed invocations or throttles:

    • ExecutionEnvironmentGPUMemoryUsed approaching the accelerator memory you requested

    • GPUAvailable staying at zero under load

    For more information about monitoring GPU-accelerated functions, see Monitoring GPU-accelerated functions.