View a markdown version of this page

GPU base images for Lambda Managed Instances - AWS Lambda

GPU base images for Lambda Managed Instances

Lambda provides container base images that you can use to build GPU-accelerated functions on Lambda Managed Instances. Each image includes the CUDA runtime libraries, the framework the image is named for (such as PyTorch, vLLM, or SGLang) with its dependencies, and the Lambda runtime interface client.

Note

Lambda's GPU base images maintain compatibility with the NVIDIA GPU drivers on the instances. We recommend building your GPU-accelerated functions from these images unless your workload uses a language or framework that they do not provide.

Available images

Lambda publishes the GPU base images in the Amazon ECR Public Gallery at public.ecr.aws/lambda/deep-learning-containers/<framework>:<tag>.

Image Framework : Tag What's included
Base base:cuda CUDA toolkit
CuPy cupy:cuda CuPy, CUDA toolkit
PyTorch pytorch:cuda PyTorch, cuDNN, torchvision, FFmpeg, s3torchconnector, CUDA toolkit
vLLM vllm:cuda vLLM inference server, PyTorch, Transformers, s3torchconnector, CUDA toolkit
SGLang sglang:cuda SGLang inference server, PyTorch, Transformers, s3torchconnector, CUDA toolkit

Base

The base:cuda image is a minimal image that includes the CUDA toolkit and the Lambda Python runtime. Use this image when you use custom CUDA kernels or frameworks that Lambda's other base images don't include (such as GROMACS or Blender). The following Dockerfile shows how to add your own packages to base:cuda.

FROM public.ecr.aws/lambda/deep-learning-containers/base:cuda RUN dnf install -y gcc gcc-c++ cmake make && dnf clean all RUN pip install --no-cache-dir your-custom-package WORKDIR /var/task COPY handler.py . CMD ["handler.handler"]

CuPy

The cupy:cuda image is meant for GPU-accelerated numerical computing with a NumPy-compatible interface. Use the CuPy image for workloads that benefit from GPU-accelerated array operations and don't need a full ML framework. Common use cases include scientific computing, signal processing, and financial modeling. The following Dockerfile shows how you can add a custom handler to this base image.

FROM public.ecr.aws/lambda/deep-learning-containers/cupy:cuda WORKDIR /var/task COPY handler.py . CMD ["handler.handler"]

The following handler code uses CuPy to perform Monte Carlo option pricing.

import cupy as cp import math def handler(event, context): # Option parameters from the request spot = event.get("spot_price", 100.0) strike = event.get("strike_price", 105.0) rate = event.get("risk_free_rate", 0.05) volatility = event.get("volatility", 0.2) expiry = event.get("years_to_expiry", 1.0) n_simulations = event.get("simulations", 10_000_000) # Monte Carlo simulation on GPU z = cp.random.standard_normal(n_simulations, dtype=cp.float32) price_at_expiry = spot * cp.exp( (rate - 0.5 * volatility ** 2) * expiry + volatility * math.sqrt(expiry) * z ) # European call option payoff payoffs = cp.maximum(price_at_expiry - strike, 0) call_price = float(cp.exp(-rate * expiry) * cp.mean(payoffs)) # European put option payoff put_payoffs = cp.maximum(strike - price_at_expiry, 0) put_price = float(cp.exp(-rate * expiry) * cp.mean(put_payoffs)) return { "call_price": round(call_price, 4), "put_price": round(put_price, 4), "simulations": n_simulations, "spot": spot, "strike": strike, }

PyTorch

The pytorch:cuda image can be used for general-purpose ML inference with PyTorch and CUDA, such as image classification, object detection, speech synthesis, and diffusion models, or for LLM inference where you need full control over the model pipeline. The following example performs image classification using mobilenet. The example Dockerfile installs the model weights at build time. Building the weights into the image is fine for small models, up to about 1 GB like the one in this example. For larger models, stream the weights from Amazon S3 at initialization instead of building them into the image, which keeps the image small and shortens startup. For more information about optimizing startup time, see Optimizing startup time for GPU-accelerated functions.

FROM public.ecr.aws/lambda/deep-learning-containers/pytorch:cuda # Download weights at build time so the function has no runtime internet dependency RUN python3 -c "from torchvision import models; models.mobilenet_v2(weights=models.MobileNet_V2_Weights.DEFAULT)" \ && cp /tmp/torch/hub/checkpoints/mobilenet_v2-7ebf99e0.pth /var/task/mobilenet_v2.pth RUN curl -s https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt -o /var/task/imagenet_classes.txt WORKDIR /var/task COPY handler.py . CMD ["handler.handler"]

The sample handler code is shown below:

import torch import io import base64 import os from torchvision import models, transforms from PIL import Image model = models.mobilenet_v2() model.load_state_dict(torch.load("/var/task/mobilenet_v2.pth", map_location="cpu", weights_only=True)) model.eval() preprocess = models.MobileNet_V2_Weights.DEFAULT.transforms() model.cuda() with open("/var/task/imagenet_classes.txt") as f: labels = f.read().strip().split("\n") def handler(event, context): img_bytes = base64.b64decode(event["image_base64"]) img = Image.open(io.BytesIO(img_bytes)).convert("RGB") tensor = preprocess(img).unsqueeze(0).cuda() with torch.inference_mode(): output = model(tensor) probs = torch.softmax(output, dim=1)[0] top5 = probs.topk(5) return { "predictions": [ {"label": labels[idx], "confidence": round(prob.item(), 4)} for prob, idx in zip(top5.values, top5.indices) ] }

vLLM

Integration with pre-trained models

The vllm:cuda image is meant for serving large language models behind an OpenAI-compatible API. Use it for chat, completion, and embedding workloads where you want the server to handle batching and GPU memory management for you. The image includes the vLLM framework, which you can use to serve models directly without writing code to load the model, batch requests, or manage GPU memory. Configure the MODEL_ID environment variable to point to your model on the local file system, in an Amazon S3 bucket, or in a Hugging Face or ModelScope repository. When retrieving models from Hugging Face, we recommend that you authenticate with the HF_TOKEN environment variable for higher download rate limits and to access private or gated models. By default, the image serves Qwen/Qwen2.5-0.5B-Instruct.

FROM public.ecr.aws/lambda/deep-learning-containers/vllm:cuda

To use a different model, update the MODEL_ID environment variable on the function and publish a new function version. You don't need to rebuild the image.

aws lambda update-function-configuration \ --function-name my-gpu-function \ --environment 'Variables={MODEL_ID=Qwen/Qwen2.5-1.5B-Instruct,HF_TOKEN=<your-hugging-face-token>}'
Note

update-function-configuration replaces the entire environment-variable map, so include any variables you already set (for example VLLM_LOAD_FORMAT).

The image starts the vLLM server during initialization and checks its health while the execution environment runs. If the server exits or fails three consecutive health checks, Lambda replaces the execution environment.

The image automatically handles model downloading, GPU memory management, and serving. Invoke payloads follow an OpenAI-compatible format. For text models, supply an input like the following:

{ "prompt": "Explain quantum computing in one sentence.", "max_tokens": 256 }

Model Streaming and Ephemeral Storage

For production workloads, we recommend copying models to an S3 bucket in the same Region as your capacity provider and loading them from there. This avoids cross-Region data transfer costs and downloads faster than the Hugging Face Hub.

huggingface-cli download Qwen/Qwen2.5-1.5B-Instruct --local-dir ./model aws s3 sync ./model s3://amzn-s3-demo-bucket/models/Qwen2.5-1.5B-Instruct/

You can then refer to your S3 prefix in the MODEL_ID environment variable. By default, the vllm:cuda image downloads models to /tmp. We recommend that you stream models directly to memory using an additional environment variable.

aws lambda update-function-configuration \ --function-name my-gpu-function \ --environment 'Variables={MODEL_ID=s3://amzn-s3-demo-bucket/models/Qwen2.5-1.5B-Instruct/,VLLM_LOAD_FORMAT=runai_streamer}'

Alternatively, you can increase your EphemeralStorage storage setting.

Custom handlers

To use a pretrained model with custom business logic, you can provide your own handler code. Your handler code should load the model into memory once while creating multiple Python processes to serve multi-concurrent requests. For more information about customizing request handling, see Customize request handling with your own handler.

SGLang

The sglang:cuda image, like the vllm:cuda image, includes a built-in model serving framework that requires no code. It provides structured generation capabilities and efficient prefix caching via RadixAttention. Use it for LLM workloads that benefit from structured output (JSON mode, regex-constrained generation), multi-turn conversations with prefix sharing, or fine-grained generation control.

FROM public.ecr.aws/lambda/deep-learning-containers/sglang:cuda

Like the vLLM image, configure it with environment variables.

aws lambda update-function-configuration \ --function-name my-gpu-function \ --environment 'Variables={MODEL_ID=Qwen/Qwen2.5-1.5B-Instruct,HF_TOKEN=<your-hugging-face-token>}'
Note

update-function-configuration replaces the entire environment-variable map, so include any variables you already set.

When overriding the default handler, the same handler pattern applies to the sglang:cuda image. The image starts the SGLang server during initialization; your handler forwards requests to http://127.0.0.1:8000, the same pattern as the vLLM custom handler. Use the SGLANG_* environment variables (SGLANG_MEM_FRACTION, SGLANG_ATTENTION_BACKEND, SGLANG_SERVER_ARGS) in place of the VLLM_* ones.

Build your own base image

A GPU-accelerated function can run a container image built from any base image, in any language, as long as the image meets the following requirements:

  • Runtime interface client – The image must include a Lambda runtime interface client that supports multi-concurrent execution on Lambda Managed Instances. The image's ENTRYPOINT and CMD must start it with your handler. For the supported clients and how to add one to an image, see Lambda Managed Instances runtimes and Create a Lambda Event Function using a container image.

  • CUDA runtime libraries – Install a CUDA 13.x or earlier runtime and framework versions that match it. To use a newer CUDA version, bundle the matching cuda-compat package in the image. You do not need to install an NVIDIA driver within your base image as Lambda installs and maintains these on the GPU instances in your capacity provider. For more information about CUDA compatibility, see CUDA Compatibility in the NVIDIA documentation.

  • Writable paths – /tmp is the only writable directory in the execution environment. Point any cache your framework writes at runtime, such as a kernel cache, to a path under /tmp.