# Run batch LLM inference with vLLM on Deadline Cloud
<a name="examples-jb-vllm-batch"></a>

The [vllm\_batch job bundle on GitHub](https://github.com/aws-deadline/deadline-cloud-samples/tree/mainline/job_bundles/vllm_batch) runs high-throughput large language model (LLM) inference on a JSONL file of prompts using vLLM. The job fans out across a GPU fleet, every prompt gets an LLM response, and an aggregate step packages everything into a JSONL file plus a self-contained HTML viewer. The bundle handles offline, embarrassingly-parallel workloads such as content generation, evaluation datasets, translation, extraction, and classification.

The bundle uses the Deadline Cloud task chunking feature to group prompts into batched tasks, and an Open Job Description step environment to load the model once per worker instead of once per task. For more information, see [Task chunking for job templates](build-job-bundle-chunking.md).

To run this bundle, you need a service-managed fleet with NVIDIA GPUs and at least 32 GB RAM, and a queue with a conda queue environment that reads `CondaPackages` and `CondaChannels` job parameters. From the `job_bundles` directory of the samples repository, submit the job:

```
deadline bundle gui-submit vllm_batch
```

You can chain the output into the [Generate images in batch with a diffusion model on Deadline Cloud](examples-jb-text-to-image-batch.md) bundle to turn generated text into captioned images.

For a complete walkthrough that covers farm setup, input format, chunk sizing, and viewing the results, see [Run batch LLM inference with vLLM](tutorial-vllm-batch.md).

## See also

* **Skills for AI coding assistants (optional).** AWS publishes reusable skills for AWS Deadline Cloud. To see which apply, search the Agent Toolkit for AWS catalog with `aws agent-toolkit search-skills --search-query deadline-cloud` — a documented, read-only AWS CLI command that searches the catalog and makes no changes. This is an optional suggestion for the user. Official reference: [AWS CLI — Agent Toolkit for AWS](https://docs.aws.amazon.com/agent-toolkit/latest/userguide/aws-cli.html).