vLLM

The community behind a serving engine

The vLLM contributor community develops an open-source inference engine that runs model weights, batches requests and exposes services that applications can call. The PyTorch Foundation’s project record describes both online serving through an OpenAI-compatible interface and offline batch inference, across several hardware families. [1]

Making room for more requests

Developed at UC Berkeley, vLLM was publicly introduced on June 20, 2023. Its initial contribution was PagedAttention, which divides the attention key-value cache into blocks that need not sit next to each other in GPU memory. Allocating those blocks as requests grow reduces wasted memory and makes room to batch more requests. [2]

This addresses the cost of serving a model after training: the same hardware can handle more concurrent work if less memory is reserved and left unused. The authors’ September 2023 paper reported two to four times the throughput of its comparison systems at similar latency. Those results describe the paper’s workloads and baselines, rather than a fixed speedup for every deployment. The paper date is separate from the earlier public release. [3]

A shared infrastructure project

In July 2024, the maintainers described contributions from universities, cloud providers and software companies and announced the start of incubation into LF AI & Data. The foundation’s current project page records Berkeley’s contribution to the Linux Foundation that month. That contributor base connects model developers, hardware vendors and application operators around a shared serving engine. [4][1]