vLLM publicly releases its language-model serving engine

The UC Berkeley project released an open-source inference engine using PagedAttention to reduce wasted attention-cache memory and batch more requests. Its authors reported throughput gains on LLaMA models; these were their own tests, not a universal speedup.