vLLM made single-node inference fast. It didn't solve what happens once you're running it in production at real traffic — a model too big for one GPU, requests queueing behind whatever replica they land on, and cache work getting redone instead of reused.
llm-d is the open-source answer — an orchestration layer that turns a fleet of GPUs into one coordinated serving system, not a pile of independent ones.
None of these are bugs in vLLM — they're outside what a model server is meant to own. Someone still has to solve them.
Frontier-scale models don't fit in one accelerator's memory. Serving them at all means splitting the model across a cluster — something vLLM alone doesn't orchestrate for you.
Without cluster-aware routing, a request can land on a busy replica while an idle one sits a rack away — and the KV-cache that would make it instant sits on a different GPU entirely.
Long conversations and shared prefixes recompute context a smarter router would have kept warm — burning GPU-seconds on work that's already been done once.
llm-d is a CNCF Sandbox project, accepted March 2026, founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It runs on standard Kubernetes and speaks the same APIs as vLLM and SGLang — nothing about your model server changes. What changes is everything around it.
source: github.com/llm-d/llm-d
Every request is routed to whichever replica is free, or whichever one already has the matching prefix cached, instead of round-robin luck.
Warm context offloads to CPU and disk behind a global index instead of getting evicted, so long conversations and shared prefixes don't get recomputed.
For the largest models, prefill and decode run on separate pools with wide expert-parallelism, instead of one GPU doing both jobs serially.
None of this changes what a GPU-hour costs. It changes how many GPU-hours the same traffic needs. A fleet that's routing intelligently and reusing cache — instead of sitting idle or recomputing work — needs fewer replicas to hit the same latency target for the same number of users.
That's the entire economic argument for running an orchestration layer at all, and it's the reason five separate infrastructure vendors are backing the same one instead of five competing ones.
No. llm-d is an orchestration layer that runs on top of vLLM (or SGLang) — you still deploy vLLM as the model server; llm-d coordinates many instances of it as one fleet.
Yes. It's fully open source and joined the Cloud Native Computing Foundation as a Sandbox project in March 2026.
Yes — llm-d is built to run on Kubernetes and assumes that as the deployment substrate, alongside standard accelerator APIs.
Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA founded the project together.
llm-d doesn't publish one universal percentage, and neither do we. The saving comes from needing fewer replicas to hit the same latency and throughput target — which depends on your traffic pattern, model size, and prompt shapes. The architectural case (routing, cache reuse, disaggregation) is documented on the project's GitHub; benchmarking it against your own workload is how you get a real number.
The GitHub repo has the quickstart and full docs. This page is the plain-English map of why each piece exists before you go read them.
Building AI infrastructure at your company? Eight Labs consults for enterprise, SMEs and startups. Details at theaihow.com.