When DeepSeek open-sourced its production inference engine, the repository named three components a complete serving stack needs and does not ship: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Those three pieces separate a fast engine from a running service. On October 20, at the PyTorch Conference, Google, Red Hat, and IBM shipped them — free, vendor-neutral, and already running in production. The project is llm-d 1.0, a CNCF Sandbox stack built by more than 500 contributors across 25-plus organizations, founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA.
What vendor-neutral actually buys is simple: you keep your engine, model, and cluster. llm-d orchestrates above model servers like vLLM and SGLang rather than replacing them. The project’s own framing is blunt: “Serving a frontier model fast is no longer the hard part. Running it reliably, for many tenants, through upgrades, under real load, is.” The 1.0 release delivers multi-tenant flow control, zero-downtime upgrades via DisaggregatedSet, session-aware routing for agents, and a stable v1 API — portable inference plumbing that does not care whose GPUs sit underneath.
The production roster is the evidence. Vendor-reported figures show Google Cloud’s GKE Inference Gateway running on the llm-d router, with the Gemini platform doubling its prefix cache hit rate from 35 to 70 percent and cutting time-to-first-token by more than 35 percent for coding agents. AWS reports up to 70 percent more tokens per second on SageMaker HyperPod. DigitalOcean serves the trillion-parameter Kimi K3 model across AMD and NVIDIA hardware. Tesla and Red Hat report 3x output throughput on Llama 3.1 70B. Mistral runs it behind its regional inference offering. JetBrains uses it for Junie. And Red Hat ships llm-d as the distributed inference stack in Red Hat AI. The IBM newsroom release lists the founders and adopters spanning clouds, accelerator vendors, and enterprises. At Snap, engineering lead Vinay Kola reports “prefix cache hit rates ranging up to 75-80%” through llm-d’s prefix-cache-aware routing.
The agentic workload is where the architecture earns its keep. Mistral’s inference engineers describe the problem plainly: “Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers.” llm-d 1.0 addresses this with session-aware routing, turn-priority scheduling, and native Responses API state so agents reuse context across turns. The TorchTPU integration shipped the same day as Google and Meta’s open-source release, with co-operative time-slicing that the project reports raising accelerator duty cycles from a 40 percent baseline to above 70 percent during reinforcement learning rollouts — vendor-reported, no independent verification yet.
IBM’s cost results deserve careful reading because the two figures are not the same claim. In one, IBM Research used open frontier models on llm-d to explore quantum error correction hardware at 11.5 times lower inference cost than an all-proprietary configuration, with roughly 98 percent prefix-cache hit rate and a blended cost of about $0.11 per million tokens. In a separate demonstration, IBM deployed GLM-5.2 — an approximately 753-billion-parameter open-weight model — on NVIDIA GPUs and reports over 6.6 million output tokens per minute at peak, serving up to 3,000 concurrent coding agents with zero preemptions, at costs that “would have been 5 to 10 times less per token than equivalent commercial API pricing, with the largest savings on the input-heavy traffic patterns typical of agentic workloads.” Both results are vendor-reported. The 5-10x figure originates in an IBM Research blog post titled “How llm-d makes the most of the hardware you already have,” which returned a 403 CloudFront block at fetch and is named here without a link.
The value capture is structural. The sponsors and early adopters of llm-d sell the managed tiers that sit directly above the free layer: GKE Inference Gateway on Google Cloud, Red Hat AI on OpenShift, SageMaker HyperPod on AWS, DigitalOcean Serverless Inference. Standardizing the serving layer is also distributing it. As Red Hat’s Brian Stevens puts it: “the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads.” The disaggregated, session-aware serving that agents require is exactly what these managed tiers sell. This is alignment of interests, not coordination.
What that means for builders: the cost of serving open-weight models just dropped, and the money is moving to the managed layer above. The four labs that gated commercial deployment at $20 million in revenue are now competing against a free serving stack that makes self-hosting the open models they cannot gate. The engine costs nothing. The plumbing now costs nothing. What remains priced is the complexity of running agents in production — and that complexity is where the clouds, the platform vendors, and the consultancies intend to collect.
