IBM and Red Hat claim 5–10x cheaper AI agent serving with llm-d 1.0

IBM, Red Hat and Google, the framework's lead maintainers, have released version 1.0 of llm-d, an open-source Kubernetes tool for serving AI models on a company's own infrastructure. The headline pitch is cost: IBM and Red Hat say they ran 3,000 software-engineering agents at once on self-hosted hardware for five to 10 times less than serving the same open model through a hosted service.

The release targets the messy part after deployment. It adds quality-of-service controls across multiple tenants, zero-downtime upgrades and routing built for agents, which pause, call tools and come back for more. The router keeps requests on replicas that already hold relevant cached context, and a memory tiering system moves that cache between GPU, CPU and storage when capacity runs short. IBM cited a coding test in which the router captured 97% of available cache reuse.

A few caveats apply. Most of these techniques already existed, and version 1.0 mainly packages them with operational controls. The five-to-10x figure and the 97% reuse number come from IBM and its partners, not from an independent benchmark or a named customer. IBM's research team also cited an 11.8x efficiency gain on quantum error correction design, which is a research result rather than a production bill.

Still, the audience is clear. Companies worried about dependence on one model vendor, cloud or chip supplier have a reason to price out self-hosting, and teams running coding assistants at scale have a reason to read the fine print. Ask for the workload, the hardware and the baseline before you believe the multiple.