Google Cloud releases llm-d 1.0 open-source inference stack for LLM serving fleets
85% lower median TTFT
Vendor-reported: median TTFT down over 85% and 2.4x throughput on an Ironwood TPU coding workload
Google Cloud and partners released llm-d 1.0, a stable Kubernetes-native layer that coordinates vLLM or SGLang model servers into one pool. A vendor-reported Ironwood TPU coding benchmark showed median time-to-first-token cut by more than 85% and 2.4x throughput.
Read the full story at MarkTechPost