SageMaker AI Routes Requests Using Container-Reported GPU Cache State

Amazon Web Services introduced KV cache-aware routing for Amazon SageMaker AI real-time endpoints on November 17, 2026, using container-reported cache state to place requests. AWS reports up to 97% lower median time to first token and 234% higher throughput than random routing in long-prompt tests under load. The KV cache holds prior prompt computations in GPU memory, so a model can reuse them rather than repeat the initial processing, or prefill, of matching input tokens. That makes cache…

This article has been indexed from Unite.AI