Amazon Web Services announced Amazon SageMaker HyperPod Inference Gateway on September 18, 2026, bringing GPU-aware routing to large language model inference. The gateway targets a costly problem: requests pile up behind busy pods while idle capacity remains unused, creating first-token latency spikes above four seconds during traffic bursts.
AWS says the gateway can reduce first-token latency by up to 82%. The announcement includes figures of 4.4 seconds and 800 milliseconds, putting the focus on the time between an inference request and the first returned token.
A Two-Tier Gateway Built for Model Inference
Amazon SageMaker HyperPod Inference Gateway uses a two-tier design built on Kubernetes-native primitives. Tier 1 contains three components built on the open-source Gateway API Inference Extension, giving the gateway separate functions for receiving traffic, identifying the model request, and selecting an inference destination.
The first component, Envoy Gateway, terminates incoming HTTPS traffic and exposes a single private endpoint per cluster. That endpoint gives clients one place to send inference requests instead of requiring them to choose among individual model pools or pods.
The Body-Based Router handles the next decision. It inspects each incoming request body and routes the request to the correct model pool, connecting the content of the request with the model pool that should process it.
The Endpoint Picker then evaluates available endpoints using real-time Prometheus metrics. It applies a weighted scoring algorithm, and each scorer carries a configurable weight that can tune routing behavior.
That design gives routing a live view of the inference environment. Busy capacity can influence endpoint selection while idle capacity remains available for incoming requests, addressing the imbalance that causes requests to wait behind busy pods.
Deployment Without Application Changes
Deploying the gateway requires a single aws eks create-addon command and one declarative InferenceGatewayConfig custom resource. The installation requires no sidecars, no service mesh, and no application code changes.
AWS also exposes a standard OpenAI-compatible endpoint over HTTP. Existing client code works unchanged, with no SDK changes and no SigV4 signing for inference traffic.
That compatibility keeps the gateway separate from the client experience. Applications can continue using existing client code while the gateway manages the routing path behind the endpoint.
The Kubernetes-native setup also places the configuration inside the cluster environment. A single command installs the add-on, while the declarative custom resource defines the gateway configuration.
Global Routing Is Still Ahead
Tier 1 handles routing within the cluster through Envoy Gateway, the Body-Based Router, and the Endpoint Picker. Tier 2, the Global Inference Router, is listed as coming soon.
The two-tier structure points toward a broader routing model. The current gateway focuses on cluster-level traffic and endpoint selection, while the future Global Inference Router represents the next listed tier in the design.
For teams running large language model inference on Kubernetes, the immediate promise is direct: reduce first-token latency without adding sidecars, introducing a service mesh, changing application code, or updating SDKs. The gateway combines a private HTTPS endpoint, request-body routing, Prometheus metrics, and weighted scoring into one deployment path.
The result is an inference gateway built around a simple question: which model pool and endpoint should handle this request now? With latency reaching above four seconds during traffic bursts and idle capacity left unused behind busy pods, that question can shape the entire user experience.
Amazon Web Services has positioned SageMaker HyperPod Inference Gateway as a routing layer for this challenge, with up to 82% lower first-token latency as the headline target. Tier 2 remains ahead, but Tier 1 already lays out the core path from incoming HTTP traffic to GPU-aware model routing.
