Scale back LLM latency with prefix-aware routing on Amazon SageMaker Inference

Whenever you construct an software on high of a big language mannequin (LLM), the immediate you ship to the mannequin sometimes has two elements. There’s a hard and fast half that units up context (directions, reference paperwork, dialog historical past) and a variable half that comprises the precise consumer enter. Take a customer support bot for instance. Every request begins with the identical block of textual content: “You’re a assist agent for AnyCompany. Listed here are our insurance policies…” adopted by regardless of the buyer typed. The directions on the high may be 3,000 tokens. The shopper’s query on the backside may be 50 tokens.

Which means throughout lots of or hundreds of requests, your mannequin is processing that very same 3,000-token starting over and over.

LLM serving frameworks like vLLM and TensorRT-LLM have an answer for this. They cache the computed key-value (KV) pairs for immediate prefixes which were seen earlier than. When the identical starting exhibits up in a brand new request, the mannequin reuses the cached computation and solely processes the brand new tokens on the finish. That is referred to as prefix caching, and it could scale back time-to-first-token (TTFT) considerably.

However there’s an issue while you scale past a single occasion. When you have a fleet of machines behind an endpoint, requests get distributed throughout all of them. That very same 3,000-token prefix lands on occasion A for one request, occasion B for the subsequent, occasion C after that. Every occasion computes it from scratch as a result of none of them see it often sufficient to construct a dependable cache. The prefix caching characteristic is there, however the routing layer spreads requests too thinly for it to assist.

Right now, Amazon SageMaker Inference introduces prefix-aware routing. It’s a new routing technique that appears at first of every request and constantly sends requests with the identical starting to the identical occasion. The KV cache on that occasion really builds up and will get reused. In our benchmarks on Llama 3.1 70B, this lowered P50 TTFT by as much as 77 p.c and elevated throughput by as much as 16 p.c. It additionally pushed KV cache hit charges from roughly 25 p.c to over 80 p.c.

What prefix-aware routing does

When a request arrives at your endpoint, Amazon SageMaker seems to be at first of the payload and makes use of it to determine which occasion ought to deal with it. The identical starting goes to the identical occasion. Completely different beginnings unfold throughout totally different cases. If 10 requests share a prefix or starting, all 10 go to the identical machine, and that machine’s cache stays heat for that prefix.

You don’t must tag requests or handle affinity your self. The endpoint handles it primarily based on the content material of the request.

There are two built-in safeguards:

Overload safety. If one prefix is extraordinarily common and the goal occasion is already at capability, the endpoint routes the request to a much less busy occasion as an alternative. You configure the concurrency restrict, and the endpoint respects it. You would possibly miss a cache hit on that one request, however you keep away from overwhelming a single machine.

Secure habits throughout scaling. Whenever you add or take away cases, most requests proceed going to the identical occasion they have been going to earlier than. Solely a small fraction of visitors shifts to account for the modified fleet. Your caches don’t get invalidated each time you scale.

Efficiency benchmarks

We benchmarked prefix-aware routing in opposition to the default random routing baseline utilizing Llama 3.1 70B Instruct on 7 ml.p5.48xlarge cases with vLLM (prefix caching enabled). We ran 16 check configurations masking single mannequin endpoints, inference element endpoints, the native Invoke API, and the OpenAI-compatible API. All checks accomplished with one hundred pc success price.

Lengthy context workloads

8,000-token shared prefixes, sustained over 1 hour:

P90 TTFT: lowered by 33–37 p.c.
P50 TTFT: lowered by 71–77 p.c.
KV cache hit price: from roughly 25–82 p.c.
Throughput: elevated 15–16 p.c.

Brief context workloads

Variable-length ShareGPT-style conversations, half-hour:

P90 TTFT: lowered by 24–37 p.c.
P50 TTFT: lowered by 13–16 p.c.
KV cache hit price: from roughly 30–80 p.c.
Throughput: elevated 1.7–2.0 p.c.

The longer your shared prefix, the larger the win. Lengthy context workloads profit probably the most as a result of there’s extra computation to skip on every cache hit. Brief context workloads nonetheless profit, however the shared prefixes are smaller so the financial savings per request are proportionally smaller.

Routing overhead

The prefix-aware routing logic provides 1.3–1.9 milliseconds per request. Mannequin TTFT in these checks ranged 63–280 milliseconds. The routing value is negligible.

Site visitors distribution remained balanced throughout all eventualities. Every of the 7 cases obtained 13.3–15.4 p.c of requests, inside 1 p.c of a really perfect even cut up. No scorching spots.

Routing methods on SageMaker Inference

With this launch, Amazon SageMaker Inference gives three routing methods for real-time endpoints:

RANDOM (default): Distributes requests uniformly throughout cases. Really useful for general-purpose workloads, non-LLM fashions, or a state of affairs the place requests are interchangeable and there’s no profit to sending particular requests to particular cases.

LEAST_OUTSTANDING_REQUESTS: Sends every request to the occasion with the fewest in-flight requests. Really useful when request processing occasions differ and also you wish to preserve all cases equally busy. Helps forestall sluggish requests from piling up on one machine whereas others sit idle.

PREFIX_AWARE (new): Sends requests sharing the identical immediate prefix to the identical occasion. Really useful for LLM workloads the place many requests share frequent textual content at first and your serving framework has prefix caching enabled.

You set the technique per manufacturing variant in your endpoint configuration. You’ll be able to change between them by updating the endpoint configuration with out redeploying your mannequin.

When to make use of prefix-aware routing

The characteristic delivers worth when your requests share textual content at first. Listed here are the patterns the place it has probably the most affect:

Retrieval Augmented Technology (RAG) purposes. You retrieve a doc and prepend it earlier than the consumer’s query. When a number of customers ask questions on the identical doc, all of them share that doc as a prefix. Prefix-aware routing sends them to the identical occasion, the place the KV cache for that doc is already heat.

Multi-turn conversations. Every flip in a dialog consists of the complete historical past of earlier turns. Because the dialog grows, that shared historical past turns into an extended and dearer prefix. Routing on that prefix retains the dialog’s cache on one occasion throughout turns.

Templated bots and assistants. Bots with lengthy, structured directions (insurance policies, formatting guidelines, persona definitions) ship those self same directions with each request. Solely the consumer message on the finish modifications. Prefix-aware routing implies that costly instruction block will get processed as soon as, not hundreds of occasions.

Code completion. Coding assistants embody file contents as context. Whereas a developer works in the identical file, each completion request shares that file content material as a prefix.

The way to allow it

You configure prefix-aware routing while you create your endpoint configuration. Two parameters management the habits:

PrefixLength (1024–65536): How a lot of the request to make use of for routing. For the native Amazon SageMaker Invoke API, that is bytes from the start of the request physique. For the OpenAI-compatible API, that is characters from the extracted message textual content. Set this to cowl your shared prefix plus sufficient distinctive content material to unfold totally different workloads throughout cases.

ConcurrencyThreshold (1–1024): The utmost in-flight requests on the goal occasion earlier than overflow kicks in. If the goal occasion is at this restrict, the request goes to a much less loaded occasion as an alternative.

Right here is an instance:

aws sagemaker create-endpoint-config 
    --endpoint-config-name example-llm-config 
    --production-variants '[{
        "VariantName": "AllTraffic",
        "ModelName": "example-llm-model",
        "InitialInstanceCount": 3,
        "InstanceType": "ml.p5.48xlarge",
        "RoutingConfig": {
            "RoutingStrategy": "PREFIX_AWARE",
            "PrefixAwareRoutingConfig": {
                "PrefixLength": 4096,
                "ConcurrencyThreshold": 10
            }
        }
    }]'

Then create your endpoint as common:

aws sagemaker create-endpoint 
    --endpoint-name example-llm-endpoint 
    --endpoint-config-name example-llm-config

No modifications to your mannequin container or serving framework are wanted. Prefix-aware routing operates solely on the endpoint routing layer.

Invoking the endpoint

Nothing modifications about the way you name the endpoint. The identical InvokeEndpoint and InvokeEndpointWithResponseStream APIs work precisely as earlier than:

aws sagemaker-runtime invoke-endpoint 
    --endpoint-name example-llm-endpoint 
    --content-type software/json 
    --body fileb://request.json 
    output.json

Identical for the OpenAI-compatible Chat Completion API:

from openai import OpenAI
from sagemaker.core.token_generator import generate_token

consumer = OpenAI(
    base_url=f"https://runtime.sagemaker.us-west-2.amazonaws.com"
             f"/endpoints/example-llm-endpoint/openai/v1",
    api_key=generate_token(area="us-west-2")
)
response = consumer.chat.completions.create(
    mannequin="example-model",
    messages=[
        {"role": "user", "content": "What is your return policy?"},
    ],
)

Multi-tenant prefix isolation

If totally different tenants share the identical immediate directions however you need them routed individually (to maintain cache contexts impartial), cross an non-obligatory ID:

Native Invoke API: set the X-Amzn-SageMaker-Prefix-Conscious-Id header (as much as 64 ASCII characters).

OpenAI API: embody the prompt_cache_key subject within the request physique.

This ID combines with the prefix in order that requests with equivalent prefixes however totally different IDs land on totally different cases.

Inference parts and LoRA adapters

Prefix-aware routing works with inference element endpoints and dynamic Low-Rank Adaptation (LoRA) adapters. For inference parts, it behaves the identical as single mannequin endpoints. For LoRA adapters, it operates inside the adapter’s sticky occasion set, utilizing prefix-based choice among the many cases that have already got the adapter loaded.

Sensible steering

Allow prefix caching in your serving framework. Prefix-aware routing will get repeated prefixes to the identical occasion, however your container wants prefix caching turned on to really retailer and reuse these KV pairs. In vLLM, that is enabled by default in latest variations. Different frameworks would possibly require express configuration.

Preserve request serialization constant. For the native Invoke API, PrefixLength operates on uncooked bytes. JSON whitespace, key ordering, and formatting all have an effect on routing. When you serialize the identical immediate otherwise throughout requests, they may find yourself on totally different cases. Use constant serialization.

Measurement PrefixLength fastidiously. Too brief and all requests with the identical brief prefix get funneled to at least one occasion, triggering overflow. Too lengthy and small payload variations (like temperature values) scatter requests that ought to keep collectively. Begin with the size of your shared prefix plus a modest buffer.

You want no less than two cases. With one occasion, all requests go to the identical place no matter technique.

Monitor cache hit charges. Allow SageMaker detailed observability to trace KV cache hit charges on the mannequin degree. This confirms whether or not prefix-aware routing is working in your particular workload.

Conclusion

Prefix-aware routing is obtainable at present on SageMaker real-time inference endpoints. Replace your AWS SDK or CLI to the newest model to entry the brand new RoutingStrategy and PrefixAwareRoutingConfig parameters. Check with this pocket book for examples of find out how to allow it throughout endpoint creation.