inside your individual firm and nearly any failure is reasonable: you retry, fall again, or doubtlessly even ignore it. Put that very same workflow behind a buyer’s API or MCP server and the grace is gone. Now just one factor issues: did the client get an accurate, usable consequence? Their course of depends upon yours delivering one. They, not you, now resolve what counts as delivered. At Databook we course of billions of tokens for the world’s largest enterprises; this text relies on actual knowledge from manufacturing flows at scale. I hope it presents you some helpful insights.
Delivering that result’s tougher than it appears to be like, as a result of LLMs are notoriously unreliable. They fail often, in 4 flavors: an invalid reply (empty, unparseable, or just fallacious), a tough error, no reply in any respect, or no reply in time. And the entire run solely succeeds if each step does, so the extra you chain collectively, the extra possibilities there are for one among them to fail. A workflow of individually wonderful steps can nonetheless come out a coin flip.
Inside your individual firm you’ll be able to take up each one among these, as a result of you’ve gotten slack on each axis: retry the failed step, wait out the sluggish one, spend slightly extra, loosen up the bar in the event you should. Put the identical workflow behind a buyer’s API and the slack vanishes, as a result of the run now has to clear three useful resource budgets on the similar time, none of which you set:
- Time — a window that closes whether or not or not you’re performed: a tough gateway timeout (one to 3 minutes, generally 5) that severs the connection mid-run, or one thing softer: an SLA, a caller blocked on the consequence, a course of that may solely wait so lengthy. And it doesn’t resume: when the window closes, the client simply retries, beginning the entire run over from zero.
- Price — now a margin, not a pool. Each run carries a value the client already paid, so it has to come back again worthwhile, not merely reasonably priced. And the client, not you, decides how usually it runs.
- Tokens and charge — a per-minute token price range (TPM) you share throughout each buyer directly, they usually are likely to name in the identical bursts. You hit the ceiling precisely when load is heaviest, which is strictly when latency is worst.
Below all three sits a tough ground you by no means commerce beneath: high quality. The reply needs to be proper to depend in any respect. A quick, low-cost, on-time reply that’s fallacious remains to be a failure. High quality isn’t a price range you spend down.
Any one among these you may handle by itself. The bind is that they apply collectively and pull towards one another, so the apparent repair for one spends one other. Wait out a sluggish step and also you blow the time window. Race a second copy to beat the clock and also you burn price and quota. Attain for a stronger mannequin to clear the standard ground and also you get slower. Not one of the budgets are yours to loosen, so the one transfer left is to commerce intentionally throughout all of them directly — with out ever dropping beneath the ground.
That’s what makes a customer-facing workflow a genuinely totally different factor to construct, and it generally forces a playbook that, from the within, appears to be like completely backwards:
- Kill a name that hasn’t failed
- Fireplace a replica of a name you’re already paying for
- Drop to a weaker mannequin on objective
Inside your individual partitions you’d by no means hassle. You’d simply let the sluggish step end. And the price range that punishes you most quietly is time: miss it and nothing appears to be like damaged in your aspect. An ideal reply that lands a couple of seconds late nonetheless reads as successful in your dashboards and as a failure to the client, and it’s the one restrict nothing within the stack enforces for you.
Right here’s the thesis, up entrance, as a result of every little thing else serves it: as soon as high quality clears the bar, dependable supply is a query of variance, not velocity. A predictable completion time beats a quick one with an extended tail, as a result of your clients can’t run their infrastructure in your greatest case; they must construct in your worst.
What that is — and isn’t: workflows, not free reasoning brokers
One distinction up entrance, as a result of it modifications every little thing. That is about an agentic workflow: a recognized course of movement with LLM-powered steps inside it, run by a deterministic orchestrator. It’s not a reasoning agent that decides its personal subsequent transfer at runtime. For a similar activity, a workflow is solely quicker: it already is aware of the plan, skips the deliberation, and runs each impartial step in parallel, so it reaches the identical reply in a fraction of the time and price a reasoning agent would take. Each have their place (reasoning brokers are way more versatile), however they fail in another way and also you repair them in another way. A reasoning agent’s downside is deciding what to do; a workflow’s downside (the one clients really feel) is delivering what it already is aware of how one can do, with high quality, and in time. This text is in regards to the latter.
How our system is constructed
The findings beneath come from our structure, and they need to generalize. These are unusual, direct API calls. Nonetheless, it helps to know the setup so you’ll be able to evaluate it to yours.
We run a customized orchestrator over managed third-party APIs (no self-hosted fashions on this dataset), and we run flagship fashions each straight by their suppliers (OpenAI, Anthropic, …) and thru managed platforms (Bedrock, Databricks, …), so high fashions have greater than 1 supplier. That lets us evaluate serving paths and transfer work between them.
Our workloads are a mixture: easy agent calls, deep reasoning, extractions, JSON and free textual content outputs. For a big fraction of calls we synthesize a big truth base into a solution, so massive enter and small to medium outputs. The analytics on this article maintain enter and output dimension fixed inside buckets (see appendix).
The sluggish tails we encounter are largely transient. Be aware that in case your structure is self-hosted or on devoted capability the tail could behave in another way, and can warrant one other method. Secondly, working a number of suppliers is what makes routing a hedge to a separate price range sensible. With a single supplier, fewer of those strikes can be found.
The declare, and the receipts
So right here’s the transfer that sounds backwards: we minimize a step off at 20-30 seconds even after we understand it may need answered completely slightly later — and that makes the system extra dependable, not much less.
That isn’t a hunch. It’s true on paper — the maths of heavy-tailed retries is unambiguous — and it’s true within the knowledge: a scan of properly over 1,000,000 current manufacturing LLM calls throughout our enterprise workloads — actual buyer visitors. The very first thing that visitors tells you is how unusual a single name’s timing actually is. A typical longer-output name comes again in a few dozen seconds. However one in 100 takes thirty seconds, generally a full minute or extra — for no cause linked to how a lot work it was doing.
That hole between the everyday name and the sluggish one underlies a lot of this text. The remainder of the article evaluations what to do about it.
Why the clock is unforgiving
A workflow isn’t judged on its common. It’s judged towards a deadline. On common our flows end comfortably; nonetheless outlier runs in lengthy tails don’t. These tail runs aren’t damaged. They’d return an ideal reply a bit later, and on an inside run they might depend as successes. On the client’s aspect, each one among them is a failure. Your entire tail of your latency distribution, nonetheless right, turns into an addition to your failure charge.
That’s why the quantity that issues right here isn’t common latency, it’s variance. A quick median buys you nothing in case your tail is lengthy.
The second squeeze is sunk price. The deeper you’re right into a workflow, the extra you’ve already spent: time, {dollars}, and your TPM quota. A failure on step 9 is much dearer than the identical failure on step two. You throw away every little thing the workflow constructed and you’ve gotten much less of the clock left to shift gears. We by no means restart the entire workflow ourselves, however the buyer will. If we fail, they’ll nearly definitely retry, beginning the total movement once more from the start. That compounds the issue on our aspect. It burns extra price, extra token price range, and the error price range on the SLA. And since the situations that made the run fail often haven’t modified, the retry has an analogous probability of failing. Worse, it tends to occur throughout a high-TPM window. The worst attainable time to pile additional load onto an already-strained system, and precisely when the chances of failing once more are highest.
There’s a second multiplier, and it’s straightforward to overlook. The primary is the one from the opening: reliability compounds, so a sequence of individually wonderful steps can nonetheless come out a coin flip1. However that failure is at all times advised as a narrative about correctness: getting a fallacious reply.
Right here’s what you nearly by no means hear about: the very same compounding occurs on the clock. Each step provides its personal small probability of touchdown within the sluggish tail, and people possibilities stack. So the extra steps you chain, the extra possible it’s that not less than one of them blows the deadline, even when each step is individually quick. That’s the multiplier this text is about, and it’s the one the literature leaves out. So let’s take a look at the numbers.
What an LLM reply time really appears to be like like
The everyday instances within the chart above sit in a reasonably tight band: each mannequin finishes a typical name someplace between eight and twenty seconds. The tails should not tight in any respect. One mannequin’s 99th-percentile name is available in round 30 seconds, one other’s previous 80. Comparable median, wildly totally different worst case. Promise a buyer your median and also you’re mendacity to the 1-in-20 and 1-in-100 calls within the tail, and a multi-step workflow hits these continuously. A quick typical time shouldn’t be a predictable one.
The plain objection is that the sluggish calls are simply doing extra work: larger prompts, longer solutions. They aren’t. Pin each the immediate dimension and the response size and the tail barely strikes: inside a single dimension bucket (work held mounted), p99 nonetheless runs two to seven instances the median (Determine 4). The slowness isn’t about how a lot the decision has to do — in our visitors it’s largely transient (queueing, scheduling, mid-stream competition, a supplier hiccup), which is strictly what makes it price interrupting.
One sluggish step sinks the entire run
You’d assume a workflow misses its deadline as a result of many steps had been every slightly sluggish. It nearly by no means occurs that approach. When a sequence blows its price range, it’s often one step that wandered into its tail whereas every little thing else behaved positive. Mathematically, a sequence’s overrun is dominated by its single worst step, not by the buildup of mildly sluggish ones. The entire behaves like its most, not its sum.2
That’s excellent news. You don’t want each step quick. It’s essential to cease any single step from working away. Which is the cutoff.
Sidebar — The maths, briefly (skip except you want math)
Three outcomes sit beneath the argument:
- Compounding. Simply the arithmetic of impartial steps: n steps every succeeding with chance p provides pⁿ end-to-end. At p = 0.95, ten steps ≈ 60% and twenty ≈ 36% — multiplication, no modeling. The identical compounding hits the clock: every added step is one other impartial draw towards the latency tail (the 2-7× p99/p50 we measure per name), so the chances that not less than one step blows its price range solely rise with size. Independence is the simplification — shared capability correlates actual steps — however it’s the conservative, illustrative case.
- The only large leap. LLM latency is heavy-tailed (lognormal-ish), and the lognormal is subexponential. For impartial subexponential steps the tail of the sum is simply the sum of the tails — `P(ΣX_i > t) ≈ Σ P(X_i > t) ≈ P(maxᵢ X_i > t)` as t grows. In phrases: a sequence overruns as a result of one step hit its tail, not as a result of many had been mildly sluggish.2
- Hedging, and why it really works for any failure. Fireplace n impartial makes an attempt and take the primary good one: if a single try fails with chance q, all n fail with chance qⁿ. That arithmetic doesn’t care what “fail” means — a blown deadline, a tough error, or a fallacious reply all purchase down the identical approach, which is why the identical retry/race/fallback transfer serves each taste. For the timing taste particularly it additionally shrinks unfold: for the reason that variances of impartial steps add, `Var(ΣX_i) = Σ Var(X_i)`, capping every step’s tail shrinks the entire chain’s. All of it rests on the makes an attempt being impartial (recent attracts, recent queue) — which is strictly why a parallel re-draw collapses a transient tail (or an unfortunate unhealthy reply) and does nothing for a deterministic one.3
The transfer: minimize early, then race
If a step has wandered into its tail, ready is the worst factor you are able to do — you’re spending your scarcest useful resource in your least possible payoff. So that you hand over early and check out once more in parallel: hearth a recent try and take whichever returns first. A recent try not often lands in the identical pothole, so two of them match contained in the time one caught name would have eaten — and the chances of each being sluggish are tiny (if one is sluggish with chance q, two are each sluggish with chance q²).3
The median barely strikes: about 10 seconds as a substitute of 12. The tail does the alternative: the 99th percentile drops from roughly 60 seconds to 25, and the run-to-run unfold is greater than minimize in half. You purchase predictability for the value of some additional tokens.
That value is actual, and it pushes again. Racing doubles the token invoice on that step, and tokens are a shared, capped price range. So price is a real downward drive on how freely you retry and race. However run the arithmetic and it’s lopsided. Doubling one step prices you that step’s tokens, as soon as. Blowing the deadline throws away every little thing you’ve already paid for, and the client nearly at all times retries, re-running all N steps of the workflow, not less than as soon as, generally extra. The deeper into the movement you’re, the extra one-sided the commerce: a redundant try on step 9 is reasonable subsequent to discarding steps one by 9 and watching them run once more. So that you hedge anyway. You simply don’t hedge indiscriminately, as a result of that shared token price range bites again hardest precisely if you most need to spend it (extra on that stress shortly).
One nuance that decides which fallback to achieve for: the route has to match why the step is failing.
- Gradual for transient causes → re-draw, ideally in parallel. A recent try escapes the stall. (A plain serial retry is weaker right here on an extended step — you’d pay the lengthy technology time twice.)
- Gradual as a result of the work is genuinely large → don’t re-run the identical name. Fall down to a quicker mannequin, or to an alternate path that reaches the identical consequence extra cheaply.
- Unsuitable, not sluggish → fall up to a extra succesful mannequin. Pace gained’t repair a foul reply; functionality would possibly. (That is the standard ground from earlier, enforced at runtime.)
Reduce on the best sign
A solution time is actually two phases.4 The look ahead to the first token is usually queueing and scheduling; the technology that follows, token by token, is the remainder. Which section carries the tail decides what you set the cutoff on. And that depends upon how a lot the step writes.
For the longer steps this text is about (those that press towards a deadline), the tail lives in technology, not the first-token wait. A sluggish queue is a small slice of a forty-second name; the unfold that blows the price range is within the tokens. So minimize these on complete elapsed time, or on tokens emitted thus far towards the time you’ve gotten left, not on time-to-first-token. (For brief steps the steadiness flips: with little to generate, the first-token wait is many of the name, and time-to-first-token turns into the cleaner minimize. Measure your individual steps to see which aspect you’re on.)
Two indicators are price wiring in regardless:
- No first token in any respect, previous the cutoff? That’s caught, not sluggish. Surrender and hedge. A recent parallel try will get newly scheduled and nearly at all times wins.
- Tokens flowing however it’ll blow the price range? Don’t re-run it. You’d simply regenerate the identical size on the similar velocity. Fall to a quicker mannequin.
And one failure no clock can catch: a step that returns on time however returns junk (e.g. it’s empty, truncated, or unparseable). A latency cutoff sails proper previous it; solely a top quality examine downstream will. For any step that’s imagined to return a particular form, the most affordable such examine is a strict validation proper after the decision. Parse the consequence towards the anticipated schema or object, and deal with a validation failure precisely like every other: minimize and fall again (re-draw, or fall up to a extra succesful mannequin). It catches a significant slice of unhealthy solutions earlier than they attain the following step. Chopping early buys you predictability, not correctness. Maintain these two jobs separate.
The catch: hedging spends the price range you’re shortest on
Racing has an ungainly property. The tail is worst when the system is busy. And “busy” is strictly when your tokens-per-minute price range has the least room left. So the one transfer that fixes the tail desires to spend tokens on the exact second they’re hardest to come back by. Do it blindly and also you get a pile-on: sluggish calls set off hedges, hedges add load, load makes every little thing slower, extra calls cross their cutoff. A latency downside turns into a rate-limit downside.
Two information make this much less forgiving than it first appears to be like. The price is dedicated the moment you hearth the second name. Cancelling the loser frees your connection, however the supplier retains producing, and billing, the deserted try. There’s no clawback, so all of the management has to reside on the resolution to hedge, not after. And also you often can’t see how a lot price range is left. Estimating it’s attainable however concerned, so any scheme that “eases off because the quota fills” is difficult to run in follow.
What works in follow is cruder and extra structural:
- Ship the hedge someplace with its personal price range. Token limits are per-model and per-provider, and most of us run multiple (as famous in How our system is constructed). Routing the retry to a totally different mannequin or supplier will get a separate quota and an impartial draw. The identical transfer that escapes the stall additionally avoids spending the scarce price range twice.
- Maintain hedges uncommon by building. That is what the precomputed cutoffs already purchase you: with the edge set at every step’s measured p95, a hedge fires solely on the sluggish minority, so the additional spend stays small with no runtime accounting in any respect. (Identical cutoffs as the following part, no new equipment.)
- React to the indicators you really get. You most likely can’t learn headroom, however you’ll be able to learn 429s and climbing latency. Deal with these because the cue to hedge much less and minimize later, no more.
- At actual saturation, cease hedging. As soon as the supplier is already returning rate-limit errors, extra makes an attempt solely deepen the outlet. Downshift to a smaller, cheaper mannequin or shed the work as a substitute.
One lever we haven’t constructed, and supply solely as a route: an specific international cap that holds hedged calls to a small fraction of complete visitors, impartial of the per-step choices. It’s the principled backstop the tail-at-scale work factors to;3 we set conservative cutoffs as a substitute and haven’t wanted it, however at greater hedge charges that’s the place we’d go subsequent.
Sidebar — A budget strikes you make first
Cutoffs and hedging are insurance coverage. You purchase much less of it if the workflow is constructed properly to start with. The defaults that fireplace on each request, earlier than any reactive trick:
- Parallelism by design. Lay the movement out as a dependency graph and run each step the second its inputs exist. Then go additional — design the dependencies out. Fewer dependencies means extra steps are leaves, and a leaf can fail cheaply with out taking the remainder of the graph down.
- Don’t name the mannequin in any respect if you don’t must. Essentially the most dependable name is the one you don’t make — use code, lookups, and validators wherever the work doesn’t really need a mannequin.
- Combine fashions per step, not per workflow. Quick and low-cost the place it’s sufficient; succesful the place it isn’t.
- Cache the deterministic components. Don’t pay an LLM twice for a solution that may’t change.
The purpose right here: spend your reliability price range on construction first, so the clock work has much less to repair.
When do you really pull the set off?
The cutoff is a knob, not a relentless. How exhausting you flip it comes down to 3 plain questions on every step:
- How a lot does the reply want this step? Good-to-have: let it go. Should-have: defend it.
- How a lot is ready on it? If nothing depends upon it, let it run to the deadline. If half the workflow is queued behind it, end it sooner, and ensure it’s proper, as a result of a fallacious reply right here poisons every little thing downstream.
- How a lot time is left? A lot: retry calmly. Nearly out: minimize quick and fall again.
The extra a step is must-have, load-bearing, and brief on time, the sooner you hearth the backup and the extra you’ll spend to hedge it. An elective, terminal, early step will get none of that. (“Early or late within the movement” was by no means the actual axis. It was a proxy for the way a lot nonetheless depends upon this step.)
And also you don’t guess the quantity. You run the workflow many instances, measure every step’s latency curve (P95), and set the cutoff from that curve. Under the step’s worst case, weighted by the three questions. A step that often solutions in 20 seconds will get minimize at 30, although it may need succeeded at 60.
Why nearly no person does this
This isn’t exhausting. It’s nuanced, and most groups don’t have the engine for it.
The favored workflow instruments, the Airflows and Temporals, had been constructed to make pipelines sturdy: retry, resume, don’t lose state, they usually’re excellent at it. Their timeout recommendation follows from that purpose: set a per-step timeout longer than the slowest run and retry till it succeeds.5 That’s the best intuition when the job is to sturdy completion, and it’s precisely the fallacious recommendation when the job is to complete in time. Your workflow engine will fortunately retry a step many instances; it has no notion of a step’s measured typical time and downstream implications, so it might’t minimize early and swap fashions. That isn’t a flaw. It’s by design.
The distributed-systems fundamentals are already on our aspect: work from a deadline price range, match every timeout to measured latency.6 We’re not contradicting that. We’re making use of it to a case these instruments don’t assume: a brief, non-resumable price range the place the best transfer on the cutoff is a quicker various, not the identical name once more. Identical precept, inverted route.
Takeaway
One factor, in the event you hold nothing else: a predictable completion time beats a quick one with an extended tail. Low variance beats low latency. You possibly can’t promise a buyer a median, solely a sure. Every part right here serves that sure. Chopping early, hedging, racing, designing out dependencies: every trades slightly common velocity for lots much less variance. You hand over the best tail to purchase the left.
In a customer-facing agentic workflow, reliability is the product. The craft isn’t proudly owning a bag of retries and fallbacks, these are desk stakes. It’s deciding, per step, whether or not to hedge and when to surrender, from the constraints and the measured habits of your individual system.
APPENDIX
In regards to the creator
Frank Wittkampf is Head of Utilized AI Engineering at Databook. His crew architects, builds, and operates a completely customized AI stack together with deep reasoning, an agentic workflow engine, AI asset technology, agentic harnesses, information base & context graph, AI pre-processing, multi-tenant AI configuration administration, and many others. This AI infrastructure powers the GTM groups of high Enterprise firms like Microsoft, Salesforce, Amazon, Databricks, and lots of others.
A be aware on the information
The latency figures right here come from current (June 2026), anonymized manufacturing visitors throughout enterprise buyer workloads — roughly 1.2 million LLM calls over a 30-day window, not artificial benchmarks or a public hint. As described in How our system is constructed, these are direct calls to managed third-party APIs, which is a part of why the sluggish tail is essentially transient. The numbers within the textual content describe the longer calls (output ≥ 600 tokens), since these are those that really press towards a deadline; shorter calls are quicker and fewer variable. All through, a “tail ratio” (p99/p50) holds name dimension mounted inside a bucket except said in any other case. Fashions are labeled by household and serving path solely; predictability depends upon the serving path (e.g. a managed API vs. a direct one), not simply the mannequin, so these are intentionally not a mannequin rating. Durations had been bucketed in one-second bins; a tough 90-second ceiling truncates solely the final ~0.2% of longer calls, so the tail you see is actual, not an artifact of the cap.
Isn’t the tail simply the larger calls?
The honest objection to Determine 4: every row is a token bucket, not a hard and fast token depend, so possibly the sluggish calls inside a cell are merely the bigger ones — extra to prefill, extra to generate — and the tail is simply dimension, not something transient.
It isn’t, and the information’s personal form reveals why. If dimension drove the within-cell tail, two issues would comply with: the tail ratio would develop with the quantity of labor, and probably the most tightly bounded cells would have nearly no tail. Neither holds.
Two issues to learn off it. First, the tail ratio is flat at roughly 2–4× throughout each output-size column — it doesn’t climb because the work grows, so the tail doesn’t scale with the work. Second, and decisively, take a look at the leftmost column: these calls emit at most 50 output tokens, so technology time bodily can’t range by greater than a few second — but the tail there may be nonetheless ~3.5×. There isn’t a dimension variable massive sufficient to supply that. The residual unfold is transient (queueing, scheduling, a momentary supplier hiccup), which is strictly what a recent try escapes.
Why these numbers look smaller than the two–7× quoted earlier: the column figures listed below are volume-weighted averages throughout many cells, which clean out the unfold, whereas the two–7× within the physique is the per-call envelope — the vary particular person cells really span. Identical knowledge, two totally different cuts: the averages present the tail doesn’t scale with work; the envelope reveals how huge it will get on any given name.
Notes & Footnotes
Be aware: All photos created by the creator.
1: Ten steps at 95% every ≈ 60% end-to-end; twenty ≈ 36% (assuming independence).
2: The lognormal lies within the subexponential class, the place the tail of a sum of impartial phrases is asymptotically the sum of the person tails: `P(S_n > t) ∼ Σ_i P(X_i > t) ∼ P(max_i X_i > t)` as t → ∞ — the “single large leap” precept (Foss, Korshunov & Zachary, An Introduction to Heavy-Tailed and Subexponential Distributions, Springer, 2nd ed. 2013, eqs. 1.3 & 1.6). It’s an asymptotic assertion and assumes independence, so deal with it because the instinct for why one sluggish step dominates, not a plug-in system.
3: If every impartial try is sluggish with chance q, two parallel makes an attempt are each sluggish with chance q²; n makes an attempt, qⁿ. The basic hedged-request consequence (Dean & Barroso, “The Tail at Scale,” CACM 2013); in an agent setting, Winston et al. (arXiv:2605.21470, ICML 2026) select between serial, parallel, and hedged execution from measured latency curves. On our manufacturing knowledge, racing two makes an attempt minimize p99 on longer steps by greater than half (≈60s→25s) whereas sequential re-draw on complete time didn’t.
4: The break up is normal in inference work: “time to first token” (queue + prefill) versus per-token technology. See e.g. Agrawal et al., Taming the Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv:2403.02310, 2024). In our manufacturing visitors the tail for longer calls sits within the technology section, not the first-token wait — which is why we minimize lengthy steps on complete elapsed time moderately than time-to-first-token.
5: Temporal’s exercise timeouts are designed to complete ultimately, together with retries — therefore Begin-To-Shut set above the sluggish tail.
6: Google SRE, gRPC deadlines, and Spanner all advise propagating a complete price range and dropping work that may now not assist the caller. We prolong the identical precept to a sync, non-resumable buyer price range.
