Lengthy Context Isn’t Free — I Constructed a Protected Immediate-Pruning Layer That Makes LLM Techniques Work

I’ve labored on, dialog state tends to develop rapidly over time. It’s frequent to resend massive parts of the historical past on every flip—together with older device outputs, repeated RAG retrievals, and context that’s not related. As this accumulates, prompts can grow to be considerably bigger, which can improve inference value and latency, and in some circumstances have an effect on reasoning efficiency.

I constructed a deterministic pipeline that prunes this redundant state earlier than the immediate ever reaches the mannequin. The model I applied avoids LLM calls, embeddings, and exterior dependencies. Relying strictly on normal library parts ensures each pruning resolution stays totally deterministic and reproducible.

State monitoring executes in three distinct passes: Expired Context Elimination, Duplicate Context Elimination, and Dependency Restoration. The third go is what helps make the primary two safer in observe. It ensures that nothing a later message is dependent upon is by chance eliminated.

Whereas constructing it, I bumped into two bugs that modified the design. My first benchmark corpus used a hard and fast variety of duplicates and off device calls, which made discount percentages shrink as conversations grew. That didn’t replicate what I might anticipate from real-world habits.

My dependency restoration logic additionally went fully untested at first as a result of my artificial information by no means created a case the place a required message was really eliminated. Each points are coated right here, together with how fixing them modified the outcomes.

After correcting the pipeline, I benchmarked it throughout three workloads: plain chat, a RAG assistant, and a tool-heavy agent. Every was examined at 5 dialog sizes, for a complete of 15 configurations on two totally different machines. Throughout these runs, all labeled required information had been preserved. The system additionally reached a steady mounted level after a single go, which suggests pruning an already pruned immediate produces no additional adjustments.

Token discount is dependent upon the workload. It’s about 2 to 4 p.c for plain chat, 27 to 32 p.c for a RAG assistant, and 33 to 34 p.c for a tool-heavy agent. Even at 2,000 turns and 131,000 tokens, preprocessing stayed below 50 milliseconds.

Full code, all 35 assessments, and the uncooked terminal output are included under so you possibly can run the pipeline your self.

Full code: https://github.com/Emmimal/prompt-pruning-layer/

Each immediate I ship retains getting heavier

I stored seeing the very same sample pop up throughout each long-running agent I constructed. A dialog begins out completely clear. Fifty turns later, it’s an absolute mess.

By that fiftieth flip, the immediate payload you’re delivery out on each single request contains the system immediate, the complete chat historical past, 4 device outputs (two of that are fully stale as a result of the device ran twice), six retrieved chunks (three are near-duplicates as a result of the consumer circled again to an previous subject), a SQL outcome from twenty turns in the past that’s simply sitting there, and a single consumer desire said as soon as and by no means introduced up once more.

None of these things is technically unsuitable. Each single piece made sense the precise second it was injected. The true challenge is that nothing ever will get cleared out. The immediate turns into an append-only log of each historic occasion, and also you’re dumping the entire thing onto the mannequin, each single flip, indefinitely.

That’s not only a storage downside. Reasoning efficiency measurably degrades as enter size grows, even when the added content material is irrelevant to the duty [1], and fashions are worse at utilizing data buried in the course of a protracted context than data close to the perimeters [2].

The instant knee-jerk response if you see this bloating is simply primary truncation. Slice the final N messages and drop the whole lot else. It’s actually one line of code. The catch is it silently breaks your dialog chains in methods you received’t even understand till it blows up on an actual consumer:

Flip 3:   Consumer: My most popular output format is CSV.
Flip 4:   Assistant: Acquired it.
...
Flip 47:  Consumer: Export the outcomes.

In case your window is capped on the final 20 turns, flip 3 disappears by flip 47. The assistant loses the context that the consumer requested for a CSV. That information was not stale; it was a tough dependency, and easy positional truncation can not differentiate between previous, expendable context and previous context {that a} later flip nonetheless depends on.

That’s the actual design constraint this undertaking addresses. Any mechanism that removes redundant state should distinguish between these two classes. Recency alone is inadequate.

Borrowing an thought from working programs

Right here is the reframe I constructed the remainder of this round: an working system is continually deciding which pages keep resident in RAM and which get evicted. An extended-running LLM dialog has the very same downside, besides nothing performs the position of the reminiscence supervisor. Context simply accumulates endlessly as a result of no course of owns the job of deciding what stops incomes its place within the immediate.

The Immediate Pruner on this article is that lacking piece.

Each piece of dialog state (a consumer flip, an assistant flip, a device output, a retrieved chunk) is a Message object with a job, a flip quantity, and a little bit of bookkeeping metadata:

@dataclass
class Message:
    id: str
    position: str
    content material: str
    flip: int
    tool_call_key: Elective[str] = None
    expires_after_turn: Elective[int] = None
    defines_keys: record = subject(default_factory=record)

Three passes run over that record so as. Each is a pure operate: messages in, a filtered record out. No mannequin is concerned wherever within the pruning step itself.

Why there’s no mannequin contained in the pruner itself

I may have used an embedding mannequin to attain message relevance, which might have made this fuzzier and so much simpler to write down. I didn’t, and it’s not for the sake of purity.

As soon as a pruning resolution is dependent upon a mannequin’s judgment, you lose the flexibility to cause about what a dialog will appear like on the subsequent flip. The identical enter not ensures the identical output. A pruning layer meant to make a manufacturing system extra predictable shouldn’t be the least predictable a part of it. All the pieces right here runs on dataclasses, regex, and dict lookups, which is strictly the toolset this downside wants.

Move 1: Expired Context Elimination

If a device will get referred to as greater than as soon as below the identical key (the identical search question, the identical SQL lookup, the identical file learn), solely the most recent outcome remains to be reliable. All the pieces earlier below that secret’s expired.

def _pass1_expired_context_elimination(self, messages):
    last_occurrence = {}
    for m in messages:
        if m.tool_call_key:
            last_occurrence[m.tool_call_key] = m.id

    stored, eliminated = [], []
    for m in messages:
        if m.tool_call_key and last_occurrence[m.tool_call_key] != m.id:
            eliminated.append(m)
        else:
            stored.append(m)
    return stored, eliminated

Move 2: Duplicate Context Elimination

Retrieval pipelines continually pull up similar or near-duplicate passages, particularly when a consumer loops again to an earlier subject. This go normalizes the whitespace and casing, retains solely the primary incidence, and drops each duplicate after it.

def _pass2_duplicate_context_elimination(self, messages):
    seen, stored, eliminated = {}, [], []
    for m in messages:
        if m.position == ROLE_RETRIEVED_DOC:
            norm = " ".be a part of(m.content material.decrease().break up())
            if norm in seen:
                eliminated.append(m)
                proceed
            seen[norm] = m.id
        stored.append(m)
    return stored, eliminated

Move 3: Dependency Restoration (and the bug that made me construct it correctly)

This go is the rationale the primary two are protected to run in any respect. If Move 1 or Move 2 drops a message that occurs to be the one place a still-referenced truth obtained outlined, this go catches it and places it again.

The mechanism is deliberately easy: a message marks itself with a literal DEFINE: tag, and a later message references it utilizing REF:. If a REF survives into the ultimate stored set however its matching DEFINE obtained dropped upstream, Dependency Restoration restores that lacking message to the record.

def _pass3_dependency_restoration(self, all_messages, kept_messages, removed_messages):
    kept_ids = {m.id for m in kept_messages}
    by_id = {m.id: m for m in all_messages}

    key_definer = {}
    for m in all_messages:
        for key in m.defines_keys:
            key_definer[key] = m.id

    referenced_keys = set()
    for m in kept_messages:
        referenced_keys.replace(m.references())

    restored = []
    for key in referenced_keys:
        definer_id = key_definer.get(key)
        if definer_id and definer_id not in kept_ids:
            restored_msg = by_id[definer_id]
            kept_messages.append(restored_msg)
            kept_ids.add(definer_id)
            restored.append(restored_msg)

    kept_messages.type(key=lambda m: (m.flip, m.id))
    return kept_messages, restored

Right here is the bug. I ran my first full benchmark throughout all three workloads and 5 sizes, and each single row printed “Restored (deps): 0.” Each one. My first response was that this appeared nice—an ideal security file. It wasn’t. It meant Move 3 had by no means really restored a single message on any run at any dimension.

I went again into my corpus generator to seek out out why, and the reply was embarrassing as soon as I noticed it. My artificial conversations solely ever hooked up DEFINE markers to plain consumer messages, whereas Move 1 and Move 2 solely ever take away device outputs and retrieved paperwork. The 2 classes by no means overlapped. Dependency Restoration was sitting within the codebase, totally written, however fully untested by my very own benchmark as a result of nothing I generated ever gave it a cause to fireplace.

The repair was to let some device outputs additionally outline a dependency, the identical manner an actual “get consumer settings” device name would possibly floor a truth the dialog is dependent upon later, although that actual device name may get outdated and marked expired by Move 1. As soon as I added that, the numbers modified instantly: 2 restorations on the smallest tool-agent dialog, climbing to 127 on the largest. Required information stayed at one hundred pc preserved the complete time, however now that quantity really meant one thing, as a result of the go being examined was able to failing and didn’t.

I wish to be direct in regards to the limitation that’s nonetheless right here even after the repair: dependency detection is literal identifier matching, not understanding. It catches an actual REF matching an actual DEFINE. It won’t catch a consumer paraphrasing, asking “what format did I point out earlier” with no matching tag. A semantic dependency resolver would wish an embedding mannequin or an LLM name, and that’s explicitly exterior what this deterministic pipeline does. I’d moderately ship a narrower assure I can show than a broader one I can’t.

Design targets, and why every one exists

Deterministic. Identical enter all the time yields the identical output. Conserving the mannequin out of the loop eliminates run-to-run variance and any threat of hallucinating what ought to stay within the immediate.

Dependency-safe. It by no means silently drops a truth {that a} later flip nonetheless wants. Positional truncation fully lacks this property, which makes it non-negotiable right here. A pruner saving 40 p.c of tokens that sometimes breaks a dialog is a worse trade-off than one saving 4 p.c that by no means breaks something.

Idempotent. Working the pruner a second time does nothing. If that isn’t true, you can not safely re-prune on each single flip of a rising dialog with out risking compounding drift.

Light-weight. The pruning step ought to by no means grow to be the precise bottleneck it was constructed to get rid of.

The benchmark: three workloads, and a mistake I nearly shipped

My first model of the artificial corpus generator picked a hard and fast variety of duplicate passages and repeated device calls (six device calls and eight duplicates) no matter how lengthy the dialog was. I ran it at 5 sizes and the discount share went down because the dialog obtained longer: 9.9 p.c at 50 turns, dropping to 0.3 p.c at 2,000 turns. That runs backward from precise manufacturing site visitors, and it’s not defensible. If the quantity of waste in a benchmark is only a fixed I hand-picked, anybody studying it’s proper to ask whether or not I constructed the benchmark to show the algorithm works.

So I threw that generator out and rebuilt it round an specific workload mannequin as an alternative: retrieval-per-turn, retrieval overlap charge, device name charge, and gear repetition charge. All of those had been mounted earlier than operating a single benchmark, based mostly on what appeared like believable manufacturing habits, moderately than adjusted afterward to hit a quantity I favored. Three workloads got here out of that:

Regular chat. No retrieval, occasional device calls, largely odd back-and-forth.

RAG assistant. Retrieves paperwork on each flip, with an actual probability any given passage overlaps one thing retrieved just lately, as a result of customers revisit subjects and retrieval re-surfaces the identical chunks.

Software agent. Frequent calls throughout 5 device sorts (search, SQL, calculator, filesystem, internet fetch), excessive repetition charge, modeling one thing that re-plans and re-queries continually.

Each artificial corpus additionally ships with floor reality: each message some later message is dependent upon will get labeled required up entrance. So “did pruning preserve the whole lot it wanted to” is a examine towards identified labels, not a guess.

Right here is the entire output. All 15 configurations, not a slice of it:

Each single row says Info stored: Sure and Idempotent: Sure. Not most rows. All fifteen.

The sample by workload is smart when you have a look at the place the waste really comes from. Regular chat barely retrieves something and infrequently repeats a device name, so there may be nearly nothing for Move 1 or Move 2 to catch; it stays round 4 p.c irrespective of how lengthy the dialog runs. The RAG assistant retrieves each flip with actual overlap, so Duplicate Context Elimination carries many of the weight, touchdown round 32 p.c. The Software agent combines each issues (frequent device repetition and retrieval overlap) and hits the very best discount at 33 to 34 p.c.

Completely different workloads accumulate totally different sorts of waste. The pruner responds on to no matter waste is sitting in entrance of it, moderately than producing a flat quantity that will recommend the benchmark was reverse-engineered to hit a goal.

If I solely obtained to maintain one outcome from this entire benchmark, it’s the security property: 15 out of 15 configurations preserved one hundred pc of required information. Zero lacking dependencies, throughout three structurally totally different workloads and a 40x vary in dialog size. Truncation by place can not provide that. For me, this was crucial sign when evaluating whether or not the method could be usable in manufacturing.

That quantity can be computed, not hand-counted. The benchmark script itself tallies how lots of the 15 (workload, dimension) pairs preserved each required truth and what number of reached the idempotent mounted level, printing an mixture abstract block after the 15 detailed runs:

============================================================
SUMMARY
============================================================
Configurations run:               15 (3 workloads x 5 sizes)
Required information preserved:         15/15
Reached mounted level (idempotent): 15/15

Workload            Token discount vary    Info preserved   Idempotent
Regular chat          1.9-4.1%                YES               YES
RAG assistant        27.0-32.5%               YES               YES
Software agent           33.1-33.7%               YES               YES

I wrote this examine as a result of I caught myself hand-counting desk rows for an early draft. That metric belongs within the script output, not my very own eyes squinting at a terminal log. If a take a look at run ever hits something lower than 15/15, it means I broke the pruner and have a regression to search out, not a typo to edit within the publish.

I left one particular metric out of the ultimate numbers: tokens eliminated per millisecond of execution overhead. The code computes it—it peaks at round 4,000 tokens per millisecond on small tool-agent runs and lands between 900 and 1,700 tokens at bigger scales. It stays within the codebase as an inside subject as a result of it helps monitor scaling prices, however it belongs exterior the principle desk. Readers can not act on it the way in which they’ll with uncooked token rely, discount share, or millisecond overhead. Three direct metrics exhibiting a transparent trade-off are higher than a fourth that acts as a novelty.

The idempotence result’s the half I favored monitoring essentially the most. Proving prune(prune(x)) == prune(x) means the pipeline hits a steady mounted level on the primary go. Working it once more on an already-pruned immediate adjustments nothing:

That guidelines out oscillation. It additionally guidelines out cumulative shrinkage throughout turns when you re-pruner on each single message of a rising dialog, which is strictly how this runs in manufacturing.

Reproducing it on a second machine

I ran the complete benchmark on a Linux container operating Python 3.12.3, then once more on Home windows 11 in PyCharm utilizing Python 3.12 in a separate venv. Each token rely and message rely matched precisely throughout each machines. Solely the millisecond timings moved, which is normal for various {hardware}, and even these stayed below 50 milliseconds for the biggest, most tool-heavy dialog on each setups.

One factor I observed whereas evaluating the 2 runs and wish to be straight about: prompt-build time (the time to serialize the ultimate message record right into a string) sometimes got here out slower after pruning than earlier than on the Regular Chat workload. Not by a lot—below a millisecond—however the path was backward.

My learn is that Regular Chat solely removes 2 to 4 p.c of messages, so the earlier than and after record sizes are practically similar. At sub-two-millisecond operations, system jitter and rubbish assortment pauses simply swamp the precise sign. Including a warm-up name and utilizing the median of 30 runs largely stabilized the metric, however the anomaly nonetheless pops up on that workload. It’s noise at a scale the place the delta is smaller than the measurement error, so I left it uncooked moderately than massaging the information.

What this benchmark intentionally doesn’t measure

This benchmark isolates token discount and pruning overhead as a result of these are the metrics the pipeline really controls. Finish-to-end LLM latency is a totally separate variable. It is dependent upon supplier structure, batching, regional caching, and community circumstances that this undertaking can not see. Attempting to transform a token discount share straight right into a latency delta means inventing an arbitrary conversion fixed.

The baseline actuality is straightforward: chopping 30 to 34 p.c of enter tokens means the mannequin does much less work per name. Normally, inference value and latency have a tendency to extend with immediate dimension [4], making this a helpful value lever. However a real latency quantity requires a reside validation go towards your particular supplier. Publishing a generic latency determine right here would imply making claims about infrastructure I don’t management, moderately than evaluating the pruning layer itself.

How this matches into an actual agent loop

The pruner lives in precisely one spot: proper after the dialog historical past is pulled collectively for a flip, and simply earlier than it will get serialized into the ultimate immediate string.

from prompt_pruning import PromptPruner, PromptBuilder

pruner = PromptPruner()
builder = PromptBuilder()

def handle_turn(conversation_state, new_user_message):
    conversation_state.append(new_user_message)

    pruned_messages, report = pruner.prune(conversation_state)
    immediate = builder.construct(pruned_messages)
    response = call_llm(immediate)

    conversation_state.append(make_assistant_message(response))
    return response

As a result of the pipeline is idempotent, calling prune() each flip of a rising dialog is protected. Working the pruner ten instances on a historical past pruned 9 instances yields the very same outcome as a clear run from scratch. This makes “run it each flip” a protected default, eliminating the necessity to monitor state or cause about earlier passes.

The one integration resolution left is how REF and DEFINE tags get hooked up to messages. Right here they’re literal markers contained in the message content material, which is the only mechanism for a prototype. A manufacturing system would probably connect them as structured metadata on the message object so the tags by no means leak into the uncooked textual content the mannequin reads. Move 3’s logic stays the identical both manner. An upstream course of nonetheless has to find out what counts as a dependency price tagging, as a result of Move 3 can solely restore what it’s explicitly instructed to trace.

What this doesn’t cowl

Dependency detection is literal, not semantic. If a reference is paraphrased and lacks an identical tag, the script will miss it.

These workloads are additionally fully artificial. I selected the three parameter units based mostly on believable manufacturing habits, not actual telemetry. When you have manufacturing logs, regenerating these three workload classes from precise utilization is the apparent subsequent step. The numbers will shift relying on what your precise site visitors seems like.

This pipeline omits semantic compression, embeddings, and LLM-scored pruning. These are legitimate, various approaches, however avoiding them retains this implementation totally deterministic and dependency-free. LLMLingua is a primary instance of the discovered various; it makes use of a small language mannequin to attain and drop tokens, reaching a lot increased compression ratios than this script [3]. Selecting between them is a direct trade-off: you trade determinism and zero-dependency execution for tighter compression.

The token counts are additionally approximations. The script makes use of a whitespace and punctuation-boundary heuristic as an alternative of a manufacturing subword tokenizer like tiktoken. As a result of the heuristic runs constantly earlier than and after pruning, the relative discount percentages stay correct, even when absolutely the numbers don’t completely match an official tokenizer.

Lastly, there isn’t any direct latency measurement for the infrastructure causes detailed earlier.

The place I’d take this subsequent

Two extensions make sense right here moderately than increasing the present three passes.

The primary is a hybrid method. Maintain these three deterministic passes as a quick, protected first stage, then hand the output to an embedding-aware or LLM-scored compression device like LLMLingua. This catches semantic redundancy that literal identifier matching misses, comparable to two passages saying the identical factor in several phrases.

Working the deterministic passes first preserves the protection assure. If the discovered stage misbehaves, the dialog drops again to the deterministic baseline as an alternative of failing unpredictably. Architecturally, the discovered go solely operates on the output of Move 3. A mannequin bug within the compression step would possibly shrink a immediate too aggressively, however it can not reintroduce a dependency failure that the deterministic passes already cleared.

The second extension is closing the hole between artificial information and manufacturing actuality. The subsequent step is regenerating these identical three workload classes from precise manufacturing traces. Conserving the benchmark methodology similar—the identical mounted parameters, ground-truth dependency labels, and 15-configuration sweep—ensures the outcomes stay straight comparable to those printed figures whereas changing guesses with actual telemetry.

Precise utilization logs will probably shift the RAG and tool-agent metrics in both path. Actual-world site visitors doesn’t mechanically imply increased compression. As an illustration, a manufacturing system with aggressive upstream deduplication would possibly already get rid of the waste this artificial mannequin assumes. That will be a extremely helpful discovering in its personal proper, moderately than a failure.

A 3rd, smaller level to notice: the REF/DEFINE conference used right here is only a placeholder. A manufacturing system ought to derive these tags mechanically from structured information, device name arguments, session variables, or specific consumer settings, moderately than counting on handbook textual content markers.

Whereas the deterministic logic in Move 3 stays similar both manner, the precise worth of this pipeline relies upon fully on how cleanly you possibly can generate correct dependency tags upstream. That’s an architecture-specific integration downside moderately than one thing a general-purpose pruning library can remedy out of the field.

Manufacturing serving programs already deal with immediate bloat as a reminiscence administration downside on the infrastructure layer, evicting and sharing KV cache pages very like an working system handles bodily reminiscence [4]. You possibly can consider this undertaking as working one layer above that, on the immediate development part, earlier than the request ever reaches the infrastructure.

The 2 layers don’t compete. A smaller, deduplicated immediate supplies a greater enter to a well-managed cache moderately than appearing as a alternative for it.

The Core Takeaway

If a crucial truth is preserved, the pipeline must show that retention moderately than assume it based mostly on the absence of apparent errors. This precept guided the design of the system.

Lengthy-running conversations don’t all the time require a bigger, smarter mannequin to determine what to recollect. Typically, the extra sturdy answer is a predictable, three-pass system that may programmatically show what it didn’t lose.

Sources

[1] Levy, M., Jacoby, A., & Goldberg, Y. (2024). Identical process, extra tokens: The impression of enter size on the reasoning efficiency of huge language fashions. In Proceedings of the 62nd Annual Assembly of the Affiliation for Computational Linguistics (Quantity 1: Lengthy Papers) (pp. 15339–15353). Affiliation for Computational Linguistics. https://doi.org/10.18653/v1/2024.acl-long.818

[2] Liu, N. F., Lin, Ok., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Misplaced within the center: How language fashions use lengthy contexts. Transactions of the Affiliation for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638

[3] Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023). LLMLingua: Compressing prompts for accelerated inference of huge language fashions. In Proceedings of the 2023 Convention on Empirical Strategies in Pure Language Processing (pp. 13358–13376). Affiliation for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.825

[4] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Environment friendly reminiscence administration for giant language mannequin serving with PagedAttention. In Proceedings of the twenty ninth Symposium on Working Techniques Rules (SOSP ’23) (pp. 611–626). Affiliation for Computing Equipment. https://doi.org/10.1145/3600006.3613165

All code, benchmark numbers, and take a look at outcomes on this article are my very own, generated by operating the included codebase straight and reproduced on two separate machines. No proprietary datasets, copyrighted textual content, or third-party code had been utilized in constructing or benchmarking this method. All code, the corpus generator, the pruner, the benchmark harness, and all 35 assessments, is accessible within the repository linked under.

https://github.com/Emmimal/prompt-pruning-layer