The rising measurement of Massive Language Fashions (LLMs) makes environment friendly inference difficult, primarily because of the reminiscence calls for of the autoregressive Key-Worth (KV) cache. Present eviction or compression strategies scale back price however depend on heuristics, equivalent to recency or previous consideration scores, which serve solely as oblique proxies for a token’s future utility and introduce computational overhead. We reframe KV cache eviction as a reinforcement studying (RL) drawback: studying to rank tokens by their predicted usefulness for future decoding. To this finish, we introduce KV Coverage (KVP), a framework of light-weight per-head RL brokers educated on pre-computed era traces utilizing solely key and worth vectors. Every agent learns a specialised eviction coverage guided by future utility, which evaluates the standard of the rating throughout all cache budgets, requiring no modifications to the underlying LLM or further inference. Evaluated throughout two completely different mannequin households on the long-context benchmark RULER and the multi-turn dialogue benchmark OASST2-4k, KVP considerably outperforms baselines. Moreover, zero-shot checks on normal downstream duties (e.g., LongBench, BOOLQ, ARC) point out that KVP generalizes nicely past its coaching distribution and to longer context lengths. These outcomes display that studying to foretell future token utility is a robust and scalable paradigm for adaptive KV cache administration.
