EAGLE<\/a> is the state-of-the-art technique for speculative decoding in massive language mannequin (LLM) inference, however its autoregressive drafting creates a hidden bottleneck: the extra tokens that you just speculate, the extra sequential ahead passes the drafter wants. Ultimately these overhead eats into your positive factors. P-EAGLE removes this ceiling by producing all Okay draft tokens in a single ahead go, delivering as much as 1.69x speedup over vanilla EAGLE-3 on actual workloads on NVIDIA B200.<\/p>\n You may unlock this efficiency acquire by downloading (or coaching) a parallel-capable drafter head, including On this put up, we clarify how P-EAGLE works, how we built-in it into vLLM ranging from v0.16.0<\/a> (PR#32887), and the right way to serve it with our pre-trained checkpoints. Right here is the checklist of artifacts used:<\/p>\n Determine 1: P-EAGLE over different strategies on SPEED-BENCH with Concurrency of 1 on one NVIDIA B200 card.<\/p>\n<\/p><\/div>\n You may allow parallel drafting with a single configuration change within the Right here\u2019s an instance command in vLLM to allow parallel drafting with P-EAGLE as drafter:<\/p>\n EAGLE achieves 2\u20133\u00d7 speedups over commonplace autoregressive decoding and is extensively deployed in manufacturing inference frameworks together with vLLM, SGLang, and TensorRT-LLM.EAGLE drafts tokens autoregressively. To provide Okay draft tokens, it requires Okay ahead passes by way of the draft mannequin. As drafter fashions get higher at drafting lengthy outputs, this drafting overhead turns into important\u2014the drafter\u2019s latency scales linearly with hypothesis depth, constraining how aggressively we will speculate.<\/p>\n We current P-EAGLE, which transforms EAGLE from autoregressive to parallel draft era. On B200 GPUs, P-EAGLE achieves 1.05\u00d7\u20131.69\u00d7 speedup over vanilla EAGLE-3 on GPT-OSS 20B over MT-Bench, HumanEval, and SpeedBench. It’s now built-in into vLLM to unlock parallel speculative decoding, and able to speed up real-world deployments.<\/p>\n P-EAGLE generates the Okay draft tokens in a single ahead go. Determine 2 exhibits the structure, which consists of two steps.<\/p>\n Step 1: Prefilling<\/strong>. The goal mannequin processes the immediate and generates a brand new token, as it could throughout regular inference. Alongside the way in which, P-EAGLE captures the mannequin\u2019s inner hidden states: Step 2: P-EAGLE Drafter<\/strong>. The drafter constructs inputs for every place in parallel. Every enter consists of a token embedding concatenated with a hidden state.<\/p>\n For immediate positions, the enter pairs every immediate token embedding For place 1, Subsequent-Token-Prediction (NTP), the enter pairs the newly generated token embedding emb(new) with Positions go collectively by way of N transformer layers, then by way of the language mannequin head to foretell draft tokens t1, t2, t3, and t4 in a single ahead go.<\/p>\n Determine 2: P-EAGLE structure overview.<\/p>\n<\/p><\/div>\n Trendy reasoning fashions produce lengthy outputs. As proven in Determine 3, GPT-OSS 120B generates sequences (together with prompts) with a median size of three,891 tokens and P90 of 10,800 tokens on the UltraChat dataset. Draft fashions should be educated on matching context lengths to be efficient at inference.<\/p>\n Determine 3: Sequence size (immediate + era) distribution on UltraChat dataset with GPT-OSS 120B. Reasoning degree: Medium.<\/p>\n<\/p><\/div>\n A key problem is that parallel drafting amplifies reminiscence necessities throughout coaching. Coaching Okay parallel teams on a sequence of size N creates N \u00d7 Okay complete positions. With N = 8,192 and Okay = 8, a single coaching instance incorporates 65,536 positions. Consideration requires every place to attend to each legitimate place\u201465K \u00d7 65K means over 4 billion parts, consuming 8GB in bf16.<\/p>\n Place sampling [An et al., 2025<\/a>] reduces reminiscence by randomly skipping positions, however skipping too aggressively degrades draft high quality. Gradient accumulation is the usual resolution for memory-constrained coaching, however it splits throughout totally different coaching examples. When a single sequence exceeds reminiscence, there\u2019s nothing to separate.<\/p>\n P-EAGLE introduces a sequence partition algorithm for intra-sequence splitting. The algorithm divides the N \u00d7 Okay place sequence into contiguous chunks, maintains right consideration dependencies throughout chunk boundaries, and accumulates gradients throughout chunks of the identical sequence. For particulars, see the P-EAGLE paper<\/a>.<\/p>\n In lots of speculative decoding setups, drafting and verification share the identical per-request token format. That\u2019s largely true for EAGLE: the drafter consumes a window that already matches what the verifier will test; Okay drafted tokens and one further sampled token.<\/p>\n Parallel drafting breaks that consistency. To foretell Okay tokens in a single drafter ahead go, we append MASK placeholders (for instance, [token, MASK, MASK, \u2026]). These further positions exist just for drafting, so the draft batch form not matches the verification batch form. As a result of we will\u2019t reuse verification metadata, we should rebuild the batch metadata. We increase the enter token IDs, hidden states, and positions to insert slots for masks tokens\/embeddings, increment positions per request, then recompute the slot mapping and per-request begin indices from the up to date positions.<\/p>\n To offset the overhead of rebuilding the batch metadata, we implement a fused Triton kernel that populates the drafter\u2019s enter batch on-GPU by copying and increasing the target-model batch. In a single go, the kernel copies the earlier token IDs and positions from the goal batch into new vacation spot slots and inserts the per-request bonus token sampled by the goal mannequin. It then fills the additional parallel-drafting slots with a particular MASK token ID. Lastly, it generates light-weight metadata: a rejected-token masks, a masked-token masks for parallel drafting slots, new-token indices for sampling draft tokens, and a hidden-state mapping.<\/p>\n This logic would in any other case be many GPU ops (copy\/scatter + insert + fill + masks + remap). Fusing it into one kernel reduces launch overhead and further reminiscence site visitors, retaining the drafting setup low cost.<\/p>\n For EAGLE-based strategies that go hidden states to the draft mannequin, parallel drafting handles populating these fields individually. Since hidden states are considerably bigger than the remainder of the enter batch, we cut up the work: the Triton kernel outputs a mapping, and a devoted copy kernel broadcasts the realized hidden state placeholder into the masks token slots.<\/p>\n The For KV cache slot mapping, legitimate tokens obtain regular slot project whereas rejected tokens are mapped to We practice P-EAGLE on GPT-OSS-20B and consider throughout three benchmarks: MT-Bench<\/a> for multi-turn instruction following, SPEED-Bench<\/a> Code for long-term code era, and HumanEval<\/a> for function-level code synthesis. P-EAGLE delivers 55\u201369% greater throughput at low concurrency (c=1), with positive factors of 5\u201325% sustained at excessive concurrency (c=64), in comparison with the publicly obtainable vanilla EAGLE-3 checkpoint<\/a>. Outcomes are proven in Determine 4-6.<\/p>\n The P-EAGLE drafter is a light-weight 4-layer mannequin educated to foretell as much as 10 tokens in parallel. To judge efficiency, we sweep hypothesis depths Okay \u2208 {3,5,7} throughout concurrency ranges C \u2208 {1,2,4,8,16,32,64}. Our purpose is to establish the suitable deployment configuration for each P-EAGLE and vanilla EAGLE-3. Linear drafting is used for each P-EAGLE and vanilla EAGLE-3. On this context, \u201cfinest P-EAGLE\u201d and \u201cfinest EAGLE-3\u201d confer with the configurations that obtain peak throughput. These are measured in tokens per second (TPS), for a given hypothesis depth Okay. For every technique, we choose Okay that maximizes TPS beneath the given serving situations.<\/p>\n A constant sample emerges. P-EAGLE achieves peak TPS at Okay=7 throughout all concurrency ranges. In distinction, vanilla EAGLE-3 reaches its highest TPS at Okay=3, with its improved depth often shifting towards greater values relying on concurrency. This conduct displays a elementary benefit of parallel drafting. P-EAGLE generates all Okay draft tokens in a single ahead go, permitting it to learn from deeper hypothesis with out incurring further sequential overhead. Autoregressive drafters, in contrast, should generate speculative tokens step-by-step, which limits their capability to effectively scale to bigger Okay.<\/p>\n All experiments are performed on one NVIDIA B200 (Blackwell) GPU utilizing vLLM with the next serving configuration.<\/p>\n\u201cparallel_drafting\u201d: true<\/code> on you vLLM serving pipeline. Pre-trained P-EAGLE heads are already obtainable on HuggingFace for GPT-OSS 120B<\/a>, GPT-OSS 20B<\/a>, and Qwen3-Coder 30B<\/a>, so you can begin as we speak.<\/p>\n![\"Figure](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/03\/12\/ML-20619-image-1.png\")
<\/p>\nFast begin P-EAGLE:<\/strong><\/h2>\n
SpeculativeConfig<\/code> class:<\/p>\n# vllm\/config\/speculative.py\n parallel_drafting: bool = True<\/code><\/pre>\n<\/p><\/div>\n vllm serve openai\/gpt-oss-20b \n --speculative-config '{\"technique\": \"eagle3\", \"mannequin\": \"amazon\/gpt-oss-20b-p-eagle\", \"num_speculative_tokens\": 5, \"parallel_drafting\": true}'<\/code><\/pre>\n<\/p><\/div>\nEAGLE\u2019s Drafting Bottleneck<\/strong><\/h2>\n
Our Strategy: Parallel-EAGLE (P-EAGLE)<\/strong><\/h2>\n
h_prompt<\/code> for every immediate place, and h_context<\/code> for the newly generated token. These hidden states encode what the goal mannequin \u201cis aware of\u201d at every place and can information the drafter\u2019s predictions. This step is similar to autoregressive EAGLE.<\/p>\nemb(p)<\/code> with its corresponding h_prompt<\/code> from the goal mannequin. Following the identical conference as autoregressive EAGLE, positions are shifted by one. Place i receives the token and hidden state from place i-1, enabling it to foretell the token at place i.<\/p>\nh_context<\/code>. This place operates identically to the usual autoregressive EAGLE.For positions 2 by way of Okay, Multi-Token-Prediction (MTP), the required inputs\u2014the token embedding and hidden state\u2014don’t but exist. P-EAGLE fills these with two learnable parameters: a shared masks token embedding emb(masks)<\/code> and a shared hidden state h_shared<\/code>. These are mounted vectors realized throughout coaching that function impartial placeholders.<\/p>\n![\"Figure](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/03\/13\/ml-20619-img1-new.png\")
<\/a><\/p>\nCoaching P-EAGLE on Lengthy Sequences<\/strong><\/h3>\n
![\"Figure](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/03\/13\/ML-20619-IMG2-1.png\")
<\/a><\/p>\nImplementation in vLLM<\/strong><\/h3>\n
Parallel drafting challenges<\/strong><\/h3>\n
The Triton Kernel<\/strong><\/h2>\n
Hidden State Administration<\/strong><\/h2>\n
# Copy goal hidden states to their new positions\n self.hidden_states[out_hidden_state_mapping] = target_hidden_states\n \n# Fill masked positions with the realized Parallel Drafting hidden state\n masks = self.is_masked_token_mask[:total_num_output_tokens]\n torch.the place(\n masks.unsqueeze(1),\n self.parallel_drafting_hidden_state_tensor,\n self.hidden_states[:total_num_output_tokens],\n out=self.hidden_states[:total_num_output_tokens],\n )<\/code><\/pre>\n<\/p><\/div>\nparallel_drafting_hidden_state_tensor<\/code> is loaded from the mannequin\u2019s mask_hidden<\/code> buffer, a realized illustration that tells the mannequin these positions ought to predict future tokens.<\/p>\nPADDING_SLOT_ID (-1)<\/code> to forestall spurious cache writes. For CUDA graphs, we lengthen the seize vary by Okay \u00d7 max_num_seqs<\/code> to accommodate the bigger draft batch launched by parallel drafting.<\/p>\nvLLM Benchmarking on P-EAGLE<\/strong><\/h2>\n
VLLM_USE_FLASHINFER_MOE_MXFP4_MXFP8=1 \nvllm serve openai\/gpt-oss-20b \n --speculative-config '{\n \"technique\": \"eagle3\",\n \"mannequin\": \"amazon\/GPT-OSS-20B-P-EAGLE\",\n \"num_speculative_tokens\": 7,\n \"parallel_drafting\": true}' \n --port 8000 \n --max-num-seqs 1024 \n --max-model-len 100000 \n --max-num-batched-tokens 100000 \n --max-cudagraph-capture-size 4096 \n --no-enable-prefix-caching \n --no-enable-chunked-prefill \n --kv-cache-dtype fp8 \n --async-scheduling \n --stream-interval 20<\/code><\/pre>\n<\/p><\/div>\n
![\"Figure](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/03\/12\/ML-20619-image-4.png\")
<\/p>\n![\"Figure](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/03\/12\/ML-20619-image-5.png\")
<\/p>\n![\"Figure](\"https:\/\/d2908q01vomqb2.cloudfront.net\/f1f836cb4ea6efb2a0b1b99f41ad8b103eff4b59\/2026\/03\/12\/ML-20619-image-6.png\")
<\/p>\n