Programs Engineering Playbook: Optimizing Qwen 3.5-397B MoE on Ironwood (TPU7x)

Govt Abstract

Deploying and serving a Combination-of-Specialists (MoE) mannequin like Qwen3.5-397B on specialised {hardware} accelerators presents vital techniques engineering challenges. Loading the 400 GB weight footprint into Excessive Bandwidth Reminiscence (HBM) and maximizing {hardware} utilization requires a disciplined, first-principles engineering methodology over empirical trial-and-error modifications.

Crucially, because the panorama of open-weights fashions grows in complexity, engineering groups can not afford to spend months optimizing every new mannequin household in isolation. To resolve this scalability problem, our efficiency workforce has pioneered a modular, model-agnostic optimization technique. Relatively than tackling fashions as monolithic techniques, we decompose them into self-contained, impartial constructing blocks (similar to Batched RPA, Grouped GEMMs, and SparseCore unpermutation) accompanied by hardware-aware value fashions. When a brand new structure arrives, these pre-optimized modules are ported with near-zero engineering friction. This enables our engineers to ship state-of-the-art serving efficiency properly forward of preliminary projections, shifting our focus from localized mannequin optimization to world, platform-level scalability.

This technical report particulars how we systematically utilized this world optimization playbook to Qwen 3.5 MoE on the Ironwood (TPU v7x) platform. By leveraging our library of reusable JAX/Pallas kernels and focusing on solely Qwen 3.5’s novel parts—similar to Gated DeltaNet (GDN) linear consideration and Consideration Knowledge Parallelism—our workforce achieved vital efficiency uplift for each decode-heavy and prefill-heavy workloads.

The optimizations mentioned under allowed us to enhance inference efficiency by roughly 3.1x for Decode-heavy and by roughly 4.7x for Prefill-heavy workloads (512 Concurrency tier) between April and June 2026. Moreover, by integrating these modular optimizations natively into open-source serving frameworks like vLLM and SGLang, now we have neutralized legacy software program obstacles, offering a seamless, production-ready migration path for world enterprise workloads at scale.

1. Qwen 3.5 Structure Overview & Mannequin Configuration

The mannequin consists of 397 billion complete parameters, however leverages a extremely sparse routing scheme that prompts precisely 17 billion parameters per token per ahead move. This sparse configuration represents a 4.3% routing activation ratio, enabling the mannequin to ship the expressive capability and intelligence of a 400B-class mannequin whereas sustaining the inference footprint and execution velocity of a a lot smaller 20B-class system.

The official mannequin weights and configuration may be accessed straight by way of the Qwen3.5-397B-A17B Hugging Face Repository. For a complete structural evaluation of Qwen 3.5’s hybrid linear consideration and gating parts, see the technical deep-dives on Qwen3.5: No person Agrees on Consideration Anymore (Hugging Face Weblog) and Gated DeltaNet for Linear Consideration (Sebastian Raschka, PhD).

Architectural Structure & Interleaving

The community consists of 60 layers in complete, hidden dimension D=4096, and a padded vocabulary measurement of 248,320 tokens. Relatively than using a uniform Transformer layer stack, Qwen 3.5 employs a extremely custom-made hybrid format composed of 15 repeating structural blocks. Every block is organized in a 3:1 ratio:

• Gated DeltaNet Layers (75% of layers): 3 consecutive layers that mix Gated DeltaNet (linear consideration) with routed sparse MoE.
• Grouped Question Consideration (GQA) Layers (25% of layers): 1 layer of ordinary GQA coupled with routed sparse MoE.

This repeating sequence may be expressed as:

Key Mathematical Parts

The mannequin’s hybrid nature integrates three distinct mathematical formulations:

1. Gated DeltaNet (GDN)

Customary self-attention mechanisms scale quadratically with sequence size (O(S²)), making a computational bottleneck for long-context era. GDN solves this by computing linear consideration, using 64 linear consideration heads for Values (V) and 16 heads for Queries and Keys (QK) with a head dimension of 128. As an alternative of establishing a pairwise softmax consideration matrix, GDN maintains a constant-sized hidden state matrix per head (matching the d_okayd_v key-value dimensions) that features as a recurrent reminiscence.

At every token step t, the state matrix is up to date utilizing the delta rule:

The place q_t, okay_t, and v_t are the question, key, and worth vectors, and t is a discovered gating parameter. This recurrent replace is preceded by a causal 1D convolution (Okay=4) to seize native spatial dependencies. This recurrent formulation permits the context window to scale linearly (O(S)) in reminiscence, protecting the recurrent state footprint fixed.

2. Grouped Question Consideration (GQA)

To anchor the linear consideration retrieval, the mannequin makes use of normal GQA in 25% of its layers. GQA makes use of 32 question heads (N_q=32) and precisely 2 key-value (KV) heads (N_kv=2) globally, with a head dimension of 256 and a Rotary Place Embedding (RoPE) dimension of 64. This excessive GQA format compresses the KV cache footprint throughout era, however imposes strict hardware-level sharding constraints, as detailed in Part 3.

3. Sparse Combination-of-Specialists (MoE)

The feed-forward community (FFN) layers are sharded into 512 small specialists with an intermediate professional dimension of 1024. Throughout execution, a router gate tasks token representations and selects the highest 10 routed specialists by way of a soft-max chance distribution. Crucially, the mannequin additionally incorporates one shared professional path that’s at all times executed, functioning as a typical illustration layer:

This native multimodal MoE structure natively processes textual content, picture, and video inputs by way of an early-fusion coaching paradigm on trillions of multimodal tokens. The context window helps a local context size of 262,144 tokens, extensible to over 1,010,000 tokens utilizing YaRN RoPE scaling.

2. Benchmark Setup & Workload Configurations

To systematically isolate, profile, and resolve compiler and kernel bottlenecks, the techniques engineering workforce established a rigorous multi-dimensional analysis matrix primarily based on real-world, uneven workloads.

The Multi-Dimensional Analysis Matrix

Our benchmarking sweeps throughout uneven workloads designed to emphasize separate {hardware} execution subsystems:

1. Prefill-Heavy Workloads (8K Enter / 1K Output): Characterised by lengthy enter immediate sequences and quick token era outputs. These workloads are compute-bound and closely stress the bodily floating-point matrix multiplication capability of the TPU’s TensorCore Matrix Execution Models (MXUs).
2. Decode-Heavy Workloads (1K Enter / 8K Output): Characterised by quick enter prompts and extended token-by-token era phases. These workloads are memory-bound, because the system should repeatedly stream all 400 GB of parameters from Excessive Bandwidth Reminiscence (HBM) to the execution cores to generate a single token per request.
3. Concurrency Tiers: To look at system scaling curves and establish {hardware} queuing/reminiscence bottlenecks beneath load, each workloads are evaluated throughout 4 concurrency tiers: 64, 128, 256, and 512 concurrent requests.

Combined-Engine Orchestration & Topology

The benchmark was executed on an enterprise-grade single-host cluster:

• Accelerator Topology: A single bodily host housing 4 bodily Ironwood chips. Every bodily chip consists of two logical chiplets, exposing a logical topology of 8 distinct execution cores (units) interconnected by way of a high-speed, sub-microsecond Inter-Chip Interconnect (ICI) airplane.
• Inference Server Engine: Engineered utilizing vllm-project/tpu-inference. For the ultimate optimized runs using Consideration DP, the server execution loop was configured with --max-num-batched-tokens=1024 and --max-num-seqs=64 per core (in comparison with --max-num-batched-tokens=8192 and --max-num-seqs=512 utilized in early tensor-parallel baselines).
• Metrics Monitoring: Efficiency is tracked and reported as Token-Throughput-per-Chip (TPS/chip), calculated as the entire tokens processed (enter + output) divided by the execution length and the variety of bodily chips (4).

3. Sharding Methods & Distributed Collectives

The precise architectural constraints of Qwen 3.5—specifically, having precisely 2 KV heads within the GQA layers and 512 specialists within the MoE layers—invalidate conventional uniform sharding approaches.

Parallelism Commerce-offs: Tensor Parallelism vs. Knowledge Parallelism

In normal Consideration Tensor-Parallel (TP) + Skilled MoE configurations, consideration weights are sliced and sharded throughout the machine dimension. Nevertheless, trying to shard the GQA layers with a tensor parallelism measurement of 8 (TP=8) forces fractional head sharding (2/8 = 0.25 heads per machine), which is bodily unattainable on {hardware}.

Replicating the heads regionally throughout 8 cores duplicates the bodily KV cache reminiscence footprint on each machine, neutralizing the memory-saving advantages of GQA. This reminiscence redundancy severely restricts the HBM headroom out there for energetic KV caches beneath high-load workloads. This capability limitation forces the server engine to cap the precise achieved concurrency far under anticipated targets—limiting the system to roughly ~200 concurrent requests as a substitute of the deliberate 512.

To remove this bottleneck, we co-designed a hybrid sharding scheme (PR #2577): 8-way Consideration Batch Sharding (Knowledge Parallelism, DP=8) mixed with 8-way Skilled Parallelism (EP=8) within the MoE layers.

Replicating GQA and GDN weights throughout all 8 units permits every core to course of consideration regionally with the complete 2 KV heads, preserving native KV cache consistency and eliminating intra-attention sharding communication. Within the feed-forward MoE layers, we swap to Skilled Parallelism (EP=8). The 512 routed specialists are distributed evenly (64 specialists per machine), which avoids duplicating the 400 GB parameter footprint throughout all nodes whereas protecting collective payload sizes manageable.

Deep Dive into Distributed Collective Sequences

Transitioning between Consideration DP and MoE EP requires cross-device token routing. In designing our Combination-of-Specialists (MoE) routing layer, we evaluated two main structural approaches to deal with this cross-device transition:

• Possibility A (All-to-All Shuffling): This method makes use of an All-to-All -> Native MoE -> All-to-All pipeline. Tokens are dynamically shuffled throughout the community to the precise chips internet hosting their goal specialists, computed regionally, and shuffled again. Whereas this minimizes redundant computation, it incurs huge, unpredictable community routing overhead as a consequence of world All-to-All steps beneath variable workloads.
• Possibility B (Full Token Replication): This method makes use of an All – Collect -> Native MoE -> Scale back-Scatter pipeline. An All-Collect replicates all token vectors throughout all units. Every chip then filters and computes inputs solely for its native specialists, aggregating the outputs later by way of a Scale back-Scatter. This utterly bypasses the unpredictable All-to-All routing penalties at the price of increased native reminiscence consumption.

As a result of deterministic latency is essential for real-world serving, we opted for Possibility B and subsequently developed low-level communication fusions to optimize its collective pathways.

1. The three-to-2 All-Collect Optimization

Beneath a naive Possibility B implementation, getting ready for native MoE computation requires broadcasting three distinct items of information throughout the cluster to each machine rank. Assuming an area tensor slice with a form of [1024,4096] for token hidden dimensions, we sometimes should carry out three separate collective operations:

1. All-Collect 1: The token hidden dimensions ([1024,4096]).
2. All-Collect 2: The chosen professional indices ([1024,10], assuming topk=10).
3. All-Collect 3: The gating topk weights ([1024,10]).

Each collective communication name carries a set kernel launch and community synchronization latency penalty on the TPU. To optimize Skilled Parallelism (EP) effectivity, we consolidated these three All-Gathers down to 2 in PR #2836. As a result of the professional indices (integers) and the topk weights (floats) share an identical tensor shapes ([1024,10]), we stack, bitcast, and pack them collectively alongside a brand new dimension right into a single dense 32-bit integer array (blob). This enables us to run a single All-Collect throughout the info dimension (ShardingAxisName.MLP_DATA) for each routing metadata blocks, unpacking them regionally and halving the routing metadata collective latency.

2. Hierarchical Scale back-Scatter

After professional execution, token outputs should return to their data-parallel ranks. A typical All-Scale back over the 8-device mesh is very inefficient. We changed this with a customized, TPU-native Hierarchical Scale back-Scatter written in Pallas/Mosaic (see PR #2679. The collective runs in two pipelined phases:

• Intra-chip Scale back-Scatter: Logical chiplets on the identical bodily chip change and sum their information utilizing quick, native shared-memory transfers (that are 6x sooner than chip-to-chip ICI bandwidth).
• Inter-chip Scale back-Scatter: Partially decreased information is exchanged throughout bodily chips utilizing a recursive-doubling hypercube algorithm over the TPU’s bodily ICI hyperlinks.

To forestall VMEM Out-of-Reminiscence (OOM) errors, the info is sliced into 2 to 4 micro-batches. The kernel pipelines distant DMA transfers of micro-batch i whereas the TensorCore is performing vector additions for micro-batch i-1, hiding the communication latency behind the compute.

4. Prefill vs. Decode Roofline Evaluation

To establish the theoretical bounds of our techniques engineering and perceive the place execution stalls occurred, we performed a first-principles roofline evaluation for the Qwen 3.5 workload beneath a typical 8K/1K configuration at 64 concurrency.

Ironwood {Hardware} Specs

• Tensor Core (TC) Frequency: 2.2 GHz
• Tensor Cores per chip: 2
• MXUs (Matrix Execution Models) per TC: 2 (complete 4 MXUs per chip)
• Peak BF16 efficiency: 2,307 TFLOPS/chip ((262,144 FLOP/cycle/MXU × 2.2 GHz × 4 MXUs = 2,307 TFLOPS))
• Peak FP8 efficiency: 4,614 TFLOPS/chip

1. Prefill Part (Compute-Sure)

Through the prefill section, a batch of 64 prompts with 8,192 enter tokens every yields 524,288 tokens processed in parallel.

• Arithmetic Depth: The GEMM operations within the projection layers scale quadratically with sequence size and batch measurement. The arithmetic depth (FLOPs/Byte) is extraordinarily excessive, inserting execution deep within the compute-bound regime of the roofline mannequin.
• Operational Boundary: Bounded by the height floating-point execution capability of the TPU v7 TensorCore MXUs (4,614 TFLOPS in FP8).
• Programs Bottlenecks: MXU underutilization happens primarily as a consequence of ragged token distribution throughout specialists. If one professional receives considerably extra tokens than others in a given batch, the corresponding machine turns into a straggler. Minimizing padding in our Grouped GEMM kernels was essential to closing the hole between precise TFLOPS and the theoretical peak.

2. Decode Part (Reminiscence-Sure)

Through the decode section, the mannequin processes 64 tokens per step (1 token per energetic request).

• Arithmetic Depth: To generate one token, the system should stream all 400 GB of mannequin weights from HBM to the processor. The arithmetic depth is near-unit (~1FLOP/Byte), inserting the workload squarely within the memory-bound regime.
• Operational Boundary: Bounded by the HBM reminiscence bandwidth.

• Programs Bottlenecks: The first latency contributors are HBM switch latency for mannequin parameters, VPU indexing stalls throughout sparse KV cache retrieval, and recurrent state replace round-trips within the Gated DeltaNet (GDN) layers.

Quantifying our Roofline Boundaries (BS=64)

To translate these first-principles {hardware} constraints into plannable software program engineering metrics, we modeled our normal analysis workload (Concurrency 64, with an 8K/1K prefill-heavy and 1K/8K decode-heavy sequence size format) utilizing our end-to-end roofline mannequin. This evaluation establishes absolutely the, application-level throughput bounds (in tokens-per-second per bodily chip) for every serving section:

• Prefill-Heavy Part (8K Enter / 1K Output): As a result of the prefill section is compute-bound, it’s restricted by the height floating-point execution capability of the TensorCore Matrix Execution Models (MXUs) (4,614TFLOPS FP8 per chip). Taking into consideration the quadratic scaling of GQA consideration operations over 8,192 tokens and normal {hardware} execution overheads, our mannequin establishes an estimated most theoretical roofline throughput of 5,170 tokens/s/chip (undiscounted), and 4,500 tokens/s/chip beneath normal scheduling de-rate components.
• Decode-Heavy Part (1K Enter / 8K Output): As a result of producing one token per energetic stream is memory-bound, efficiency is strictly restricted by the HBM interface bandwidth. The entire execution latency throughout all 60 layers is calculated at 16.36 ms per token step. This leads to a peak theoretical throughput of 978 tokens/s/chip (undiscounted) and a practical, discounted serving roofline restrict of 850 tokens/s/chip.

Kernel Optimizations

“By authoring customized kernels utilizing the JAX customized kernel language, Pallas, we bypassed the usual XLA decreasing path to manage VMEM format, registers, and reminiscence scheduling straight throughout the three main execution tracks:

By authoring customized kernels utilizing the JAX customized kernel language, Pallas, we bypassed the usual XLA decreasing path to manage VMEM format, registers, and reminiscence scheduling straight.

A. Consideration Monitor: Ragged Web page Consideration (RPA)

Managing the KV cache for the 25% GQA layers requires dynamic reminiscence allocation. We make use of Ragged Web page Consideration (RPA) to index non-contiguous reminiscence blocks in HBM (see #PR 2632).

1. KV Web page Dimension Tuning

Traditionally, a block measurement of 16 tokens was used to reduce reminiscence fragmentation. Nevertheless, on TPU, smaller block sizes lead to huge indexing overhead, inflicting the Vector Processing Unit (VPU) to stall throughout the decode section. We resolved this by coarse-graining the indexing to a KV web page measurement of 256 (enabled by way of the server command --block-size=256). This coarse-grained indexing decreased the decode step latency beneath Concurrency-512 from 428µs to 283µs, reaching a 33.8% kernel-level speedup.

2. Batched RPA

To additional saturate the reminiscence bus, we designed batched RPA kernels. This design teams a number of decode streams collectively right into a single compiled Pallas kernel (#PR 2632), amortizing VPU instruction dispatch latency, breaking the info dependency stalls of sequential requests, and enhancing reminiscence alignment.

B. MoE Monitor: SparseCore & TensorCore Co-Design

The fine-grained routing issue of top_k=10 in Qwen 3.5 introduces non-power-of-two tensor dimensions. Permuting and unpermuting these arrays on the TensorCore beforehand resulted in closely padded, unaligned HBM reminiscence writes. We resolved this by way of a SparseCore-TensorCore co-design circulate:

1. Customized SparseCore Ragged Collect Kernel

We authored a customized Pallas/Mosaic kernel that offloads token routing to the TPU’s SparseCore (SC), a {hardware} unit optimized for oblique addressing (see PR #2137). The SC reads the routing indices, performs an oblique DMA collect of token embeddings straight from HBM, and writes them right into a contiguous digital buffer. This bypasses the materialization of heavily-padded, unaligned intermediate tensors in HBM, saving huge reminiscence bandwidth.

2. Grouped GEMM (GMM) V2 with Fused Activation

Within the GMM V2 kernel, we fused the SwiGLU activation features straight into the principle matrix multiplication loops (gating and up-projection are packed and processed in a single tile by way of twin DMA reads), avoiding register spills to HBM. Moreover, we carried out dynamic bounded slices to course of the variable token payloads of every professional with minimal padding. We transitioned to 512 subchannel activation quantization for FP8 operations to remove VREG spills and reminiscence load stalls, doubling vector arithmetic throughput on the VPU.

3. Fused Ragged Collect Scale back Kernel

Offloads the token-unpermutation and native discount operations totally to the SparseCore. By performing oblique collect and native discount straight on the SC, we bypassed the materialization of intermediate, padded activation tensors in HBM, lowering the HBM learn necessities from 20 right down to 10 and writes from 15 down to five, slashing MoE overhead.

To maximise {hardware} effectivity, our implementation leverages a chunk-level pipelined structure reasonably than performing native discount and the 8-device Scale back-Scatter sequentially on the complete [81920,4096] tensor. The workload is partitioned into 4 distinct chunks. As quickly as Chunk 1 completes its native unpermutation and gather-reduce on the SparseCore, it asynchronously kicks off its Scale back-Scatter collective throughout bodily ICI hyperlinks. Concurrently, the SparseCore begins the native gather-reduce for Chunk 2. This strict chunk-level pipelining successfully overlaps and hides the cross-device community latency of the Scale back-Scatter behind the native compute of subsequent chunks.

C. GDN Monitor: Gated DeltaNet Optimization

The recurrent state updates within the 75% Gated DeltaNet (GDN) layers are extremely prone to reminiscence bandwidth bottlenecks as a consequence of fixed recurrent state updates

To optimize this monitor, the efficiency workforce carried out a sequence of algorithmic fusions and precision co-designs:

1. Causal Conv1D Fusion

The GDN recurrent replace is preceded by a causal 1D convolution (Okay=4). Initially, this was compiled as an impartial operation, forcing the intermediate convolution outputs to be written to and browse from HBM. We designed a register-level sliding window algorithm that caches historic token states straight throughout the TPU’s VPU registers. Fusing the 1D convolution and the GDN recurrent state replace right into a single execution block eradicated 6 redundant HBM round-trips (see PR #2823).

2. Algebraic Identification Optimizations

We restructured the linear consideration replace equations to take advantage of algebraic identities. By mathematically rearranging the operations, we utterly skipped the costly post-rank-1 matrix multiplication within the fused GDN kernels, lowering the computational footprint (see PR #2498).

Moreover, to additional saturate the Vector Processing Unit (VPU), we transitioned the recurrent State House Mannequin (SSM) state variables from Float32 to BFloat16 precision. This doubled the vector arithmetic throughput on the VPU with out compromising numerical convergence or output high quality.

3. Ragged Sequence Dealing with & Chunked GDN

To forestall padding overhead from losing MXU FLOPs throughout batched prefill execution, we optimized JAX-native chunked layouts in and launched specialised sequence-handling routines that natively course of ragged inputs in PR #2218, guaranteeing that variable sequence lengths don’t introduce processing stragglers.

4. Absolutely-Fused Conv1D and GDN Kernel

Relatively than counting on separate execution phases, we designed and merged a fully-fused Pallas kernel in PR #3016 that compiles the causal 1D convolution and your entire GDN recurrent linear consideration block right into a single, unified execution unit on the VPU. By caching intermediate sequence and recurrent states straight throughout the native registers, this kernel utterly bypasses the necessity to learn and write intermediate activation tensors to VMEM or HBM.

This register-level fusion eliminates register-to-memory synchronization latency and gives a essential efficiency enhance for each serving phases:

• Prefill Part: It considerably reduces the reminiscence bandwidth footprint when processing lengthy enter sequence prompts, maximizing TensorCore MXU floating-point effectivity.
• Decode Part: It eliminates memory-bound round-trip stalls throughout extended token-by-token era.

D. Reminiscence Monitor: Hybrid Consideration KV Structure Optimization

Serving Qwen 3.5 requires managing two heterogeneous consideration state constructions: the fixed-size recurrent linear consideration state of Gated DeltaNet (GDN) and the dynamically rising normal consideration Key-Worth (KV) cache of Grouped Question Consideration (GQA). As a result of the TPU v7 options 192GB of HBM capability per chip (e.g. in comparison with the 288GB out there on Blackwell GB300 GPUs – a ~50% capability distinction), HBM footprint optimization beneath excessive concurrency is a extreme techniques constraint. In PR #2416, we launched a customized reminiscence format designed to align and retailer these hybrid consideration states collectively in HBM. This format minimizes padding and prevents reminiscence fragmentation, straight reclaiming essential HBM headroom. This optimization will increase the utmost supportable batch sizes, permitting the TPU to scale easily and maintain excessive serving throughput beneath heavy consumer concurrencies.

6. Present Efficiency Outcomes

The techniques engineering optimizations have been validated beneath rigorous, empirical benchmarking situations. Under, we current the uncooked throughput outcomes with our optimized JAX/Pallas stack on Ironwood TPU throughout 4 concurrency tiers.

Closing the Headroom Hole: Precise Throughput vs. Roofline Limits

Putting our empirical serving outcomes side-by-side with our first-principles roofline limits on the baseline Concurrency 64 tier demonstrates the real-world effectivity of our customized Pallas kernels and DP+EP sharding topology:

• Prefill-Heavy Effectivity: Beneath the 8K/1K prefill-heavy workload, our JAX serving stack delivers an precise throughput of 3,707 tokens/s/chip. In comparison with our estimated prefill roofline restrict of 4,500 tokens/s/chip (discounted), our customized SparseCore and TensorCore co-designed GEMMs efficiently extract 82.4% of absolutely the compute capability of the TPU v7 TensorCores.
• Decode-Heavy Effectivity: Beneath the 1K/8K decode-heavy workload, our stack delivers an precise throughput of 677 tokens/s/chip. In comparison with our memory-bound decode roofline restrict of 850 tokens/s/chip (discounted), our Ragged Web page Consideration (RPA) and Gated DeltaNet (GDN) fusions efficiently obtain 79.6% of the theoretical HBM bandwidth restrict.

This shut alignment demonstrates that our low-level compiler and kernel fusions push the TPU {hardware} near its bodily execution limits, leaving minimal remaining headroom and proving the intense effectivity of the open-source software program stack.

Rigorous Numerical Verification & Correctness

Working large-scale Combination-of-Specialists fashions at scale requires not simply uncooked throughput, however strict mathematical correctness. Beneath excessive concurrency, gating and routing matrices are extremely delicate to low-precision accumulation errors. In designing our customized JAX/Pallas gating kernels, the techniques engineering workforce integrated a devoted Numerical Verification Layer to audit accumulation precision throughout our FP8 scaling blocks. By repeatedly monitoring the softmax distribution ranges and professional load balances, we verified that our Pallas-lowered gating weights keep zero deviation from the high-precision Float32 reference path (see PR #2328 and PR #2674), guaranteeing excessive throughput alongside strict output high quality.

7. Future Optimizations Roadmap

To remove the remaining bottlenecks, our engineering workforce has structured an energetic optimization roadmap divided into two main technical tracks:

1. Collectives Optimization Monitor

• Low-Bandwidth FP8 All-Collect Collectives: We’re designing low-bandwidth FP8 collectives for the Token/Metadata All-Collect step. Quantizing the routing metadata to FP8 previous to cross-node transmission will halve the communication quantity over the bodily ICI hyperlinks, straight lowering the routing latency barrier.
• Hierarchical Scale back-Scatter Tuning: We’ll proceed to refine the block sizes and micro-batch pipelining parameters throughout the customized Hierarchical Scale back-Scatter kernel. Particularly, we goal to implement dynamic, token-dependent micro-batch sizing to optimize bandwidth utilization beneath variable routing distributions.

2. Kernel & Gating Fusion Monitor

• Router Gate & High-Okay Fusion: We plan to fuse the routing gate projection and the next top_k choice kernel straight on the VPU. At present, routing logits are computed on the TensorCore and transferred to the VPU for top_k choice, introducing a serialization bottleneck. Fusing these operations will preserve the routing pipeline native to the VPU.

Conclusion

Optimizing huge open-weights fashions like Qwen 3.5 on trendy accelerators is a problem that can not be solved by brute-force empirical modifications. It requires a disciplined, first-principles techniques engineering method: mapping theoretical {hardware} limits by way of Roofline Modeling, systematically isolating bottlenecks by way of profile traces, and bridging the remaining gaps with hand-scheduled customized Pallas kernels and hardware-aware sharding topologies.

The optimization playbook developed for this deployment does extra than simply speed up Qwen 3.5; it establishes a hardened, reusable open-source software program stack that makes Google Cloud TPUs a extremely general-purpose and aggressive engine for the following era of sparse Combination-of-Specialists architectures.

To discover our full technical guides, entry ready-to-run code templates, and study extra about optimizing frontier-class fashions on Google Cloud {hardware}, take a look at our new Google Cloud TPU Developer Hub.

Acknowledgements

This optimization work was an enormous collaborative effort throughout a number of engineering, compiler, and administration groups. We wish to categorical our gratitude to the next contributors who drove this initiative to success:

• Engineering Crew: Alyssa Nie, Amy Zhang, Clemens Schaefer, Daniel Ning, Daybreak Han, Donghyun Cho, Gai Liu, George Polovets, Guangxiang Du, Guowei Jiang, Haowen Ning, Jacob Platin, Jaehong Kim, Jevin Jiang, Jiaxin Cao, Kunjan Patel, Kyuyeun Kim, Liqun Cheng, Mani Ananth, Ming Liu, Muhuan Huang, Patrick Ji, Pritha Doddahosahally Narayanappa, Qi Zhou, Qiliang Cui, Renee Zhu, Sanjay Gupta, Seher Ellis, Songyi Han, Srinath Mandalapu, Tomas Longeri, Vipan Nalla, Wangyuan Zhang, WenXin Dong, Wonpyo Park, Xiongfei Wei, Yijia Jin, Yixiu Liu, Yuyan Peng.
• Program & Product Crew: Brittany Rockwell, Krunal Sharma, Sayce Falk, Santosh D, Vivek Sharma and Max Sapo.