![\"Gemini_Generated_Image_rhl8wrrhl8wrrhl8\"\/](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Gemini_Generated_Image_rhl8wrrhl8wrrhl8.original.jpg\")
<\/p>\n
Govt Abstract<\/b><\/h3>\n
Deploying and serving a Combination-of-Specialists (MoE) mannequin like Qwen3.5-397B on specialised {hardware} accelerators presents vital techniques engineering challenges. Loading the 400 GB weight footprint into Excessive Bandwidth Reminiscence (HBM) and maximizing {hardware} utilization requires a disciplined, first-principles engineering methodology over empirical trial-and-error modifications.<\/p>\n
Crucially, because the panorama of open-weights fashions grows in complexity, engineering groups can not afford to spend months optimizing every new mannequin household in isolation. To resolve this scalability problem, our efficiency workforce has pioneered a modular, model-agnostic optimization technique. Relatively than tackling fashions as monolithic techniques, we decompose them into self-contained, impartial constructing blocks (similar to Batched RPA, Grouped GEMMs, and SparseCore unpermutation) accompanied by hardware-aware value fashions. When a brand new structure arrives, these pre-optimized modules are ported with near-zero engineering friction. This enables our engineers to ship state-of-the-art serving efficiency properly forward of preliminary projections, shifting our focus from localized mannequin optimization to world, platform-level scalability.<\/p>\n
This technical report particulars how we systematically utilized this world optimization playbook to Qwen 3.5 MoE on the Ironwood (TPU v7x) platform. By leveraging our library of reusable JAX\/Pallas kernels and focusing on solely Qwen 3.5\u2019s novel parts\u2014similar to Gated DeltaNet (GDN) linear consideration and Consideration Knowledge Parallelism\u2014our workforce achieved vital efficiency uplift for each decode-heavy and prefill-heavy workloads.<\/p>\n
The optimizations mentioned under allowed us to enhance inference efficiency by roughly 3.1x for Decode-heavy and by roughly 4.7x for Prefill-heavy workloads (512 Concurrency tier) between April and June 2026. Moreover, by integrating these modular optimizations natively into open-source serving frameworks like vLLM and SGLang, now we have neutralized legacy software program obstacles, offering a seamless, production-ready migration path for world enterprise workloads at scale.<\/p>\n<\/div>\n
![\"Qwen](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Qwen_3.5_497B_Throughput_Chart.original.png\")
<\/p><\/div><\/div>\n
1. Qwen 3.5 Structure Overview & Mannequin Configuration<\/b><\/h2>\n
The mannequin consists of 397 billion complete parameters, however leverages a extremely sparse routing scheme that prompts precisely 17 billion parameters per token per ahead move. This sparse configuration represents a 4.3% routing activation ratio, enabling the mannequin to ship the expressive capability and intelligence of a 400B-class mannequin whereas sustaining the inference footprint and execution velocity of a a lot smaller 20B-class system.<\/p>\n
The official mannequin weights and configuration may be accessed straight by way of the Qwen3.5-397B-A17B Hugging Face Repository<\/a>. For a complete structural evaluation of Qwen 3.5’s hybrid linear consideration and gating parts, see the technical deep-dives on Qwen3.5: No person Agrees on Consideration Anymore (Hugging Face Weblog)<\/a> and Gated DeltaNet for Linear Consideration (Sebastian Raschka, PhD)<\/a>.<\/p>\n The community consists of 60 layers in complete, hidden dimension D=4096, and a padded vocabulary measurement of 248,320 tokens. Relatively than using a uniform Transformer layer stack, Qwen 3.5 employs a extremely custom-made hybrid format composed of 15 repeating structural blocks. Every block is organized in a 3:1 ratio:<\/p>\n This repeating sequence may be expressed as:<\/p>\n<\/div>\n Key Mathematical Parts<\/p>\n The mannequin’s hybrid nature integrates three distinct mathematical formulations:<\/p>\n Customary self-attention mechanisms scale quadratically with sequence size (O(S<\/i>2<\/sup><\/i>)<\/i>), making a computational bottleneck for long-context era. GDN solves this by computing linear consideration, using 64 linear consideration heads for Values (V) and 16 heads for Queries and Keys (QK) with a head dimension of 128. As an alternative of establishing a pairwise softmax consideration matrix, GDN maintains a constant-sized hidden state matrix per head (matching the d<\/i>okay<\/sub><\/i>d<\/i>v<\/sub><\/i> key-value dimensions) that features as a recurrent reminiscence.<\/p>\n At every token step t<\/i>, the state matrix is up to date utilizing the delta rule:<\/p>\n<\/div>\n The place q<\/i>t<\/sub><\/i>, okay<\/i>t<\/sub><\/i>, and v<\/i>t<\/sub><\/i> are the question, key, and worth vectors, and t is a discovered gating parameter. This recurrent replace is preceded by a causal 1D convolution (Okay<\/i>=4) to seize native spatial dependencies. This recurrent formulation permits the context window to scale linearly (O(S)<\/i>) in reminiscence, protecting the recurrent state footprint fixed.<\/p>\n To anchor the linear consideration retrieval, the mannequin makes use of normal GQA in 25% of its layers. GQA makes use of 32 question heads (N<\/i>q<\/sub><\/i>=32) and precisely 2 key-value (KV) heads (N<\/i>kv<\/sub><\/i>=2) globally, with a head dimension of 256 and a Rotary Place Embedding (RoPE) dimension of 64. This excessive GQA format compresses the KV cache footprint throughout era, however imposes strict hardware-level sharding constraints, as detailed in Part 3.<\/p>\n The feed-forward community (FFN) layers are sharded into 512 small specialists with an intermediate professional dimension of 1024. Throughout execution, a router gate tasks token representations and selects the highest 10 routed specialists by way of a soft-max chance distribution. Crucially, the mannequin additionally incorporates one shared professional path that’s at all times executed, functioning as a typical illustration layer:<\/p>\n<\/div>\n This native multimodal MoE structure natively processes textual content, picture, and video inputs by way of an early-fusion coaching paradigm on trillions of multimodal tokens. The context window helps a local context size of 262,144 tokens, extensible to over 1,010,000 tokens utilizing YaRN RoPE scaling.<\/p>\n<\/div>\n To systematically isolate, profile, and resolve compiler and kernel bottlenecks, the techniques engineering workforce established a rigorous multi-dimensional analysis matrix primarily based on real-world, uneven workloads.<\/p>\n Our benchmarking sweeps throughout uneven workloads designed to emphasize separate {hardware} execution subsystems:<\/p>\n The benchmark was executed on an enterprise-grade single-host cluster:<\/p>\n The precise architectural constraints of Qwen 3.5\u2014specifically, having precisely 2 KV heads within the GQA layers and 512 specialists within the MoE layers\u2014invalidate conventional uniform sharding approaches.<\/p>\n In normal Consideration Tensor-Parallel (TP) + Skilled MoE configurations, consideration weights are sliced and sharded throughout the machine dimension. Nevertheless, trying to shard the GQA layers with a tensor parallelism measurement of 8 (TP=8) forces fractional head sharding (2\/8 = 0.25 heads per machine), which is bodily unattainable on {hardware}.<\/p>\n Replicating the heads regionally throughout 8 cores duplicates the bodily KV cache reminiscence footprint on each machine, neutralizing the memory-saving advantages of GQA. This reminiscence redundancy severely restricts the HBM headroom out there for energetic KV caches beneath high-load workloads. This capability limitation forces the server engine to cap the precise achieved concurrency far under anticipated targets\u2014limiting the system to roughly ~200 concurrent requests as a substitute of the deliberate 512.<\/p>\n To remove this bottleneck, we co-designed a hybrid sharding scheme (PR #2577<\/a>): 8-way Consideration Batch Sharding (Knowledge Parallelism, DP=8) mixed with 8-way Skilled Parallelism (EP=8) within the MoE layers.<\/p>\n<\/div>\n Replicating GQA and GDN weights throughout all 8 units permits every core to course of consideration regionally with the complete 2 KV heads, preserving native KV cache consistency and eliminating intra-attention sharding communication. Within the feed-forward MoE layers, we swap to Skilled Parallelism (EP=8). The 512 routed specialists are distributed evenly (64 specialists per machine), which avoids duplicating the 400 GB parameter footprint throughout all nodes whereas protecting collective payload sizes manageable.<\/p>\n Transitioning between Consideration DP and MoE EP requires cross-device token routing. In designing our Combination-of-Specialists (MoE) routing layer, we evaluated two main structural approaches to deal with this cross-device transition:<\/p>\n As a result of deterministic latency is essential for real-world serving, we opted for Possibility B<\/b> and subsequently developed low-level communication fusions to optimize its collective pathways.<\/p>\n Beneath a naive Possibility B implementation, getting ready for native MoE computation requires broadcasting three distinct items of information throughout the cluster to each machine rank. Assuming an area tensor slice with a form of [1024,4096] for token hidden dimensions, we sometimes should carry out three separate collective operations:<\/p>\n Each collective communication name carries a set kernel launch and community synchronization latency penalty on the TPU. To optimize Skilled Parallelism (EP) effectivity, we consolidated these three All-Gathers down to 2 in PR #2836<\/a>. As a result of the professional indices (integers) and the topk weights (floats) share an identical tensor shapes ([1024,10]), we stack, bitcast, and pack them collectively alongside a brand new dimension right into a single dense 32-bit integer array (blob). This enables us to run a single All-Collect throughout the info dimension ( After professional execution, token outputs should return to their data-parallel ranks. A typical All-Scale back over the 8-device mesh is very inefficient. We changed this with a customized, TPU-native Hierarchical Scale back-Scatter written in Pallas\/Mosaic (see PR #2679<\/a>. The collective runs in two pipelined phases:<\/p>\n To forestall VMEM Out-of-Reminiscence (OOM) errors, the info is sliced into 2 to 4 micro-batches. The kernel pipelines distant DMA transfers of micro-batch i whereas the TensorCore is performing vector additions for micro-batch i<\/i>-1, hiding the communication latency behind the compute.<\/p>\n<\/div>\n To establish the theoretical bounds of our techniques engineering and perceive the place execution stalls occurred, we performed a first-principles roofline evaluation for the Qwen 3.5 workload beneath a typical 8K\/1K configuration at 64 concurrency.<\/p>\n Through the prefill section, a batch of 64 prompts with 8,192 enter tokens every yields 524,288 tokens processed in parallel.<\/p>\n Through the decode section, the mannequin processes 64 tokens per step (1 token per energetic request).<\/p>\n To translate these first-principles {hardware} constraints into plannable software program engineering metrics, we modeled our normal analysis workload (Concurrency 64, with an 8K\/1K prefill-heavy and 1K\/8K decode-heavy sequence size format) utilizing our end-to-end roofline mannequin. This evaluation establishes absolutely the, application-level throughput bounds (in tokens-per-second per bodily chip) for every serving section:<\/p>\n “By authoring customized kernels utilizing the JAX customized kernel language, Pallas, we bypassed the usual XLA decreasing path to manage VMEM format, registers, and reminiscence scheduling straight throughout the three main execution tracks:<\/p>\n By authoring customized kernels utilizing the JAX customized kernel language, Pallas<\/a>, we bypassed the usual XLA decreasing path to manage VMEM format, registers, and reminiscence scheduling straight.<\/p>\n Managing the KV cache for the 25% GQA layers requires dynamic reminiscence allocation. We make use of Ragged Web page Consideration (RPA) to index non-contiguous reminiscence blocks in HBM (see #PR 2632<\/a>).<\/p>\n Traditionally, a block measurement of 16 tokens was used to reduce reminiscence fragmentation. Nevertheless, on TPU, smaller block sizes lead to huge indexing overhead, inflicting the Vector Processing Unit (VPU) to stall throughout the decode section. We resolved this by coarse-graining the indexing to a KV web page measurement of 256 (enabled by way of the server command To additional saturate the reminiscence bus, we designed batched RPA kernels. This design teams a number of decode streams collectively right into a single compiled Pallas kernel (#PR 2632<\/a>), amortizing VPU instruction dispatch latency, breaking the info dependency stalls of sequential requests, and enhancing reminiscence alignment.<\/p>\n The fine-grained routing issue of top_k=10<\/i> in Qwen 3.5 introduces non-power-of-two tensor dimensions. Permuting and unpermuting these arrays on the TensorCore beforehand resulted in closely padded, unaligned HBM reminiscence writes. We resolved this by way of a SparseCore-TensorCore<\/b> co-design circulate:<\/p>\n<\/div>\n We authored a customized Pallas\/Mosaic kernel that offloads token routing to the TPU’s SparseCore (SC), a {hardware} unit optimized for oblique addressing (see PR #2137<\/a>). The SC reads the routing indices, performs an oblique DMA collect of token embeddings straight from HBM, and writes them right into a contiguous digital buffer. This bypasses the materialization of heavily-padded, unaligned intermediate tensors in HBM, saving huge reminiscence bandwidth.<\/p>\n Within the GMM V2 kernel, we fused the SwiGLU activation features straight into the principle matrix multiplication loops (gating and up-projection are packed and processed in a single tile by way of twin DMA reads), avoiding register spills to HBM. Moreover, we carried out dynamic bounded slices to course of the variable token payloads of every professional with minimal padding. We transitioned to 512 subchannel activation quantization for FP8 operations to remove VREG spills and reminiscence load stalls, doubling vector arithmetic throughput on the VPU.<\/p>\n Offloads the token-unpermutation and native discount operations totally to the SparseCore. By performing oblique collect and native discount straight on the SC, we bypassed the materialization of intermediate, padded activation tensors in HBM, lowering the HBM learn necessities from 20 right down to 10 and writes from 15 down to five, slashing MoE overhead.<\/p>\n To maximise {hardware} effectivity, our implementation leverages a chunk-level pipelined structure reasonably than performing native discount and the 8-device Scale back-Scatter sequentially on the complete [81920,4096] tensor. The workload is partitioned into 4 distinct chunks. As quickly as Chunk 1 completes its native unpermutation and gather-reduce on the SparseCore, it asynchronously kicks off its Scale back-Scatter collective throughout bodily ICI hyperlinks. Concurrently, the SparseCore begins the native gather-reduce for Chunk 2. This strict chunk-level pipelining successfully overlaps and hides the cross-device community latency of the Scale back-Scatter behind the native compute of subsequent chunks.<\/p>\n The recurrent state updates within the 75% Gated DeltaNet (GDN) layers are extremely prone to reminiscence bandwidth bottlenecks as a consequence of fixed recurrent state updates<\/p>\n<\/div>\n To optimize this monitor, the efficiency workforce carried out a sequence of algorithmic fusions and precision co-designs:<\/p>\n The GDN recurrent replace is preceded by a causal 1D convolution (Okay<\/i>=4). Initially, this was compiled as an impartial operation, forcing the intermediate convolution outputs to be written to and browse from HBM. We designed a register-level sliding window algorithm that caches historic token states straight throughout the TPU’s VPU registers. Fusing the 1D convolution and the GDN recurrent state replace right into a single execution block eradicated 6 redundant HBM round-trips (see PR #2823<\/a>).<\/p>\n We restructured the linear consideration replace equations to take advantage of algebraic identities. By mathematically rearranging the operations, we utterly skipped the costly post-rank-1 matrix multiplication within the fused GDN kernels, lowering the computational footprint (see PR #2498<\/a>).<\/p>\n Moreover, to additional saturate the Vector Processing Unit (VPU), we transitioned the recurrent State House Mannequin (SSM) state variables from Float32 to BFloat16 precision. This doubled the vector arithmetic throughput on the VPU with out compromising numerical convergence or output high quality.<\/p>\n To forestall padding overhead from losing MXU FLOPs throughout batched prefill execution, we optimized JAX-native chunked layouts in and launched specialised sequence-handling routines that natively course of ragged inputs in PR #2218<\/a>, guaranteeing that variable sequence lengths don’t introduce processing stragglers.<\/p>\n Relatively than counting on separate execution phases, we designed and merged a fully-fused Pallas kernel in PR #3016<\/a> that compiles the causal 1D convolution and your entire GDN recurrent linear consideration block right into a single, unified execution unit on the VPU. By caching intermediate sequence and recurrent states straight throughout the native registers, this kernel utterly bypasses the necessity to learn and write intermediate activation tensors to VMEM or HBM.<\/p>\n This register-level fusion eliminates register-to-memory synchronization latency and gives a essential efficiency enhance for each serving phases:<\/p>\n Serving Qwen 3.5 requires managing two heterogeneous consideration state constructions: the fixed-size recurrent linear consideration state of Gated DeltaNet (GDN) and the dynamically rising normal consideration Key-Worth (KV) cache of Grouped Question Consideration (GQA). As a result of the TPU v7 options 192GB of HBM capability per chip (e.g. in comparison with the 288GB out there on Blackwell GB300 GPUs – a ~50% capability distinction), HBM footprint optimization beneath excessive concurrency is a extreme techniques constraint. In PR #2416<\/a>, we launched a customized reminiscence format designed to align and retailer these hybrid consideration states collectively in HBM. This format minimizes padding and prevents reminiscence fragmentation, straight reclaiming essential HBM headroom. This optimization will increase the utmost supportable batch sizes, permitting the TPU to scale easily and maintain excessive serving throughput beneath heavy consumer concurrencies.<\/p>\n<\/div>\n The techniques engineering optimizations have been validated beneath rigorous, empirical benchmarking situations. Under, we current the uncooked throughput outcomes with our optimized JAX\/Pallas stack on Ironwood TPU throughout 4 concurrency tiers.<\/p>\n<\/div>\n Closing the Headroom Hole: Precise Throughput vs. Roofline Limits<\/i><\/p>\n Putting our empirical serving outcomes side-by-side with our first-principles roofline limits on the baseline Concurrency 64<\/b> tier demonstrates the real-world effectivity of our customized Pallas kernels and DP+EP sharding topology:<\/p>\n This shut alignment demonstrates that our low-level compiler and kernel fusions push the TPU {hardware} near its bodily execution limits, leaving minimal remaining headroom and proving the intense effectivity of the open-source software program stack.<\/p>\n Working large-scale Combination-of-Specialists fashions at scale requires not simply uncooked throughput, however strict mathematical correctness. Beneath excessive concurrency, gating and routing matrices are extremely delicate to low-precision accumulation errors. In designing our customized JAX\/Pallas gating kernels, the techniques engineering workforce integrated a devoted Numerical Verification Layer<\/b> to audit accumulation precision throughout our FP8 scaling blocks. By repeatedly monitoring the softmax distribution ranges and professional load balances, we verified that our Pallas-lowered gating weights keep zero deviation from the high-precision Float32 reference path (see PR #2328<\/a> and PR #2674<\/a>), guaranteeing excessive throughput alongside strict output high quality.<\/p>\n<\/div>\n To remove the remaining bottlenecks, our engineering workforce has structured an energetic optimization roadmap divided into two main technical tracks:<\/p>\n Optimizing huge open-weights fashions like Qwen 3.5 on trendy accelerators is a problem that can not be solved by brute-force empirical modifications. It requires a disciplined, first-principles techniques engineering method: mapping theoretical {hardware} limits by way of Roofline Modeling, systematically isolating bottlenecks by way of profile traces, and bridging the remaining gaps with hand-scheduled customized Pallas kernels and hardware-aware sharding topologies.<\/p>\n The optimization playbook developed for this deployment does extra than simply speed up Qwen 3.5; it establishes a hardened, reusable open-source software program stack that makes Google Cloud TPUs a extremely general-purpose and aggressive engine for the following era of sparse Combination-of-Specialists architectures.<\/p>\nArchitectural Structure & Interleaving<\/b><\/h3>\n
\n
![\"Repeating](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Repeating_Sequence_Formula.original.png\")
<\/p><\/div><\/div>\n![\"image](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/image_7.original.png\")
<\/p><\/div><\/div>\n\n
![\"The](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/The_Delta_Rule_Formula.original.png\")
<\/p><\/div><\/div>\n2. Grouped Question Consideration (GQA)<\/i><\/h4>\n
3. Sparse Combination-of-Specialists (MoE)<\/i><\/h4>\n
![\"Representation](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Representation_Layer_Forumla.original.png\")
<\/p><\/div><\/div>\n2. Benchmark Setup & Workload Configurations<\/b><\/h2>\n
The Multi-Dimensional Analysis Matrix<\/h3>\n
\n
Combined-Engine Orchestration & Topology<\/b><\/h3>\n
\n
vllm-project\/tpu-inference<\/code>. For the ultimate optimized runs using Consideration DP, the server execution loop was configured with --max-num-batched-tokens=1024<\/code> and --max-num-seqs=64<\/code> per core (in comparison with --max-num-batched-tokens=8192<\/code> and --max-num-seqs=512<\/code> utilized in early tensor-parallel baselines).<\/li>\n3. Sharding Methods & Distributed Collectives<\/b><\/h2>\n
Parallelism Commerce-offs: Tensor Parallelism vs. Knowledge Parallelism<\/b><\/h3>\n
![\"Attention](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Attention_Layer.original.png\")
<\/p><\/div><\/div>\nDeep Dive into Distributed Collective Sequences<\/b><\/h3>\n
\n
1. The three-to-2 All-Collect Optimization<\/i><\/h4>\n
\n
ShardingAxisName.MLP_DATA<\/code>) for each routing metadata blocks, unpacking them regionally and halving the routing metadata collective latency.<\/p>\n2. Hierarchical Scale back-Scatter<\/i><\/h4>\n
\n
4. Prefill vs. Decode Roofline Evaluation<\/b><\/h2>\n
Ironwood {Hardware} Specs<\/b><\/h3>\n
\n
![\"Theoretical](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Theoretical_Bounds.original.png\")
<\/p><\/div><\/div>\n1. Prefill Part (Compute-Sure)<\/b><\/h3>\n
\n
2. Decode Part (Reminiscence-Sure)<\/b><\/h3>\n
\n
![\"Theoretical](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Theoretical_Decode_Throughput_Formula.original.png\")
<\/p><\/div><\/div>\n\n
Quantifying our Roofline Boundaries (BS=64)<\/b><\/h3>\n
\n
Kernel Optimizations<\/b><\/h2>\n
A. Consideration Monitor: Ragged Web page Consideration (RPA)<\/b><\/h3>\n
1. KV Web page Dimension Tuning<\/i><\/h4>\n
--block-size=256<\/code>). This coarse-grained indexing decreased the decode step latency beneath Concurrency-512<\/code> from 428\u00b5s to 283\u00b5s, reaching a 33.8%<\/b> kernel-level speedup.<\/p>\n2. Batched RPA<\/i><\/h4>\n
B. MoE Monitor: SparseCore & TensorCore Co-Design<\/b><\/h3>\n
![\"SparseCore-TensorCore](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/SparseCore-TensorCore_co-design_flow.original.png\")
<\/p><\/div><\/div>\n1. Customized SparseCore Ragged Collect Kernel<\/i><\/b><\/h4>\n
2. Grouped GEMM (GMM) V2 with Fused Activation<\/i><\/b><\/h4>\n
3. Fused Ragged Collect Scale back Kernel<\/i><\/b><\/h4>\n
C. GDN Monitor: Gated DeltaNet Optimization<\/b><\/h3>\n
<\/p><\/div><\/div>\n1. Causal Conv1D Fusion<\/i><\/b><\/h4>\n
2. Algebraic Identification Optimizations<\/i><\/b><\/h4>\n
3. Ragged Sequence Dealing with & Chunked GDN<\/i><\/b><\/h4>\n
4. Absolutely-Fused Conv1D and GDN Kernel<\/i><\/h4>\n
\n
D. Reminiscence Monitor: Hybrid Consideration KV Structure Optimization<\/b><\/h3>\n
6. Present Efficiency Outcomes<\/b><\/h2>\n
![\"Current](\"https:\/\/storage.googleapis.com\/gweb-developer-goog-blog-assets\/images\/Current_Performance_Results.original.png\")
<\/p><\/div><\/div>\n\n
Rigorous Numerical Verification & Correctness<\/i><\/h4>\n
7. Future Optimizations Roadmap<\/b><\/h2>\n
1. Collectives Optimization Monitor<\/i><\/b><\/h3>\n
\n
2. Kernel & Gating Fusion Monitor<\/i><\/b><\/h3>\n
\n
top_k<\/code> choice kernel straight on the VPU. At present, routing logits are computed on the TensorCore and transferred to the VPU for top_k<\/code> choice, introducing a serialization bottleneck. Fusing these operations will preserve the routing pipeline native to the VPU.<\/li>\n<\/ul>\nConclusion<\/b><\/h2>\n