Guard the Router and Quantization Hurts Less: An Accuracy-Cost Map for Selective NVFP4 Quantization in MoE

This post will be useful for cloud and AI engineers who are serving Mixture-of-Experts (MoE) models on H200 or Blackwell-class clusters, or who are evaluating whether to adopt a quantization policy. The paper introduced today formalizes a policy that, instead of uniformly compressing an entire MoE model to NVFP4 4-bit precision, selectively protects the router (gating network) and low-traffic “rare” experts at full precision while compressing only the rest to 4 bits. It also reports, without hiding it, that the planned empirical experiments could not be run due to an infrastructure fault.

The Problem: Uniform Quantization Assumes Every Parameter Compresses Equally

MoE architectures decouple parameter count from per-token compute by activating only a small subset of expert subnetworks for each token. But in low-batch settings such as interactive serving, the bottleneck is not compute but bandwidth. Because tokens are scattered across many experts, each expert’s weight matrix has to be freshly streamed from HBM just to process a handful of tokens, and the GPU spends most of its time waiting on memory rather than computing. Ultra-low-precision formats like NVFP4, which compress weights to 4 bits, attack this bottleneck head-on, but an approach that quantizes the entire model uniformly implicitly assumes that every parameter block tolerates compression equally well.

The authors argue this assumption is wrong. MoE has two structurally special components. One is the router. Because the router’s output feeds into a discrete routing decision through an argmax, it does not simply absorb noise the way other hidden states do; even a small perturbation to its logits can send a token to a completely different expert. The other is the low-traffic, “rare” experts. These are streamed infrequently and contribute little to overall bandwidth cost, yet the small number of tokens routed to them depend on that expert’s specialized representation. From a traffic-weighted perspective, the router and rare experts account for a small share of the bytes being streamed, so the paper’s core insight is that the cost of protecting them is small while the stability gained is large.

A conceptual traffic distribution based on the Switch-Base-8 MoE architecture. This is not measured data but an illustrative figure to explain the paper’s policy design, showing the per-layer median-split boundary used by RASQ after normalizing traffic rank.

Core Contribution: The RASQ Policy and an Honestly Reframed Comparison Question

The paper calls this policy Router-Aware Selective NVFP4 Quantization, or RASQ for short. The definition is simple. For each MoE layer, experts are ranked by traffic, and using the median as the cutoff, the top half with the most traffic is quantized to 4 bits while the bottom half with the least traffic is kept at full precision (16-bit). The router’s linear layer is always fixed at full precision, without exception. The authors are also explicit that this median split is not a sophisticated method for isolating genuinely “long-tail rare experts” but a coarse approximation that protects half the experts in each layer, and they leave room to extend it into more refined variants, such as protecting only the bottom 10 percent, or into knapsack-style budget-constrained allocation.

To make this policy actually comparable, the paper presents a storage-bit cost model as a formula. Layers quantized to 4 bits are computed as 4-bit weights plus per-channel scale and zero-point overhead, while layers kept at full precision are computed as plain 16-bit. Within this framework, uniform 4-bit quantization reduces exactly to a special case of RASQ obtained by pushing RASQ’s threshold to its extreme, so that every expert is quantized to 4 bits. In other words, uniform quantization and RASQ are simply two points on the same single-parameter threshold family.

A conceptual storage-cost breakdown computed from the cost model in Section 3 of the paper. Uniform NVFP4 quantizes all expert linear layers to 4 bits, while RASQ keeps the router and rare-expert layers at 16 bits and compresses only high-traffic experts to 4 bits.

This is where the paper is especially honest. The authors themselves point out that under this static storage metric, RASQ is structurally bound to store more bits than uniform 4-bit quantization. The moment the low-traffic half is protected at full precision, the number of quantized parameters necessarily decreases. This makes it arithmetically impossible for RASQ to claim strict Pareto dominance over uniform quantization on both axes, and the authors state plainly that attempting to verify such a claim is itself the wrong question. Instead, the well-posed question the paper poses is interpolation efficiency: how much of the accuracy gap between uniform quantization and the full-precision baseline RASQ recovers, set alongside the ratio of the storage premium paid for it. To this end, the paper uses google/switch-base-8, an actual public Switch-Transformer model, as a proxy, and specifies a fully reproducible evaluation protocol across three settings (full-precision baseline, uniform 4-bit, and selective 4-bit RASQ) that approximates the accuracy effects of NVFP4 in a hardware-independent way using a per-channel affine 4-bit fake-quantization operator.

To actually run this protocol, the authors submitted a Kueue custom-route job to an H200 GPU cluster, but the job failed before the first forward pass even started, with an error that the GPU context tkai-prod-compute-h200 did not exist at submission time. Rather than dressing this up as a negative experimental result, the authors report it plainly as “not measured, but skipped.” As a result, what this paper actually delivers is not empirical measurements but a formalized policy, a cost model, and a complete protocol that can be run as-is the moment the cluster is restored.

Contributions Across Company, Society, and Science

The paper divides its contributions into three layers. At the platform and industry layer, it proposes RASQ as a policy that batch schedulers like Kueue could adopt as a default when placing MoE inference jobs on H200 or Blackwell-class clusters. Because the precision map becomes static once traffic is profiled offline a single time, it integrates cleanly with model-loading and admission logic without requiring a runtime control loop. At the societal layer, by lowering per-token inference cost and energy use, it aims to widen access so that resource-constrained organizations or individuals can use powerful open MoE models without bearing the cost of full-precision serving. At the scientific layer, it poses, in an empirically testable form, the question of how far selective precision allocation based on expert traffic distribution can improve the accuracy-cost Pareto curve relative to uniform NVFP4 quantization, and it adds to the existing PTQ literature on uniform and block-wise NVFP4 quantization a storage cost model, a rigorous reduction of uniform quantization to a special case, and a path for extension into budget-constrained allocation.

A conceptual diagram showing how RASQ interpolates on the accuracy-versus-storage-cost Pareto frontier between uniform 4-bit (minimum storage, maximum error) and the full-precision baseline (maximum storage, minimum error). As Section 4’s limitations state, this is not measured data (no actual measurement was performed), and the accuracy axis is the cross-entropy loss on a 16-sentence probe set, while the storage axis is static, whole-model storage bits.

In place of measurements, the paper offers explicitly conditional predictions grounded in the literature. Drawing on prior work showing that perturbing router logits destabilizes routing itself, on utilization-aware precision research showing that expert traffic is long-tailed so uniform allocation wastes budget, and on scaling-law research that treats low-bit quantization as a Pareto problem of quality against bit budget, the paper states that it expects RASQ to be an efficient midpoint that recovers a large share of the accuracy gap relative to uniform 4-bit quantization for only a small storage premium. However, this remains strictly a prediction, and the paper does not present any of it as a finding.

Limitations

The most fundamental limitation is that there are no measurements at all. Because the job failed before the forward pass due to a cluster infrastructure fault, every numerical statement in the paper is a definition or a prediction, not a result. Second, the per-channel affine 4-bit fake-quantization operator used to measure accuracy is merely a proxy whose numerical scheme and rounding differ from NVFP4’s two-level microblock scaling. This proxy was necessary because H200 is a Hopper-generation chip with no native FP4 tensor cores, so the protocol can only approximate accuracy effects and cannot measure NVFP4’s actual throughput gains. Third, the protocol targets only a single small Switch-Transformer model using top-1 routing, so it is uncertain whether the results transfer as-is to larger MoE models, top-k routing, or architectures with a different number of experts. Fourth, the static, offline traffic profile that RASQ relies on risks drifting once deployment workloads change. As prior utilization-aware research reports, expert traffic can shift depending on the workload as to which experts run hot, meaning a median split computed once may end up protecting the wrong experts by the time of deployment. Finally, the authors themselves note as a weakness that both traffic profiling and accuracy evaluation reuse the same small 16-sentence probe set, coupling the two measurements together.

Paper detail page: https://huggingface.co/datasets/thaki-AI/daily-paper-2026-07-12-nvfp4-moe-selective-quant