MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM

Dongwei Wang ^*

Jinhee Kim ^*

Seokho Han ^*

Denis Gudovskiy

Yohei Nakata

Tomoyuki Okuno

KhayTze Peong

Kang Eun Jeon ^†

Jong Hwan Ko ^†

Yiran Chen ^†

Huanrui Yang ^†

NeurIPS 2026

^*Equal contribution., ^†Corresponding author.

Paper Code Poster

Abstract

Dynamic runtime latency and memory constraints necessitate flexible large language model (LLM) deployment, where an LLM can be inferred with various quantization precisions based on available computational resources. Recent work on such any-precision quantization either relies on hardware-inefficient vector quantization or induces additional scaling factors when switching between bit-widths. Meanwhile, existing post-training quantization (PTQ) methods calibrated for a fixed low precision show poor generalizability under runtime precision change. In this work, we attribute the source of poor generalization across bit-widths to a precision-dependent outlier migration phenomenon where the distribution of PTQ-sensitive tokens changes across precisions. Motivated by this observation, we propose MoBiQuant, a novel any-precision Mixture-of-Bits quantization framework that adjusts weight precision for flexible LLM inference based on token sensitivity. Specifically, we propose a many-in-one recursive residual quantization that can iteratively reconstruct higher-precision weights at runtime and mitigates outlier migration with a token-aware router to dynamically select the optimal inference precision of each token. Extensive experiments show that MoBiQuant matches or surpasses frontier single-precision PTQ while exhibiting strong elasticity, achieving significant memory savings and throughput gains of up to $1.34\times$ over state-of-the-art any-precision methods.

Motivation: Why existing PTQ cannot generalize across bits

A model calibrated at one bit-width does not necessarily generalize to another. For example, on LLaMA-3-8B, using 3-bit calibration parameters for 4-bit inference increases WikiText-2 perplexity by 2.65 points compared with a model calibrated at 4-bit, as shown in Figure 1 (left).

To examine the underlying cause, we plot the per-token quantization error distributions for the same query at 3-bit and 4-bit weight precision in Figure 1 (right) and observe markedly different distributions. Specifically, tokens that are accurately represented by 4-bit weights may emerge as dominant outliers for 3-bit weights, whereas outliers at 4-bit may no longer be the primary source of quantization error at 3-bit. We define this phenomenon as outlier migration, which occurs across different PTQ calibration methods.

Interestingly, we found that keeping the top 10% of token outliers from 3-bit calibration at 3-bit during otherwise 4-bit inference partially recovers performance. This observation suggests a promising direction: token-adaptive precision adjustment can improve the generalization of PTQ calibration parameters across bit-widths.

Calibration-inference precision mismatch and token-wise quantization error at 3-bit and 4-bit in LLaMA3-8B. — **Figure 1.** (Left) Existing PTQ methods poorly generalize under calibration-inference precision mismatch, e.g., inference at 4-bit with 3-bit calibrated parameters (green vs. red). Incorporating token-aware bit adjustment partially mitigates this degradation (pink). Our **MoBiQuant** improves generalization across precisions (orange). (Right) Per-token quantization error distribution at layer 5 in LLaMA3-8B: quantization outliers are highly non-uniform at each bit-width. Tokens well-fitted at 4-bit can become outliers at 3-bit, while 4-bit outliers may not be the primary error source at 3-bit.

MoBiQuant: Bit Slicing Meets Token-Adaptive Routing

These observations suggest that improving PTQ generalization requires two capabilities: the quantized model must offer weight representations at different bit-widths, and it must learn which representation works best for each token. MoBiQuant provides both through MoBiSlice, which builds the different precision options, and MoBiRoute, which learns to route each token to a suitable option within a compute budget.

MoBiSlice: build multiple precisions from shared slices. Instead of storing a separate model for every bit-width, MoBiSlice breaks each weight matrix into small, reusable bit slices. The first slice approximates the original weights; each additional slice captures what the previous slices missed. Adding more slices gives a higher-precision representation. With four 2-bit slices, the model can construct 2-, 4-, 6-, and 8-bit weights. All slices share a common set of calibration parameters, so each precision does not need its own separately learned scales.

MoBiRoute: choose the right precision for each token. Having multiple precision options is only half the solution: the model also needs to learn which one to use. MoBiRoute selects the bit slices for each token while keeping the average bit-width within a target budget. Every token uses the first shared slice, and the router learns which additional slices suit that token. During calibration, the original LLM weights stay frozen while the quantization parameters and router are optimized together. Routing decisions gradually become binary: each additional slice is either used or skipped.

MoBiQuant linear block, recursive residual bit slicing in MoBiSlice, and soft-to-binary token routing in MoBiRoute. — **Figure 2.** **MoBiQuant** enables token-adaptive any-precision inference for linear blocks in LLMs (a). It consists of two main components: **MoBiSlice** that supports multiple precisions by decomposing FP16 weights into bit slices via recursive quantization (b); and **MoBiRoute** with token-adaptive precision adjustment that is trained to route each token to the optimal bit slice under a budget (c).

At inference time, a globally adjustable routing threshold controls the number of active slices per token. Increasing the threshold reduces effective precision; decreasing it activates more slices. This enables fine-grained runtime adaptation without repeated weight repacking.

Efficient Any-Precision Kernels

Flexible precision also needs efficient execution. MoBiQuant stores weights in independently accessible bit-planes, allowing the kernel to fetch only the planes needed at the selected precision. During prefilling and batched decoding, tokens are permuted after routing to group those sharing active slices, improving memory locality and load balancing.

MoBiQuant improves kernel efficiency in two aspects. First, unlike the bit-plane transposes and table lookups of AnyPrecisionLLM, or the multiple scales used by AnyBCQ, MoBiQuant performs direct bit-plane operations with shared scales, enabling efficient shift-and-add dequantization. Second, a fixed 2-bit kernel avoids relaunching distinct kernels when changing precision.

Comparison of AnyPrecisionLLM, AnyBCQ, and MoBiQuant bit-plane storage and dequantization pipelines. — **Figure 3.** (a) AnyPrecisionLLM with the inefficient bit-plane transposes and table lookups, (b) AnyBCQ with the multi-scale storage and dequantization overhead, and (c) our **MoBiQuant** that performs direct bit-plane operations with the shared scales for efficient any-precision dequantization.

Experimental Results

End-to-End Comparison with Any-Precision Baselines

As shown in Table 3, MoBiQuant delivers higher decoding throughput and generally better model performance than existing any-precision baselines.

Table 3: WikiText-2 perplexity, MMLU accuracy, and decoding throughput for FP16, AnyPrecisionLLM, AnyBCQ, and MoBiQuant across four models at 2-, 3-, and 4-bit precision. — **Table 3.** End-to-end comparison with the any-precision baselines AnyPrecisionLLM (AP) and AnyBCQ (ABC). **MoBiQuant** achieves higher decoding throughput while maintaining competitive perplexity and accuracy.

Causal Link Between Token Routing and Outlier Migration

To examine whether token routing mitigates outlier migration, we compare MoBiRoute’s router scores with the per-token error increases when switching OmniQuant from 4-bit calibration to 3-bit inference. In Figure 5 (left), tokens with larger error increases tend to receive higher router scores and higher precision, while less sensitive tokens are routed to 2-bit computation.

Figure 5 (right) shows more consistent token-wise error distributions across bit-widths than static PTQ in Figure 1. Together, these observations support that token-adaptive routing assigns higher precision to vulnerable tokens, reducing outlier migration and the calibration-inference mismatch.

Figure 5: Router scores track token-wise error increases (left); MoBiQuant reduces outlier migration across bit-widths (right). — **Figure 5. Visualization of outlier migration mitigation.** MoBiQuant router scores correlate with the actual token-wise error increments under precision switching (left), and it produces more consistent token-wise error distributions across bit-widths with reduced outlier migration (right).

Figure 1, right: static PTQ produces different token-wise quantization error distributions at 4-bit and 3-bit, illustrating outlier migration. — **Figure 5. Visualization of outlier migration mitigation.** MoBiQuant router scores correlate with the actual token-wise error increments under precision switching (left), and it produces more consistent token-wise error distributions across bit-widths with reduced outlier migration (right).

Welcome to cite us

@article{wang2026mobiquant,
  title={MoBiQuant: Mixture-of-Bits Quantization for Token-Adaptive Any-Precision LLM},
  author={Wang, Dongwei and Kim, Jinhee and Han, Seokho and Gudovskiy, Denis and Nakata, Yohei and Okuno, Tomoyuki and Peong, KhayTze and Jeon, Kang Eun and Ko, Jong Hwan and Chen, Yiran and others},
  journal={arXiv preprint arXiv:2602.20191},
  year={2026}
}