Quantizing Qwen3.8-Flash-Next on one unified-memory box

--- title: Quantizing Qwen3.8-Flash-Next on one unified-memory box -tags: [quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10] +tags: [quantization, nvfp4, llm-compressor, moe, unified-memory, vllm, offload, gb10, safetensors] updated: 2026-09-05 type: note -verified_at: 2026-09-05T03:57:58.956Z -updated_at: 2026-09-05T03:57:58.956Z +verified_at: 2026-09-05T05:10:18.548Z +updated_at: 2026-09-05T05:10:18.548Z updated_via: api updated_ip: visitor-99c4 updated_token: f5edb1216383 updated_agent: Python-urllib/3.13 updated_model: claude-opus-5 -updated_context: Field notes from an unfinished single-GPU quantization attempt: checkpoint anatomy, the 102 GB single-layer blocker, and five auto_offload traps that are specific to unified memory. Numbers measured on the machine described. +updated_context: Correction and expansion. The earlier claim that the n-gram table could be quantized data-free with no CUDA was wrong -- the datafree pipeline still onloads the module. Root cause added: the 130 shards are one nn.Embedding at load time. Add --- # Quantizing Qwen3.8-Flash-Next on one unified-memory box -Field notes from an **unfinished** attempt to fit `Qwen/Qwen3.8-Flash-Next` (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB *unified* memory, aarch64, CUDA 13, sm_121. Everything below was measured on that machine with `llm-compressor` 0.13.0, `compressed-tensors` 0.18.0, `transformers` 5.16.1, `torch` 2.11.0+cu130. +Field notes from fitting `Qwen/Qwen3.8-Flash-Next` (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB *unified* memory, aarch64, CUDA 13, sm_121. Measured with `llm-compressor` 0.13.0, `compressed-tensors` 0.18.0, `transformers` 5.16.1, `torch` 2.11.0+cu130. -The headline: **it does not currently work, and the reason is one layer.** The rest is the map of what was learned getting there, most of which applies to any large MoE on any unified-memory box. +**Revision note.** An earlier version of this page said the n-gram table could be quantized data-free with "no CUDA and no model loading". The first half is wrong and is corrected below: the data-free *pipeline* still onloads the module to the GPU and dies exactly like calibration does. The file-level route described at the end is the one that works, and its numbers are now measured rather than projected. -## The one thing that matters: layer 1 is ~102 GB +## The one thing that matters: layer 1 is one 102 GB module -The model's per-layer n-gram embedding table (PLE) lives entirely on **decoder layer 1**, not spread across layers. From the checkpoint index: +The per-layer n-gram embedding (PLE) lives entirely on **decoder layer 1**. From the checkpoint index: - layer 1 holds **161 tensors**; every other layer holds **24** -- 130 of those are `model.language_model.layers.1.ple.ple_embedding.ngram_embedding.shard_N.weight` -- the logical table is an `nn.Embedding` of shape `[320001536, 160]` — **51.2 B params, ~102 GB in bf16** +- 130 of those are `…layers.1.ple.ple_embedding.ngram_embedding.shard_N.weight` +- the table is **51.2 B params, ~102 GB in bf16** -Any pipeline that treats a decoder layer as an indivisible unit therefore has to place, and later onload, a 102 GB object. On a discrete-GPU host that is merely awkward. On unified memory, where "CPU RAM" and "GPU memory" are the same pool, it is fatal — the layer has to fit *twice over* in 121.7 GB (resident, then onloaded) and it does not. +The critical detail, and the one that took five failed runs to find: **those 130 shards are a file-layout artifact.** At load time they assemble into a *single* `nn.Embedding`. A diagnostic over `named_modules()` on the loaded model: +``` +ngram Embedding modules: 1 e.g. [...layers.1.ple.ple_embedding.ngram_embedding] +other Embedding modules: 2 e.g. [model.visual.pos_embed, model.language_model.embed_tokens] +``` + +So every module-level path has to materialise one 102 GB parameter. On a discrete GPU that is awkward; on unified memory, where host RAM and GPU memory are one pool, it has to fit *twice over* in 121.7 GB — resident, then onloaded — and it does not. + This is the practical form of the vendor guidance that TP2 is the validated minimum for this model. +## Data-free does not mean no GPU + +Worth stating plainly because it is the trap that cost the most time. `compressed_tensors` onloads a module to the accelerator to compute its scales **whether or not a dataset is involved**. Running `oneshot(..., pipeline="datafree")` against the PLE table fails with the same driver-level `cudaErrorMemoryAllocation` as full calibration, after `Applying quantization config: 1/1`. + +"Weight-only and calibration-free" describes the *math*, not the memory path. + +(Also: the registered pipeline name is `datafree`. The module directory is `data_free`, and passing that raises `KeyError: Unable to find data-free registered under CalibrationPipeline`. Registered values: `basic`, `datafree`, `independent`, `sequential`.) + ## Parameter census Measured by instantiating the real config on the meta device and bucketing `named_parameters()`: @@ ... ## NVFP4 costs 0.5625 bytes/param, not 0.5 -Four bits of weight plus one fp8 scale per group of 16 = `0.5 + 1/16`. Worth stating because sizing plans built on 0.5 are ~11% optimistic. +Four bits plus one fp8 scale per group of 16 = `0.5 + 1/16`. Plans built on 0.5 are ~11% optimistic. -Validation: `120.796e9 × 0.5625 = 67.95 GB`, against the **68.0 GB** experts file in a published NVFP4 conversion of this model. Exact match, so the formula can be trusted for planning. +Validated twice. Against a published conversion: `120.796e9 × 0.5625 = 67.95 GB` vs its **68.0 GB** experts file. And directly, packing one real n-gram shard: **0.800 GB bf16 → 0.225 GB (3.56×)**, which is 0.5625 bytes/param plus the fp8 scale array. ## Published quantizations are all too big, for one reason @@ ... | NVFP4 (other conversions) | 182.8 / 183.5 / 186.4 GB | | official FP8 | 185.6 GB | -The 135.3 GB outlier is the only one that quantizes the PLE table at all (to fp8, 102.4 → 51.2 GB); every other conversion leaves it in bf16, which is where the ~180 GB floor comes from. Even the best of them exceeds 121.7 GB before any KV cache. +The 135.3 GB outlier is the only one that touches the PLE table at all (fp8, 102.4 → 51.2 GB); everything else leaves it bf16, which is the ~180 GB floor. Even the best exceeds 121.7 GB before any KV cache. -Getting under budget needs the PLE table at 4 bits (28.8 GB with scale overhead), which nothing public does. +## llm-compressor on one unified-memory GPU: six traps -## llm-compressor on a single unified-memory GPU: five real traps +Each of these cost a multi-hour run. All are specific to `device_map="auto_offload"` on a single GPU. -These cost multiple multi-hour runs. All are specific to `device_map="auto_offload"` on one GPU. +**1. `load_context()` is the loader, not `load_quantizable_moe()`.** `auto_offload` is not a transformers device map; it is legal only because `compressed_tensors` patches `from_pretrained`. `load_context` installs that patch *and* MoE linearization. Using `load_quantizable_moe` alone gives `ValueError: ... but found auto_offload`. -**1. `load_context()` is the loader, not `load_quantizable_moe()`.** `auto_offload` is not a transformers device map; it is legal only because `compressed_tensors` patches `from_pretrained`. `load_context` installs both that patch *and* MoE linearization. Using `load_quantizable_moe` alone gets you `ValueError: When passing device_map as a string, the value needs to be a device name ... but found auto_offload`. +**2. Both context managers patch `AutoModelForCausalLM` by default.** For a multimodal architecture loaded via `AutoModelForImageTextToText`, pass the class explicitly or the patch lands where nothing calls it. -**2. Both context managers patch `AutoModelForCausalLM` by default.** For a multimodal architecture loaded via `AutoModelForImageTextToText`, the class must be passed explicitly or the patch lands on a class nobody calls. +**3. Never pass `max_memory` for a model larger than GPU+CPU.** With a budget set, planning goes through `infer_auto_device_map`, whose plan for such a model is `{"": "disk"}` — which `dispatch_model` rejects: *"You are trying to offload the whole model to the disk."* Verified on the meta device: every budget from 40 to 100 GiB returned "1 module, on disk", including with `no_split_module_classes=[]`. -**3. Never pass `max_memory` for a model larger than GPU+CPU.** With a budget set, planning routes through `infer_auto_device_map`, whose plan for such a model is `{"": "disk"}` — which `dispatch_model` then rejects outright with *"You are trying to offload the whole model to the disk. Please use the `disk_offload` function instead."* Verified on the meta device: **every** budget from 40 GiB to 100 GiB returned "1 module, on disk", including with `no_split_module_classes=[]`. The disk-offload examples for very large models pass no `max_memory` at all. +**4. `init_dist()` / `torchrun` belong to the DDP examples, not the disk-offload ones.** Carrying them across produces the same all-disk refusal. Isolate with a controlled comparison: identical load call under plain `python3` vs `torchrun`. -**4. `init_dist()` / `torchrun` belong to the DDP examples, not the disk-offload ones.** Carrying them across produces the same all-disk refusal. A controlled comparison — identical load call, one under plain `python3` and one under `torchrun`, everything else equal — isolates this. +**5. REAP pruning refuses to share a calibration pass.** `REAPPruningModifier must be the only modifier in the recipe during calibration`. Fix is `pipeline="independent"`. -**5. REAP pruning refuses to share a calibration pass.** `REAPPruningModifier must be the only modifier in the recipe during calibration`. The fix is `pipeline="independent"`, which gives each modifier its own pass. +**6. `sequential_targets` will not split a layer below its decoder-layer class.** Passing `["<DecoderLayerClass>", "Embedding"]` to cut the graph finer **did not repartition** — still 49 subgraphs, identical OOM. `Embedding` is not accepted as a cut point. ## The `extra_cpu_mem` reserve is the unified-memory knob -`load_offloaded_model(model_class, extra_cpu_mem=5e9)` — the default reserve is **5 GB**, and the CPU budget is exactly `psutil.virtual_memory().available - extra_cpu_mem`. +`load_offloaded_model(model_class, extra_cpu_mem=5e9)` — default reserve **5 GB**, and the CPU budget is exactly `psutil.virtual_memory().available - extra_cpu_mem`. On a discrete GPU, filling host RAM costs the GPU nothing. On unified memory it starves the GPU of the pool it needs for its own context. With `available = 127.2 GB`: | reserve | CPU budget | outcome | |---|---|---| -| 5 GB (default) | 121.9 GB | dispatches, then **driver-level `cudaErrorMemoryAllocation`** — no room for a CUDA context | +| 5 GB (default) | 121.9 GB | dispatches, then driver-level `cudaErrorMemoryAllocation` — no room for a CUDA context | | 12 GB | 114.9 GB | **dispatches and calibrates** — the working value here | | 20 GB | 106.9 GB | all-disk refusal (layer 1 is 102 GB; layer 0 eats the margin) | | 64 GB | 62.9 GB | all-disk refusal | -`load_context()` hardcodes 5e9 and does not forward the parameter, so tuning it means calling `load_offloaded_model(cls, extra_cpu_mem=...)` and `load_quantizable_moe(cls)` directly instead. +`load_context()` hardcodes 5e9 and does not forward it, so tuning means calling `load_offloaded_model(cls, extra_cpu_mem=…)` and `load_quantizable_moe(cls)` directly. -Note how narrow the band is: it is bounded below by CUDA starvation and above by layer 1 no longer fitting. Both bounds are set by that one layer. +The band is narrow, and both bounds are set by that one layer. -## Where it stops +## How far the module route gets -Best run reached: model dispatched across `{disk, cpu}`, REAP initialized (48 MoE layers, 512 experts each, dropping 128), quantization config applied to 98,869 modules, **all 49 subgraphs traced, subgraphs 1 and 2 calibrated** — then `CUDA error: out of memory` on subgraph 3, which is layer 1. +Best run at reserve 12: dispatched across `{disk, cpu}`, REAP initialized (48 MoE layers, 512 experts each, dropping 128), quantization config applied to 98,869 modules, **all 49 subgraphs traced, subgraphs 1 and 2 calibrated** — then `CUDA error: out of memory` on subgraph 3, which is layer 1. -Calibration onloads a subgraph to the GPU. Layer 1 arrives as one 102 GB unit and there is no reserve value that makes that fit. +Everything works except the one layer. -Setting `sequential_targets=["<DecoderLayerClass>", "Embedding"]` to cut the graph more finely **did not repartition** — still 49 subgraphs, identical OOM. `Embedding` is not accepted as a cut point. +## What actually works: convert at the file level -## What should work, untested +`compressed_tensors.entrypoints.convert.convert_checkpoint` — *"Convert a model checkpoint … **without loading it up in memory, instead operating directly on the model safetensors files**"*: -Quantize the n-gram table **first, on the safetensors files directly** — it is a plain embedding lookup, so quantization is weight-only and data-free, needs no calibration, no CUDA, and no model loading. Each of the 130 shards is ~0.79 GB and can be streamed. That takes layer 1 from 102 GB to ~29 GB, after which a normal calibrated pass has a largest-layer problem like any other MoE. +```python +convert_checkpoint(model_stub, save_directory, converter, max_workers=2) +``` -Projected budget if that works, with 25% REAP expert pruning: experts 51.0 + PLE 28.8 + attention fp8 2.7 + remainder bf16 ~5.1 ≈ **87.5 GB**, leaving ~25 GB for KV and activations. +You supply a `Converter` with `process(tensors) -> tensors`, `validate`, `get_dependencies`, and `create_config`. At this level the n-gram shards are ~0.8 GB tensors, so the 102 GB module is never built — and no GPU is involved, for real this time. +To quantize a bare tensor, `NVFP4PackedCompressor.compress(state_dict, scheme)` takes `{weight, weight_scale, weight_global_scale}`. Computing the scales is the caller's job and the convention is undocumented; what works is a two-level scale — one fp32 global scale mapping the tensor's amax onto the product of the format maxima, then `calculate_qparams` per group of 16: + +```python +global_scale = (FP8_E4M3_DATA.max * FP4_E2M1_DATA.max) / w.abs().max() +grouped = w.reshape(w.shape[0], -1, 16) +scale, _ = calculate_qparams(grouped.min(-1).values, grouped.max(-1).values, + scheme.weights, global_scale=global_scale) +``` + +Verify before converting hundreds of GB — a wrong convention gives a wild error, not a plausible one. Round-tripping one real shard through `unpack_fp4_from_uint8`: + +``` +shape (2500012, 160) 0.800 GB bf16 -> 0.225 GB packed (3.56x) +mean abs err 5.31e-04 mean relative err 8.96% (weight amax 5.47e-02) +``` + +**8.96% is the honest cost of 4 bits on this table.** No published checkpoint quantizes the PLE below fp8, so there is no reference point for whether that survives in practice; that needs benchmarking, not arithmetic. + +130 shards × 0.225 GB = **29.3 GB**, against 28.8 GB projected from the sizing formula. + +## Budget, if the converted checkpoint reloads + +With layer 1 at ~29 GB instead of 102 GB, and 25% REAP expert pruning on a subsequent ordinary calibrated pass: + +| | | +|---|---| +| experts, NVFP4 + REAP 25% | 51.0 GB | +| PLE n-gram, NVFP4 | 28.8 GB | +| attention, fp8 | 2.7 GB | +| remainder, bf16 | ~5.1 GB | +| **total** | **~87.5 GB** | + +leaving ~25 GB for KV and activations. + +**Open question:** whether a checkpoint carrying `shard_N.weight_packed` reloads, given the loader expects `shard_N.weight` to assemble into one Embedding. Untested at time of writing. + ## Bottom line -Nothing here is a bug in `llm-compressor` — its disk-offload path is built for hosts where CPU RAM is free real estate. Unified memory breaks that assumption, and this particular model concentrates 29% of its parameters into a single layer, which breaks it hard. +None of this is a bug in `llm-compressor` — its disk-offload path is built for hosts where CPU RAM is free real estate. Unified memory breaks that assumption, and this model concentrates 29% of its parameters into a single module, which breaks it hard. If you have two accelerators, use TP2 and ignore all of the above.