# Quantizing Qwen3.8-Flash-Next on one unified-memory box Field notes from fitting `Qwen/Qwen3.8-Flash-Next` (177.4 B params, 360 GB bf16) onto a single GB10-class machine — 121.7 GB *unified* memory, aarch64, CUDA 13, sm_121. Measured with `llm-compressor` 0.13.0, `compressed-tensors` 0.18.0, `transformers` 5.16.1, `torch` 2.11.0+cu130. **Revision note.** An earlier version of this page said the n-gram table could be quantized data-free with "no CUDA and no model loading". The first half is wrong and is corrected below: the data-free *pipeline* still onloads the module to the GPU and dies exactly like calibration does. The file-level route described at the end is the one that works, and its numbers are now measured rather than projected. ## The one thing that matters: layer 1 is one 102 GB module The per-layer n-gram embedding (PLE) lives entirely on **decoder layer 1**. From the checkpoint index: - layer 1 holds **161 tensors**; every other layer holds **24** - 130 of those are `…layers.1.ple.ple_embedding.ngram_embedding.shard_N.weight` - the table is **51.2 B params, ~102 GB in bf16** The critical detail, and the one that took five failed runs to find: **those 130 shards are a file-layout artifact.** At load time they assemble into a *single* `nn.Embedding`. A diagnostic over `named_modules()` on the loaded model: ``` ngram Embedding modules: 1 e.g. [...layers.1.ple.ple_embedding.ngram_embedding] other Embedding modules: 2 e.g. [model.visual.pos_embed, model.language_model.embed_tokens] ``` So every module-level path has to materialise one 102 GB parameter. On a discrete GPU that is awkward; on unified memory, where host RAM and GPU memory are one pool, it has to fit *twice over* in 121.7 GB — resident, then onloaded — and it does not. This is the practical form of the vendor guidance that TP2 is the validated minimum for this model. ## Data-free does not mean no GPU Worth stating plainly because it is the trap that cost the most time. `compressed_tensors` onloads a module to the accelerator to compute its scales **whether or not a dataset is involved**. Running `oneshot(..., pipeline="datafree")` against the PLE table fails with the same driver-level `cudaErrorMemoryAllocation` as full calibration, after `Applying quantization config: 1/1`. "Weight-only and calibration-free" describes the *math*, not the memory path. (Also: the registered pipeline name is `datafree`. The module directory is `data_free`, and passing that raises `KeyError: Unable to find data-free registered under CalibrationPipeline`. Registered values: `basic`, `datafree`, `independent`, `sequential`.) ## Parameter census Measured by instantiating the real config on the meta device and bucketing `named_parameters()`: | component | params | bf16 | |---|---|---| | MoE experts | 120.80 B | 241.6 GB | | PLE n-gram table | 51.20 B | 102.4 GB | | linear attention (Gated DeltaNet) | 2.09 B | 4.2 GB | | other | 0.70 B | 1.4 GB | | embed_tokens | 0.64 B | 1.3 GB | | lm_head | 0.64 B | 1.3 GB | | full attention | 0.62 B | 1.2 GB | | vision tower | 0.45 B | 0.9 GB | | shared experts | 0.24 B | 0.5 GB | | PLE projections/norms | 0.03 B | 0.1 GB | | **total** | **177.39 B** | **354.8 GB** | Only 12 of 48 layers are full attention (`full_attention_interval: 4`); the other 36 are linear attention with constant-size state, so KV cache is unusually cheap for the parameter count. ## NVFP4 costs 0.5625 bytes/param, not 0.5 Four bits plus one fp8 scale per group of 16 = `0.5 + 1/16`. Plans built on 0.5 are ~11% optimistic. Validated twice. Against a published conversion: `120.796e9 × 0.5625 = 67.95 GB` vs its **68.0 GB** experts file. And directly, packing one real n-gram shard: **0.800 GB bf16 → 0.225 GB (3.56×)**, which is 0.5625 bytes/param plus the fp8 scale array. ## Published quantizations are all too big, for one reason | checkpoint | size | |---|---| | NVFP4 (PLE cast to fp8) | **135.3 GB** | | W4A16 | 179.9 GB | | NVFP4 (other conversions) | 182.8 / 183.5 / 186.4 GB | | official FP8 | 185.6 GB | The 135.3 GB outlier is the only one that touches the PLE table at all (fp8, 102.4 → 51.2 GB); everything else leaves it bf16, which is the ~180 GB floor. Even the best exceeds 121.7 GB before any KV cache. ## llm-compressor on one unified-memory GPU: six traps Each of these cost a multi-hour run. All are specific to `device_map="auto_offload"` on a single GPU. **1. `load_context()` is the loader, not `load_quantizable_moe()`.** `auto_offload` is not a transformers device map; it is legal only because `compressed_tensors` patches `from_pretrained`. `load_context` installs that patch *and* MoE linearization. Using `load_quantizable_moe` alone gives `ValueError: ... but found auto_offload`. **2. Both context managers patch `AutoModelForCausalLM` by default.** For a multimodal architecture loaded via `AutoModelForImageTextToText`, pass the class explicitly or the patch lands where nothing calls it. **3. Never pass `max_memory` for a model larger than GPU+CPU.** With a budget set, planning goes through `infer_auto_device_map`, whose plan for such a model is `{"": "disk"}` — which `dispatch_model` rejects: *"You are trying to offload the whole model to the disk."* Verified on the meta device: every budget from 40 to 100 GiB returned "1 module, on disk", including with `no_split_module_classes=[]`. **4. `init_dist()` / `torchrun` belong to the DDP examples, not the disk-offload ones.** Carrying them across produces the same all-disk refusal. Isolate with a controlled comparison: identical load call under plain `python3` vs `torchrun`. **5. REAP pruning refuses to share a calibration pass.** `REAPPruningModifier must be the only modifier in the recipe during calibration`. Fix is `pipeline="independent"`. **6. `sequential_targets` will not split a layer below its decoder-layer class.** Passing `["", "Embedding"]` to cut the graph finer **did not repartition** — still 49 subgraphs, identical OOM. `Embedding` is not accepted as a cut point. ## The `extra_cpu_mem` reserve is the unified-memory knob `load_offloaded_model(model_class, extra_cpu_mem=5e9)` — default reserve **5 GB**, and the CPU budget is exactly `psutil.virtual_memory().available - extra_cpu_mem`. On a discrete GPU, filling host RAM costs the GPU nothing. On unified memory it starves the GPU of the pool it needs for its own context. With `available = 127.2 GB`: | reserve | CPU budget | outcome | |---|---|---| | 5 GB (default) | 121.9 GB | dispatches, then driver-level `cudaErrorMemoryAllocation` — no room for a CUDA context | | 12 GB | 114.9 GB | **dispatches and calibrates** — the working value here | | 20 GB | 106.9 GB | all-disk refusal (layer 1 is 102 GB; layer 0 eats the margin) | | 64 GB | 62.9 GB | all-disk refusal | `load_context()` hardcodes 5e9 and does not forward it, so tuning means calling `load_offloaded_model(cls, extra_cpu_mem=…)` and `load_quantizable_moe(cls)` directly. The band is narrow, and both bounds are set by that one layer. ## How far the module route gets Best run at reserve 12: dispatched across `{disk, cpu}`, REAP initialized (48 MoE layers, 512 experts each, dropping 128), quantization config applied to 98,869 modules, **all 49 subgraphs traced, subgraphs 1 and 2 calibrated** — then `CUDA error: out of memory` on subgraph 3, which is layer 1. Everything works except the one layer. ## What actually works: convert at the file level `compressed_tensors.entrypoints.convert.convert_checkpoint` — *"Convert a model checkpoint … **without loading it up in memory, instead operating directly on the model safetensors files**"*: ```python convert_checkpoint(model_stub, save_directory, converter, max_workers=2) ``` You supply a `Converter` with `process(tensors) -> tensors`, `validate`, `get_dependencies`, and `create_config`. At this level the n-gram shards are ~0.8 GB tensors, so the 102 GB module is never built — and no GPU is involved, for real this time. To quantize a bare tensor, `NVFP4PackedCompressor.compress(state_dict, scheme)` takes `{weight, weight_scale, weight_global_scale}`. Computing the scales is the caller's job and the convention is undocumented; what works is a two-level scale — one fp32 global scale mapping the tensor's amax onto the product of the format maxima, then `calculate_qparams` per group of 16: ```python global_scale = (FP8_E4M3_DATA.max * FP4_E2M1_DATA.max) / w.abs().max() grouped = w.reshape(w.shape[0], -1, 16) scale, _ = calculate_qparams(grouped.min(-1).values, grouped.max(-1).values, scheme.weights, global_scale=global_scale) ``` Verify before converting hundreds of GB — a wrong convention gives a wild error, not a plausible one. Round-tripping one real shard through `unpack_fp4_from_uint8`: ``` shape (2500012, 160) 0.800 GB bf16 -> 0.225 GB packed (3.56x) mean abs err 5.31e-04 mean relative err 8.96% (weight amax 5.47e-02) ``` **8.96% is the honest cost of 4 bits on this table.** No published checkpoint quantizes the PLE below fp8, so there is no reference point for whether that survives in practice; that needs benchmarking, not arithmetic. 130 shards × 0.225 GB = **29.3 GB**, against 28.8 GB projected from the sizing formula. ## Budget, if the converted checkpoint reloads With layer 1 at ~29 GB instead of 102 GB, and 25% REAP expert pruning on a subsequent ordinary calibrated pass: | | | |---|---| | experts, NVFP4 + REAP 25% | 51.0 GB | | PLE n-gram, NVFP4 | 28.8 GB | | attention, fp8 | 2.7 GB | | remainder, bf16 | ~5.1 GB | | **total** | **~87.5 GB** | leaving ~25 GB for KV and activations. **Open question:** whether a checkpoint carrying `shard_N.weight_packed` reloads, given the loader expects `shard_N.weight` to assemble into one Embedding. Untested at time of writing. ## Bottom line None of this is a bug in `llm-compressor` — its disk-offload path is built for hosts where CPU RAM is free real estate. Unified memory breaks that assumption, and this model concentrates 29% of its parameters into a single module, which breaks it hard. If you have two accelerators, use TP2 and ignore all of the above.