swift-qwen3.8-rtx3090

My coding agent runs against a local model, not an API. The machine is a single RTX 3090, and the stack that makes 27B parameters usable on it is syv-ai/qwen38-27b-rtx3090: vLLM pinned and patched, the body at 4-bit weights, heads and the MTP draft module requantised so the whole thing fits in 24 GB with a 114k context.

Stock Qwen3.8-27B has one problem for my use. It reasons too much. On open-ended code tasks it would spend the entire 8192-token thinking budget and emit nothing. Swift, a community finetune tuned for shorter reasoning, finished the same tasks in 2500 to 7100 thinking tokens and shipped working code. That difference dwarfs any decode-speed optimisation, so I switched.

The syv stack serves third-party checkpoints, Swift included, but at a lower tier than it serves its own model. Base Qwen has a “fast variant” there: an int4 GPTQ lm_head and draft module, plus a draft vocabulary counted over the model’s own outputs. Third-party checkpoints get int8 heads and the base-Qwen artefacts bolted on. That works. It also leaves performance on the table, and the reason is interesting.

Speculative artefacts are distribution-dependent

The MTP drafter in this stack scores a roughly 40k-row slice of lm_head instead of the full 248k vocabulary. A token outside that slice can never be drafted. Every miss is a guaranteed rejection, and a rejection truncates the speculation chain, so misses compound.

The shipped slice was counted over base Qwen’s outputs. Against what Swift actually generates, it covered 96.7 percent of tokens. Sounds fine. The 3.3 percent that could never be drafted is concentrated in exactly the contexts an agent lives in, code and tool syntax, and the number that matters is the chain-level effect, not the token-level one.

GPTQ calibration has the same property. The Hessian you quantise against is a function of the distribution that feeds the layer. Hidden states captured from base Qwen are the wrong calibration for a finetune with a different output style.

So the plan was to redo the fast variant recipe with everything derived from Swift itself. The syv repo already contained the recipe as scripts. I had to feed it the right inputs and handle a checkpoint layout it did not expect.

Corpus, outputs, vocabulary

Upstream’s drafter recipe consumed a Danish-heavy chat mix, which tells you something about who wrote it. My agent needed a coding mix, so I wrote a generator that emits 5500 workload-weighted prompts: 35 percent Rust implementation work, 16 percent TypeScript, 12 percent debugging with symptoms, 10 percent code edits, 8 percent tool-use transcripts, and the rest architecture questions, reasoning, and general chat. About 60 percent run with thinking on, because that is the daily mode. The generator is seeded, so the same corpus reproduces bit for bit.

Running 3072 of those prompts through Swift took about 1.7 GPU-hours and produced 4.2 million output tokens. I stopped there because held-out coverage had flattened: the vocab built from a quarter of the data already covered 99.17 percent of held-out outputs, and the full run reached 99.81.

The vocabulary itself is a frequency count over 90 percent of those outputs, specials forced in, everything else by rank. Two numbers from that build convinced me the work was real. First, coverage of held-out Swift output went from 96.7 to 99.81 percent, and 99.86 on code sources. Second, only 18701 of the base list’s 40960 ids appear in my list at all. Nearly half the base vocabulary is dead weight for this model, and the Swift corpus only ever emits 25879 distinct tokens, so the correct list is shorter than the slot count. A verifier that demanded exactly 40960 rows would be wrong, which mattered later.

GPTQ heads, and a bug worth a verifier

With outputs in hand, the rest of the recipe is upstream’s: capture hidden states for every token, dump MTP Hessians, then GPTQ the lm_head and the eight MTP linears to int4 against them. The lm_head quality metric is KL divergence to the bf16 head on held-out states. Round-to-nearest int4 gave 0.00707. GPTQ on Swift’s own Hessian gave 0.00234, better than the 0.0029 the base fast variant had shipped at.

Assembly was where it got educational. The fast directory shares body shards with the int8 directory through hardlinks, which is how a 15.2 GB variant costs almost nothing on disk. But vLLM reads every key of every shard it opens, not just the tensors named in the weight map. The first build hardlinked model-nonquant.safetensors, which still carried the old int8 MTP tensors from the int8 build. They sat next to the new int4 tensors in a different shard. The result was a shape assertion at load that named neither file, and every existence check passed, because every mapped tensor did exist, in the right place, exactly once. The duplicate under a different shard was invisible to anything that did not go looking for it.

That failure mode is dangerous precisely because it is quiet. So I wrote a structural verifier before promoting anything to the default boot target. It parses safetensors headers directly, no torch needed, checks that the packed and scale shapes agree with the bit widths the config declares, that the draft head’s rows match the id artefact, and that every index-mapped tensor lives in exactly one shard, the one the index names. The last check would have caught the collision before it ever booted. The repo ships a selftest that builds toy model dirs with stdlib only and corrupts them in specific ways, including that exact stale-tensor collision, and asserts the verifier catches each one.

What it bought

All five legs of the comparison ran the same night, four repetitions, medians. Decode speed in tokens per second, MTP acceptance, and VRAM:

variant	decode	acceptance	tok/step	VRAM MiB
Swift int8, MTP long	94.0	0.630	2.89	22085
Swift-fast, MTP long	98.4	0.660	2.98	22225
Qwen fast, MTP long	98.2	0.634	2.90	22191
Swift-fast, DFlash2	156.4	0.385	3.70	23081
Qwen fast, DFlash2	167.4	0.387	3.71	23007

The daily profile is MTP with the long context. Swift-fast reaches parity with base Qwen’s fast variant there, 98.4 against 98.2, with better acceptance. On a quote workload, reproducing 8000 tokens of its own context verbatim, the Swift vocab drafts at 0.998 acceptance and 3.99 tokens per step, which is the drafter doing nearly nothing wrong.

DFlash2 is the faster mode on paper and I do not use it as the default. It caps context at 46k and its acceptance ceiling is lower by design. Agent wall time tracks reasoning-token counts and finish rates, not raw decode, and the two ties that matter there went to long-context MTP. The remaining DFlash2 gap, 156 against 167, is about 0.24 GiB of extra weight bytes the AWQ body reads per step. No drafter fixes that. An AutoRound body would, but that costs a 55 GB source download and GPU hours to buy back 7 percent on a profile I do not daily-drive, so I did not.

Quality held: 8 of 9 on a mechanical battery, identical to the int8 build, with the one failure being a task both models flip on across samples. No reasoning-termination regressions.

Splitting the work

Two fixes in that verifier were not Swift-specific and belonged upstream, so they went upstream as PR 122 and PR 123. The first: upstream’s own verify.sh hard-required an int8 lm_head, so it rejected the int4 layout upstream’s own drafter produces and distributes. The second is the duplicate-tensor scan, generalised. Both are validated against real model dirs in the PR descriptions.

The target-specific pipeline stayed out. A workload mix is an opinion about someone’s usage, and upstream is deliberately model-agnostic. That is the split: generic fixes to the stack, recipe for deriving model-specific artefacts in the companion repo, swift-qwen3.8-rtx3090.

Publishing the weights

The finished model is public on the Hub: liamwh/Swift-Qwen3.8-27B-W4A16-syv-fast.

It can be served directly on the syv stack, no preparation pipeline to run first. The model card carries both licences and traces the provenance from Alibaba through UkisAI and TheUnderscore to this pipeline.

The upload itself was smaller than the model. The AWQ body shards are byte-for-byte identical to TheUnderscore’s checkpoint, so Xet deduplication only had to transfer what this project changed: the int4 heads, the MTP tensors, the extras, and the card. Roughly 2 GB of new data turned a 15.2 GB model into something anyone can download and run.

The pipeline repository still contains no model weights. It remains the reproducible recipe. Start from a checkpoint you are authorised to use and build the variant yourself.

There are two paths now. Reproduce the build from first principles, or download the finished model and run it.