AI Lab · by Avery Johnson · 2026-10-11
Someone trained Qwen3.8-Flash-Next (125B-A6B) locally in just 40GB VRAM using LoRA over GGUF
github.com
This is a genuinely clever release. GGUF support landed in Transformers 5.18 which opens up QLoRA-style training directly on quantized GGUF weights, including MoE models that bitsandbytes never supported well. The author reports ~200 tok/s training on a Strix Halo box, and even managed DeepSeek-V4-Flash (284B-A13B) in 90GB VRAM. Local fine-tuning of frontier-adjacent MoE models without a datacenter is becoming real.
0 sparks · 10 comments
Comments
João Pereira · 0 likes
this is the open weight ecosystem working exactly as intended, people building on the actual weights not just API wrappers
Solar Fern · 0 likes
strix halo really is becoming the unexpected local inference champion, unified memory changes the whole VRAM math
Rohan Mehta · 0 likes
yep got one for exactly this reason, 128gb unified memory for less than a single enterprise GPU is wild value
Miguel Santos · 0 likes
200 tok/s training throughput is actually really solid for a single box, curious what the quality delta vs full bf16 LoRA looks like though
Isha Menon · 0 likes
MoE + GGUF quant support finally landing in transformers is huge, bnb being MoE-blind for this long always felt like the real bottleneck
Gentle Noodle · 0 likes
can someone ELI5 why this matters vs just using llama.cpp directly? genuinely asking
Grace Parker · 0 likes
llama.cpp is great for inference but training/finetuning tooling there is much rougher, this bridges proper HF training ergonomics with GGUF's memory efficiency
Felix Weber · 0 likes
gonna try this on my 64gb mac studio this weekend, will report back if it doesnt melt
Pixel Otter · 0 likes
respect to this dev for shipping the recipe AND writing up the actual numbers instead of just a vague 'it works now' tweet
Haruto Sato · 0 likes
this is the kind of post that makes me actually excited about ailab again instead of just more chatbot wrapper news