Run a 27B model in 6.2GB, no GPU (Qwen3.8 setup)

Qwen3.8-27B is the #1 trending model on Hugging Face right now, and the reason it fits on a normal laptop is quantization: Unsloth's dynamic quants squeeze the full 27B down to a fraction of its 55GB full-precision size, and llama.cpp runs the result on CPU. Same model, same weights, stored coarser. The smaller the file, the more quality you trade.

It is also a vision model. It reads images and long videos, not just text, which needs one extra file (covered below).

Pick your size first

One rule: take the biggest file that fits in your RAM with about 2GB to spare for context and the OS. These sizes are from the repo's own file listing, not the description.

File	Size on disk	Machine it suits	The trade
Qwen3.8-27B-UD-IQ1_S	6.19 GB	8 GB RAM, tight but real	1-bit, the ragged edge of quality
Qwen3.8-27B-UD-IQ2_XXS	7.27 GB	8 to 12 GB RAM	2-bit, the sensible floor
Qwen3.8-27B-UD-IQ3_XXS	10.93 GB	16 GB RAM	3-bit, noticeably closer to full
Qwen3.8-27B-UD-Q4_K_M	16.46 GB	24 GB RAM	4-bit, the usual sweet spot
BF16 (two shards)	54.66 GB	server territory	none, this is the reference

The RAM column is a rule of thumb (file size plus headroom, since llama.cpp memory-maps the weights), not a published spec. If the load fails or the machine swaps, go one row up.

Install llama.cpp

Prebuilt releases exist for Linux, macOS and Windows at github.com/ggml-org/llama.cpp/releases. Download, unzip, done. Or via a package manager:

brew install llama.cpp        # macOS
winget install llama.cpp      # Windows

Get the model

Everything lives in one repo: huggingface.co/unsloth/Qwen3.8-27B-GGUF. Pull the one file you picked above:

pip install -U huggingface_hub
hf download unsloth/Qwen3.8-27B-GGUF Qwen3.8-27B-UD-IQ2_XXS.gguf --local-dir ./qwen38

Swap the filename for whichever row you chose.

Run it

llama-cli -m ./qwen38/Qwen3.8-27B-UD-IQ2_XXS.gguf -p "why is the sky blue?"

Or as a local server with a web UI at localhost:8080:

llama-server -m ./qwen38/Qwen3.8-27B-UD-IQ2_XXS.gguf

Give it eyes

The vision half needs the projector file from the same repo (about 0.93 GB):

hf download unsloth/Qwen3.8-27B-GGUF mmproj-F16.gguf --local-dir ./qwen38
llama-server -m ./qwen38/Qwen3.8-27B-UD-IQ2_XXS.gguf --mmproj ./qwen38/mmproj-F16.gguf

Then attach images in the web UI. Without the mmproj file the model is text-only and will not tell you why.

Where it will disappoint you

Nobody has published CPU speeds for this model yet, and a 27B on CPU will not feel like an app. I am not going to invent a tokens-per-second number here. Measure your own before judging it:

llama-bench -m ./qwen38/Qwen3.8-27B-UD-IQ2_XXS.gguf

1-bit is the aggressive end. The 6.2GB file is the headline and it is real, but every bit you shave costs answer quality. If your machine holds the 2-bit or 3-bit, take it. Unsloth's claim is that their dynamic method beats other quants at the same size, not that 1-bit is free.

8GB of RAM means a tight fit. The 6.19GB file plus context plus your browser is most of the machine. Close things, keep the context modest, and expect the first load to take a while from a slow disk.

Source

huggingface.co/Qwen/Qwen3.8-27B, released 5 August 2026, Apache 2.0, 12,672 likes and about 2.9M downloads on the base repo, 7.3M more on the GGUF repo, as of 26 August 2026.

---

What machine are you on, and how much RAM?

Ask me in The Rabbit Hole and I will tell you which file to download. I read every one.

It is the room where I keep the parts that don't fit on a page like this: the full run of numbers, what broke the first time, and the ones that looked good and then didn't work at all.

Two catches, up front, because that's the rule here. It's private, so you have to be inside to read anything. And it's new, so you'd be early rather than joining something busy.

Free: skool.com/the-rabbit-hole-3206