Fine tune an 8B model on a 4GB laptop GPU (Soup setup)

This is the setup for Soup, the project from the reel. Everything below is from its own README, its benchmarks folder and the author's replies on Hacker News, read on 7 September 2026.

What it actually is: a fine tuning CLI driven by one YAML file. Its trick is layer streaming: while LoRA trains, the frozen base model never lives in VRAM. It sits in host RAM and streams into a small pool of VRAM buffers one decoder layer at a time, prefetched one ahead. Peak VRAM stops being the whole model.

The number, and exactly what it means

What	Figure
Card	RTX 3050 Laptop, 4GB
Model	Llama 3.1 8B Instruct, NF4
Peak VRAM	3.32GB
Speed	119.6 tok/s
Check	bit exact against a normal resident run, reproduced on an H100 at 113.00 tok/s in the same 3.32GB

Three things to know before you trust that row, all from the project itself.

It was measured on v0.72.2. The README says so: the tok/s figure predates the v0.73.0 correctness repair and has not been re run on a 4GB card since.

The current release probably uses less, and nobody has checked. v0.74.0 (4 September 2026) found that every SFT load had been upcasting the frozen base to fp32. Fixing that cut peak VRAM 2.59x on an H100 resident run, 48,241 MiB to 18,658 MiB. The 4GB streamed run has not been re measured on that build. Treat 3.32GB as a ceiling from an older version and measure your own.

Why the floor still depends on model size. The author, answering exactly that question on Hacker News: streaming only removes the decoder stack. The embeddings and the LM head stay resident, 2.10GB of the 3.32GB on 8B, and the logits tensor scales with batch times sequence times vocabulary. So the rule is "embeddings plus one layer plus logits must fit", which is why 8B works and why he did not try 14B.

Step 0, install

Python 3.10 to 3.12 only. Double quotes around the extra, in every shell.

pipx install "soup-cli[train]"

Already inside a venv, Colab or Docker? Plain pip with the same name works.

pip install "soup-cli[train]"

Step 1, the config

soup init --template chat

Then make soup.yaml look like this. The one line that turns streaming on is stream_layers: true under training. It is opt in and the README marks it BETA.

base: meta-llama/Llama-3.1-8B-Instruct
task: sft
data:
  train: ./data/train.jsonl
  format: alpaca
  val_split: 0.1
training:
  stream_layers: true
  epochs: 3
  lr: 2e-5
  batch_size: auto
  lora:
    r: 64
    alpha: 16
  quantization: 4bit
output: ./output

Llama 3.1 is a gated model on Hugging Face, so your account needs access to it before the download will start.

Step 2, train

soup train --config soup.yaml

The pre flight prints what it is about to do: on the README's own 8B run that is a 3.60GB base store pinned in RAM across 32 layers and two 113MB VRAM buffers. Which is the part people miss: system RAM matters more than VRAM here. The base sits in host memory and has to page lock. The author's floor for 8B is about 16GB of RAM.

Step 3, talk to it, merge it, ship it

soup chat --model ./output
soup merge --adapter ./output
soup export --model ./output --format gguf --quant q4_k_m

Check it yourself on a free Colab T4

The repo ships notebooks/proof-4gb.ipynb. It caps the process at 4GB, then asserts that a streamed model is bit identical to a normal one. The T4 crash that used to stop this on the free tier was fixed in v0.74.0.

What to expect, honestly

Slower than a card that fits the model. At 0.5B, where the author could measure both, resident training was 1.43x faster than streaming. His own words: do not buy a 4GB card for this. If you are buying, buy VRAM. Streaming is for the card you already own.

Opt in, and BETA. The VRAM table on the site is the normal resident path. Streaming is the flag above.

Old adapters can be inert. Anything trained with stream_layers: true on v0.72.0 saved empty tensors. Fixed in v0.72.1, so re run or re save if that was you.

Version pins bite. The declared torch>=2.5.0 floor does not work with trl>=0.29; a fresh install resolves a newer torch and is fine, a pinned 2.5.x environment is not.

The source

The project: https://github.com/MakazhanAlpamys/Soup

5,594 stars, 840 forks, Apache 2.0, first commit February 2026, v0.74.0 released 4 September 2026. The measurement records are in benchmarks/, and the method is a Zenodo preprint, version 3 of 13 August 2026: https://doi.org/10.5281/zenodo.21771064. Every claim on this page is from those, read on 7 September 2026. If it has changed since, trust the repo over this page.

What did your peak say?

Run one on whatever card you have, then tell me in The Rabbit Hole what your pre flight and your peak actually were, because the one number this page cannot give you is the one from your machine on the current release.

It is the room where I keep the parts that don't fit on a page like this: the full run of numbers, what broke the first time, and the ones that looked good and then didn't work at all.

Two catches, up front, because that's the rule here. It's private, so you have to be inside to read anything. And it's new, so you'd be early rather than joining something busy.

Free: https://www.skool.com/the-rabbit-hole-3206/about