Run a 70B model on a 4GB card (AirLLM setup)

The whole idea: AirLLM never loads the full model. It puts one layer on the GPU, runs it, swaps it out, then loads the next one. So the memory you need depends on the biggest single layer, not the size of the model.

That means roughly 4GB for a 70B, 8GB for a 405B, and about 12GB for DeepSeek V3 at 671B. No quantization, so it's the real model rather than a shrunk copy.

Install

pip install airllm

On Apple silicon it runs through mlx, so you'll also want pip install mlx torch.

The whole thing

from airllm import AutoModel

model = AutoModel.from_pretrained("Qwen/Qwen2.5-7B-Instruct")

input_tokens = model.tokenizer(["what is 2+2"], return_tensors="pt").input_ids.cuda()
output = model.generate(input_tokens, max_new_tokens=20)

print(model.tokenizer.decode(output[0]))

Swap the model name for whatever you want to run. It handles Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan and InternLM.

Which model to start with

• 4 to 6GB card: start with a 7B or 8B just to confirm it works end to end, then jump straight to a 70B. The small one is the sanity check, not the destination.
• 8GB or more: 70B is the point of this whole thing. Start there.
• Apple silicon: same idea through mlx.

There's an optional block-wise compression mode (4-bit or 8-bit) that the project says gives about a 3x speedup. Get it working plain first, then turn that on.

Two things the repo doesn't warn you about

The download is enormous. First run pulls the entire model to disk. A 70B is a lot of gigabytes. Start it before you need it, not while someone is watching.

It is slow, by design. Every single token walks through every layer, and the layers are being moved in and out. This is a tool for running a model you could not otherwise run at all. It is not a tool for chatting.

If you want fast local inference on small hardware, a quantized smaller model through llama.cpp or Ollama will beat this every time. Different problem, different tool.

Source

github.com/lyogavin/airllm, Apache 2.0.

---

Tell me which card you're running and I'll tell you what to try first.