Run Microsoft BitNet on your CPU

First, the catch from the Reel: 100B at 6.58 tokens per second was a benchmark configuration tested on an Apple M2 Ultra with 64GB RAM. It is not Microsoft's downloadable model.

The official release is BitNet-b1.58-2B-4T, a 2.4B-parameter model trained on 4 trillion tokens.

What you need

• Python 3.10 or newer
• CMake 3.22 or newer
• Clang 18 or newer
• Conda is recommended

Setup

git clone --recursive https://github.com/microsoft/BitNet.git
cd BitNet

conda create -n bitnet-cpp python=3.10
conda activate bitnet-cpp
pip install -r requirements.txt

huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf \
  --local-dir models/BitNet-b1.58-2B-4T

python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

python run_inference.py \
  -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf \
  -p "You are a helpful assistant" \
  -cnv

Use Microsoft's current README if a command changes.

Limits

• This setup runs the released 2.4B model, not the 100B benchmark configuration.
• BitNet is built around ternary models. It does not turn an ordinary model into a native 1.58-bit model.
• Your speed depends on CPU, memory bandwidth, thread count and kernel support.