Serve a 1.02T open model: the MiMo-V2.6-Pro setup

MiMo-V2.6-Pro is Xiaomi's open model. Its Hugging Face repo went up on 21 September under an MIT license, and on 2 October it sat first on benchlm's open-weight ranking (75.5, with Qwen3.8 Max second at 72.1). That ranking is third party and it can change, so read it as a snapshot.

It needs a multi-GPU server. The commands, hardware figures and flags below come from the model card, the vLLM recipe and the repo's own files, checked on 4 October 2026, and where I add arithmetic of my own I say so.

What it is

• A sparse mixture of experts: 384 routed experts per layer, 8 used for each token, 70 layers.
• 1.02T parameters in total, 42B of them active per token.
• It takes text, images, video and audio, with a context of up to 1M tokens.
• It ships with a built-in 5 layer speculative decoder, which the EAGLE flags below switch on.

Why it needs a server

The weights are stored as MXFP4, a 4-bit block format. That is how 1.02T parameters come to 566 GB (the total_size in the repo's weight index) instead of about 2 TB in bf16. The whole repo, with the separate drafter and audio tokenizer, is 573 GB.

A mixture of experts only computes 42B parameters per token, but every expert has to sit in memory. The vLLM recipe asks for at least 680 GB of GPU memory in total, for example 8 H200s or 4 MI355Xs. The card's SGLang command spans two machines and 16 GPUs. Dividing 566 GB by 8 gives 71 GB, but that is my arithmetic and only the stored bytes split eight ways. It is a floor, not a requirement.

Try it without a cluster

The card says it is also served in Xiaomi's AI Studio, the Xiaomi MiMo API platform and on OpenRouter. On OpenRouter the id is xiaomi/mimo-v2.6-pro, listed at $0.435 per million input tokens and $0.87 per million output tokens on 4 October. That is a way to try the model before you rent GPUs. The hosted setup has the call and the Command Code config, and there's a course for that route in The Rabbit Hole: Use MiMo-V2.6-Pro without a GPU.

Serve it with SGLang

The card says to use SGLang "for best performance", with the Docker image lmsysorg/sglang:latest. latest moves, so pin a version you've tested. This is the card's command, unchanged. It spans two machines. Before you paste it, replace <node-rank> with 0 on the first machine and 1 on the second, and <node0-ip> with the first machine's reachable IP address. As written, the angle brackets are shell redirection and it won't run.

It also listens on 0.0.0.0:30000 with no API key, so anyone who can reach that port can use your GPUs. Keep it on a private network, or behind a firewall and authentication.

sglang serve \
  --trust-remote-code \
  --model-path XiaomiMiMo/MiMo-V2.6-Pro-RL \
  --tp 16 \
  --dp 2 \
  --enable-dp-attention \
  --mm-enable-dp-encoder \
  --ep 16 \
  --moe-a2a-backend deepep \
  --moe-dense-tp-size 1 \
  --mem-fraction-static 0.7 \
  --max-running-requests 128 \
  --chunked-prefill-size 32768 \
  --page-size 64 \
  --swa-full-tokens-ratio 0.3 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-multi-layer-eagle \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --host 0.0.0.0 \
  --port 30000 \
  --nnodes 2 \
  --node-rank <node-rank> \
  --dist-init-addr <node0-ip>:20000

What the key flags do:

Flag	What it does
--tp 16	one tensor parallel group of 16 GPUs, across the 2 machines
--dp 2 --enable-dp-attention	inside that group, attention runs as 2 data parallel groups. SGLang's current docs mark this flag deprecated (use --attn-dp-size 2, removal after 31 December 2026)
--ep 16 --moe-a2a-backend deepep	spreads the experts over 16 GPUs, with DeepEP moving tokens between them
--moe-dense-tp-size 1	sets tensor parallelism for the dense, non expert layers to 1
--speculative-algorithm EAGLE --enable-multi-layer-eagle	speculative decoding, using the model's own drafter to propose tokens that the full model then checks
--page-size 64	allocates the KV cache in pages of 64 tokens
--chunked-prefill-size 32768	feeds long prompts in chunks of 32,768 tokens
--nnodes 2	two machines

Or vLLM, the shorter command

The card's vLLM command uses 8 GPUs and far fewer flags, and it is the same command the vLLM recipe gives for 8 H200s. The card itself still points at the V2.5 recipe and image. The V2.6 recipe says stable vLLM (0.29.0 and older) can't load these MXFP4 weights, so use vllm/vllm-openai:mimo-v26, or a nightly build from after 20 September 2026. The same warning about the open port applies.

vllm serve XiaomiMiMo/MiMo-V2.6-Pro-RL \
  --tensor-parallel-size 8 \
  --trust-remote-code \
  --gpu-memory-utilization 0.95 \
  --max-model-len auto \
  --reasoning-parser mimo \
  --tool-call-parser mimo \
  --enable-auto-tool-choice \
  --generation-config vllm

Sampling the card recommends: temperature=1.0, top_p=0.95.

Where it will disappoint you

The card has no speed numbers. It publishes accuracy tables and no tokens per second, so treat any speed you read elsewhere with care. I'm not going to guess one.

The benchmark table is Xiaomi's own. It has the model ahead of Claude Opus 5 on some tests (Terminal Bench 2.1, 89.9 to 89.1, and AutomationBench, 53.1 to 50.3) and well behind on others (Terminal Bench 4.0, 34.9 to 49.0, and ExploitBench, 47.9 to 70.0). Check a test that looks like your work before you believe the headline.

Price it before you rent. Compare what a hosted route charges for your workload with what 8 to 16 GPUs cost per hour.

Source

huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL: the model card, config.json and the weight index, read on 4 October 2026, with the vLLM recipe and SGLang's DP attention guide. Ranking: benchlm.ai/best/open-source, as of 2 October 2026. OpenRouter prices: its public model list, 4 October 2026.

Where the longer version lives

The Rabbit Hole is the community where I post the setups, the ones that worked and the ones that didn't. Tell me what you want to run it for and I'll say whether you need the cluster at all. It's free to join; Premium ($5 a month) adds weekly office hours. skool.com/the-rabbit-hole-3206

There's a course for the hosted route there too, Use MiMo-V2.6-Pro without a GPU, and the other open courses are on the courses page.