Instructions to use facebook/MobileMoE-M-QAT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use facebook/MobileMoE-M-QAT with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="facebook/MobileMoE-M-QAT", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("facebook/MobileMoE-M-QAT", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use facebook/MobileMoE-M-QAT with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "facebook/MobileMoE-M-QAT"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "facebook/MobileMoE-M-QAT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/facebook/MobileMoE-M-QAT

SGLang

How to use facebook/MobileMoE-M-QAT with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "facebook/MobileMoE-M-QAT" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "facebook/MobileMoE-M-QAT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "facebook/MobileMoE-M-QAT" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "facebook/MobileMoE-M-QAT",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use facebook/MobileMoE-M-QAT with Docker Model Runner:
```
docker model run hf.co/facebook/MobileMoE-M-QAT
```

You need to agree to share your contact information to access this model

The information you provide will be collected, stored, processed and shared in accordance with the Meta Privacy Policy.

MobileMoE-M (QAT) Model Card

MobileMoE is a family of on-device Mixture-of-Experts (MoE) language models with sub-billion active parameters, designed to push the quality–efficiency Pareto frontier for on-device LLMs, including three model scales (S/M/L): 0.3B/0.5B/0.9B active parameters (1.3B/2.8B/5.3B total), with <3 GB INT4 weight footprints to fit in mobile DRAM. Each scale is released in three variants: a Base model (pre-training + mid-training), an SFT model (supervised fine-tuning), and a QAT model (quantization-aware training). You are currently in the MobileMoE-M-QAT repository — the INT4 quantization-aware-trained 0.5B-active model, built on top of MobileMoE-M-SFT and intended for on-device deployment.

	S	M	L
Active / total params	272M / 1.3B	528M / 2.8B	922M / 5.3B
Layers	20	26	32
Model dimension	768	1024	1280
Heads (Q / KV)	12 / 4	16 / 4	20 / 4
Routed experts	60	60	60
Top-k	4	4	4
Projected INT4 weight memory	0.68 GB	1.48 GB	2.75 GB
Base	MobileMoE-S-Base	MobileMoE-M-Base	MobileMoE-L-Base
SFT	MobileMoE-S-SFT	MobileMoE-M-SFT	MobileMoE-L-SFT
QAT (INT4)	MobileMoE-S-QAT	MobileMoE-M-QAT	MobileMoE-L-QAT

For the detailed technical report: 📝 MobileMoE: Scaling On-Device Mixture of Experts

For more versions, check out the 🤗 MobileMoE Collection

MobileMoE establishes a new Pareto frontier for on-device LLMs. Average benchmark accuracy, computed over 14 benchmarks spanning commonsense, knowledge, science, comprehension, and reasoning, is plotted against (a) per-token inference compute F_inf = 2N_act (GFLOPs) and (b) total parameters N_total (B); in (b), x-axis tick labels show total params (B) | projected INT4 memory (GB). Accuracy is shown for the instruction-tuned models.

Key Features

A new Pareto frontier for on-device LLMs. Across 14 foundational benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs at 2–4× fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters.
Scaling-law-derived architecture. The architecture is derived from an on-device MoE scaling law that jointly optimizes under mobile memory and compute constraints, identifying an on-device sweet spot: moderate sparsity, with fine-grained experts and shared expert.
Four-stage recipe. Pre-training → mid-training → instruction fine-tuning → INT4 quantization-aware training, all on open-source datasets.

Model Information

Model: MobileMoE-M-QAT (INT4, quantization-aware trained)
Active Parameters: 528M
Total Parameters: 2.8B
Layers: 26
Model Dimension: 1024
Attention Heads: 16
KV Heads: 4 (GQA)
Head Dimension: 64
Routed Experts: 60 (fine-grained, FFN hidden dim 512 each)
Active Experts per Token: 4 (top-k sigmoid routing, with normalization)
Shared Expert: 1, always on (FFN hidden dim 2048)
Vocabulary Size: 128,256
Other Features: QK-Norm, tied input/output embeddings, RoPE (θ = 500,000)
Input Modality: Text
Output Modality: Text
Languages: English
Chat Template: Yes (end-of-turn token <|eot|>)
Training Stages: Pre-training → mid-training → SFT → INT4 QAT
Context Length: 8,192 tokens
Precision: INT4 weights (symmetric, group size 32; FP16 scales); router stored and routing computed in FP32; BF16 norms and dense compute
Projected INT4 Weight Memory: 1.48 GB
Model Developer: Meta
Model Release Date: Aug 2026
License: MobileMoE is FAIR NC licensed

Quantization

INT4 quantization-aware training is applied to the SFT checkpoint. Weights are quantized as

W̃g = sg · clamp(round(Wg / sg), −8, 7),    sg = 2·max(|Wg|) / (2^b − 1),   b = 4

where Wg is a contiguous group of 32 weights sharing one scale sg.

component	precision
Attention (q/k/v/o), routed experts, shared expert, embeddings	INT4, symmetric, group size 32
Router	Stored and routing computed in FP32
RMSNorm weights, expert bias	BF16

Note: Projected INT4 weight memory is a weight-only estimate. Packaged checkpoint files (model.safetensors) are 0.71 / 1.59 / 3.00 GB for S / M / L, including FP16 group scales, FP32 router weights, and BF16 norms/biases. Runtime memory is backend-dependent and additionally includes the KV cache, activations, and other overhead.

Results

All numbers below are for INT4-quantized models, re-evaluated under identical settings with greedy decoding using lm-eval; few-shot counts are given in parentheses after the benchmark name, and benchmarks shown without one are evaluated 0-shot. All three MobileMoE scales are the exact weights released in their respective QAT repositories, which include brief fine-tuning with self-identity beyond the QAT checkpoints in the technical report. This produces small differences against the report: averages of 43.9 / 52.7 / 58.1 here versus 44.0 / 52.5 / 57.8 for S / M / L in the report.

Foundational benchmarks (INT4)

Benchmark	MobileLLM-Pro	MobileMoE-S	MobileMoE-M	MobileMoE-L
Active / total params	1.1B	272M / 1.3B	528M / 2.8B	922M / 5.3B
Commonsense Reasoning
HellaSwag	64.7	53.3	63.7	70.7
PIQA	75.6	73.9	76.4	78.9
SIQA	47.4	45.3	48.7	53.4
WinoGrande	62.8	56.6	61.6	65.4
Knowledge
MMLU (5-shot)	30.4	39.0	52.2	57.4
NaturalQuestions (5-shot)	13.9	8.6	4.7	19.3
TriviaQA (5-shot)	39.9	25.2	42.7	52.7
Science
ARC-Challenge (25-shot)	51.6	42.0	51.5	55.6
ARC-Easy	75.2	69.3	79.2	80.6
OpenBookQA	42.8	30.6	36.6	40.4
Reading
BoolQ	76.8	71.2	77.1	78.3
DROP (3-shot)	20.5	26.3	43.5	47.3
Reasoning
BIG-Bench Hard (3-shot)	31.4	30.8	37.0	39.5
GSM8K (8-shot)	4.1	42.9	63.5	73.2
Average	45.5	43.9	52.7	58.1

Training

MobileMoE uses a four-stage recipe. This checkpoint is the output of stage 4 (INT4 QAT).

MobileMoE four-stage training recipe: pre-training (PT) → mid-training (MT) → instruct supervised fine-tuning (SFT) → quantization-aware training (QAT) with INT4 precision.

	Pre-training	Mid-training	SFT	QAT
Context length	2,048	8,192	8,192	8,192
Total tokens	~6T	~500B	~126B	~21B
Peak learning rate	4×10^-4	4×10^-5	4×10^-6	4×10^-6
LR schedule	Cosine	Linear	Cosine	Cosine
Token dispatch	drop-and-pad	drop-and-pad	dropless	dropless

How to use

MobileMoE uses a custom architecture (model_type: mobilemoe) that is not yet part of upstream transformers, so trust_remote_code=True is required. The modeling code ships in this repo (configuration_mobilemoe.py, modeling_mobilemoe.py), and it handles INT4 dequantization automatically.

Requirements

pip install "torch>=2.1" "transformers>=4.57" "safetensors>=0.4" "accelerate>=1.0"

Verified with the following versions:

Package	Version
`torch`	2.8.0 (cu128)
`transformers`	4.57.6
`tokenizers`	0.22.2
`safetensors`	0.7.0
`accelerate`	1.13.0

Chat

This model is instruction-tuned (QAT is applied on top of the SFT checkpoint), so use the chat template:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "facebook/MobileMoE-M-QAT"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    dtype=torch.bfloat16,
)
model.to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

messages = [{"role": "user", "content": "Why are open-source on-device language models great?"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    max_new_tokens=1024,
    do_sample=False,
    temperature=None,
    top_p=None,
    pad_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

For multi-turn conversations, append each generated reply to messages with the assistant role. This ensures that each subsequent prompt includes the complete conversation history:

messages = []

for user_message in ["Who are you?", "Why are open-source on-device language models great?"]:
    messages.append({"role": "user", "content": user_message})
    input_ids = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt"
    ).to(model.device)
    outputs = model.generate(
        input_ids,
        attention_mask=torch.ones_like(input_ids),
        max_new_tokens=1024,
        do_sample=False,
        temperature=None,
        top_p=None,
        pad_token_id=tokenizer.eos_token_id,
    )
    reply = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True).strip()
    messages.append({"role": "assistant", "content": reply})
    print(reply)

Known issues.

Loading the tokenizer on transformers 4.57.6 prints a fix_mistral_regex=True warning. Please ignore it and do not set the flag, as MobileMoE uses the Llama-3 tokenizer whose default tokenization is already correct.
The checkpoint stores qweight / weight_scale and creates dense weights on the first forward pass, so a "newly initialized" warning before then is expected.
vLLM does not support this repository's INT4 format directly because its Transformers fallback replaces the custom layers and ignores the INT4 tensors. Convert the checkpoint to BF16 as described below, or use MobileMoE-M-SFT instead.

Exporting the INT4 Checkpoint to BF16

No conversion is needed for general use (e.g., with transformers) because the custom modeling code dequantizes the INT4 checkpoint to BF16 on the first forward pass. This section exports a numerically identical BF16 checkpoint to support other runtimes (e.g., vLLM) that bypass the custom loading logic and cannot use qweight / weight_scale directly.

import os, shutil, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

SRC, DST = "facebook/MobileMoE-M-QAT", "./MobileMoE-M-QAT-bf16"

model = AutoModelForCausalLM.from_pretrained(SRC, trust_remote_code=True, dtype=torch.bfloat16)
model(torch.tensor([[128000]]))                 # first forward materializes INT4 -> BF16

state = {k: v for k, v in model.state_dict().items()
         if not (k.endswith("qweight") or k.endswith("_scale"))}
model.config.quantization = None                # mark the copy as plain BF16
model.save_pretrained(DST, state_dict=state, safe_serialization=True)
AutoTokenizer.from_pretrained(SRC).save_pretrained(DST)

# a trust_remote_code architecture needs its modeling files alongside the weights
for f in ("modeling_mobilemoe.py", "configuration_mobilemoe.py"):
    src_f = os.path.join(SRC, f) if os.path.isdir(SRC) else __import__(
        "huggingface_hub").hf_hub_download(SRC, f)
    shutil.copy2(src_f, os.path.join(DST, f))

Citation

@article{chen2026mobilemoe,
  title={MobileMoE: Scaling On-Device Mixture of Experts},
  author={Chen, Yanbei and Huang, Hanxian and Chang, Ernie and Szwejbka, Jacob and Desai, Digant and Liu, Zechun and Chandra, Vikas and Krishnamoorthi, Raghuraman},
  journal={arXiv preprint arXiv:2605.27358},
  year={2026}
}