gianlucamazza/SmolLM2-360M-Instruct-ort-genai-int4-cpu

SmolLM2-360M-Instruct: ONNX Runtime GenAI, int4, CPU

An int4 ONNX export of HuggingFaceTB/SmolLM2-360M-Instruct for ONNX Runtime GenAI on the CPU execution provider.

All model weights and training are by the Hugging Face SmolLM team. This repo contains only a converted, quantized copy. It is not a new model and it was not fine-tuned.

Who it is for

Developers who want to try a small instruct model with ONNX Runtime GenAI on a plain CPU, without running the conversion themselves. It is a test artifact, not a model for production use.

Status and limits

• Experimental. No quality evaluation of this quantization is published. Expect some quality loss compared with the original fp16/bf16 model. Check outputs on your own task before you rely on it.
• The base model "primarily understands and generates content in English" and its output may be inaccurate or biased (base model card, Limitations).
• The context length set in genai_config.json is 8192.
• I made this conversion while building xllama, local LLM inference on Xbox Series S. This file has not been run on an Xbox, and its speed has not been measured. Independent project, not affiliated with or endorsed by Microsoft. Xbox is a trademark of the Microsoft group of companies. Not affiliated with Hugging Face.
• This repo matches xllama's catalogue entry smollm2-360m-cpu-int4 in base model, int4 precision and ORT CPU runtime. It is not the file xllama ships, so xllama's throughput figures do not describe this repo.

Files

model.onnx + model.onnx.data (int4 weights), genai_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja (ChatML).

How to run

pip install onnxruntime-genai huggingface_hub
hf download gianlucamazza/SmolLM2-360M-Instruct-ort-genai-int4-cpu --local-dir smollm2-int4

import onnxruntime_genai as og

model = og.Model("smollm2-int4")
tokenizer = og.Tokenizer(model)
prompt = "<|im_start|>user\nWhat is the capital of Italy?<|im_end|>\n<|im_start|>assistant\n"

params = og.GeneratorParams(model)
params.set_search_options(max_length=256)
generator = og.Generator(model, params)
generator.append_tokens(tokenizer.encode(prompt))
stream = tokenizer.create_stream()
while not generator.is_done():
    generator.generate_next_token()
    print(stream.decode(generator.get_next_tokens()[0]), end="", flush=True)

License

Apache-2.0, the same as the base model. Keep the base model's attribution when you redistribute.

Upstream contributions (xllama)

Running xllama on an Xbox Series S in Developer Mode surfaced problems in the open-source ONNX Runtime projects. I opened these pull requests. Status as shown on GitHub, checked 2 Nov 2026:

• onnxruntime-genai #2280: falls back to the system D3D12 runtime when the Agility SDK device factory cannot create a device. Merged, 13 Jul 2026.
• onnxruntime-genai #2300: lets a model opt out of DirectML graph capture through a provider option. Merged, 20 Jul 2026.
• onnxruntime-genai #2316: adds tests and documentation for the #2300 opt-out. Open.
• onnxruntime #29732: reduces the Windows ReadFileIntoBuffer chunk size to 16 MB (AppContainer error 1450). Open.

Issues reported against onnxruntime. Both were closed for inactivity, without a fix:

• #29739: wrong decoder logits on the DirectML execution provider on Xbox Series S, correct on the CPU execution provider. Closed for inactivity, 18 Sep 2026.
• #29730: large ReadFileIntoBuffer chunks can fail with ERROR_NO_SYSTEM_RESOURCES (1450) inside a Windows AppContainer. Pull request #29732 proposes the change. Closed for inactivity, 18 Sep 2026.

This work came from xllama on the Xbox. This repo's CPU conversion has not been run on an Xbox (see Status and limits). The onnxruntime.dll and onnxruntime-genai.dll that xllama ships are patched by xllama and are not official builds.

Credits

• Base model: SmolLM2 by Hugging Face (Allal et al., 2025, SmolLM2: When Smol Goes Big).
• Conversion: Gianluca Mazza (GitHub). I take LLM systems into production: state, recovery, eval, cost.