Instructions to use halt95/Hy4-flash-W4A16-Merlin with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Merlin
How to use halt95/Hy4-flash-W4A16-Merlin with Merlin:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
halt95/Hy4-flash-W4A16-Merlin
Hy4-flash W4A16, powered by Merlin: a 4-bit-expert checkpoint, 152.6 GB, with the full 262,144-token context on four 24 GB GPUs. A 609,398-token KV pool in VRAM, 75.7 tok/s one-stream decode at 4K, and 1,608 tok/s prefill on a 100K prompt.
A 4-bit-expert build of Hy4-flash, Tencent's MoE model with 220B total parameters (including 63.5B Engram parameters), served here at 262,144 tokens of context on four 24 GB GPUs. It is 152.6 GB. The routed experts are W4A16 (4-bit weights, 16-bit activations). The attention projections Tencent ships in FP8 keep their FP8 bytes. The four Engram tables are FP8 and live in pinned host RAM instead of on the GPUs. The DFly2 drafter's linear layers are INT4, and everything else is unchanged BF16. Merlin serves it for text only: the vision encoder's weights are in the files, but Merlin does not load them, so image input is not supported. It is served by the Merlin engine, a vLLM fork tuned for Ampere and Ada. It needs Merlin 3.0.2.2 or later; earlier Merlin versions and stock vLLM cannot serve it. Measured on RTX 3090s.
Numbers: Merlin 3.0.2.2 on 4× RTX 3090 without NVLink, 220 W, P2P on, measured 2026-11-21 (UTC). KV pool: FP8 (fp8_ds_mla), all in VRAM. Decode: one stream, thinking on at reasoning effort high, T=0, each send a cold prefill, median of 3 sends. Prefill: the median of 3 cold sends, prompt tokens / time to first token. Each figure is also in a table below.
The checkpoint
| Component | Format |
|---|---|
| Routed experts (512 per layer, 16 active) | W4A16, round-to-nearest (RTN), symmetric, group size 128, no calibration |
Attention q_b_proj, o_proj and linear_gate of layers 1-41 |
FP8 block-128: Tencent's bytes from tencent/Hy4-flash-FP8, copied unchanged |
| Engram tables (4) | FP8, one scale per row, held in pinned host RAM at serve time (EngramConfig cpu_offload=True) |
| DFly2 drafter (5 layers) | Linear layers INT4 (W4A16, group size 128, symmetric RTN); its norms and small projections stay BF16 |
| Everything else: layer 0, the other attention projections, shared experts, latent projections, routers, indexers, norms, lm_head (shared with the drafter), token embedding, vision encoder | BF16, unchanged; the token embedding is held in host RAM at serve time |
| KV cache | FP8 (fp8_ds_mla) |
The checkpoint uses the compressed-tensors layout. We also measured an INT8 drafter-head variant (904 MiB) against this INT4 drafter and did not ship it: it accepted fewer draft tokens and used more memory on the four-card layout.
Two builds
| FP8 Engram (default) | BF16 Engram | |
|---|---|---|
| Branch | main |
engram-bf16 |
| Weights without Engram (on the GPUs) | 88.0 GB | 88.0 GB (the same files) |
| Engram tables (in host RAM) | 64.7 GB (FP8, one scale per row) | 126.7 GB (Tencent's BF16 rows, unchanged) |
| Download in all | 152.6 GB | 214.7 GB |
| Pinned host RAM for Engram | about 60 GiB | about 118 GiB |
| Host RAM available before the start | about 140 GiB | about 198 GiB |
| Status | Everything on this card was measured on it, including the KL divergence against BF16 in Quality | Untested on hardware: our four-card host has too little host RAM; no speed or quality rows |
The Engram tables are 660 million rows of 96 values: 98 bytes a row in FP8 (96 values and a 2-byte scale), 192 bytes in BF16. Both builds share the same W4A16 experts and every other tensor; only the Engram tables differ. To get the BF16 Engram build, download it with hf download halt95/Hy4-flash-W4A16-Merlin --revision engram-bf16. Select it with ENGRAM=bf16 for tools/merlin/launch.py, or with ./setup.sh --model hy4flash --engram bf16. The host RAM, speed and quality figures elsewhere on this card are for the FP8 Engram default.
Files in this repository (branch main)
| Files | What they hold | Size |
|---|---|---|
model-00001-of-00148.safetensors … model-00147-of-00148.safetensors (147 shards) |
Routed experts (W4A16), FP8 attention projections, Engram tables, the BF16 tensors and the vision encoder | 152.09 GB |
model-dfly-int4.safetensors |
DFly2 drafter (INT4) | 527.1 MB |
model-expert-shapes.safetensors |
The shapes of the packed expert weights | 7.7 MB |
model.safetensors.index.json |
Which file holds each of the 191,005 tensors | 18.1 MB |
config.json, generation_config.json, processor_config.json, tokenizer.json, tokenizer_config.json |
Model, quantisation, generation and tokenizer settings | small (in git, not LFS) |
chat_template.jinja |
The chat template Merlin serves: thinking at high, or off with no_think |
small (in git, not LFS) |
LICENSE |
Tencent Hy License 1.0, verbatim copy of Tencent's file | small (in git, not LFS) |
README.md |
This card | small (in git, not LFS) |
152.64 GB (152,642,614,330 bytes) over the 150 LFS files. Branch engram-bf16 has the same file names; its shards hold the BF16 Engram tables, 214.68 GB (214,683,216,043 bytes) in all.
Speed
| Prompt | Decode, one stream | Time to first token |
|---|---|---|
| 4K (3,840 tokens) | 75.7 tok/s | 2.55 s |
| 128K (130,816 tokens) | 67.1 tok/s § | 84.0 s |
| 262K (261,888 tokens) | 89.3 tok/s § | 189.3 s |
4× RTX 3090, thinking on at reasoning effort high, 256 tokens out (128 at 262K), median of 3 sends; decode tok/s = (completion tokens − 1) / (last − first token time). 4K is measured at the shipped DFly2 draft length, K=3 (sends 73.9–80.2). § Measured with K=5, not the shipped K=3: 128K sends 66.3–71.4, 262K sends 77.6–89.7; at 4K, K=5 gave 75.3 tok/s against K=3's 75.7. Decode speed depends on how many draft tokens the model accepts for the text it is writing, which is why the 262K row is faster than the 128K row. With DFly2 off, 4K decode is 45.1 tok/s.
| Cold prefill, median of 3 cold sends | Prefill | Time to first token |
|---|---|---|
| 10,240 tokens | 1,672 tok/s | 6.13 s |
| 102,400 tokens | 1,608 tok/s | 63.68 s |
| 261,888 tokens | 1,384 tok/s | 189.3 s |
Each figure is the median of 3 cold sends, a fresh prompt each send (no cached prefix): prompt tokens / time to first token. Measured with K=5.
Several requests at once (agents) and several long sessions held together were not measured on 3.0.2.2.
How it was measured
- Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, P2P on, no NVLink.
- Engine: Merlin 3.0.2.2, started by
tools/merlin/launch.py --profile hy4flash-quad-3090with K=3 (the shipped default) for the 4K figure; K=5 for the 128K, 262K and prefill rows. - Layout: TP2 × PP2 with decode context parallel 2 and expert parallel on four cards, FP8 KV (
fp8_ds_mla), CUDA graphs on, reasoning efforthigh. - Protocol: one client, serial sends, a fresh prompt each send; decode rows generate 256 tokens (128 at 262K), median of 3 sends; streams are client-timed.
- Measured on the Merlin 3.0.2.2 engine code on four RTX 3090s.
Quality
Quantisation checks of this checkpoint, from the pack's own verification against the BF16 weights:
| Check | Result |
|---|---|
| W4 routed experts, relative RMS error | mean 0.111, worst 0.124 (all down_proj) |
| Engram tables FP8, relative L2 error | 0.0258–0.0260 (about 31.7 dB), 0 zero rows |
These are weight-space errors of the quantisation, not task scores. The experts are plain round-to-nearest with no calibration data.
This checkpoint against the BF16 weights, served on the Merlin engine:
| Check | Result | Setting |
|---|---|---|
| KL divergence of the next-token distribution, served checkpoint vs BF16 | mean 0.1457, p95 0.5855 | 31 prompts, 9,623 scored positions, up to 4,608 tokens |
| Top-1 agreement with BF16 | 90.20 % | the same positions |
The BF16 reference is a layer-by-layer forward pass of Tencent's BF16 weights. The served side is this checkpoint on Merlin 3.0.2.2 with DFly2 off. Task benchmarks (GSM8K, needle retrieval) were not run on this checkpoint for 3.0.2.2, so how much task accuracy the 4-bit experts keep is not measured here.
Tencent's base-model and instruct results are on the upstream card.
What it fits on
| Measured on | Context per request | KV pool | Held at once |
|---|---|---|---|
| 4× RTX 3090, 24 GB (🟡 testing) | 262,144 tokens | 609,398 tokens, FP8, all in VRAM | 2 × 262K or 4 × 128K by the engine's pool arithmetic (not run as concurrent sessions) |
| 8× RTX 3090, 24 GB (⚫ untested on hardware) | 1,048,576 tokens | 4,196,751 tokens (an estimate from the four-card row, not a boot) | 4 × 1M by the same arithmetic |
Profile hy4flash-quad-3090: TP2 × PP2, decode context parallel 2, expert parallel, FP8 KV (fp8_ds_mla), cards at 220 W. Full-model runs passed on four RTX 3090s (boot, 262K prefill and decode with no out-of-memory, and a serial-prompt token comparison against the stock kernels); a kernel parity qualification is still pending. Profile hy4flash-octo-3090 (TP4 × PP2) ships but has not run on eight cards. 🟢 measured by us · 🟡 testing · ⚫ a profile ships, not yet run by us.
| Need | Details |
|---|---|
| Host RAM | At least 140 GiB available before the start (what the four-card boots needed): about 60 GiB for the pinned FP8 Engram tables plus about 18 GiB for the engine processes, with headroom. Merlin's setup checks it. |
| Disk | About 152.6 GB for the checkpoint, plus the engine environment and caches |
| Engine | Merlin 3.0.2.2 or later. Earlier Merlin versions cannot serve this checkpoint. |
| Engine host | Linux x86-64, NVIDIA driver R580 or later (CUDA 13), Python 3.13; GPUs headless with no other CUDA process, and PCIe P2P on among the cards. |
| KV RAM cache | Not supported on this checkpoint yet: Merlin refuses --kv-ram-cache, --swarm and MERLIN_KV_HOST_TIER_GIB above 0 for it. |
Use it
git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
./setup.sh --model hy4flash # FP8 Engram (the default)
./setup.sh --model hy4flash --engram bf16 # or: BF16 Engram
Setup checks the host (GPUs, P2P, host RAM, memlock, disk), installs the engine, downloads the pinned revision of this repository and checks every file against the sha256 and size the hub states, then starts the server. Add --dry-run to see the plan without changing anything.
Or by hand:
git clone --branch merlin-v3.0.2.2 --depth 1 https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.2.2/vllm-0.33.0+merlin3.0.2.2.c56eeed392-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.33.0+merlin3.0.2.2.c56eeed392-cp313-cp313-linux_x86_64.whl" bash release/install-env.sh ./venv
venv/bin/hf download halt95/Hy4-flash-W4A16-Merlin --revision 45a685b169f418b0f8e22867fc631ab68d107610 \
--local-dir ~/models/hy4-flash
venv/bin/python tools/merlin/launch.py --profile hy4flash-quad-3090 -- ~/models/hy4-flash
For the BF16 Engram build, download --revision fb3973c7bec167e1fe79e1ebf7a07d8feb35b885 and start with ENGRAM=bf16. No checksum file ships: hf download checks each LFS file against the sha256 the hub stores.
launch.py serves an OpenAI-compatible API on 127.0.0.1:8000; requests name the model hy4flash. Leave the profile's knobs as they are. SERVE_PRINT_ENV_AND_EXIT=1 prints the resolved command and environment without starting anything. The first start compiles kernels; the server is ready when the Ready: line prints. On eight RTX 3090s, use --profile hy4flash-octo-3090 (untested on hardware).
Settings
| Mode | temperature | top_p |
|---|---|---|
Thinking (high) or no_think |
0.9 | 1.0 |
Tencent's recommended settings from the upstream card, which states no top_k and no penalties.
- Thinking control: pass
"chat_template_kwargs": {"reasoning_effort": "no_think"}to turn thinking off for a request. Merlin serveshighby default; the model acceptsno_thinkandhighonly. - Chat template: the repository's
chat_template.jinjais the one Merlin serves. - Tool calls use the standard
toolsfield. Merlin serves thehy_v4reasoning and tool-call parsers, the ones Tencent's card names (--reasoning-parser hy_v4,--tool-call-parser hy_v4). - Input is text only on Merlin: the checkpoint's vision encoder is not loaded, and image requests are not supported.
- To reach it from other machines, start it with
HOST=0.0.0.0andVLLM_API_KEYset; put a TLS reverse proxy in front before opening it beyond your own network. - DFly2 drafter (speculative decoding): on by default with a draft length of K=3;
DFLY=0turns it off.
The base model
Hy4-flash is Tencent's MoE model with 220B total parameters (including 63.5B Engram parameters). It takes text and image input, and 11.3B text parameters (plus 0.43B vision) are active per token. What it uses, in short:
- Hybrid sparse and sliding-window attention, at a ratio of 1:3.
- LatentMoE. 512 routed experts with 16 active, plus 1 shared expert.
- Engram tables. Four Engram tables add parameter capacity as lookups rather than as extra experts. Merlin keeps them in pinned host RAM, so the VRAM goes to the KV cache.
- DFly2 drafter. A drafter head for speculative decoding, quantised to INT4 in this checkpoint.
| Property | Hy4-flash |
|---|---|
| Type | MoE, text and image input |
| Parameters | 220B total (including 63.5B Engram; Tencent's headline rounds it to 64B): 155.5B excluding Engram, ViT and DFly2; Engram 63.5B; ViT 0.43B; DFly2 drafter 0.85B (5 layers) |
| Activated per token | 11.3B text + 0.43B vision |
| Layers | 42 (1 dense FFN + 41 MoE) |
| Experts | 512 routed, 16 active, plus 1 shared |
| Attention | Hybrid sparse and sliding-window attention (1:3) |
| Context | 1M (1,048,576) native. Merlin serves 262,144 tokens on the four-card hy4flash-quad-3090 profile; the eight-card hy4flash-octo-3090 profile is configured for 1M (untested). |
As reported by Tencent on the upstream card.
Other builds
| Checkpoint | Base | GPUs |
|---|---|---|
| Hy4-flash W4A16 (this card) | tencent/Hy4-flash | 4× 3090 |
| Qwen3.8-Flash-Next | Qwen3.8-Flash-Next, 125B MoE | 4× 3090 |
| Swift1.5 | Flash-Next fine-tune | 4× 3090 |
| Swift1.5-Huihui | Swift1.5, abliterated | 4× 3090 |
| Qwen3.8-27B | Qwen3.8-27B, dense | 1×, 2× or 4× 3090 |
| ThinkingCap | Qwen3.8-27B fine-tune | 2× or 4× 3090 |
| Nemotron 3.5 Super VL | Nemotron 3.5 Super VL, 120B-A12B MoE | 4× 3090 |
| Ornith 1.5 | Qwen3.5-122B-A10B fine-tune (ornith-ai), 122B MoE | 4× 3090 |
Our other Merlin checkpoints; each card has its own measurements.
Good to know
- Every figure we measured for this checkpoint comes from one setup: 4× RTX 3090 at 220 W, P2P on, no NVLink. Upstream results use their own stated setups.
- The Engram tables are FP8 and the experts are round-to-nearest 4-bit without calibration; the quantisation errors are in Quality.
- Text only on Merlin: the vision encoder ships in the files (unchanged BF16) but is not loaded, so image input is not supported.
- The four-card profile is in testing: full-model runs passed, and the kernel parity qualification is pending. The eight-card profile is untested on hardware.
Changes
| Date | Change to this repository |
|---|---|
| 2026-11-21 | Weights published: FP8 Engram on main, BF16 Engram on engram-bf16 |
Credits and licence
Tencent (model; Tencent Hy License 1.0, see LICENSE).
Licence. This repository is distributed under the Tencent Hy License 1.0, which is Apache-2.0-derived with an added Section 10. LICENSE is a verbatim copy of Tencent's file. Tencent ships no NOTICE file with the upstream model.
Modifications (Section 4(b)). The weights in this repository are a modified version of tencent/Hy4-flash: we quantized them. The routed experts were converted to W4A16 (round-to-nearest, symmetric, group size 128, no calibration), the Engram tables were converted to FP8, the DFly2 drafter's linear layers were quantised to INT4, the FP8 attention projections were taken from Tencent's FP8 release, and the weights were repacked in the compressed-tensors layout. These modifications were made by halt95, not by Tencent.
Section 10 carries through. Section 10 of the Tencent Hy License 1.0 (a separate licence for businesses that offer the model as a service, "MaaS"; contact hy_partners@tencent.com) applies to this derivative in the same way as to the original model.
- Downloads last month
- -
Model tree for halt95/Hy4-flash-W4A16-Merlin
Base model
tencent/Hy4-flash