halt95/Hy4-flash-W4A16-Merlin

Hy4-flash W4A16, powered by Merlin: a 4-bit-expert checkpoint, 152.6 GB, with the full 262,144-token context on four 24 GB GPUs. A 609,398-token KV pool in VRAM, 75.7 tok/s one-stream decode at 4K, and 1,608 tok/s prefill on a 100K prompt.

A 4-bit-expert build of Hy4-flash, Tencent's MoE model with 220B total parameters (including 63.5B Engram parameters), served here at 262,144 tokens of context on four 24 GB GPUs. It is 152.6 GB. The routed experts are W4A16 (4-bit weights, 16-bit activations). The attention projections Tencent ships in FP8 keep their FP8 bytes. The four Engram tables are FP8 and live in pinned host RAM instead of on the GPUs. The DFly2 drafter's linear layers are INT4, and everything else is unchanged BF16. Merlin serves it for text only: the vision encoder's weights are in the files, but Merlin does not load them, so image input is not supported. It is served by the Merlin engine, a vLLM fork tuned for Ampere and Ada. It needs Merlin 3.0.2.2 or later; earlier Merlin versions and stock vLLM cannot serve it. Measured on RTX 3090s.

Numbers: Merlin 3.0.2.2 on 4× RTX 3090 without NVLink, 220 W, P2P on, measured 2026-11-21 (UTC). KV pool: FP8 (fp8_ds_mla), all in VRAM. Decode: one stream, thinking on at reasoning effort high, T=0, each send a cold prefill, median of 3 sends. Prefill: the median of 3 cold sends, prompt tokens / time to first token. Each figure is also in a table below.

Buy me a coffee on Ko-fi Discord

The checkpoint

Component Format
Routed experts (512 per layer, 16 active) W4A16, round-to-nearest (RTN), symmetric, group size 128, no calibration
Attention q_b_proj, o_proj and linear_gate of layers 1-41 FP8 block-128: Tencent's bytes from tencent/Hy4-flash-FP8, copied unchanged
Engram tables (4) FP8, one scale per row, held in pinned host RAM at serve time (EngramConfig cpu_offload=True)
DFly2 drafter (5 layers) Linear layers INT4 (W4A16, group size 128, symmetric RTN); its norms and small projections stay BF16
Everything else: layer 0, the other attention projections, shared experts, latent projections, routers, indexers, norms, lm_head (shared with the drafter), token embedding, vision encoder BF16, unchanged; the token embedding is held in host RAM at serve time
KV cache FP8 (fp8_ds_mla)

The checkpoint uses the compressed-tensors layout. We also measured an INT8 drafter-head variant (904 MiB) against this INT4 drafter and did not ship it: it accepted fewer draft tokens and used more memory on the four-card layout.

Two builds

FP8 Engram (default) BF16 Engram
Branch main engram-bf16
Weights without Engram (on the GPUs) 88.0 GB 88.0 GB (the same files)
Engram tables (in host RAM) 64.7 GB (FP8, one scale per row) 126.7 GB (Tencent's BF16 rows, unchanged)
Download in all 152.6 GB 214.7 GB
Pinned host RAM for Engram about 60 GiB about 118 GiB
Host RAM available before the start about 140 GiB about 198 GiB
Status Everything on this card was measured on it, including the KL divergence against BF16 in Quality Untested on hardware: our four-card host has too little host RAM; no speed or quality rows

The Engram tables are 660 million rows of 96 values: 98 bytes a row in FP8 (96 values and a 2-byte scale), 192 bytes in BF16. Both builds share the same W4A16 experts and every other tensor; only the Engram tables differ. To get the BF16 Engram build, download it with hf download halt95/Hy4-flash-W4A16-Merlin --revision engram-bf16. Select it with ENGRAM=bf16 for tools/merlin/launch.py, or with ./setup.sh --model hy4flash --engram bf16. The host RAM, speed and quality figures elsewhere on this card are for the FP8 Engram default.

Files in this repository (branch main)

Files What they hold Size
model-00001-of-00148.safetensors … model-00147-of-00148.safetensors (147 shards) Routed experts (W4A16), FP8 attention projections, Engram tables, the BF16 tensors and the vision encoder 152.09 GB
model-dfly-int4.safetensors DFly2 drafter (INT4) 527.1 MB
model-expert-shapes.safetensors The shapes of the packed expert weights 7.7 MB
model.safetensors.index.json Which file holds each of the 191,005 tensors 18.1 MB
config.json, generation_config.json, processor_config.json, tokenizer.json, tokenizer_config.json Model, quantisation, generation and tokenizer settings small (in git, not LFS)
chat_template.jinja The chat template Merlin serves: thinking at high, or off with no_think small (in git, not LFS)
LICENSE Tencent Hy License 1.0, verbatim copy of Tencent's file small (in git, not LFS)
README.md This card small (in git, not LFS)

152.64 GB (152,642,614,330 bytes) over the 150 LFS files. Branch engram-bf16 has the same file names; its shards hold the BF16 Engram tables, 214.68 GB (214,683,216,043 bytes) in all.

Speed

Speed on 4x RTX 3090, Merlin 3.0.2.2. Decode, one stream, thinking on (effort high): 75.7 tok/s at 4K with K=3; 67.1 tok/s at 128K and 89.3 at 262K with K=5. Cold prefill, median of 3 cold sends: 1,672 tok/s at 10K, 1,608 at 100K, 1,384 at 262K. The same figures are in the tables below.
Prompt Decode, one stream Time to first token
4K (3,840 tokens) 75.7 tok/s 2.55 s
128K (130,816 tokens) 67.1 tok/s § 84.0 s
262K (261,888 tokens) 89.3 tok/s § 189.3 s

4× RTX 3090, thinking on at reasoning effort high, 256 tokens out (128 at 262K), median of 3 sends; decode tok/s = (completion tokens − 1) / (last − first token time). 4K is measured at the shipped DFly2 draft length, K=3 (sends 73.9–80.2). § Measured with K=5, not the shipped K=3: 128K sends 66.3–71.4, 262K sends 77.6–89.7; at 4K, K=5 gave 75.3 tok/s against K=3's 75.7. Decode speed depends on how many draft tokens the model accepts for the text it is writing, which is why the 262K row is faster than the 128K row. With DFly2 off, 4K decode is 45.1 tok/s.

Cold prefill, median of 3 cold sends Prefill Time to first token
10,240 tokens 1,672 tok/s 6.13 s
102,400 tokens 1,608 tok/s 63.68 s
261,888 tokens 1,384 tok/s 189.3 s

Each figure is the median of 3 cold sends, a fresh prompt each send (no cached prefix): prompt tokens / time to first token. Measured with K=5.

Several requests at once (agents) and several long sessions held together were not measured on 3.0.2.2.

How it was measured
  • Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, P2P on, no NVLink.
  • Engine: Merlin 3.0.2.2, started by tools/merlin/launch.py --profile hy4flash-quad-3090 with K=3 (the shipped default) for the 4K figure; K=5 for the 128K, 262K and prefill rows.
  • Layout: TP2 × PP2 with decode context parallel 2 and expert parallel on four cards, FP8 KV (fp8_ds_mla), CUDA graphs on, reasoning effort high.
  • Protocol: one client, serial sends, a fresh prompt each send; decode rows generate 256 tokens (128 at 262K), median of 3 sends; streams are client-timed.
  • Measured on the Merlin 3.0.2.2 engine code on four RTX 3090s.

Quality

Quantisation checks of this checkpoint, from the pack's own verification against the BF16 weights:

Check Result
W4 routed experts, relative RMS error mean 0.111, worst 0.124 (all down_proj)
Engram tables FP8, relative L2 error 0.0258–0.0260 (about 31.7 dB), 0 zero rows

These are weight-space errors of the quantisation, not task scores. The experts are plain round-to-nearest with no calibration data.

This checkpoint against the BF16 weights, served on the Merlin engine:

Check Result Setting
KL divergence of the next-token distribution, served checkpoint vs BF16 mean 0.1457, p95 0.5855 31 prompts, 9,623 scored positions, up to 4,608 tokens
Top-1 agreement with BF16 90.20 % the same positions

The BF16 reference is a layer-by-layer forward pass of Tencent's BF16 weights. The served side is this checkpoint on Merlin 3.0.2.2 with DFly2 off. Task benchmarks (GSM8K, needle retrieval) were not run on this checkpoint for 3.0.2.2, so how much task accuracy the 4-bit experts keep is not measured here.

Tencent's base-model and instruct results are on the upstream card.

What it fits on

Measured on Context per request KV pool Held at once
4× RTX 3090, 24 GB (🟡 testing) 262,144 tokens 609,398 tokens, FP8, all in VRAM 2 × 262K or 4 × 128K by the engine's pool arithmetic (not run as concurrent sessions)
8× RTX 3090, 24 GB (⚫ untested on hardware) 1,048,576 tokens 4,196,751 tokens (an estimate from the four-card row, not a boot) 4 × 1M by the same arithmetic

Profile hy4flash-quad-3090: TP2 × PP2, decode context parallel 2, expert parallel, FP8 KV (fp8_ds_mla), cards at 220 W. Full-model runs passed on four RTX 3090s (boot, 262K prefill and decode with no out-of-memory, and a serial-prompt token comparison against the stock kernels); a kernel parity qualification is still pending. Profile hy4flash-octo-3090 (TP4 × PP2) ships but has not run on eight cards. 🟢 measured by us · 🟡 testing · ⚫ a profile ships, not yet run by us.

Need Details
Host RAM At least 140 GiB available before the start (what the four-card boots needed): about 60 GiB for the pinned FP8 Engram tables plus about 18 GiB for the engine processes, with headroom. Merlin's setup checks it.
Disk About 152.6 GB for the checkpoint, plus the engine environment and caches
Engine Merlin 3.0.2.2 or later. Earlier Merlin versions cannot serve this checkpoint.
Engine host Linux x86-64, NVIDIA driver R580 or later (CUDA 13), Python 3.13; GPUs headless with no other CUDA process, and PCIe P2P on among the cards.
KV RAM cache Not supported on this checkpoint yet: Merlin refuses --kv-ram-cache, --swarm and MERLIN_KV_HOST_TIER_GIB above 0 for it.

Use it

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine
./setup.sh --model hy4flash                 # FP8 Engram (the default)
./setup.sh --model hy4flash --engram bf16   # or: BF16 Engram

Setup checks the host (GPUs, P2P, host RAM, memlock, disk), installs the engine, downloads the pinned revision of this repository and checks every file against the sha256 and size the hub states, then starts the server. Add --dry-run to see the plan without changing anything.

Or by hand:

git clone --branch merlin-v3.0.2.2 --depth 1 https://github.com/halt95/merlin-engine.git && cd merlin-engine
curl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.2.2/vllm-0.33.0+merlin3.0.2.2.c56eeed392-cp313-cp313-linux_x86_64.whl
MERLIN_WHEEL="$PWD/vllm-0.33.0+merlin3.0.2.2.c56eeed392-cp313-cp313-linux_x86_64.whl" bash release/install-env.sh ./venv

venv/bin/hf download halt95/Hy4-flash-W4A16-Merlin --revision 45a685b169f418b0f8e22867fc631ab68d107610 \
  --local-dir ~/models/hy4-flash
venv/bin/python tools/merlin/launch.py --profile hy4flash-quad-3090 -- ~/models/hy4-flash

For the BF16 Engram build, download --revision fb3973c7bec167e1fe79e1ebf7a07d8feb35b885 and start with ENGRAM=bf16. No checksum file ships: hf download checks each LFS file against the sha256 the hub stores.

launch.py serves an OpenAI-compatible API on 127.0.0.1:8000; requests name the model hy4flash. Leave the profile's knobs as they are. SERVE_PRINT_ENV_AND_EXIT=1 prints the resolved command and environment without starting anything. The first start compiles kernels; the server is ready when the Ready: line prints. On eight RTX 3090s, use --profile hy4flash-octo-3090 (untested on hardware).

Settings

Mode temperature top_p
Thinking (high) or no_think 0.9 1.0

Tencent's recommended settings from the upstream card, which states no top_k and no penalties.

  • Thinking control: pass "chat_template_kwargs": {"reasoning_effort": "no_think"} to turn thinking off for a request. Merlin serves high by default; the model accepts no_think and high only.
  • Chat template: the repository's chat_template.jinja is the one Merlin serves.
  • Tool calls use the standard tools field. Merlin serves the hy_v4 reasoning and tool-call parsers, the ones Tencent's card names (--reasoning-parser hy_v4, --tool-call-parser hy_v4).
  • Input is text only on Merlin: the checkpoint's vision encoder is not loaded, and image requests are not supported.
  • To reach it from other machines, start it with HOST=0.0.0.0 and VLLM_API_KEY set; put a TLS reverse proxy in front before opening it beyond your own network.
  • DFly2 drafter (speculative decoding): on by default with a draft length of K=3; DFLY=0 turns it off.

The base model

Hy4-flash is Tencent's MoE model with 220B total parameters (including 63.5B Engram parameters). It takes text and image input, and 11.3B text parameters (plus 0.43B vision) are active per token. What it uses, in short:

  • Hybrid sparse and sliding-window attention, at a ratio of 1:3.
  • LatentMoE. 512 routed experts with 16 active, plus 1 shared expert.
  • Engram tables. Four Engram tables add parameter capacity as lookups rather than as extra experts. Merlin keeps them in pinned host RAM, so the VRAM goes to the KV cache.
  • DFly2 drafter. A drafter head for speculative decoding, quantised to INT4 in this checkpoint.
Property Hy4-flash
Type MoE, text and image input
Parameters 220B total (including 63.5B Engram; Tencent's headline rounds it to 64B): 155.5B excluding Engram, ViT and DFly2; Engram 63.5B; ViT 0.43B; DFly2 drafter 0.85B (5 layers)
Activated per token 11.3B text + 0.43B vision
Layers 42 (1 dense FFN + 41 MoE)
Experts 512 routed, 16 active, plus 1 shared
Attention Hybrid sparse and sliding-window attention (1:3)
Context 1M (1,048,576) native. Merlin serves 262,144 tokens on the four-card hy4flash-quad-3090 profile; the eight-card hy4flash-octo-3090 profile is configured for 1M (untested).

As reported by Tencent on the upstream card.

Other builds

Checkpoint Base GPUs
Hy4-flash W4A16 (this card) tencent/Hy4-flash 4× 3090
Qwen3.8-Flash-Next Qwen3.8-Flash-Next, 125B MoE 4× 3090
Swift1.5 Flash-Next fine-tune 4× 3090
Swift1.5-Huihui Swift1.5, abliterated 4× 3090
Qwen3.8-27B Qwen3.8-27B, dense 1×, 2× or 4× 3090
ThinkingCap Qwen3.8-27B fine-tune 2× or 4× 3090
Nemotron 3.5 Super VL Nemotron 3.5 Super VL, 120B-A12B MoE 4× 3090
Ornith 1.5 Qwen3.5-122B-A10B fine-tune (ornith-ai), 122B MoE 4× 3090

Our other Merlin checkpoints; each card has its own measurements.

Good to know

  • Every figure we measured for this checkpoint comes from one setup: 4× RTX 3090 at 220 W, P2P on, no NVLink. Upstream results use their own stated setups.
  • The Engram tables are FP8 and the experts are round-to-nearest 4-bit without calibration; the quantisation errors are in Quality.
  • Text only on Merlin: the vision encoder ships in the files (unchanged BF16) but is not loaded, so image input is not supported.
  • The four-card profile is in testing: full-model runs passed, and the kernel parity qualification is pending. The eight-card profile is untested on hardware.

Changes

Date Change to this repository
2026-11-21 Weights published: FP8 Engram on main, BF16 Engram on engram-bf16

Credits and licence

Tencent (model; Tencent Hy License 1.0, see LICENSE).

Licence. This repository is distributed under the Tencent Hy License 1.0, which is Apache-2.0-derived with an added Section 10. LICENSE is a verbatim copy of Tencent's file. Tencent ships no NOTICE file with the upstream model.

Modifications (Section 4(b)). The weights in this repository are a modified version of tencent/Hy4-flash: we quantized them. The routed experts were converted to W4A16 (round-to-nearest, symmetric, group size 128, no calibration), the Engram tables were converted to FP8, the DFly2 drafter's linear layers were quantised to INT4, the FP8 attention projections were taken from Tencent's FP8 release, and the weights were repacked in the compressed-tensors layout. These modifications were made by halt95, not by Tencent.

Section 10 carries through. Section 10 of the Tencent Hy License 1.0 (a separate licence for businesses that offer the model as a service, "MaaS"; contact hy_partners@tencent.com) applies to this derivative in the same way as to the original model.

Downloads last month
-
Safetensors
Model size
90B params
Tensor type
BF16
·
F8_E4M3
·
I32
·
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for halt95/Hy4-flash-W4A16-Merlin

Quantized
(44)
this model