Qwen3-4B-Diffusion-Base

Qwen3-4B-Diffusion-Base is a 4.41B-parameter masked-diffusion language-model backbone initialized from Qwen/Qwen3-4B-Base. It is the Stage 0 diffusion base used for the 4B-scale Continuous Interaction Diffusion (CID) training path.

This repository contains the diffusion-converted Qwen3 backbone at CID Stage 0, before Stage A / Stage B training. It serves as the 4B CID initialization checkpoint and as a standalone artifact for diffusion-LM research, reproduction, and ablation studies.

Summary

Property	Value
Base model	Qwen/Qwen3-4B-Base
Parameters	4,411,424,256
Objective	masked diffusion
Attention	bidirectional
Sequence length	2,048
Published checkpoint	step 3,300
Training tokens in published checkpoint	648,806,400
Global batch size	96 sequences
Learning rate	2e-5, constant
Mask-ratio range	0.001–1.0
Weight format	safetensors, 3 shards
Quantization	none

The model adds a dedicated <|cid_mask|> token and uses a diffusion-aware remote-code loader. Generation follows masked-diffusion semantics with bidirectional attention.

Files

• model-00001-of-00003.safetensors … model-00003-of-00003.safetensors — model weights.
• model.safetensors.index.json — shard index and parameter metadata.
• config.json — Qwen3 configuration adapted for bidirectional masked-diffusion use.
• diffusion_config.json — diffusion objective, mask token, training corpus, and checkpoint metadata.
• modeling_cid_diffusion.py — Hugging Face remote-code implementation with bidirectional forward, denoising, and diffusion-aware generate().
• tokenizer files — tokenizer inherited from the Qwen3 base with the CID mask token.
• logs/ — sanitized training and evaluation metrics plus public run metadata.
• SHA256SUMS — checksums for release files.

Optimizer state, FSDP rank-local checkpoint state, raw stdout/stderr, hostnames, local absolute paths, process IDs, and private job metadata are not part of this release.

Usage

The repository can be loaded through Transformers with trust_remote_code=True:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "fwerkor/Qwen3-4B-Diffusion-Base"

tokenizer = AutoTokenizer.from_pretrained(
    model_id,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)

inputs = tokenizer("Diffusion language models can", return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    diffusion_steps=64,
    block_length=16,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

The loader also exposes denoise() for directly resolving existing mask tokens. Direct generations reflect base-model behavior; instruction following and full CID behavior are introduced in later training stages.

For CID Stage A / Stage B training and runtime integration, use the main CID repository:

https://github.com/fwerkor/continuous-interaction-diffusion

Stage 0 training

Stage 0 converts the autoregressive Qwen3 base into a native masked-diffusion backbone by continued pretraining with bidirectional attention and a LLaDA-style masked-diffusion objective.

The published checkpoint uses:

• 2 × NVIDIA GeForce RTX 3090 (24 GiB)
• 2-way FSDP with CPU offload
• micro-batch size 4 per rank
• global batch size 96
• gradient accumulation: 12
• sequence length: 2,048
• AdamW, learning rate 2e-5
• weight decay 0.1
• constant learning-rate schedule, no warmup
• gradient checkpointing
• mask-ratio sampling over [0.001, 1.0]
• checkpoint interval: 100 optimizer steps
• evaluation interval: 250 steps

The public checkpoint is step 3,300, corresponding to 648,806,400 training tokens. Public training logs are truncated at this checkpoint so that the released metrics correspond exactly to the published weights.

Training data

The Stage 0 stream mixes four public corpora:

Source	Weight	Pinned revision
openbmb/UltraX-Preview / UltraX-Ultra-FineWeb	0.70	a88527587389fd4ab352e9ad1273f4c0a234d8df
openbmb/Ultra-FineWeb / Chinese split	0.15	02c85641e3d19a854be2e09139c25adaa9518063
openbmb/UltraData-Code	0.10	85182d829f2ce7ea07cca72ebfc509deea1d9f5f
openbmb/UltraData-Math	0.05	fe10db8efd35597fd7fcff8ff576b5ec4ea5ff87

Exact source metadata is also preserved in diffusion_config.json.

Training logs

Sanitized release logs are under logs/:

• logs/train_metrics.jsonl — rank-0 training metrics through the published step-3300 checkpoint.
• logs/eval_metrics.jsonl — periodic Stage 0 evaluation loss.
• logs/run_config.json — public training configuration and provenance.
• logs/manifest.json — checksums and privacy metadata for the released logs.
• logs/README.md — field and sanitization notes.

The public logs contain numerical training statistics only. Raw process logs are intentionally excluded because they may contain local paths, host information, process metadata, and transient infrastructure details.

At the published checkpoint, the rank-0 logged training loss is 3.0079. The latest evaluation before the checkpoint is step 3,250 with loss 2.8881. These values are raw run metrics and should be interpreted within this training run.

Checkpoint provenance

• Published source checkpoint: step 3,300
• Published tokens seen: 648,806,400
• Release loader source commit: 3be85aa0ab31b7465a7fa4a28ae33af9988d6b0c
• Exact training script SHA-256: 60808056f8fa01fb7a88c8ee66bee6a34fdc711bdde7f819c73d159dd1dd57f0
• Exported model implementation SHA-256: 9658daf8fd5b0b6db3c8fbb546425334ca6fe83110f8a2e8d87c39c685298230

The training source tree had local changes relative to the recorded Git commit, so the exact training-script hash above is the authoritative code provenance for this run.

Intended use and limitations

This checkpoint is intended for:

• diffusion-language-model research;
• CID Stage A initialization;
• training/reproduction studies;
• diffusion decoding and runtime experiments;
• ablations on AR-to-diffusion conversion.

This release corresponds specifically to the Stage 0 diffusion base that initializes the later 4B CID training path. CID Stage A/B checkpoints add the task-specific training and runtime behavior used by the full system.

Stage 0 loss alone is insufficient for downstream benchmark claims.

Paper and citation

This checkpoint accompanies the CID paper: Continuous Interaction Diffusion: A Diffusion-Native Architecture for Asynchronous Tool-Augmented Reasoning (arXiv:2608.10438, DOI).

If you use this model or the CID runtime, please cite:

@article{cao2026continuous,
  title   = {Continuous Interaction Diffusion: A Diffusion-Native Architecture for Asynchronous Tool-Augmented Reasoning},
  author  = {Cao, Yuhang and Mu, Yanzhou and Fang, Chunrong and Chen, Zhenyu},
  journal = {arXiv preprint arXiv:2608.10438},
  year    = {2026},
  doi     = {10.48550/arXiv.2608.10438},
  url     = {https://arxiv.org/abs/2608.10438}
}

License

This checkpoint is derived from Qwen/Qwen3-4B-Base, which is released under the Apache License 2.0. This release uses the same Apache-2.0 license designation.