CL-Vanilla-1B

CL-Vanilla-1B is a KDA hybrid language model released with Competing Learning in Hybrid Architectures.

Recurrent mixers and softmax attention can compete over retrieval. When recurrent layers acquire retrieval earlier, attention receives less learning signal to develop the same ability.

The Vanilla baseline trains every block throughout pretraining. The paired BLD model uses the same architecture, data, token budget, and optimization schedules, while freezing recurrent blocks early to give attention an earlier opportunity to learn retrieval.

This is a base language model. It was pretrained on DCLM and is not instruction tuned.

Model details

Field	Value
Total unique parameters, including embeddings	1,014,512,856
Shared input embedding / LM-head parameters	235,008,000 (counted once)
Non-embedding parameters	779,504,856
Published size label	1B
Hidden layers	24
Hidden width	1536
Vocabulary size	153000
Attention layer indices (zero based)	[0, 4, 8, 12, 16, 20]
Block pattern	Attention → Recurrent → Recurrent → Recurrent
Position encoding	NoPE
Training corpus	DCLM
Checkpoint	Final pretrained weights
Total tokens implied by recorded update counts	17,600,348,160

The published names use the established rounded total-parameter labels. Exact counts are given above. The checkpoint is copied byte for byte from the original safetensors file, retaining BF16 weights and FP32 auxiliary parameters.

Installation and loading

Use Linux, an NVIDIA GPU with BF16 support, and a CUDA-enabled environment. The validated environment and installation commands are recorded in ENVIRONMENT.md. This custom architecture requires the included Python files and trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "xingyushenai/CL-Vanilla-1B"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda").eval()

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Provenance and validation

provenance.json records checkpoint provenance, exact parameter counts, training updates, and the weight SHA-256. validation.json records release smoke tests; these tests are not a benchmark rerun. Input embeddings and the output head share one parameter and must not be counted twice.

For the method, training setup, and evaluation instructions, see the Competing Learning repository.