MuseMesh/mume-english-125m

Mume English 125M

Mume English 125M is a 134M-parameter English language model trained from scratch on 1.33B tokens of FineWeb. With Muon, rotary positions, QK-norm, an untied head and squared ReLU it reads FineWeb validation text 3.9% better than its plain-AdamW calibration run (v0.1.0) on the same data and steps, and beats the plain-AdamW GPT-2 124M reference at equal tokens by 3.7%.

From Muse Mesh (Hugging Face). Part of the Muse Mesh English and Math models (collections). Every training run, including the ones not released, is logged at saansar.com, the Sansar Lab.

Model

Architecture	decoder-only transformer, pre-LayerNorm (LayerNorm without bias), no biases anywhere; GPT-2-small shape
Layers / heads / width	12 / 12 / 768 (head dim 64); MLP 4x = 3,072
Positions	rotary (base 10,000; half-split pairing), no position table
Attention	causal SDPA; queries and keys RMS-normalised per head (QK-norm)
MLP activation	squared ReLU
Output head	separate (untied), zero-initialised
Vocabulary	32,000 (MuseMesh/mume-tokenizer-32k v0.1.0, bundled)
Context	1,024 tokens
Parameters, total	134,105,856
Parameters, non-embedding	84,953,856
Token embedding	24,576,000
Output head	24,576,000
Weights in this repo	bfloat16 safetensors (trained as fp32 master weights under bf16 autocast)

Why this run: The EN1 recipe screens (Muon, an architecture bundle, both, both plus warm-down and value embeddings; 10,000 steps each against a same-schedule AdamW control) picked Muon + the architecture bundle; this is that recipe on EN0's exact data, steps and schedule.

Evaluation

Metric: bits per UTF-8 byte (lower is better): the model's negative log-likelihood of a text divided by the text's UTF-8 byte count, so models with different tokenizers share a denominator. Each set's records are joined with </s> into one stream and scored teacher-forced in 1,024-token windows with stride 512 (every scored token after the first window has at least 512 tokens of context); </s> carries loss but no bytes. Scored with scripts/train/eval_bpb.py on a GPU in bf16 autocast. The sets are frozen and published as MuseMesh/mume-eval-suites.

set	what	this model	v0.1.0 (EN0)	change
fineweb_val	FineWeb validation: the first 10,485,760 GPT-2 tokens of llm.c's FineWeb validation shard (14,983 web documents), the set the GPT-2 speedrun scores	1.10625	1.15174	-3.9%
fineweb_val_clean	fineweb_val minus the 3,935 documents that share a copied passage with the training data (11,048 documents)	1.12346	1.16785	-3.8%
wikitext103_test	WikiText-103 test, raw, grouped into its 62 articles	1.16525	1.21418	-4.0%
enwik8_test	enwik8 bytes 95,000,000-100,000,000 (the standard test split; Wikipedia XML)	1.38144	1.42598	-3.1%
text8_test	text8 characters 95,000,000-100,000,000 (lowercase letters and spaces)	1.25837	1.30127	-3.3%
pooled	byte-weighted over the five sets	1.13758	1.18255	-3.8%

With 512-token windows (stride 256) the same sets read fineweb_val 1.1146, fineweb_val_clean 1.1306, wikitext103_test 1.2040, enwik8_test 1.4031, text8_test 1.2700 (eval/bpb_ctx512.json).

Against GPT-2 124M at the same token budget

The pass mark was fixed before training: the plain-AdamW GPT-2 124M run of the modded-nanogpt speedrun (record 2024-06-06; GPT-2 tokenizer, 524,288 tokens per step), read off its validation curve at our budget of 1,332,314,112 tokens: 1.1755 bits per byte on the same FineWeb validation shard (val loss 3.6143 nats per GPT-2 token, converted with the measured 4.43604 bytes per GPT-2 token). Their number is over non-overlapping 1,024-token windows, so the like-for-like comparison is our run's own stream validation, scored the same way; the sliding-window fineweb_val above gives every token more context and reads lower.

model	tokens	FineWeb val, like for like	note
GPT-2 124M, plain AdamW (modded-nanogpt)	1.33B	1.1755	the pass mark
GPT-2 124M, plain AdamW (modded-nanogpt)	2.62B	1.135	its last evaluation
GPT-2 124M, llm.c (published)	10B	1.07
v0.1.0 (EN0, AdamW)	1.33B	1.1781	+0.2% against the mark
this model (EN1-full)	1.33B	1.1322	-3.7% against the mark

As recorded in the experiment log (EN1-full): stream val 1.1322 vs EN0 1.1781 (-3.90%); frozen fineweb_val 1.1062 vs 1.15174 (-3.95%); every set -3 to -4%. Step for step against EN0 (same data, schedule and steps) the gap is -6.3% at step 2,000, narrowing to -3.9% at the end. It beats the plain-AdamW GPT-2 124M reference at equal tokens (1.1322 vs 1.1755, -3.7%) and matches it at twice the tokens (1.135 at 2.62B).

For context, the published GPT-2 117M zero-shot numbers (OpenAI 2019, trained on WebText, about 10x our tokens): enwik8 1.16 bits per byte, text8 1.17 bits per character, WikiText-103 perplexity 37.5 (about 0.997 bits per byte). enwik8 is raw Wikipedia XML, far from FineWeb's text, which is why both models read high there.

Contamination: FineWeb contains copies of some evaluation text. Measured GPT-2-paper style (lowercase word 8-grams of each set looked up in the whole training slice): fineweb_val 3.7% of 8-grams, 3,935 of 14,983 records with a copied passage (a run of 5+ consecutive hits), wikitext103_test 1.7% of 8-grams, 18 of 62 records with a copied passage (a run of 5+ consecutive hits). fineweb_val stays as the like-for-like set (llm.c and the speedrun train on FineWeb with the same copies); fineweb_val_clean drops every document with a copied passage and is the honest generalisation number. GPT-2 measured 0.7% for WikiText-103 test against WebText.

How the recipe was chosen (EN1 screens)

Each screen changed one thing on top of EN0 and ran 10,000 steps (491.5M tokens) with an annealed schedule; it is read against an AdamW control on the same schedule (stream validation, bits per byte, at step 10,000).

screen	change	stream val	vs control	decision	prior work
EN1-ctrl	AdamW, same schedule	1.2441		the bar	Hägele et al. 2024, arXiv:2405.18392
EN1-muon	Muon on the block matrices (NS5, Nesterov 0.95, lr 0.02)	1.2254	-1.50%	keep	Jordan et al. 2024 (modded-nanogpt, Muon); Liu et al. 2025, arXiv:2502.16982
EN1-arch	rotary positions, QK-norm, untied zero-init head, squared ReLU	1.2143	-2.40%	keep	Su et al. 2021, arXiv:2104.09864; Henry et al. 2020, arXiv:2010.04245; So et al. 2021, arXiv:2109.08668; Press & Wolf 2017, arXiv:1608.05859
EN1-muon_arch	both	1.1860	-4.67%	keep: they stack
EN1-muon_arch_wd	+ warm-down (WSD) + value embeddings	1.2154	-2.31% (+2.48% vs muon_arch)	discard (confounded)	Hu et al. 2024, arXiv:2404.06395; Zhou et al. 2024, arXiv:2410.17897

EN1-full (this version) is muon_arch at EN0's full length. The untied head adds 24.6M parameters, which is part of the gain; the parameter table above reports totals with and without it.

Checked before release

• modeling_mume.py against the training code (scripts/train/model.py), fp32 on CPU, 6 inputs (one evaluation text per set, cropped to 1,024 tokens, and 1,024 random ids): with the same bf16 weights the logits are identical (max |difference| 0); against the fp32 training weights the bf16 storage moves logits by at most 0.269.
• Bits per byte with eval_bpb.py's scoring on the first 20 records of each set (each cut to 20,000 characters): 1.18894 (training checkpoint, fp32) vs 1.18901 (this repo, bf16 weights) = +0.0058%.
• The whole wikitext103_test set through this repo's model (bf16 weights, fp32 on CPU): 1.16528, against 1.16525 in the table (training checkpoint, GPU bf16 autocast), +0.003%.
• Tokenizer: AutoTokenizer(..., trust_remote_code=True) gives the training ids on 62/62 sample texts and decodes them back exactly on 62/62; on all 58,338 texts of the tokenizer's own check the same holds (see MuseMesh/mume-tokenizer-32k).
• Left-padded batches give the same logits as single sequences (max |difference| 2.8e-05).

Quick start

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "MuseMesh/mume-english-125m"
tok = AutoTokenizer.from_pretrained(repo, revision="v0.2.0", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, revision="v0.2.0", trust_remote_code=True)

inputs = tok("The history of the city begins", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=60, do_sample=True, temperature=0.8, top_k=100)
print(tok.decode(out[0], skip_special_tokens=True))

Needs transformers (tested with 5.18.0), torch and sentencepiece. trust_remote_code=True runs two small files from this repository, modeling_mume.py (the network; the Sansar model code with the classes renamed) and tokenization_mume.py (SentencePiece, so the ids are exactly the training ids); read them before you run them. Without trust_remote_code the tokenizer falls back to tokenizer.json, which agrees on short texts but splits a few near-tie words differently in long documents. The weights are stored in bfloat16; pass dtype=torch.float32 for fp32, usually the faster choice on a CPU. The tokenizer adds no BOS/EOS: training documents were separated by </s> only.

What this checkpoint wrote for that prompt (CPU, torch.manual_seed(0), 60 new tokens):

The history of the city begins sometime around 1170. It is a great city for a number of reasons:
- It is well known for being the home of the first two thefts - and the historical and cultural center of the city. The 100th anniversary of the attack on the

and with greedy decoding:

The history of the city begins with the 18th century, when the city was founded by the kings of the kings of the kings of the kings of the kings of the kings of the kings of the kings of the kings of the

To score text instead of generating it, call the model with labels=input_ids; loss is the mean negative log-likelihood per token in nats.

Training data

FineWeb sample-10BT files 001-004: 4,180,465 web documents, 2.16B words, 12.9 GB. File 000 is not used: llm.c's validation shard (our fineweb_val) is cut from its head, and the slice builder proved the cut aligned (decoded validation record 0 = parquet row 0). Files 005 and 007 were in the tokenizer's sample only.

Preparation (scripts/train/prep_data.py): Unicode NFC; documents longer than 20,000 characters cut into parts at whitespace (4,257,824 records); tokenized with the shared 32k tokenizer; one </s> after every record. A 0.5% validation stream by hash of each record's text (21,358 records); the rest is training: 4,236,466 records, 3,253,318,552 tokens, 12.84 GB (3.95 bytes per token).

Every training document is listed (source file, row, id; no text) in MuseMesh/mume-eval-suites, config training_manifests, split english_pretrain, with the parts that went to validation, so the slice can be rebuilt from the public source.

Training

Optimizer, block matrices	Muon (84,934,656 params): momentum 0.95 (Nesterov), 5 Newton-Schulz steps in bf16, peak lr 0.02, update scaled by sqrt(max(1, rows/cols)), no weight decay
Optimizer, everything else	AdamW: embedding + head with weight decay 0.1, LayerNorm gains without; peak lr 0.001, betas (0.9, 0.95), eps 1e-8
Learning-rate schedule	linear warm-up over 500 steps, then cosine decay to 0.1 x peak at step 27,106
Batch	12 sequences x 4 accumulation steps x 1,024 tokens = 49,152 tokens per step
Steps	27,106
Tokens seen	1,332,314,112 (0.41 passes over 3,253.3M training tokens)
Gradient clipping	global norm 1.0
Precision	bf16 autocast, fp32 master weights, torch.compile
Seed	1337
Data sampling	1,024-token windows at uniformly random offsets of the token stream (documents separated by </s>)
Hardware	1x NVIDIA RTX 3060 12 GB, power-capped at 150 W
Wall time	891 min (14.9 h), 25.2k tokens/s; two power cuts, resumed from checkpoints
Compute cost	own hardware, no cloud cost
Final validation	1.1322 bits per byte on the run's own validation stream (0.5% of documents, non-overlapping windows: the like-for-like number in the comparisons above)

Limitations

• Base model. It continues text. It has not been instruction-tuned or aligned: it does not follow instructions, answer questions or chat, and it will not refuse anything.
• It makes things up. Fluent text with invented facts, names, numbers, citations. Do not use it as a source of facts.
• Small and short. 134M parameters and a 1,024-token context. This implementation has no key/value cache, so long generations are slow (each new token re-reads the context).
• Web English. FineWeb is filtered Common Crawl text: the model reproduces its topics, styles, biases and errors, including offensive content that passed the filters. English only; other languages appear only where FineWeb's English pages contain them.
• Contamination. Some evaluation text occurs in the training data (measured above); prefer the _clean sets.
• Tokenizer. 32k pieces shared with Sanskrit (in SLP1 transliteration) and math; FineWeb text takes 13% more tokens than with GPT-2's tokenizer (1.52 vs 1.35 per word).

Versions

Each version is a git tag on this repository; main is version v0.2.0. Pin one with revision=. A version is one training run, named by its run id in the training log.

version	where	date	training run	key numbers
v0.1.0	tag v0.1.0	2026-09-22	en0_shared32k_d125m_ctx1024 (EN0)	fineweb_val 1.1517, clean 1.1679
v0.2.0	tag v0.2.0, main	2026-09-24	en1_full_d125m_ctx1024 (EN1-full)	fineweb_val 1.1062, clean 1.1235

Files

file	what
model.safetensors	the weights, bfloat16 (no optimizer state)
config.json, generation_config.json	architecture and default generation settings
configuration_mume.py, modeling_mume.py	the network for transformers (auto_map, trust_remote_code)
tokenizer.model, tokenizer.json, tokenization_mume.py, tokenizer_config.json, special_tokens_map.json	the tokenizer (MuseMesh/mume-tokenizer-32k v0.1.0)
eval/bpb_ctx1024.json, eval/bpb_ctx512.json, eval/contamination_overlap.json, eval/heldout_meta.json, eval/reference_gpt2_124m_adamw.json, eval/smoke_test.json, eval/v0.1.0_en0_bpb_ctx1024.json, eval/verification.json	the evaluation outputs quoted above, the release checks and the sample generations
training/data_meta.json, training/slice_summary.json, training/summary.json	every training argument, the validation points and the data preparation record
LICENSE, LICENSE-CODE, CHANGELOG.md	licences and version history

Attribution

data	Hugging Face	reference	licence
FineWeb	HuggingFaceFW/fineweb	Penedo et al. 2024, The FineWeb Datasets, arXiv:2406.17557	ODC-By 1.0; use is also subject to the Common Crawl Terms of Use
WikiText-103	Salesforce/wikitext	Merity et al. 2016, Pointer Sentinel Mixture Models, arXiv:1609.07843	CC BY-SA 4.0 (per its dataset card)
enwik8 / text8	n/a	Matt Mahoney, mattmahoney.net/dc/textdata.html; the first 10^8 bytes of the English Wikipedia dump of 2006-03-03	no licence stated by the distributor; the text is English Wikipedia (GFDL at the time, CC BY-SA 3.0 since the 2009 relicensing)

The GPT-2 124M reference curve is from Keller Jordan's modded-nanogpt (MIT), record 2024-06-06, and the FineWeb validation shard is llm.c's (Karpathy), as re-hosted in kjj0/fineweb10B-gpt2.

Licence

This release is for research and non-commercial use.

• Weights (model.safetensors), tokenizer.model and tokenizer.json: CC BY-NC 4.0 (LICENSE). Non-commercial use (research, teaching, non-profit work) with attribution to "Muse Mesh Private Limited".
• Code (configuration_mume.py, modeling_mume.py, tokenization_mume.py): Apache-2.0 (LICENSE-CODE).
• Commercial use: contact kushal@muse-mesh.com.

Why non-commercial: The weights saw only permissively licensed text: FineWeb (ODC-By 1.0). The tokenizer is the issue: the shared 32k tokenizer was trained on a 1.2 GB sample of which 400 MB is Sanskrit from the Sansar corpus, and part of that Sanskrit is licensed for non-commercial use only (GRETIL, CC BY-NC-SA 4.0; Muktabodha, CC BY-NC 4.0) or carries no licence statement. The tokenizer, and the weights that only work with it, are therefore released under CC BY-NC 4.0, the same terms as the Sansar Sanskrit tokenizer. An Apache-2.0 line is planned as separate repositories: a new tokenizer trained without those sources, and the models retrained on it.

Citation

@misc{mume_english_125m_2026,
  title  = {Mume English 125M},
  author = {Muse Mesh},
  year   = {2026},
  note   = {v0.2.0},
  url    = {https://huggingface.co/MuseMesh/mume-english-125m}
}

Contact: kushal@muse-mesh.com