Berna R7 v3
1.2B parameter language model with self-growing cell architecture.
| Parameters | 1,199,439,360 |
| Cells | 896 active / 1680 total |
| Data | 2.66B tokens (eng/math/code) |
| Tokenizer | vocab 32000 (custom BPE) |
| Status | ✅ Training complete (step 10,200 / 10,200) |
| Final Loss | 2.35 |
| Final Checkpoint | step_00010200 |
Cell Growth Dynamics
| Step | Active Cells | Event |
|---|---|---|
| 0 | 112 | Initial |
| 300 | 224 | Split wave 1 |
| 600 | 448 | Split wave 2 |
| 900 | 896 | Split wave 3 |
| 900+ | 896 | Stable (9300+ steps) |
Growth: 8x (112 → 896), then natural stop.
Why it stopped:
- All cells hit min-neurons limit (256)
- Saturation S = 0.8578 (just above threshold 0.85)
- Data domains fully covered
K6 Encompassment: stable at 0.6138 (no explosion, no collapse) Births: 0 (no novel patterns — expected with current data)
Benchmarks (Final)
| Task | acc | acc_norm |
|---|---|---|
| HellaSwag | 28.19% | 29.84% |
| ARC-Easy | 47.69% | 41.71% |
| ARC-Challenge | 21.08% | 26.54% |
| PIQA | 60.50% | 57.78% |
| WinoGrande | 48.70% | — |
| OpenBookQA | 15.60% | 29.80% |
Perplexity (WikiText-103): 27.17
Perplexity (held-out English): 19.91
Comparison: GPT-2 Small (124M) achieves HellaSwag 31.4%, ARC-Easy 44.0%, PIQA 64.1%. Berna R7 v3 (1.2B) matches this at ~15x less training data.
Files
code/ Training code
config/ config.json
cells_runtime_current/ Live cells + cognitive log
cells_runtime_final/ Final cell state
checkpoints/step_00010200/ Final model (with wrapper)
checkpoints/step_00001700/ Earlier checkpoint
data/tokenized/ eng.npy, math.npy, code.npy
data/tokenizer/ tokenizer.json
Quick Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
p = "muhamedkamil/berna-r7-v3/checkpoints/step_00010200/model"
tok = AutoTokenizer.from_pretrained(p, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
p, trust_remote_code=True, torch_dtype=torch.bfloat16
).cuda().eval()
prompt = "The capital of France is"
ids = tok.encode(prompt, return_tensors="pt").cuda()
out = model.generate(ids, max_new_tokens=40, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
Critical Notes
--resume mandatory for continued training
bf16 required (else 8x slower — Tensor Cores disabled)
num_workers=8 required (else 8x slower)
Optimizer state incompatible between bnb.AdamW8bit and torch.optim.AdamW
trust_remote_code=True required for loading (custom architecture)
Performance
GPU Config tok/s ETA (2.67B)
RTX 5090 batch=1 4,630 ~140h
A100 SXM batch=16, bf16, nw=8 14,000 ~46h
H100 batch=16, bf16 12-15k ~45h
H200 batch=24, bf16 15-22k ~30-40h
License
CC BY-NC 4.0 — research and personal non-commercial use only.
Commercial use requires written permission from the author.
Full text: https://creativecommons.org/licenses/by-nc/4.0/legalcode
Citation
bibtex
@misc{berna_r7_v3_2026,
author = {Muhamed Kamil},
title = {Berna R7 v3: DNA-Kernel Plexus Language Model},
year = {2026},
howpublished = {Hugging Face: muhamedkamil/berna-r7-v3},
note = {ORCID: 0009-0002-5866-8815, DOI: 10.5281/zenodo.23015308}
}
Links
R5 Theory: https://doi.org/10.5281/zenodo.23015308
R7 v0.1: https://doi.org/10.5281/zenodo.23037065
GitHub: https://github.com/Berna-Labs
Author: Muhamed Kamil (ORCID)
Updated: 2026-10-12 — Training complete