zen-0.1

Small language model (176.2M parameters) trained from scratch on 2× NVIDIA T4 (Kaggle).

Benchmarks (0-shot, lm-evaluation-harness)

Model Params ARC-Easy ARC-Challenge HellaSwag PIQA LAMBADA Avg
KLM 1.6 99.0M 61.07% 33.53% 45.28% 70.18% 55.90% 53.19%
GPT-X3-150M 149M 59.64% 31.31% 46.16% 71.33% 50.90% 51.87%
KLM 1.5 99.0M 59.68% 32.68% 43.74% 68.55% 48.20% 50.57%
SmolLM2-135M 135M 58.63% 29.69% 43.22% 68.44% 39.20% 47.84%
LFM-2.5-230M-Base 230M 59.72% 34.98% 39.29% 63.38% 36.60% 46.79%
void.1 90.1M 49.32% 29.35% 39.46% 67.19% 43.70% 45.80%
zen-0.1 176.2M 33.96% 21.67% 26.02% 55.17% 7.28% 28.82%

zen-0.1 evaluated with lm-evaluation-harness (acc_norm where available, otherwise acc). Other models' numbers are as reported by their authors; evaluation settings may differ.

Training

Architecture Llama (GQA, RoPE, SwiGLU, RMSNorm, tied embeddings)
Layers / hidden / heads (kv) 22 / 768 / 12 (4)
Context 1024 (train)
Tokenizer SmolLM2 (49k BPE)
Tokens seen 189M
Steps 1443
Training time 10.0 h on 2× T4
Precision fp16 mixed

Data mix: HuggingFaceFW/fineweb-edu (65% → 30%), HuggingFaceTB/smollm-corpus (30% → 18%), jonathanasdf/MathGLM-dataset-500k (0% → 5%), Crownelius/Complete-FABLE.5-traces-2M (2% → 10%), Claux67/FABLE-5.2-MAX-DATABASE (1% → 7%), VINAY-UMRETHE/Sonnet-Opus-4.5-4.6-Gemini-3.0-3.1-Pro-GPT-5-5.1-5.2-GLM-4.7-MiniMax-M2.1-DeepSeek-V3.2-High (0% → 15%), Jackrong/GLM-5.1-Reasoning-1M-Cleaned (2% → 15%)

GGUF

  • zen-0.1-F16.gguf
  • zen-0.1-Q8_0.gguf
llama-cli -hf angelgodbrokenns/zen-0.1:Q8_0 -p "The capital of France is"
ollama run hf.co/angelgodbrokenns/zen-0.1:Q8_0

Transformers

from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("angelgodbrokenns/zen-0.1")
model = AutoModelForCausalLM.from_pretrained("angelgodbrokenns/zen-0.1")
ids = tok("1+8/1*10+2=", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=40)[0]))

Samples (greedy)

Prompt: '1+8/1*10+2='

Output: 1.5643333333333333333333333333333333333333333333333333333333

Prompt: 'The capital of France is'

Output: the capital of the United States. The United States is a member of the United States, and the United States is the only country in the world. The United States is the only country in the world. The United States is the only country in the world. It is the largest country in

Prompt: '<|im_start|>user\nWhat is 12*7?<|im_end|>\n<|im_start|>assistant\n'

Output: The problem asks for the number of ways to get 12*7.

Let's break down the steps:

  1. Find the number of ways to get 12 (the number of ways to get 12).
  2. **Calculate the total number

License

The license has not been determined yet: check the licenses of all training datasets before choosing one.

Limitations

Tiny model trained on a small compute budget: it hallucinates, makes arithmetic mistakes, and is not suitable for production use.

Downloads last month
179
Safetensors
Model size
0.2B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for angelgodbrokenns/zen-0.1

Quantizations
1 model

Datasets used to train angelgodbrokenns/zen-0.1