oriyonay/vega-v1

Vega v1

Vega v1 is a 1B-parameter GPT model trained from scratch. It was pretrained on 13B tokens using 8 Intel XPUs for 2 weeks, then SFT’d on 2B tokens on 8 XPUs for another 2 days.

It can have basic conversations and recall well known facts. Of course, it hallucinates very often.

Vega v1 is a baseline for my future adventures into LLM training; it is not very useful, but it's fun to play with.

This repository contains the model checkpoint and tokenizer. The included vega_v1_inference.py entry point is the reference inference program.

Vega v1 has:

• Model architecture: vega_v1
• Layers: 14
• Hidden size: 2560
• Vocabulary size: 65536

Evaluation results

Zero-shot results from lm-evaluation-harness on the core suite. Vega v1 was evaluated with lm-eval 0.4.12 on one Intel PVC XPU with batch size 1.

Benchmark	Vega v1	GPT-2 XL	Qwen3.5-0.8B Instruct
HellaSwag (acc_norm)	43.66	40.0	46.2
PIQA (acc_norm)	68.93	70.5	68.2
WinoGrande	51.22	58.3	57.3
ARC-Easy (acc_norm)	55.93	58.3	50.1
ARC-Challenge (acc_norm)	32.00	25.0	35.1
OpenBookQA (acc_norm)	35.00	22.4	34.8
BoolQ	51.10	61.8	73.3
MMLU aggregate	25.72	≈25	≈42

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('oriyonay/vega-v1', trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained('oriyonay/vega-v1', trust_remote_code=True, device_map='auto').eval()

messages = [
    {'role': 'system', 'content': 'You are Vega v1, a helpful, honest, and concise AI assistant.'},
    {'role': 'user', 'content': 'What is the capital of France?'},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors='pt',
    return_dict=True,
)

inputs = inputs.to(next(model.parameters()).device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    do_sample=True,
)

generated_tokens = outputs[0, inputs['input_ids'].shape[1]:]
response = tokenizer.decode(generated_tokens, skip_special_tokens=True)

print(response)

Because Vega v1 uses custom model code, trust_remote_code=True is required.