Instructions to use Aleph-Alpha/Kolibri-1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Aleph-Alpha/Kolibri-1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Aleph-Alpha/Kolibri-1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Aleph-Alpha/Kolibri-1")
model = AutoModelForCausalLM.from_pretrained("Aleph-Alpha/Kolibri-1", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Aleph-Alpha/Kolibri-1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Aleph-Alpha/Kolibri-1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aleph-Alpha/Kolibri-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Aleph-Alpha/Kolibri-1

SGLang

How to use Aleph-Alpha/Kolibri-1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Aleph-Alpha/Kolibri-1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aleph-Alpha/Kolibri-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Aleph-Alpha/Kolibri-1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Aleph-Alpha/Kolibri-1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Aleph-Alpha/Kolibri-1 with Docker Model Runner:
```
docker model run hf.co/Aleph-Alpha/Kolibri-1
```

Request: Quantized versions (MLX / GGUF) for Apple Silicon – happy to benchmark

by Blackbeard82 - opened 19 days ago

Discussion

Blackbeard82

19 days ago

Hi, and thanks for releasing Kolibri-1 under Apache 2.0 — a German-focused MoE with only 3.46B active parameters is exactly what many of us have been waiting for.

Would it be possible to publish (or point us to) lower-bit quantized versions for consumer hardware? We run local inference on a 64 GB Apple Silicon Mac (M2 Max). The FP8 weights (~~78 GB) don't fit, but a 2.5–4-bit quant (~~26–39 GB) would.

What would help most:

MLX quantizations (3-bit and 4-bit, ideally keeping router and embeddings at higher precision) and the mlx-lm version that supports the architecture.
GGUF quantizations (Q3_K_M / IQ3 / Q4_K_M, a Q2 variant if quality allows) and the llama.cpp version or PR that supports Kolibri.
If available, quality numbers for the quantized versions versus FP8.

Once working builds exist, we'd be glad to run a short benchmark (speed, memory use, tool calling, German test set) and share the results here.

l-salewski

Aleph Alpha org 18 days ago

Thanks for for the amazing interest in our model! Whilst we don't have official quants yet, the amazing open source community has already created a few. You can find them here: https://huggingface.co/models?other=base_model:quantized:Aleph-Alpha/Kolibri-1

Blackbeard82

17 days ago

Follow-up: I measured multi-session throughput (4 sessions ≈ 84 tok/s total on an M2 Max), memory use and a KV-cache pitfall in mlx-lm on the MLX 2-bit build here: https://huggingface.co/velaia/Kolibri-1-MLX-2bit/discussions/1

tobiasoberrauch

17 days ago

I provided a 4-bit version https://huggingface.co/tobiasoberrauch/Kolibri-1-mlx-4bit

tobiasoberrauch

17 days ago

And 8-bit version https://huggingface.co/tobiasoberrauch/Kolibri-1-mlx-8bit

albinekb

17 days ago

Getting mlx-serve (zig kernel) working over here: https://github.com/ddalcu/mlx-serve/pull/738
no real speed gains yet, using https://huggingface.co/here-be-dragons-ai/Kolibri-1-MLX-3bit on M5 Pro 64GB.

webmp3

16 days ago

I’ve published a new low-bit GGUF series for Kolibri-1:

Sakura-MicroQuality Kolibri-1

IQ2_XS: 20.94 GiB / 2.30 bpw
~59% lower KL divergence than my standard IQ2_XS baseline
90.5% top-token agreement against the near-lossless Q8_0 reference

I also released a separate 365E expert-pruned variant at 19.99 GiB / 74.4B parameters.

Q3 and Q4 variants are currently uploading and should follow within the next few hours.
Repo:
https://huggingface.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF
365E repo:
https://huggingface.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF

Happy to share more measurements or compare results on different hardware.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment