Model Card for OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output

INT4 (W4A16) quantized version of OLMo-3.1-32B-Think with 16-bit activations (output). Made with LLM Compressor using GPTQ. Maintained by TOWIN-IA (Zamora, Spain).

Cuantificación desarrollada por: Erik Crespo Pera


Model Card for OLMo-3.1-32B-Think

OLMo-3.1-32B-Think is a 32-billion-parameter language model with an explicit reasoning mode (think). It has been trained on a large multilingual corpus covering Spanish, English and code, and is designed to show its internal reasoning process before producing the final answer. The model can be queried:

  • As instruct, to answer questions, explain concepts, analyze documents, or produce structured outputs.
  • As think, to expose the chain of reasoning (visible in the content field of the response), useful for auditing and traceability scenarios.

Quantization details

This repository contains the INT4 W4A16 variant of the base model, produced with:

Parameter Value
Quantization method GPTQ (via llmcompressor)
Weights INT4 (W4), group_size=128, symmetric
Activations 16-bit (A16) — not quantized
Format compressed-tensors (pack-quantized)
lm_head Kept in BF16 (ignored)
Calibration dataset openai/gsm8k (train[:512])
Output size ~17 GB (single model.safetensors)

This W4A16 scheme keeps activations at full 16-bit precision, which preserves the reasoning quality of the think mode while reducing weight memory 4×. It is compatible out-of-the-box with vLLM.

Installation

It is recommended to use this model with vLLM:

pip install vllm
Download
python
from huggingface_hub import snapshot_download
from pathlib import Path

models_path = Path.home().joinpath("models", "OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output")
models_path.mkdir(parents=True, exist_ok=True)

snapshot_download(
    repo_id="TOWIN-AI/OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output",
    local_dir=models_path,
)
Serving with vLLM
bash
vllm serve /path/to/OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output \
  --served-model-name TORQUEMADA-INT4 \
  --max-model-len 65536 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 2
The compressed-tensors format is detected automatically. No --quantization flag is required.

Chat Example
python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

response = client.chat.completions.create(
    model="TORQUEMADA-INT4",
    messages=[
        {"role": "system", "content": "Eres un auditor experto. Responde con precisión."},
        {"role": "user", "content": "Explica brevemente qué es el RGPD."},
    ],
    max_tokens=256,
    temperature=0.7,
)

print(response.choices[0].message.content)
The model will think out loud before answering. The content field contains the reasoning trace followed by the final response. This is intrinsic to the think mode and is not a bug.

Example output:

text
Okay, the user is asking for a brief explanation of the RGPD. First, I need to
confirm what RGPD stands for...

[final answer]
El RGPD (Reglamento General de Protección de Datos) es una normativa de la UE
que regula el tratamiento de datos personales y garantiza los derechos de los
ciudadanos...
Controlling the think mode
If you want to suppress the reasoning trace and keep only the final answer, you can either:

Parse the content in your client (split on the reasoning delimiters defined in chat_template.jinja).

Use stop tokens in the request:

python
response = client.chat.completions.create(
    model="TORQUEMADA-INT4",
    messages=[...],
    stop=["<|endofthought|>"],  # adjust to the model's actual template
    max_tokens=256,
)
Limitations
W4A16 precision: the activations are kept at 16-bit, but the weights are quantized to INT4. This may introduce small degradations in very long-context tasks compared to the BF16 base model.

Moderation: the model does not include built-in moderation. Add guardrails in your application if you plan to expose it to end users.

Reasoning trace: the think mode exposes the internal reasoning in the content field. Do not show it directly to end users unless that is the intended UX.

Hardware: INT4 W4A16 requires less VRAM than BF16 but still needs ~17 GB for weights. Plan your KV cache (--kv-cache-dtype fp8) based on your context length.

License
Base model (OLMo-3.1-32B-Think) is released under Apache 2.0 by the Allen Institute for AI. This quantized derivative is distributed under the same Apache 2.0 license.

Credits & Authorship
This quantized artifact was produced by TOWIN-IA (TYBSI SL, Zamora, Spain).

Quantization Pipeline: Custom W4A16 recipe using LLM Compressor.

Calibration Dataset: openai/gsm8k (512 samples).

Hardware: NVIDIA RTX PRO 6000 Blackwell (97 GB VRAM).

Authors: Erik Crespo Pera and the TOWIN-IA engineering team.

If you use this model in your research or production stack, please cite it as:

bibtex
@misc{towinia_olmo_int4,
  author       = {TOWIN-IA},
  title        = {OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/TOWIN-AI/OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output}}
}

About TOWIN-IA
TOWIN-IA is the sovereign AI infrastructure project of TYBSI SL, based in Zamora, Spain. We build on-premise AI compute capacity for European clients under European jurisdiction, with European-origin models running on hardware we control.

Our infrastructure is operated under principles of digital sovereignty: hardware in our datacenter, models from European and aligned vendors (Mistral, OLMo), data that never leaves our racks, full compliance with EU regulations (GDPR, NIS2, AI Act).

This model is part of our TORQUEMADA agent stack.

🌐 towinia.es · ✉️ info@towinia.es
Downloads last month
78
Safetensors
Model size
32B params
Tensor type
I32
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train TOWIN-AI/OLMo-3.1-32B-Think-GPTQ-Int4-16bit-output