metadata
license: cc-by-nc-4.0
datasets:
- nvidia/Nemotron-Post-Training-Dataset-v1
base_model:
- zai-org/GLM-4.5-Air
pipeline_tag: text-generation
GLM-4.5-Air-NotaMoeQuant-Int4
Overview
We developed a weight-only quantization method specialized for the Mixture-of-Experts (MoE) architecture, and we release GLM-4.5-Air quantized with our algorithm. The quantized weights are packed using an AutoRound-based quantization format.
Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
MODEL_NAME = "nota-ai/GLM-4.5-Air-NotaMoeQuant-Int4"
# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(
MODEL_NAME,
dtype="auto",
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
model.eval()
# prepare the model input
prompt = "What is large language model?"
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
return_token_type_ids=False
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
# conduct text completion
generated_ids = model.generate(
model_inputs['input_ids'],
attention_mask=model_inputs['attention_mask'],
max_new_tokens=150
)
- Note: The gate layers were modified to use nn.Linear before quantization. Therefore, you must set trust_remote_code=True when loading this model.
- Model evaluations weres conducted using AutoRound==0.8.0.