---
license: cc-by-nc-4.0
datasets:
- nvidia/Nemotron-Post-Training-Dataset-v1
base_model:
- zai-org/GLM-4.5-Air
pipeline_tag: text-generation
---

# GLM-4.5-Air-NotaMoeQuant-Int4
## Overview
We developed a weight-only quantization method specialized for the Mixture-of-Experts (MoE) architecture, and we release GLM-4.5-Air quantized with our algorithm.
The quantized weights are packed using an AutoRound-based quantization format.

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_NAME = "nota-ai/GLM-4.5-Air-NotaMoeQuant-Int4"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME,
    dtype="auto",
    trust_remote_code=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

model.eval()

# prepare the model input
prompt = "What is large language model?"
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    return_token_type_ids=False
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    model_inputs['input_ids'],
    attention_mask=model_inputs['attention_mask'], 
    max_new_tokens=150
)

```

* Note: The gate layers were modified to use nn.Linear before quantization. Therefore, you must set trust_remote_code=True when loading this model.
* Model evaluations weres conducted using AutoRound==0.8.0.