--- license: apache-2.0 base_model: - tencent/Hy-Memory-30B-A3B base_model_relation: quantized pipeline_tag: text-generation library_name: gguf tags: - gguf - safetensors - mlx - fp8 - nvfp4 - tencent - hunyuan - memory - agent-memory - moe - llama-cpp - ollama --- [![🤗 Model Requests](https://img.shields.io/badge/%F0%9F%A4%97_Model_Requests-Submit_Here-f59e0b?style=for-the-badge&labelColor=1f2937)](https://abenzerps.com/request-a-model/) # Hy-Memory-30B-A3B Quantized GGUF, Safetensors, and MLX quantized formats of [tencent/Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B) for local memory operations and agent execution. Tencent's **Hy-Memory-30B-A3B** is a specialized Mixture-of-Experts (MoE) model (30B total parameters, 3B active per token) trained specifically as a high-performance memory writer and agent-memory operator. It excels at session summarization, context compaction, entity memory extraction, and agent memory management. ## Benchmark ![Hy-Memory-30B-A3B Benchmark](assets/benchmark.png) ## Quantized Files | **Quantization** | **File** | **Size** | | --- | --- | ---: | | FP8 | [**hy-memory-30b-a3b-fp8.safetensors**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/hy-memory-30b-a3b-fp8.safetensors) | 30.58 GB | | NVFP4 | [**hy-memory-30b-a3b-NVFP4.safetensors**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/hy-memory-30b-a3b-NVFP4.safetensors) | 17.65 GB | | MLX 4-bit | [**hy-memory-30b-a3b-MLX-4bit.safetensors**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/hy-memory-30b-a3b-MLX-4bit.safetensors) | 17.65 GB | | MLX 6-bit | [**hy-memory-30b-a3b-MLX-6bit.safetensors**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/hy-memory-30b-a3b-MLX-6bit.safetensors) | 25.04 GB | | MLX 8-bit | [**hy-memory-30b-a3b-MLX-8bit.safetensors**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/hy-memory-30b-a3b-MLX-8bit.safetensors) | 32.43 GB | | Q8_0 | [**Hy-Memory-30B-A3B-Q8_0.gguf**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/Hy-Memory-30B-A3B-Q8_0.gguf) | 31.99 GB | | Q6_K | [**Hy-Memory-30B-A3B-Q6_K.gguf**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/Hy-Memory-30B-A3B-Q6_K.gguf) | 24.71 GB | | Q5_K_M | [**Hy-Memory-30B-A3B-Q5_K_M.gguf**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/Hy-Memory-30B-A3B-Q5_K_M.gguf) | 21.37 GB | | Q4_K_M | [**Hy-Memory-30B-A3B-Q4_K_M.gguf**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/Hy-Memory-30B-A3B-Q4_K_M.gguf) | 18.24 GB | | Q4_0 | [**Hy-Memory-30B-A3B-Q4_0.gguf**](https://huggingface.co/abenzerps/Hy-Memory-30B-A3B-Quantized/blob/main/Hy-Memory-30B-A3B-Q4_0.gguf) | 17.02 GB | **Q4_K_M** is recommended for the best balance of size and quality for GGUF. **NVFP4** and **FP8** offer high performance on supported NVIDIA GPUs, while **MLX** formats are optimized for Apple Silicon. ## Usage ### 1. llama.cpp / llama-server (GGUF) #### Interactive Chat via CLI Run multi-turn interactive conversation directly in your terminal: ```bash ./llama-cli -m Hy-Memory-30B-A3B-Q4_K_M.gguf \ -cnv \ -c 8192 \ -n 512 \ --temp 0.6 \ -t 16 ``` #### OpenAI-Compatible API Server Start `llama-server` to serve OpenAI-compatible `/v1/chat/completions` endpoints: ```bash ./llama-server -m Hy-Memory-30B-A3B-Q4_K_M.gguf \ -c 16384 \ --port 8080 ``` Query via curl: ```bash curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "messages": [ { "role": "user", "content": "Extract key entities and memory facts from this text: Alice is a software architect living in Berlin who prefers dark mode and Python." } ], "temperature": 0.6 }' ``` ### 2. Ollama (GGUF) Create a `Modelfile`: ```dockerfile FROM ./Hy-Memory-30B-A3B-Q4_K_M.gguf TEMPLATE """{{- if .System }}<｜hy_User｜>{{ .System }} {{ end }}{{- range .Messages }}<｜hy_{{ if eq .Role "user" }}User{{ else }}Assistant{{ end }}｜>{{ .Content }}{{ if eq .Role "assistant" }}{{ end }}{{- end }}{{- if .Prompt }}<｜hy_User｜>{{ .Prompt }}<｜hy_Assistant｜>{{- end }}""" PARAMETER stop "" PARAMETER stop "<｜hy_User｜>" PARAMETER stop "<｜hy_Assistant｜>" PARAMETER stop "<｜hy_begin▁of▁sentence｜>" PARAMETER temperature 0.6 ``` Build and run: ```bash ollama create hy-memory -f Modelfile ollama run hy-memory ``` ### 3. Apple Silicon (MLX Safetensors) For Apple Silicon Macs, load MLX quantized weights (`4-bit`, `6-bit`, or `8-bit`): ```python import mlx.core as mx from safetensors.numpy import load_file # Load MLX-quantized weights weights = load_file("hy-memory-30b-a3b-MLX-4bit.safetensors") # Convert to native MLX array dictionary mlx_model_weights = {k: mx.array(v) for k, v in weights.items()} print(f"Loaded {len(mlx_model_weights)} MLX tensor parameters successfully.") ``` ### 4. NVIDIA FP8 & NVFP4 (Safetensors) Load the quantized weights directly via `safetensors` for TensorCore-accelerated pipelines (such as `comfy-kitchen`, `vLLM`, or custom inference): ```python import safetensors.torch import torch # Load FP8 (torch.float8_e4m3fn with per-tensor scales) fp8_weights = safetensors.torch.load_file("hy-memory-30b-a3b-fp8.safetensors") # Load NVFP4 (packed 4-bit with block-wise scales) nvfp4_weights = safetensors.torch.load_file("hy-memory-30b-a3b-NVFP4.safetensors") ``` ## Source and Build - **Source model:** [tencent/Hy-Memory-30B-A3B](https://huggingface.co/tencent/Hy-Memory-30B-A3B) - **Architecture:** HYV3ForCausalLM (MoE: 128 experts, 8 active per token, 30B total / 3B active) - **Conversion & Quantization:** [llama.cpp](https://github.com/ggml-org/llama.cpp) & [comfy-kitchen](https://github.com/comfyanonymous/comfy-kitchen) - **License:** Apache 2.0 - **Checksums:** [SHA256SUMS](./SHA256SUMS)