ARTPARK-IISc/DhVaani-0.5

DhVaani — TTS for 27 Indian languages

DhVaani is a fast, zero-shot voice-cloning text-to-speech model covering 27 Indian languages. Give it a few seconds of reference audio + its transcript and any target text, and it speaks that text in the reference voice — at 24 kHz.

Fine-tuned from ZipVoice (a 123M-parameter flow-matching TTS), with a character-level tokenizer that handles Indian script.

Quick start

# install torch/torchaudio for your machine first (GPU shown; use /whl/cpu for CPU):
pip install torch torchaudio 
pip install -r requirements.txt

transformers AutoModel (recommended)

import os,torch, soundfile as sf
from transformers import AutoModel
os.environ.setdefault("HF_TOKEN", "<your-hf-token>")
DEV = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained("ARTPARK-IISc/DhVaani-0.5",
                                  trust_remote_code=True).to(DEV).eval()

audio = model.synthesize(
    text="നമസ്‌കാരം, സുഖമാണോ?",
    prompt_wav="reference.wav",          # a few seconds of the voice to clone
    prompt_text="reference transcript",
)                                        # float32 numpy @ model.sampling_rate (24 kHz)
sf.write("out.wav", audio, model.sampling_rate)

Or standalone (no transformers)

from dhvaani import DhVaani
tts = DhVaani()                          # auto GPU/CPU
tts.synthesize(text="வணக்கம்!", prompt_wav="ref.wav",
               prompt_text="…", out_path="out.wav")

python dhvaani.py --prompt-wav ref.wav --prompt-text "…" --text "…" --out out.wav

Knobs: num_step (16 = quality, 8 = faster), guidance_scale, speed, seed.

Languages & data

Trained pooled from multiple TTS datasets like IISc SYSPIN, IndicTTS (SPRINGLab)and Rasa at 16 kHz.

well-represented	mid	low-resource
English, Hindi, Bengali, Marathi, Kannada, Telugu, Maithili, Magahi, Chhattisgarhi, Bhojpuri, Assamese, Tamil	Gujarati, Malayalam, Manipuri, Odia, Punjabi, Nepali, Sindhi, Konkani, Santali, Bodo, Urdu	Sanskrit, Dogri, Kashmiri, Rajasthani

Scripts: Devanagari, Bengali–Assamese, Tamil, Telugu, Kannada, Malayalam, Gujarati, Gurmukhi, Odia, Perso-Arabic, Ol Chiki, Latin.

What's in this repo

file	what
config.json · modeling_dhvaani.py	the AutoModel (trust_remote_code) wrapper
dhvaani.py	standalone API + CLI (no transformers needed)
model.safetensors	weights (491 MB)
model.json · tokens.txt	architecture config · 1058-char vocabulary
requirements.txt	dependencies
_backend/	the trimmed, inference-only model code DhVaani runs on
samples/	example outputs

Training

• Fine-tuned from k2-fsa/ZipVoice; the base Chinese+English text embedding (360 tokens) was re-initialised for the 1058-token Indic character vocab, the flow-matching acoustic backbone transferred.

Limitations

• Code-switching (English words inside Indic text) runs — Latin is in the vocab — but wasn't explicitly trained, so mixed-language prosody can be rough at boundaries.
• Data imbalance: low-resource languages (Rajasthani, Kashmiri, Dogri, Sanskrit) are ~10× smaller and weaker.
• Out-of-vocabulary characters (rare diacritics, emoji) are silently dropped.
• Best with a clean, single-speaker, 3–10 s reference clip.

Credits & license

Apache-2.0, following the base model. Built on ZipVoice (k2-fsa). Please also respect the licenses of the training corpora (IndicTTS, Rasa, IISc SYSPIN).