Anime-Speech-Japanese-Refiner
This model is a fine-tuned version of Qwen/Qwen3-Omni-30B-A3B-Instruct.
This is an audio processing model specialized for Japanese anime-style or game-style speech. It takes an audio input and its original transcription (text) to generate detailed descriptions (emotion, profile, etc.) and a refined transcription that includes non-speech events (e.g., breaths, sighs).
It was fine-tuned using the NandemoGHS/Galgame_Gemini_Captions dataset.
The training was conducted using the ms-swift library with the Megatron Backend.
Demo: https://huggingface.co/spaces/OmniAICreator/Anime-Speech-Japanese-Refiner-Demo
Intended Use and Limitations
This model is specifically designed for Japanese game-style or anime-style speech.
Due to the nature of its training data, it is not expected to perform well on:
- Languages other than Japanese.
- General conversational speech (e.g., meetings, casual dialogue).
How to Use (Inference)
We recommend using vLLM for inference.
vLLM Installation Requirements
This model requires building vLLM from a recent development commit as it is not yet supported in the latest stable release (v0.11.0 as of this writing).
It has been tested and confirmed to work with commit 18961c5ea62976efc50525b72e40337993c5e4f9. You must build vLLM from source:
git clone [https://github.com/vllm-project/vllm.git](https://github.com/vllm-project/vllm.git)
cd vllm
uv pip install . --torch-backend=auto -v --prerelease=allow
This requirement will likely be unnecessary after the v0.11.1 release.
Inference Example
Here is a simple inference script using vLLM:
import os
import torch
from vllm import LLM, SamplingParams
from transformers import Qwen3OmniMoeProcessor
from qwen_omni_utils import process_mm_info
if __name__ == '__main__':
# vLLM engine v1 not supported yet
os.environ['VLLM_USE_V1'] = '0'
MODEL_PATH = "NandemoGHS/Anime-Speech-Japanese-Refiner-FP8-DYNAMIC"
llm = LLM(
model=MODEL_PATH, trust_remote_code=True, gpu_memory_utilization=0.95,
tensor_parallel_size=torch.cuda.device_count(),
limit_mm_per_prompt={'audio': 1},
max_num_seqs=8,
max_model_len=8192,
seed=100,
)
sampling_params = SamplingParams(
temperature=0.6,
top_p=0.95,
top_k=20,
max_tokens=4096,
)
processor = Qwen3OmniMoeProcessor.from_pretrained(MODEL_PATH)
audio_path = "https://huggingface.co/NandemoGHS/Anime-Speech-Japanese-Refiner/resolve/main/examples/example1.wav"
original_transcription = "ใใฃใใใใใฃใๅฅฝใใๅคงๅฅฝใใงใใโฆใใใฃใจโฆใฏใใใใใฃใใฏใใใฃใใใฃใจใ๏ผ"
prompt = f"""ใใใใไธใใใใ้ณๅฃฐใฏใชใใใจใใฎๆๅญ่ตทใใใซใคใใฆใๅฃฐใฎ็นๅพดใจ่ชญใฟไธใในใฟใคใซใๆๆ
ใชใฉใใขใใใผใทใงใณใใใใใงใๆฅๆฌ่ชใฎ็ญใใญใฃใใทใงใณใง่ฆ็ดใใฆใใ ใใใ
ๅบๅใซใฏไปฅไธใฎ้
็ฎใๅซใใฆใใ ใใใ
profile: ่ฉฑ่
ใใญใใกใคใซ๏ผไพ: ใๅงใใ็ใชๅฅณๆงๅฃฐ๏ผ่ฝใก็ใใ็ทๆงๅฃฐ๏ผๅฐๅฅณๅฃฐ ็ญ๏ผ
mood: ๆๆ
ใปใ ใผใ๏ผไพ: ๆใใ๏ผ่ฝใก็ใใ๏ผ็ทๅผต๏ผๆใ๏ผๆๆ๏ผๆฒใใฟ๏ผๅฟซๆฅฝ ็ญ๏ผ
speed: ่ฉฑ้๏ผไพ: ใจใฆใ้
ใ๏ผใใ้ใ๏ผไธๅฎ๏ผ(1.2ร) ็ญ๏ผ
prosody: ๆๆใปใชใบใ ๏ผไพ: ๅนณๅฆ๏ผใกใชใใช๏ผ่ชๅฐพไธใไธใ๏ผใใๆฏๆททใใ ็ญ๏ผ
pitch_timbre: ใใใ/ๅฃฐ่ณช๏ผไพ: ้ซใ๏ผไฝใ๏ผๆฏๅคใ๏ผๅผตใใฎใใ๏ผๅใ ็ญ๏ผ
style: ็บ่ฉฑในใฟใคใซ๏ผไพ: ใใฌใผใทใงใณ้ขจ๏ผไผ่ฉฑ่ชฟ๏ผๆ่ชญ่ชฟ๏ผใใฌใผใณ่ชฟ๏ผๅใ๏ผๅใ๏ผๅๅฝ๏ผๅซใณ ็ญ๏ผ
emotion: ๆๆ
ใฟใฐ๏ผๆฌกใฎใชในใใใ1ใค้ธๆ: ["angry", "sad", "disdainful", "excited", "surprised", "satisfied", "unhappy", "anxious", "hysterical", "delighted", "scared", "worried", "indifferent", "upset", "impatient", "nervous", "guilty", "scornful", "frustrated", "depressed", "panicked", "furious", "empathetic", "embarrassed", "reluctant", "disgusted", "keen", "moved", "proud", "relaxed", "grateful", "confident", "interested", "curious", "confused", "joyful", "disapproving", "negative", "denying", "astonished", "serious", "sarcastic", "conciliative", "comforting", "sincere", "sneering", "hesitating", "yielding", "painful", "awkward", "amused", "loving", "dating", "longing", "aroused", "seductive", "ecstatic", "shy"]๏ผ
notes: ็น่จไบ้
๏ผ้ใฎๅใๆนใ็ฌใใปใใใปใใฌในใใใคใบๆใใญใน้ณใๅนๆ้ณใใใฅใ้ณ ็ญ๏ผ
caption: ไธ่จใ1ใ2ๆใปๅ
จ่ง30ใ80ๆๅญใง่ช็ถๆใซ่ฆ็ด
refined_text: ๅ
ใฎๆๅญ่ตทใใใใญในใใซใๅฟ
่ฆใซๅฟใใฆ็นๆฎใฟใฐใ้ณๅฃฐไธญใฎใคใใณใใฎๆๅใจใใฆๆ็ซ ใฎใฉใใใซๆฟๅ
ฅใใใใฎ๏ผๅฟ
่ฆใชใใใฐๅ
ใใญในใใใใฎใพใพๅบๅ๏ผใ
ๅ
ใฎๆๅญ่ตทใใใใญในใ: {original_transcription}
ๅ
ใฎ้ณๅฃฐใฏใชใใ๏ผ"""
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{"type": "audio", "audio": audio_path},
],
}
]
text = processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
audios, _, _ = process_mm_info(messages, use_audio_in_video=False)
inputs = {
'prompt': text,
'multi_modal_data': {},
}
if audios is not None:
inputs['multi_modal_data']['audio'] = audios
outputs = llm.generate([inputs], sampling_params=sampling_params)
print(outputs[0].outputs[0].text)
Example Output
This is the output generated for this example audio and its transcription.
emotion: ecstatic
profile: ใๅฌขๆง้ขจใฎๅฅณๆงๅฃฐ
mood: ๅฟซๆฅฝใ็ตถ้
speed: ้ๅใ้ๅใ
prosody: ๅใใชใใ่ฉฑใ
pitch_timbre: ้ซใใๆฏๅคใใ่ฃ่ฟใๆฐๅณ
style: ๅใ
notes: ๆฟใใๅใๅฃฐใจ่ใๆฏ้ฃใใๆททใใใๆง็ใช่ก็บใฎๆไธญใๅผทใ็คบๅใใใ
caption: ใๅฌขๆง้ขจใฎๅฅณๆงใๅฟซๆฅฝใซๅใใงใใใ้ซใ่ฃ่ฟใฃใๅฃฐใงใๆฏ้ฃใ่ใ้ๅใ้ๅใใซ่ฉฑใใ็ตถ้ ใซ่ฟใ่ๅฅฎ็ถๆ
ใ
refined_text: ๏ผๅใ๏ผใใฃใใใใใฃใๅฅฝใใๅคงๅฅฝใใงใใโฆใ๏ผๅๆฏ๏ผใใฃใจโฆใฏใใใใใฃใใฏใใใฃใใใฃใจใ๏ผ
Notebook Example
For a more detailed walkthrough, please see the inference_example.ipynb notebook. (Note: You will need to adapt the prompt for this Refiner model).
Output Format
The model outputs a structured description of the audio in Japanese, following this format:
emotion: {Emotion of the speech}
profile: {Speaker profile}
mood: {Mood of the speech}
speed: {Speaking speed}
prosody: {Prosody, rhythm}
pitch_timbre:{Pitch, voice quality}
style: {Style of utterance}
notes: {Other relevant notes}
caption: {A comprehensive caption integrating all elements}
refined_text: {Original transcription with added event tags (e.g., (ๅๆฏ))}
License
This model is licensed under CC-BY-NC-4.0 License.
Furthermore, the training data utilized outputs from Gemini 2.5 Pro. Therefore, any use that competes with or violates the terms of service of Gemini is strictly prohibited.
- Downloads last month
- 147