TRL documentation

快速入门

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

快速入门

TRL 是一个全面的库,用于使用监督微调(SFT)、组相对策略优化(GRPO)、直接偏好优化(DPO)等技术对基础模型进行后训练。

快速示例

立即使用 TRL 中最受欢迎的训练器开始上手。每个示例都使用紧凑的模型,以便快速进行实验。

监督微调

from trl import SFTTrainer
from datasets import load_dataset

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
)
trainer.train()

Group Relative Policy Optimization

accuracy_reward 需要 math-verify,您可以通过 pip install trl[math_verify] 进行安装。

from trl import GRPOTrainer
from datasets import load_dataset
from trl.rewards import accuracy_reward

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=load_dataset("trl-lib/DeepMath-103K", split="train"),
    reward_funcs=accuracy_reward,
)
trainer.train()

直接偏好优化

from trl import DPOTrainer
from datasets import load_dataset

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=load_dataset("trl-lib/ultrafeedback_binarized", split="train"),
)
trainer.train()

奖励建模

from trl import RewardTrainer
from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

trainer = RewardTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=dataset,
)
trainer.train()

命令行界面

完全跳过代码 - 直接从终端开始训练:

# SFT: Fine-tune on instructions
trl sft --model_name_or_path Qwen/Qwen2.5-0.5B \
    --dataset_name trl-lib/Capybara

# DPO: Align with preferences  
trl dpo --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name trl-lib/ultrafeedback_binarized

# Reward: Train a reward model
trl reward --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name trl-lib/ultrafeedback_binarized

接下来是什么?

📚 了解更多

🚀 扩展

💡 示例

故障排除

内存不足?

减小批量大小并启用优化:

SFT
DPO
GRPO
training_args = SFTConfig(
    per_device_train_batch_size=1,  # Start small
    gradient_accumulation_steps=8,  # Maintain effective batch size
)

损失不下降?

尝试调整学习率:

training_args = SFTConfig(learning_rate=2e-5)  # Good starting point

如需更多帮助,请在 GitHub 上提交 issue。

Update on GitHub