# How to train huggingface model with fp16?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-huggingface-model-with-fp16%2F17952%3E%5C
**Category:** Beginners\
**Created:** [May 14, 2022, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-huggingface-model-with-fp16%2F17952 "2022-05-14T12:00:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Betacat](https://avatars.discourse-cdn.com/v4/letter/b/9f8e36/32.png) [@Betacat](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBetacat)%5C
**Post date:** [May 14, 2022, 12:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-huggingface-model-with-fp16%2F17952%2F1 "2022-05-14T12:00:01Z")

</div>

Hi I am using pytorch and huggingface to train my roberta-base to RTE dataset.  
But I find that just using `torch.half` to my model will cause nan after first backward.  
Is there any way to train my model with fp16 and without using huggingface’s `Trainer` function?

---

<div class="post-metadata">

**Author:** ![dblakely](https://avatars.discourse-cdn.com/v4/letter/d/54ee81/32.png) [@dblakely](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdblakely)%5C
**Post date:** [May 23, 2022, 11:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-to-train-huggingface-model-with-fp16%2F17952%2F2 "2022-05-23T23:08:54Z")

</div>

Yes, you can do this without using the Huggingface trainer. You’ll have to directly use the PyTorch AMP feature though (and this is what the HF trainer does as well, if you specify “amp” for the [half\_precision\_backend](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L249)). You’d want to get started by reading [this](https://pytorch.org/docs/stable/amp.html) page.