# Does fp16 training compromise accuracy?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-fp16-training-compromise-accuracy%2F18014%3E%5C
**Category:** Models\
**Created:** [May 16, 2022, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-fp16-training-compromise-accuracy%2F18014 "2022-05-16T15:37:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![petarulev](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@petarulev](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpetarulev)%5C
**Post date:** [May 16, 2022, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-fp16-training-compromise-accuracy%2F18014%2F1 "2022-05-16T15:37:37Z")

</div>

Does fp16 training compromise accuracy?

---

<div class="post-metadata">

**Author:** ![petarulev](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@petarulev](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpetarulev)%5C
**Post date:** [May 16, 2022, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-fp16-training-compromise-accuracy%2F18014%2F2 "2022-05-16T15:38:03Z")

</div>

> [@petarulev](#):
>
> training compromise accuracy?

If not, why isn’t it enabled by default?

---

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [May 17, 2022, 9:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdoes-fp16-training-compromise-accuracy%2F18014%2F3 "2022-05-17T21:49:45Z")

</div>

Mixed precision training (fp16) is only possible on certain hardware and in some cases results in [training instability](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315) depending on if the model was pre-trained using bfloat16.

For older GPUs (before Volta/Turing), fp16 provides no speed up and will require more memory because both the fp16 values and fp32 values will be stored in memory. I’d recommend reading this: [Performance and Scalability: How To Fit a Bigger Model and Train It Faster](https://huggingface.co/docs/transformers/performance#fp16-training)