# Mixed precision for bfloat16-pretrained models

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 5, 2021, 8:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315 "2021-04-05T20:06:20Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![stas](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/stas/32/19289_2.png) [@stas](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstas)%5C
**Post date:** [April 21, 2021, 11:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315%2F3 "2021-04-21T23:37:27Z")

</div>

We started compiling a wiki of how different models were pre-trained, please add your knowledge there - thanks!

> [@Model pre-training precision database: fp16, fp32, bf16](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcompiling-data-on-how-models-were-pre-trained-fp16-fp32-bf16%2F5671)%3A
>
> There is an emerging need to know how a given model was pre-trained: fp16, fp32, bf16. So one won’t try to use fp32-pretrained model in fp16 regime. And most recently we are bombarded with users attempting to use bf16-pretrained (bfloat16!) models under fp16, which is very problematic since [fp16 and bf16 numerical ranges don’t overlap too well](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315). We are [discussing adding a new field to hugs models](https://github.com/huggingface/transformers/issues/11209) that will tell users how it was trained. Some papers don’t disclose how a model was trained. So …

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315)._