# LongT5 fine-tunning

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongt5-fine-tunning%2F22650%3E%5C
**Category:** Beginners\
**Created:** [September 6, 2022, 9:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongt5-fine-tunning%2F22650 "2022-09-06T21:46:09Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [September 28, 2022, 9:02pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongt5-fine-tunning%2F22650%2F3 "2022-09-28T21:02:21Z")

</div>

See here for why it failed: [Mixed precision for bfloat16-pretrained models](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmixed-precision-for-bfloat16-pretrained-models%2F5315)

tldr: t5 was trained with bf16 and you fine tuned in fp16. bf16 has a much larger range than fp16 so those large values (bf16) turned into nan (fp16)

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Flongt5-fine-tunning%2F22650)._