# Bert trainning loss suddenly explodes

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-trainning-loss-suddenly-explodes%2F8684%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 26, 2021, 8:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-trainning-loss-suddenly-explodes%2F8684 "2021-07-26T08:11:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![hong-555](https://avatars.discourse-cdn.com/v4/letter/h/dc4da7/32.png) [@hong-555](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhong-555)%5C
**Post date:** [July 26, 2021, 8:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-trainning-loss-suddenly-explodes%2F8684%2F1 "2021-07-26T08:11:58Z")

</div>

![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/a797909cb7b611293f7fcdf55a9ae89ebc594f4b.png)

Training Chinese and English language modeling MLM bert model  
–num\_train\_epochs 500   
–per\_device\_train\_batch\_size 8   
–learning\_rate 1e-4   
–warmup\_steps 5000   
–max\_seq\_length 512 \

---

<div class="post-metadata">

**Author:** ![adnenabdessaied](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adnenabdessaied/32/17916_2.png) [@adnenabdessaied](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fadnenabdessaied)%5C
**Post date:** [July 25, 2023, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-trainning-loss-suddenly-explodes%2F8684%2F2 "2023-07-25T11:07:34Z")

</div>

Hey! This is probably too late. But this is most likely due to a high learning rate. You could reduce the learning rate and/or use gradient clipping. The latter will not have a significant change in performance, but will prevent a bad mini-batch from messing up the training of your model. Good luck!