# GPT loss increasing

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-loss-increasing%2F7034%3E%5C
**Category:** Models\
**Created:** [June 21, 2021, 11:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-loss-increasing%2F7034 "2021-06-21T11:21:47Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![treeofknowledge](https://avatars.discourse-cdn.com/v4/letter/t/2acd7d/32.png) [@treeofknowledge](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftreeofknowledge)%5C
**Post date:** [June 21, 2021, 11:21am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-loss-increasing%2F7034%2F1 "2021-06-21T11:21:47Z")

</div>

Hi! I’m finetuning a GPT-2 model (360M params = size M) on a huge training set (dozens of GB) with a low learning rate (15e-6, or 0.000015). The loss value dips at the start, and then gradually increases for a long time. I haven’t seen it start to fall again yet. Is this behavior normal/expected?

 ![graph](https://us1.discourse-cdn.com/hellohellohello/original/2X/3/31c62dba89968d70282d0a1b02144e676e470482.png)

---

<div class="post-metadata">

**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flewtun)%5C
**Post date:** [June 21, 2021, 2:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt-loss-increasing%2F7034%2F2 "2021-06-21T14:55:44Z")

</div>

hey @treeofknowledge this plot suggests your learning rate is too high after 25k training steps so you could try using a learning rate scheduler like the default provided in `transformers.Trainer` (see [here](https://huggingface.co/transformers/main_classes/trainer.html?highlight=trainer#transformers.Trainer.create_scheduler))