# Loss rise and acc decline

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-rise-and-acc-decline%2F2199%3E%5C
**Category:** Models\
**Created:** [November 25, 2020, 8:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-rise-and-acc-decline%2F2199 "2020-11-25T08:54:35Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sniper](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sniper/32/578_2.png) [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [November 25, 2020, 8:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Floss-rise-and-acc-decline%2F2199%2F1 "2020-11-25T08:54:35Z")

</div>

When I train a model(code like this ,it is sample,the code is very similar to TFBertForSequenceClassification,just for deffient parameter ‘training’)

```
class BertFC(tf.keras.Model):
    def __init__ (self):
        super(BertFC, self). __init__ ()
        self.bert = TFBertModel.from_pretrained('bert-base-chinese', return_dict=True)
        self.dropout = tf.keras.layers.Dropout(0.1)
        self.dense = tf.keras.layers.Dense(20, activation='softmax')

    def call(self, inputs):
        idx, attn, ids = inputs
        hidden = self.bert(idx, attention_mask=attn, token_type_ids=ids, training=False)
        temp = hidden[1]
        temp = self.dropout(temp, training=True)
        out = self.dense(temp)
        return out

```

the learning rate is like this:

```
  class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
    def __init__ (self, d_model, warmup_steps=1000):
        super(CustomSchedule, self). __init__ ()

        self.d_model = tf.cast(d_model, tf.float32)
        self.warmup_steps = warmup_steps

    def __call__ (self, step):
        arg1 = tf.math.rsqrt(step)
        arg2 = step * (self.warmup_steps ** -1.5)

        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)

```

but when I train it , the acc first will rise,and then decline like this

 ![图片](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1bd9795de1673325ad76d60319dac794725f05c3.png)

I know about the saddle point,but the loss rise for a long time and seem like never decline  
what wrong about this?(train data is current)