# I meet the zero gradient descent
**URL:**
**Author:**  [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [November 12, 2020, 12:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F1 "2020-11-12T12:43:49Z")
I want use transformers to do text classification, I want code myself rather than use `TFBertForSequenceClassification` ,so I write the model with `TFBertModel` and `tf.keras.layers.Dense` ,but this is no gradient descent in my code, I try to find what wrong with my code but I can’t. So I submit this issues to ask for some help.
my code is here:
Model:
[](https://user-images.githubusercontent.com/30463691/98935026-e15a4180-251d-11eb-87a5-0d23bcb7cf4a.png)
[](https://user-images.githubusercontent.com/30463691/98935038-e4edc880-251d-11eb-9c3d-ee32df987590.png)
and I know train data is test data,just for quick debug.
and when I train this model ,
[](https://user-images.githubusercontent.com/30463691/98934626-5ed18200-251d-11eb-9eb3-c0330a6398bf.png)
---
**Author:**  [@thomwolf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fthomwolf)%5C
**Post date:** [November 12, 2020, 1:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F2 "2020-11-12T13:08:51Z")
maybe @jplu has some idea here.
If you could share a google colab exhibiting the behavior it would be a lot better than screen caps.
---
**Author:**  [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [November 12, 2020, 2:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F3 "2020-11-12T14:47:38Z")
thanks for your reply. I upload my code on the colab,this is the link :[https://colab.research.google.com/drive/1S4Y01Pr64R8uQi6cdNx53BzVx9iROZEH?usp=sharing](https://colab.research.google.com/drive/1S4Y01Pr64R8uQi6cdNx53BzVx9iROZEH?usp=sharing) and I have gave the edit permissions
It might be a little chaos cause I wrote it just for a test.Thank you very much for answering my question.
---
**Author:**  [@jplu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjplu)%5C
**Post date:** [November 12, 2020, 3:05pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F4 "2020-11-12T15:05:36Z")
At a first glance I would say it comes from two reasons:
1. You should not flatten the output of the BERT model
2. You take the sequence output while you have to take only the pooled output.
Look at how it is done in `TFBertForSequenceClassification`.
---
**Author:**  [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [November 12, 2020, 3:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F5 "2020-11-12T15:14:30Z")
I have seen the document of them for a hold afternoon.You mean the souce code of `TFBertForSequenceClassification`? But I’m too weak to find the souce code because there are a lot of call…How can I find the souce code.
---
**Author:**  [@jplu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjplu)%5C
**Post date:** [November 12, 2020, 3:31pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F6 "2020-11-12T15:31:01Z")
Here: [transformers/modeling\_tf\_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_bert.py#L1130)
---
**Author:**  [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [November 12, 2020, 3:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F7 "2020-11-12T15:37:52Z")
Thank you very much!!! I will read the source code tommorrow because is too late.Thanks again for your patient !！
But I still can’t understand why the gradient descent was zero.
I see the document
`last_hidden_state (tf.Tensor of shape (batch_size, sequence_length, hidden_size)) – Sequence of hidden-states at the output of the last layer of the model.`
It is the parameters of bert after all,why Dense(FC) can’t match them and do the gradient descent?
---
**Author:**  [@Sniper](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSniper)%5C
**Post date:** [November 13, 2020, 4:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fi-meet-the-zero-gradient-descent%2F2023%2F8 "2020-11-13T04:40:37Z")
I solve this problem,thanks a lot for patient teaching.