# T5 sequence classification

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-sequence-classification%2F2664%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 7, 2020, 3:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-sequence-classification%2F2664 "2020-12-07T15:11:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![manzar](https://avatars.discourse-cdn.com/v4/letter/m/da6949/32.png) [@manzar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmanzar)%5C
**Post date:** [December 7, 2020, 3:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-sequence-classification%2F2664%2F1 "2020-12-07T15:11:17Z")

</div>

I would like to do sequence classification over the encoder of the T5 model.

Which hidden state of the last layer should I use for the classification? The hidden state of the last timestep or should I take a mean over all timesteps?

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![arunwzd](https://avatars.discourse-cdn.com/v4/letter/a/90db22/32.png) [@arunwzd](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Farunwzd)%5C
**Post date:** [May 8, 2022, 6:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ft5-sequence-classification%2F2664%2F2 "2022-05-08T06:53:27Z")

</div>

> [@manzar](#):
>
> advance

are there any suggestions for this? I am not sure if `hidden[:, 0, :]` makes sense (since no `[CLS]` token in T5) but I found that using `hidden[:,0,:]` is yielding better results than `torch.mean(hidden_states, dim=1)`. Any suggestions on whats the best way to do this in T5Encoder?