# Can t5 be used to text-generation?
**URL:**
**Author:**  [@kintaro](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkintaro)%5C
**Post date:** [September 11, 2020, 1:23am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F1 "2020-09-11T01:23:33Z")
Hello to all,
I’m following this tutorial: [https://huggingface.co/blog/how-to-generate](https://huggingface.co/blog/how-to-generate)
which says: " Auto-regressive language generation is now available for `GPT2` , `XLNet` , `OpenAi-GPT` , `CTRL` , `TransfoXL` , `XLM` , `Bart` , `T5` in both PyTorch and Tensorflow \>= 2.0!"
so I wanted to try to do the same, they just change the model to T5. However even though the model runs, the output is very strange. This is the code:
```
!pip install transformers
import tensorflow as tf
from transformers import TFT5ForConditionalGeneration, T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = TFT5ForConditionalGeneration.from_pretrained("t5-small", pad_token_id=tokenizer.eos_token_id)
input_ids = tokenizer.encode('Hello, my dog is cute', return_tensors='tf')
greedy_output = model.generate(input_ids, max_length=50)
print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))
```
I get this output:
Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello Hello
Because of this, and taking into account that I have not found many text-generation examples with t5, I would like to ask if this is possible? if so, why my output is so strange?
---
**Author:**  [@kintaro](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkintaro)%5C
**Post date:** [September 11, 2020, 3:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F2 "2020-09-11T15:47:47Z")
My mistake, reading the documentation is required that the first token is the task, for example ‘summarize’.
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [May 25, 2021, 12:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F3 "2021-05-25T12:52:32Z")
Right, I know about the task preamble, but I was wondering as well what task to specify for plain old auto-regressive text generation?
---
**Author:**  [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [May 29, 2021, 8:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F4 "2021-05-29T08:33:43Z")
Yes, using summarize task works. Here is an example I used:
```auto
from transformers import T5ForConditionalGeneration, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("t5-base")
model = T5ForConditionalGeneration.from_pretrained("t5-base")
input_ids = tokenizer.encode('summarize: I enjoy walking with my cute dog', return_tensors='pt')
greedy_output = model.generate(input_ids, num_beams=7, no_repeat_ngram_size=2, min_length=50, max_length=100)
print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))
```
## Output:
I enjoy walking with my cute dog - he is a joy to walk with and is very affectionate with me. I like to spend time with him on walks with his kitty cat, leo, who is so cute!
---
**Author:**  [@Savindu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSavindu)%5C
**Post date:** [July 23, 2021, 7:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F5 "2021-07-23T19:20:51Z")
I want to fine-tune the model for my task. but when I trying to use “transformers/examples/pytorch/language-modeling/run\_clm.py” gives this error.
```auto
ValueError: Unrecognized configuration class for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of RoFormerConfig, BigBirdPegasusConfig, GPTNeoConfig, BigBirdConfig, CamembertConfig, XLMRobertaConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMConfig, CTRLConfig, ReformerConfig, BertGenerationConfig, XLMProphetNetConfig, ProphetNetConfig, BartConfig, MBartConfig, PegasusConfig, MarianConfig, BlenderbotConfig, BlenderbotSmallConfig, MegatronBertConfig.
```
Is there any solutions for this?
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [July 26, 2021, 1:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F6 "2021-07-26T13:09:31Z")
As the error indicates, you cannot use the `run_clm` script with a T5 model as there is no version of T5 with a causal LM head. The error listed all the possible architectures you can use instead.
---
**Author:**  [@Savindu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSavindu)%5C
**Post date:** [August 5, 2021, 5:33pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F7 "2021-08-05T17:33:23Z")
Hi @sgugger ,
I want to **add some small rules when generating the output text**.
For e.g:- First number should be larger than the second generating number in the generating sentence.
Is that task is feasible inT5?
---
**Author:**  [@nofuture37](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnofuture37)%5C
**Post date:** [April 26, 2023, 6:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fcan-t5-be-used-to-text-generation%2F1075%2F8 "2023-04-26T06:40:03Z")
> [@sgugger](#):
>
> As the error indicates, you cannot use the `run_clm` script with a T5 model as there is no version of T5 with a causal LM head. The error listed all the possible architectures you can use instead.
hello! @sgugger , I’m reproducing a model for a paper and need to use t5-base model for Casual Language Modeling. I find nothing about it, would it be possilble to ask you?