# PreTrain T5 from scratch in Bengali
**URL:**
**Author:**  [@sbmaruf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbmaruf)%5C
**Post date:** [June 23, 2021, 3:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-bengali%2F7121%2F1 "2021-06-23T15:28:10Z")
# T5 for Bengali
Currently, there is no T5 model that was trained from scratch for Bengali on the hub. For this project, the goal is to create a strong language generation model for Bengali using T5 Model.
## 2. Language
Bengali.
## 3. Model
A randomly initialized T5 model.
## 4. Datasets
One can make use of OSCAR the dataset is also available through the datasets library here: [oscar · Datasets at Hugging Face](https://huggingface.co/datasets/oscar). The total Bengali resource in OSCAR is 11 GB.
Another source can be the mC4 dataset which is available in _[AllenAI](https://github.com/allenai/allennlp/discussions/5265)_. The resource size is 29GB.
## 5. Training scripts
A causal language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_clm_flax.py). It can be tweaked for training T5.
## 6. Challenges
- Adapt the training script to T5
- Fix a good tokenizer that covers Bengali vocabulary properly and make sure that the LM doesn’t become character-level LM.
## 7. Desired project outcome
The desired project output is a T5 model that is able to generate Bengali language.
## 8. Reads
The most important read would be the following colab:
- [Google Colaboratory](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/causal_language_modeling_flax.ipynb)
Apart from that we may need to look at the `seqio` library and source code of T5 here,
- [seqio](https://github.com/google/seqio)
- [T5 github](https://github.com/google-research/text-to-text-transfer-transformer)
---
**Author:**  [@tasnim](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftasnim)%5C
**Post date:** [June 23, 2021, 3:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-bengali%2F7121%2F2 "2021-06-23T15:57:44Z")
I am also a Bengali speaker. I am in!
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 29, 2021, 2:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-bengali%2F7121%2F3 "2021-06-29T14:36:09Z")
Sounds great, let’s finalize it!
---
**Author:**  [@trtm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftrtm)%5C
**Post date:** [April 4, 2022, 9:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-bengali%2F7121%2F4 "2022-04-04T09:45:07Z")
Any update on this? Is there a repo for this project?
---
**Author:**  [@StephennFernandes](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FStephennFernandes)%5C
**Post date:** [May 17, 2022, 4:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-bengali%2F7121%2F5 "2022-05-17T16:00:29Z")
@sbmaruf any update on this ? were you able to use seqio for T5 in the huggingface training script ?
---
**Author:**  [@sbmaruf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbmaruf)%5C
**Post date:** [July 26, 2022, 10:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-t5-from-scratch-in-bengali%2F7121%2F6 "2022-07-26T22:13:45Z")
We trained T5 with hggingface flax script, but the performance of the the language model on downstream task was very poor.
> **[sbmaruf/bengali\_t5\_base at main](https://huggingface.co/sbmaruf/bengali_t5_base/tree/main)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.
With the same data, the causal model worked fine.
More on T5 convergence issues I found later on.
[https://github.com/huggingface/transformers/issues/13335](https://github.com/huggingface/transformers/issues/13335)