# PreTrain RoBERTa from scratch in Marathi
**URL:**
**Author:**  [@nipunsadvilkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnipunsadvilkar)%5C
**Post date:** [July 6, 2021, 4:37pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F1 "2021-07-06T16:37:24Z")
# RoBERTa/BERT for Marathi
Currently, there are no BERT-like models for Marathi on the hub. For this project, the goal is to create a RoBERTa/BERT model for just the Marathi language.
## Model
A randomly initialized RoBERTa/BERT model
## Datasets
> **[The C4 Multilingual Dataset · allenai allennlp · Discussion #5265](https://github.com/allenai/allennlp/discussions/5265)**
>
> The wait has been long, but we are finally able to release the C4 multilingual dataset! We now have almost 27TB of clean-ish data, in 101 different languages (plus the "undetected" language). Here ...
One can make use [A colossal, cleaned version of Common Crawl’s web crawl corpus - Marathi](https://www.tensorflow.org/datasets/catalog/c4) dataset. Marathi corpus is ~70GB so should be great start for training a new language model.
## Available training scripts
A masked language modeling script for Flax is available - huggingface/transformers repo: examples/flax/language-modeling/run\_mlm\_flax.py. It can be used pretty much without any required code changes.
## (Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Marathi.
## (Optional) Links to read upon
The most important read would be the following colab:
- Download C4 dataset - allenai/allennlp/discussions/5056
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [July 6, 2021, 4:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F2 "2021-07-06T16:44:12Z")
We are pre-training RoBERTa for Hindi… maybe I can help…
How are we gonna evaluate this? Any benchmark datasets available for Marathi?
---
**Author:**  [@nipunsadvilkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnipunsadvilkar)%5C
**Post date:** [July 6, 2021, 4:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F3 "2021-07-06T16:49:50Z")
Found some relevent work on Marathi language:
> **[GitHub - goru001/nlp-for-marathi: State of the Art Language models and...](https://github.com/goru001/nlp-for-marathi)**
>
> State of the Art Language models and Classifier for Marathi, which is spoken predominantly by Marathi people of Maharashtra, India - GitHub - goru001/nlp-for-marathi: State of the Art Language mode...
> **[IndicBERT | AI4Bharat IndicNLP](https://indicnlp.ai4bharat.org/indic-bert/)**
>
> IndicBERT
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 6, 2021, 5:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F4 "2021-07-06T17:10:54Z")
Let’s define it
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [July 6, 2021, 5:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F5 "2021-07-06T17:13:44Z")
It would be a good start I guess. I will help 🙂
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [July 7, 2021, 1:08pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F6 "2021-07-07T13:08:26Z")
@nipunsadvilkar, [mc4](https://huggingface.co/datasets/mc4) is now available in HF datasets API as well… We can directly use it.
And we got access to TPU VM as well. I have created this channel `#roberta-pretraining-marathi` on the discord. Let’s talk more there.
---
**Author:**  [@nipunsadvilkar](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnipunsadvilkar)%5C
**Post date:** [July 7, 2021, 1:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F7 "2021-07-07T13:11:50Z")
> [@hassiahk](#):
>
> #roberta-pretraining-marathi
Great! Would you mind talking over slack channel? I will create `#roberta-pretraining-marathi` on slack then
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [July 7, 2021, 1:13pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-marathi%2F8101%2F8 "2021-07-07T13:13:36Z")
That’s fine as well :))