# PreTrain RoBERTa from scratch in Hindi
**URL:**
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 11:32am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F1 "2021-06-23T11:32:56Z")
# RoBERTa/BERT for Hindi
Currently, there is only a very limited amount of BERT-like models for Hindi on the hub: [Models – Hugging Face](https://huggingface.co/models?filter=bert,hi) . For this project, the goal is to create a RoBERTa/BERT model for just the Hindi language.
## Model
A randomly initialized RoBERTa/BERT model
## Datasets
One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [Oscar (Oscar Kjell)](https://huggingface.co/datasets/oscar).
## Available training scripts
A masked language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py). It can be used pretty much without any required code changes.
## (Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Hindi.
## (Optional) Challenges
The OSCAR dataset might be too small (it has \< 10GB of data for Hindi). Also it might be important
to find datasets the BERT-like model can be evaluated on after pretraining in Hindi. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification script [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py)
## (Optional) Links to read upon
The most important read would be the following colab:
- [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/masked_language_modeling_flax.ipynb)
---
**Author:**  [@amankhandelia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famankhandelia)%5C
**Post date:** [June 23, 2021, 5:15pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F2 "2021-06-23T17:15:06Z")
I am planning to work on this project.
So, here is my first question
As the current dataset size is too small for it to effectively train BERT and planning on building my own dataset, by scrapping website that host content in Hindi.
The first thing that comes to mind is scrapping news article, which perpetual source of data, we can also tap into some social networks which have more content in Hindi, such as ShareChat. Hindi/Hindustani Literature websites
Does this approach sounds reasonable, or is too weak because common crawl could have already covered such websites?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 23, 2021, 5:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F3 "2021-06-23T17:23:51Z")
Hi @amankhandelia
For Hindi, the OSCAR corpus has almost 8GB which will be good-enough for a BERT like model. [IndicCorp](https://indicnlp.ai4bharat.org/corpora/) also has a Hindi corpus.
And then there’s [mC4](https://github.com/allenai/allennlp/discussions/5265) which has almost 104GB of Hindi data which should be more than enough for this project 🙂
Also, here’s an interesting insight from CamemBERT paper
---
**Author:**  [@sbmaruf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbmaruf)%5C
**Post date:** [June 23, 2021, 5:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F4 "2021-06-23T17:26:12Z")
Some additional resources for hindi,
1. [mC4](https://github.com/allenai/allennlp/discussions/5265) - 105 GB
2. [CC-100](http://data.statmt.org/cc-100/) : 2.5 GB compressed.
I don’t think data will be a problem.
---
**Author:**  [@amankhandelia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famankhandelia)%5C
**Post date:** [June 23, 2021, 11:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F5 "2021-06-23T23:55:17Z")
In that case, I stand corrected.
What are some downstream tasks with corresponding dataset in Hindi on which we can test such model’s efficacy, see if increasing data volume is having an effect on performance of such tasks?
---
**Author:**  [@sbmaruf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbmaruf)%5C
**Post date:** [June 24, 2021, 12:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F7 "2021-06-24T00:13:39Z")
I can recall at least 6 datasets with `Hindi` evaluation.
See here in the XTREME Benchmark, [https://arxiv.org/pdf/2003.11080.pdf](https://arxiv.org/pdf/2003.11080.pdf)
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 24, 2021, 5:17am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F8 "2021-06-24T05:17:21Z")
There’s [IndicGLUE](https://indicnlp.ai4bharat.org/indic-glue/) benchmark here which has quite a few downstream tasks.
---
**Author:**  [@mlkorra](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmlkorra)%5C
**Post date:** [June 25, 2021, 4:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F9 "2021-06-25T16:04:26Z")
I am interested to work on this project
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [June 28, 2021, 11:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F10 "2021-06-28T11:02:26Z")
I am interested in working on this project. Can I take part?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 28, 2021, 11:12am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F11 "2021-06-28T11:12:44Z")
yes, anyone can participate 🙂
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 28, 2021, 3:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F12 "2021-06-28T15:55:44Z")
Great - let’s officially define this project then 🙂
Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing) . @sbmaruf - I’ve put you in the group for now as well, feel free to leave a comment in the google sheet to be removed if you don’t want to be in the group. I wasn’t sure 🙂
---
**Author:**  [@sbmaruf](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsbmaruf)%5C
**Post date:** [June 29, 2021, 8:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F13 "2021-06-29T08:53:47Z")
Hi! I will be doing this for Bangla. Probably it’s better I focus more there.
---
**Author:**  [@ramjaju](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Framjaju)%5C
**Post date:** [June 29, 2021, 6:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F14 "2021-06-29T18:23:46Z")
Hi, I am interested in working on this project. Can I be a participate if slots are open ? BTW I am new to JAX.
---
**Author:**  [@amankhandelia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famankhandelia)%5C
**Post date:** [June 30, 2021, 5:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F15 "2021-06-30T05:35:21Z")
Hey everyone who is participating in this project, you can join the #roberta-pretraining-hindi project over on Discord, we can start formulating how to go about it, all the datasets we will use and any downstream task on which we might wish to check the performance of our trained model
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [June 30, 2021, 8:09am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F16 "2021-06-30T08:09:14Z")
Yes! Added you to the team!
And don’t worry about JAX, we have an awesome [speaker line-up](https://github.com/huggingface/transformers/tree/master/examples/research_projects/jax-projects#talks) that will cover JAX and we and the community will be there to help with any JAX-related questions 🙂
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [June 30, 2021, 1:44pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F17 "2021-06-30T13:44:34Z")
Hey. Which Discord? Can you share me an invite?
---
**Author:**  [@amankhandelia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famankhandelia)%5C
**Post date:** [June 30, 2021, 2:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F18 "2021-06-30T14:03:10Z")
Yeah sure, [Flax-HuggingFace-Community-Week](https://discord.gg/ybwxhkhx)
---
**Author:**  [@hassiahk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhassiahk)%5C
**Post date:** [June 30, 2021, 2:06pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F20 "2021-06-30T14:06:44Z")
Thanks man 🙂
---
**Author:**  [@amankhandelia](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Famankhandelia)%5C
**Post date:** [June 30, 2021, 2:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F21 "2021-06-30T14:07:11Z")
Hey, I updated link, shared the generic link, this one will directly land you in the group
---
**Author:**  [@skylord](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fskylord)%5C
**Post date:** [June 30, 2021, 2:21pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091%2F22 "2021-06-30T14:21:07Z")
I am interested. Pls share the link to join. Thanks.
[Next page](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-hindi%2F7091.md%3Fpage%3D2)