# PreTrain RoBERTa from scratch in Thai
**URL:**
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [June 23, 2021, 11:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-thai%2F7092%2F1 "2021-06-23T11:41:24Z")
# RoBERTa/BERT for Thai
Currently, there is only a very limited amount of BERT-like models for Thai on the hub: [Models – Hugging Face](https://huggingface.co/models?filter=bert,th) . For this project, the goal is to create a RoBERTa/BERT model for just the Thai language.
## Model
A randomly initialized RoBERTa/BERT model
## Datasets
One can make use [OSCAR](https://oscar-corpus.com/) the dataset is also available through the `datasets` library here: [Oscar (Oscar Kjell)](https://huggingface.co/datasets/oscar).
## Available training scripts
A masked language modeling script for Flax is available [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py). It can be used pretty much without any required code changes.
## (Optional) Desired project outcome
The desired project output is a strong RoBERTa/BERT model in Thai.
## (Optional) Challenges
The OSCAR dataset might be too small (it has \< 20GB of data for Thai). Also it might be important
to find datasets the BERT-like model can be evaluated on after pretraining in Thai. Having found a dataset to fine-tune the pretrained BERT-like model on, one can make use of the text-classification script [here](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_mlm_flax.py)
## (Optional) Links to read upon
The most important read would be the following colab:
- [Google Colab](https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/masked_language_modeling_flax.ipynb)
---
**Author:**  [@sakares](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsakares)%5C
**Post date:** [June 30, 2021, 3:00am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-thai%2F7092%2F2 "2021-06-30T03:00:43Z")
Hi there. I have tried with this [blog](https://huggingface.co/blog/how-to-train) for PyTorch before and would love to do this in Flax/Jax too 🙂
I will take a look and update the status back soon.
---
**Author:**  [@patrickvonplaten](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpatrickvonplaten)%5C
**Post date:** [July 2, 2021, 3:36pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-thai%2F7092%2F3 "2021-07-02T15:36:21Z")
Let’s define it
---
**Author:**  [@sakares](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsakares)%5C
**Post date:** [July 2, 2021, 4:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretrain-roberta-from-scratch-in-thai%2F7092%2F4 "2021-07-02T16:55:34Z")
Alright, as far as I know once LM is ready, we will start with checking the downstream task to an existing benchmark like this [one](https://github.com/kobkrit/nlp_thai_resources#thai-text-classification-benchmarks)