# Fine tune Masked Language Model on custom dataset
**URL:**
**Author:**  [@smalltoken](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmalltoken)%5C
**Post date:** [August 18, 2020, 1:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-masked-language-model-on-custom-dataset%2F747%2F1 "2020-08-18T13:10:35Z")
Hi, I am new in Bert, I want to fine tune a plain text corpus(only text, without any label or other information) to get hidden state layer for specific word/document embedding.
Thus I think the suitable model for my task is the masked language model, however I found this only example demo [https://huggingface.co/blog/how-to-train](https://huggingface.co/blog/how-to-train), which is not universal and clear.
The original examples are here, but no mlm model available [https://huggingface.co/transformers/master/custom\_datasets.html](https://huggingface.co/transformers/master/custom_datasets.html) Does anyone have clear example for this case?
To be specific, I don’t know how to pass true label in MLM task for loss and gradient update. Also while using the Trainer, where should I insert -mlm command?
Thank you in advance
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 20, 2020, 3:25pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-masked-language-model-on-custom-dataset%2F747%2F2 "2020-08-20T15:25:27Z")
Hi @smalltoken, what is the issue with [https://huggingface.co/blog/how-to-train](https://huggingface.co/blog/how-to-train) ?
This colab should help you. It walks you through,
1. How to to train tokenizer from scratch
2. Create `RobertaModel` using the config
3. use the `DataCollatorForLanguageModeling`, which handle the masking
4. and train using `Trainer`.
---
**Author:**  [@smalltoken](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmalltoken)%5C
**Post date:** [August 20, 2020, 3:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-masked-language-model-on-custom-dataset%2F747%2F3 "2020-08-20T15:46:58Z")
Thanks for your kind reply.
I put the code and error message in this open issue on github.[https://github.com/huggingface/transformers/issues/6616](https://github.com/huggingface/transformers/issues/6616) My problem is mainly about ‘index out of range in self’ error, given that I alredy set a very low max\_length value in tokenizer. I was also confused using native PyTorch to train before, in which I don’t know how to handle mask and pass labels to model. But if DataCollatorForLanguageModeling and Trainer works well, that doesn’t matter.
Could you help verify if there is any mistake in my piece of code? Thank you!
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 20, 2020, 3:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-masked-language-model-on-custom-dataset%2F747%2F4 "2020-08-20T15:52:28Z")
Sure, what is `./bert-large-cased` in the code, is it a pre-trained bert or did you create it ? If yes, can you post the config ?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [August 20, 2020, 3:52pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-masked-language-model-on-custom-dataset%2F747%2F5 "2020-08-20T15:52:52Z")
Also `DataCollatorForLanguageModeling` handles masking
---
**Author:**  [@smalltoken](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsmalltoken)%5C
**Post date:** [August 20, 2020, 9:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffine-tune-masked-language-model-on-custom-dataset%2F747%2F6 "2020-08-20T21:09:39Z")
Because of proxyerror in my working company, I mannually downloaded the pre trained model from [s3.amazonaws.com](http://s3.amazonaws.com).(So maybe I ignored some items to be download?) And `./bert-large-cased` is a folder which contains these following items :
1. bert-large-cased-tf\_model.h5
2. config.json
3. pytorch\_model.bin
4. vocab.txt
Here is what inside the config:
- attention\_probs\_dropout\_prob:0.1
- directionality:“bidi”
- hidden\_act:“gelu”
- hidden\_dropout\_prob:0.1
- hidden\_size:1024
- initializer\_range:0.02
- intermediate\_size:4096
- max\_position\_embeddings:512
- num\_attention\_heads:16
- num\_hidden\_layers:24
- pooler\_fc\_size:768
- pooler\_num\_attention\_heads:12
- pooler\_num\_fc\_layers:3
- pooler\_size\_per\_head:128
- pooler\_type:“first\_token\_transform”
- type\_vocab\_size:2
- vocab\_size:28996