# BertForMaskedLM on a fine-tuned base model
**URL:**
**Author:**  [@ndharap](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fndharap)%5C
**Post date:** [September 7, 2020, 8:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertformaskedlm-on-a-fine-tuned-base-model%2F1028%2F1 "2020-09-07T20:20:36Z")
Hello,
Is there a way for me to fine-tune the base bert/roberta architecture on a task like sequence classification, and then use the fine-tuned model as a base model for MLM predictions? I tried this by copying the state dict over from the sequence classification task into the MLM architecture, but that did not work at all. Seems like the weights that I swap from the sequence prediction task do not play well with the MLM objective.
Here is a code snippet -
#Load the fine-tuned ‘roberta-base’ model into RobertaForMaskedLM
roberta\_mlm\_model = RobertaForMaskedLM.from\_pretrained(MODEL\_FILE)
#load the default model
default\_model = RobertaForMaskedLM.from\_pretrained(‘roberta-base’)
#swap the weights for the head
roberta\_mlm\_model.lm\_head.load\_state\_dict(default\_model.lm\_head.state\_dict())
Can someone tell me if I am thinking in the right direction here?
Nikhil
---
**Author:**  [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [September 7, 2020, 9:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertformaskedlm-on-a-fine-tuned-base-model%2F1028%2F2 "2020-09-07T21:28:47Z")
This will be problematic because the heads are not compatible. In other words, you can fine tune the model and use the weights from one model in the other, but you still have the issue that the heads are different and cannot be mapped. So after fine tuning for sequenence classification, saving model, and loading that model in a MLM version of that architecture the LMHead will not have pretrained weights.
---
**Author:**  [@ndharap](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fndharap)%5C
**Post date:** [September 7, 2020, 9:58pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbertformaskedlm-on-a-fine-tuned-base-model%2F1028%2F3 "2020-09-07T21:58:44Z")
Thanks @BramVanroy! That is what I thought. The weights will not be relevant. How do you suggest I solve this? I was thinking of trying to train both the sequence classification task and the mlm task on my dataset in a multi-task setting. Not sure it you or anyone has any pointers