# Framework for Continual Pretraining

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fframework-for-continual-pretraining%2F50989%3E%5C
**Category:** 🤗Transformers\
**Created:** [August 16, 2023, 4:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fframework-for-continual-pretraining%2F50989 "2023-08-16T16:48:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![theotherkhan](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@theotherkhan](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftheotherkhan)%5C
**Post date:** [August 16, 2023, 4:48pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fframework-for-continual-pretraining%2F50989%2F1 "2023-08-16T16:48:44Z")

</div>

I am conducting continual pretraining / [domain adaptive pretraining](https://arxiv.org/pdf/2004.10964.pdf) on bert-base-uncased using domain data in order to increase the accuracy of the model on domain specific tasks downstream. Currently, I am not seeing any improvements in those tasks after using a domain adapted model, even after fine-tuning. I want to confirm that I’m using Transformers correctly. Specifically:

1. Should I use _BertForMaskedLM_ or _BertForPreTraining_ as my starting model?
2. Should I train a tokenizer from scratch on the domain data or use a preexisting one?
3. Is having ~1M text records (each around a paragraph in length) roughly enough for continual pretraining?

Thanks!