# Pretraining ALBERT

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretraining-albert%2F5607%3E%5C
**Category:** Intermediate\
**Created:** [April 19, 2021, 6:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretraining-albert%2F5607 "2021-04-19T06:55:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![stoddur](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@stoddur](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fstoddur)%5C
**Post date:** [April 19, 2021, 6:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretraining-albert%2F5607%2F1 "2021-04-19T06:55:54Z")

</div>

Hello!

I have a question regarding using the transformers library to pretrain ALBERT. I have been using RoBERTa for some while now which I have pretrained with custom data with run\_mlm.py from the examples directory which is fine since RoBERTa only uses MLM loss when pretraining. However, ALBERT adds sentence order prediction (SOP) which is not implemented in run\_mlm.py. Are there any examples on implementing SOP which I have overlooked in the transformers library? If not, anyone care to share a code example of how to implement this? If not I would have to dive a bit deeper to implement from scratch but I’m hoping I won’t have to 🙂

Thanks!

Edit:  
I found a DataCollatorForSOP which appears to solve this task: [transformers/data\_collator.py at d9c62047a8d75e18d2849d345ab3394875a712ef · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/src/transformers/data/data_collator.py#L515)

Would be great to have a seperate file in examples which implements the pretraining for ALBERT.

---

<div class="post-metadata">

**Author:** ![hdm](https://avatars.discourse-cdn.com/v4/letter/h/8797f3/32.png) [@hdm](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhdm)%5C
**Post date:** [February 6, 2022, 1:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretraining-albert%2F5607%2F2 "2022-02-06T13:26:53Z")

</div>

May I ask how you used that DataCollator? I’m also trying to pretrain ALBERT but I am also facing these difficulties.

---

<div class="post-metadata">

**Author:** ![ibraheemmoosa](https://avatars.discourse-cdn.com/v4/letter/i/a587f6/32.png) [@ibraheemmoosa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fibraheemmoosa)%5C
**Post date:** [February 16, 2022, 3:00pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpretraining-albert%2F5607%2F3 "2022-02-16T15:00:08Z")

</div>

I have pretrained an ALBERT model last year. You do not need a special DataCollator for SOP. Just use the DataCollatorForLanguageModeling.