# SpanBERT, ELECTRA, MARGE from scratch?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%3E%5C
**Category:** Beginners\
**Created:** [January 4, 2022, 4:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374 "2022-01-04T16:57:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![HenryAI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/henryai/32/3449_2.png) [@HenryAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHenryAI)%5C
**Post date:** [January 4, 2022, 4:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%2F1 "2022-01-04T16:57:03Z")

</div>

Hey everyone! I am incredibly grateful for this tutorial on training a language model from scratch: [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)

I really want to expand this to contiguous masking of longer token sequences (e.g. [mask-5], [mask-8]). I have begun looking into how to write a custom DataCollator for this, but suspect I will also need to make some changes to the model as well.

Has anyone looked into this and can point me to any resources?

Thank you!

---

<div class="post-metadata">

**Author:** ![HenryAI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/henryai/32/3449_2.png) [@HenryAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHenryAI)%5C
**Post date:** [January 4, 2022, 10:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%2F2 "2022-01-04T22:18:29Z")

</div>

Found something useful on StackOverflow for this:

> <https://stackoverflow.com/questions/60990897/best-way-of-using-hugging-faces-mask-filling-for-more-than-1-masked-token-at-a>

---

<div class="post-metadata">

**Author:** ![HenryAI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/henryai/32/3449_2.png) [@HenryAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FHenryAI)%5C
**Post date:** [January 6, 2022, 2:07pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%2F3 "2022-01-06T14:07:26Z")

</div>

This tutorial on Keras Code Examples is the most useful thing I have found so far on this:

> **[Keras documentation: End-to-end Masked Language Modeling with BERT](https://keras.io/examples/nlp/masked_language_modeling/)**
>
> Keras documentation

---

<div class="post-metadata">

**Author:** ![merve](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/merve/32/49809_2.png) [@merve](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmerve)%5C
**Post date:** [January 8, 2022, 9:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%2F4 "2022-01-08T21:45:58Z")

</div>

Hello Connor 🤗 Great to see you here! I will not be able to help you, I’m going to ping @nielsr here, maybe he could help. Sorry for delay!

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [January 9, 2022, 12:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%2F5 "2022-01-09T12:50:20Z")

</div>

Hi,

I’ve seen that SpanBERT models are on the [hub](https://huggingface.co/SpanBERT), but we haven’t added the model itself yet to the library.

This would be a great project actually:

- contribute SpanBERT to HuggingFace Transformers, based on the [modeling file](https://github.com/facebookresearch/SpanBERT/blob/main/pretraining/fairseq/models/pair_bert.py). This will be relatively easy, as the authors already used HuggingFace’s implementation of BERT and tweaked it a little bit. The only difference is [this class](https://github.com/facebookresearch/SpanBERT/blob/0670d8b6a38f6714b85ea7a033f16bd8cc162676/pretraining/fairseq/models/pair_bert.py#L425). We could then call the model `SpanBertModel` in the library, and add a `SpanBertForPreTraining` similar to `BertForPreTraining` that includes the heads necessary for pre-training.
- add a script to the [examples](https://github.com/huggingface/transformers/tree/master/examples/pytorch/language-modeling) directory, which could be called run\_span\_mlm.py (similar to run\_mlm.py). This can be based on the files defined [here](https://github.com/facebookresearch/SpanBERT/tree/main/pretraining#important-files) (Facebook open-sourced everything!).

If anyone is interested in contributing, let me know!

---

<div class="post-metadata">

**Author:** ![Aloka](https://avatars.discourse-cdn.com/v4/letter/a/cdc98d/32.png) [@Aloka](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAloka)%5C
**Post date:** [July 22, 2023, 1:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspanbert-electra-marge-from-scratch%2F13374%2F6 "2023-07-22T01:35:46Z")

</div>

@nielsr I am also interested in fine-tuning BERT (or any BERT like pre-trained model) using span masking. Can I know whether this is supported via transformers library. If so can you refer me to any resource available?