# Sequence masking

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsequence-masking%2F17154%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 25, 2022, 12:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsequence-masking%2F17154 "2022-04-25T12:10:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitvit](https://avatars.discourse-cdn.com/v4/letter/v/b5e925/32.png) [@vitvit](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvitvit)%5C
**Post date:** [April 25, 2022, 12:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fsequence-masking%2F17154%2F1 "2022-04-25T12:10:20Z")

</div>

Hello,  
I’m using a custom tokenizer with a Roberta model.  
I want to pre-train the model using the masking task.  
Is there a way to mask a span of tokens instead of randomly selecting which to mask?  
I know that there is an option for whole word masking collator, but it is incompatible with my tokenizer and I also want more flexibility in the sequence length selection.

Thanks!