# Pass a custom mask when using RoBERTa
**URL:**
**Author:**  [@megamind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmegamind)%5C
**Post date:** [January 15, 2021, 2:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210%2F1 "2021-01-15T02:07:50Z")
Suppose I have a sequence that consists of 2 sentences separated by \ tokens like A \ B. When performing forward pass with RoBERTa model, I want tokens in sentence A only attend to tokens in sentence A and vice versa for sentence B. The mask will be look like this:
In summary, is there any way to explicitly pass a custom attention mask to the model?
Thanks in advance.
---
**Author:**  [@s4sarath](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fs4sarath)%5C
**Post date:** [January 19, 2021, 3:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210%2F2 "2021-01-19T15:40:53Z")
Attention mask is normally created from input\_mask . You cannot bypass attention mask directly. I might be wrong also.
For your purpose, create an input\_mask with 1s on First row rows and two colums and then 1s on last two rows and last two columns. Set else to 0.
---
**Author:**  [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [January 19, 2021, 9:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210%2F3 "2021-01-19T21:22:45Z")
Yes, all models take an `attention_mask` argument that you can customize.
---
**Author:**  [@megamind](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmegamind)%5C
**Post date:** [January 20, 2021, 2:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210%2F4 "2021-01-20T02:37:17Z")
From the documentation I see that model will accept 1D tensor (assume batch size equal 1) but what I need is 2D attention mask. How can I manage to do that? Thanks for your response.
**attention\_mask** ( `torch.FloatTensor` of shape `((batch_size, sequence_length))` , optional) –
Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]` :
- 1 for tokens that are **not masked** ,
- 0 for tokens that are **masked**.
---
**Author:**  [@voidful](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvoidful)%5C
**Post date:** [May 26, 2021, 8:28am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210%2F5 "2021-05-26T08:28:48Z")
you can change extended\_attention\_mask inside Roberta model, in order to make a 3D attention mask for your case.
I suggest that we can pass extended\_attention\_mask as a parameter.
---
**Author:**  [@AfonsoSousa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAfonsoSousa)%5C
**Post date:** [January 10, 2023, 5:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpass-a-custom-mask-when-using-roberta%2F3210%2F6 "2023-01-10T17:24:18Z")
I have a similar issue. Can you provide a minimal example on how to use a 3D attention mask per input sequence?