# Clarification on the attention\_mask
**URL:**
**Author:**  [@zb1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzb1)%5C
**Post date:** [October 14, 2020, 8:14pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarification-on-the-attention-mask%2F1538%2F1 "2020-10-14T20:14:59Z")
Hello everyone,
I am trying to build a domain specific RoBERTa model and I need clarification on the **attention\_mask** usage. Glossary ([https://huggingface.co/transformers/glossary.html#attention-mask](https://huggingface.co/transformers/glossary.html#attention-mask)) says “The attention mask is a binary tensor indicating the position of the padded indices so that the model does not attend to them.”
On the github page ([https://github.com/huggingface/transformers/blob/master/src/transformers/modeling\_roberta.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py))
lines 196-198:
if attention\_mask is not None:
# Apply the attention mask is (precomputed for all layers in RobertaModel forward() function)
attention\_scores = attention\_scores + attention\_mask
I would think masking would be a multiplication of attention scores with the attention\_mask, not addition. Could someone clarify this for me?
My purpose looking into attention\_mask is to be able to use it as a quick way of assigning fixed weights to certain tokens. For example, if I have a domain specific sentence like this: “ **Parasequence boundaries** may be distinguished by **differences** in **physical** and **chemical properties** ”, where I care more about the domain specific words than the common language words. I assumed attention\_mask could be used as a domain specific weighting. thanks!
---
**Author:**  [@lysandre](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flysandre)%5C
**Post date:** [October 15, 2020, 12:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarification-on-the-attention-mask%2F1538%2F2 "2020-10-15T12:47:12Z")
Hi! The attention mask is made so that it accepts `0`s and `1`s. Putting a `1` indicates that this token should be attended to, while putting a `0` indicates a value that should not be attended to.
In the models it is modified [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py#L663) ([inversion](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L203)), which means that tokens that have a `1` value will now have `0`, and tokens that have a `0` value will now have `-10000`.
This value is then summed to the attention scores [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py#L198): if tokens had `1` initially, nothing changes. If tokens has `0` initially, their attention scores becomes a very negative value, nullifying their impact on the sequence.
Unfortunately, you won’t be able to use this attention mask to assign fix weights to certain tokens. The `attention_mask` can be used to nullify the impact of certain tokens on the rest of the sequence, but that’s about it.
---
**Author:**  [@zb1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzb1)%5C
**Post date:** [October 15, 2020, 2:03pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarification-on-the-attention-mask%2F1538%2F3 "2020-10-15T14:03:20Z")
Thank you for your response, very clear. I have missed the inversion part. Now it makes perfect sense. Thanks again!
---
**Author:**  [@rahul-evenup](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frahul-evenup)%5C
**Post date:** [April 20, 2023, 4:23pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarification-on-the-attention-mask%2F1538%2F4 "2023-04-20T16:23:12Z")
If you see this, wondering if you pursued this idea @zb1 and whether you implemented something of your own or found some other workaround? Thanks!
---
**Author:**  [@rajcrimecheck](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frajcrimecheck)%5C
**Post date:** [May 3, 2024, 4:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarification-on-the-attention-mask%2F1538%2F5 "2024-05-03T16:18:07Z")
suppose my sentence is :
[‘Mr’ , ‘Obama’, ‘had’, ‘a’, ‘very’, ‘good’, ‘standing’, ‘among’, ‘world’, ‘leaders’ , ‘.’]
if I give its attention\_mask as :
[0, 1, 0, 0, 0, 0, 0, 0, 0, 0 , 0]
And since attention, in context of transformers, basically is K.QT
that is key and query ,
So my doubt is , what does ignoring mean ??
Will the attention score be only attend to Kobama . Qobama ??
or does it mean the following :
while encoding ‘Obama’ , the self attention will encode it by attending to all other words
and while encoding other words , the self attention will not look to other words/tokens in the text ??