# Bert attention mask question

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-attention-mask-question%2F60142%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 27, 2023, 9:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-attention-mask-question%2F60142 "2023-10-27T09:11:36Z")\
**Posts on this page:** 1\
**Showing post:** 5

<div class="post-metadata">

**Author:** ![kjlhlosghgupg](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kjlhlosghgupg/32/24715_2.png) [@kjlhlosghgupg](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fkjlhlosghgupg)%5C
**Post date:** [March 11, 2024, 2:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-attention-mask-question%2F60142%2F5 "2024-03-11T14:46:47Z")

</div>

another finding: which my solve your probelm, and answers why attention map could block padding token embedding during the generate of qkv, and during two attention layer:

 ![Weixin Image_20240311222039](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/5/75fd6008447098998d739f7cbf9427a2cda5bb65.jpeg)

In a word, after use attention mask(with zero score contribution on padding token when q query from key k’s padding token’s column, by q\*K), the original input 512 \* 768 and the attention output 512 \* 712, all keep their padding token related elements with the previous padding token related row  
elements. Attention Layer not change padding token’s one-one mapping(not so precisely).

The same happened in Add and LayerNorm and FFN layer, **not change’‘s padding token’'s scope of influence[this is, not change(not add or not del) their contribution in loss(in gradient)]**.

Lastly, the output of bert model( cls, hidden), cls(1_768) is the first line of bert output hidden(512_768), as you thought, all elements in hidden(512\*768) that related to padding token embedding is the last two row[-2:,768], and **cls is not related to padding token embedding, so the following pretrain head (such as MLM or NSP), which only use cls as input**( [and the original embedding as this link says](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-embeddings-for-padding-token-not-0%2F14594)%2C but let’s ignore this temporarily), their task loss will not related to padding token embeddings. That’s it!

---

_[View the full topic](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbert-attention-mask-question%2F60142)._