# How is CLS special token embedding initialized?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-cls-special-token-embedding-initialized%2F15768%3E%5C
**Category:** Intermediate\
**Created:** [March 16, 2022, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-cls-special-token-embedding-initialized%2F15768 "2022-03-16T11:07:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sgavela](https://avatars.discourse-cdn.com/v4/letter/s/e9a140/32.png) [@sgavela](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgavela)%5C
**Post date:** [March 16, 2022, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-cls-special-token-embedding-initialized%2F15768%2F1 "2022-03-16T11:07:06Z")

</div>

Hi! Does anyone know how CLS token is initialized in BERT? I mean, let’s say I would like to train a BERT model from scratch (which of course I’m not doing), how should I initialize CLS embedding? Just at random under some distribution such as uniform? How is this done in BERT?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBramVanroy)%5C
**Post date:** [March 16, 2022, 3:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fhow-is-cls-special-token-embedding-initialized%2F15768%2F2 "2022-03-16T15:46:26Z")

</div>

Just like other tokens, the CLS token is randomly initialized from a normal distribution. The only exception is the padding token, which is set to zero.

> <https://github.com/huggingface/transformers/blob/204c54d411c2b4c7f31405203533a51632f46ab1/src/transformers/models/bert/modeling_bert.py#L731-L734>