# OPT special tokens

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fopt-special-tokens%2F78667%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [March 25, 2024, 8:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fopt-special-tokens%2F78667 "2024-03-25T08:56:06Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![aljazpotocnik](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aljazpotocnik/32/18956_2.png) [@aljazpotocnik](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Faljazpotocnik)%5C
**Post date:** [March 25, 2024, 8:56am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fopt-special-tokens%2F78667%2F1 "2024-03-25T08:56:06Z")

</div>

Hello,

I can’t understand something about OPT tokenizer and its special tokens. I came across `<unk>` token in opt vocabulary. But when encoding it with opt\_tokenizer it doesn’t find it as a single token but three tokens: \<, unk, \>. How does this make sense?

Below is some code to reproduce my findings:

```auto
opt_tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")

print('<unk>' in opt_tokenizer.get_vocab())

ids = opt_tokenizer.encode("<unk>", add_special_tokens=False)

print(opt_tokenizer.convert_ids_to_tokens(ids))

```

Another funny thing is that `<unk>` token isn’t present in opt\_tokenizer.special\_tokens\_map:

```auto
print(opt_tokenizer.special_tokens_map)

```

{‘bos\_token’: ‘`</s>`’,  
‘eos\_token’: ‘`</s>`’,  
‘unk\_token’: ‘`</s>`’,  
‘pad\_token’: ‘`<pad>`’}

Best,  
AP