# Add BOS and EOS when encoding a sentence
**URL:**
**Author:**  [@alexgrishin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falexgrishin)%5C
**Post date:** [August 19, 2022, 1:39pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-bos-and-eos-when-encoding-a-sentence%2F21833%2F1 "2022-08-19T13:39:22Z")
Iām training a tokenizer from scratch. Iām using BPE tokenizer with ByteLevel pre-tokenizer.
How do I add [BOS] at the beginning of each sentence and [EOS] at the end? I can do it manually, of course, but is there a way to tell encoder do it automatically? It seems that this tokenizer with this pre-tokenizer do actually add the same token at the end of each sentence (token āÄā with token\_id=163). I would prefer to have control over how the end of sentence token looks like and its id. How do I do that?
---
**Author:**  [@lianghsun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flianghsun)%5C
**Post date:** [August 22, 2022, 3:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-bos-and-eos-when-encoding-a-sentence%2F21833%2F2 "2022-08-22T03:07:02Z")
Hi @alexgrishin, is the following code what you desire to be?
```auto
# I use tutorial code from https://huggingface.co/docs/tokenizers/quicktour as example
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]", "[BOS]", "[EOS]"]) # Adding [BOS] and [EOS] here
tokenizer.pre_tokenizer = Whitespace()
# Please using TemplateProcessing
# https://huggingface.co/docs/tokenizers/api/post-processors#tokenizers.processors.TemplateProcessing
from tokenizers.processors import TemplateProcessing
tokenizer.post_processor = TemplateProcessing(
single="[BOS] $A [EOS]",
special_tokens=[("[BOS]", 1), ("[EOS]", 2)],
)
##################################################
files = [f"datasets/wikitext-103-raw/wiki.{split}.raw" for split in ["test", "train", "valid"]]
tokenizer.train(files, trainer)
output = tokenizer.encode("Hello, y'all! How are you š ?")
print(output.ids)
# >> [1, 27255, 18, 95, 13, 5099, 7, 7963, 5114, 6220, 0, 37, 2] <-- you can see there are token [1] in the begining and token [2] at the end of the sequence
print(tokenizer.decode(output.ids))
# >> no [BOS] and [EOS] after decoding
```
---
**Author:**  [@alexgrishin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Falexgrishin)%5C
**Post date:** [August 22, 2022, 5:18pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fadd-bos-and-eos-when-encoding-a-sentence%2F21833%2F3 "2022-08-22T17:18:56Z")
Thank you! Thatās exactly what I was after!