# BART tokenizer adds two EOS (\</s\>) tokens?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-adds-two-eos-s-tokens%2F16090%3E%5C
**Category:** Beginners\
**Created:** [March 25, 2022, 6:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-adds-two-eos-s-tokens%2F16090 "2022-03-25T18:09:49Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![anon67423238](https://avatars.discourse-cdn.com/v4/letter/a/5fc32e/32.png) [@anon67423238](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanon67423238)%5C
**Post date:** [March 25, 2022, 6:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbart-tokenizer-adds-two-eos-s-tokens%2F16090%2F1 "2022-03-25T18:09:49Z")

</div>

I have the problem that the BART tokenizer adds two tokens after the first segment.

```auto
tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
tokenizer.encode("What do you think?", "Nothing.", return_tensors="pt")

```

It outputs:

> ```auto
> tensor([[ 0, 2264, 109, 47, 206, 116, 2, 2, 19847, 4,
> 2]])
> 
> ```

Let’s decode again:

```auto
tokenizer.batch_decode(x)

```

> ```auto
> ['<s>What do you think?</s></s>Nothing.</s>']
> 
> ```

Can I do anything about it?