# Microsoft/codebert-base produces two sep tokens

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmicrosoft-codebert-base-produces-two-sep-tokens%2F21366%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [August 8, 2022, 6:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmicrosoft-codebert-base-produces-two-sep-tokens%2F21366 "2022-08-08T06:51:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gostrive](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gostrive/32/5591_2.png) [@gostrive](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgostrive)%5C
**Post date:** [August 8, 2022, 6:51am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmicrosoft-codebert-base-produces-two-sep-tokens%2F21366%2F1 "2022-08-08T06:51:43Z")

</div>

Hey Guys,

this post is basically a copy of [this one](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftwo-sep-tokens-added-by-microsoft-codebert-base%2F21269). The first one did not get any attention, but I think the problem might be important and this is probably a better place to ask the question. If this is the wrong place to post this, please let me know where to post.

I noticed, that the “microsoft/codebert-base” tokenizer adds two SEP Tokens between two sentences. I don’t think this is intended behavior, but feel free to correct me.

This is the example I originally posted:

```auto
from transformers import AutoTokenizer, AutoModel
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

checkpoint = "microsoft/codebert-base"

model = AutoModel.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

from datasets import load_dataset

raw_dataset = load_dataset('json', data_files='/home/<user>/Data/<DataDir>/dataset_v1.jsonl', split='train')

def toke(example):
    return tokenizer(example["sentence1"], example["sentence2"]) 

tokenized_dataset = raw_dataset.select(list(range(10000))).map(toke, batched=True)

print(tokenized_dataset[7]['sentence1'])
print(tokenized_dataset[7]['sentence2'])
print(tokenized_dataset[7]['input_ids'])

```

Output:

```auto
train_nan_df.head()
test_df[‘ImageId’] = np.array(os.listdir (‘…/input/test_images/’))
[0, 21714, 1215, 10197, 1215, 36807, 4, 3628, 43048, 2, 2, 21959, 1215, 36807, 48759, 8532, 28081, 44403, 5457, 46446, 4, 30766, 1640, 366, 4, 8458, 41292, 31509, 49445, 46797, 73, 21959, 1215, 39472, 73, 108, 35122, 2]

```

Here is another example using publicly available data:  
In:

```auto
!pip install transformers datasets

from transformers import AutoTokenizer, AutoModel
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

checkpoint = "microsoft/codebert-base"

model = AutoModel.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

tokenizer.sep_token_id

```

Out:

```auto
2

```

In:

```auto
tokenizer("this is the first sentence", "this is the second sentence")

```

Out:

```auto
{'input_ids': [0, 9226, 16, 5, 78, 3645, 2, 2, 9226, 16, 5, 200, 3645, 2], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}

```

Also calling the respective tokenizer.tokenize and tokenizer.convert\_tokens\_to\_ids functions does not produce two SEP tokens.

---

<div class="post-metadata">

**Author:** ![vildhet](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vildhet/32/5895_2.png) [@vildhet](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvildhet)%5C
**Post date:** [September 4, 2022, 9:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmicrosoft-codebert-base-produces-two-sep-tokens%2F21366%2F2 "2022-09-04T09:53:35Z")

</div>

Looks like it’s the expected behavior for some of the tokenizers. At least according to [this github issue](https://github.com/huggingface/tokenizers/issues/983).

---

<div class="post-metadata">

**Author:** ![gostrive](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gostrive/32/5591_2.png) [@gostrive](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgostrive)%5C
**Post date:** [September 5, 2022, 6:47am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmicrosoft-codebert-base-produces-two-sep-tokens%2F21366%2F3 "2022-09-05T06:47:23Z")

</div>

Thanks for your reply! I did not expect to still get an answer but the post you shard answered my question! Merci 😃