# Two SEP Tokens added by microsoft/codebert-base

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftwo-sep-tokens-added-by-microsoft-codebert-base%2F21269%3E%5C
**Category:** Beginners\
**Created:** [August 5, 2022, 7:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftwo-sep-tokens-added-by-microsoft-codebert-base%2F21269 "2022-08-05T07:40:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gostrive](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/gostrive/32/5591_2.png) [@gostrive](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fgostrive)%5C
**Post date:** [August 5, 2022, 7:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftwo-sep-tokens-added-by-microsoft-codebert-base%2F21269%2F1 "2022-08-05T07:40:18Z")

</div>

Hey Guys,  
I have a quick question about the microsoft/codebert-base tokenizer. Why are two `</s>` tokens added to separate two strings?

My used Code:

```auto
from transformers import AutoTokenizer, AutoModel
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

checkpoint = "microsoft/codebert-base"

model = AutoModel.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

from datasets import load_dataset

raw_dataset = load_dataset('json', data_files='/home/<user>/Data/<DataDir>/dataset_v1.jsonl', split='train')

def toke(example):
    return tokenizer(example["sentence1"], example["sentence2"]) 

tokenized_dataset = raw_dataset.select(list(range(10000))).map(toke, batched=True)

print(tokenized_dataset[7]['sentence1'])
print(tokenized_dataset[7]['sentence2'])
print(tokenized_dataset[7]['input_ids'])

```

> train\_nan\_df.head()  
> test\_df[‘ImageId’] = np.array(os.listdir (‘…/input/test\_images/’))  
> [0, 21714, 1215, 10197, 1215, 36807, 4, 3628, 43048, **2, 2,** 21959, 1215, 36807, 48759, 8532, 28081, 44403, 5457, 46446, 4, 30766, 1640, 366, 4, 8458, 41292, 31509, 49445, 46797, 73, 21959, 1215, 39472, 73, 108, 35122, 2]

Thanks in advance!