# Gpt2 token of specific string

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-token-of-specific-string%2F35128%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 30, 2023, 6:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-token-of-specific-string%2F35128 "2023-03-30T18:26:47Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Towen](https://avatars.discourse-cdn.com/v4/letter/t/bcef8e/32.png) [@Towen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FTowen)%5C
**Post date:** [March 30, 2023, 6:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgpt2-token-of-specific-string%2F35128%2F1 "2023-03-30T18:26:47Z")

</div>

I have following texts to tokenize:

tokenizer = GPT2Tokenizer.from\_pretrained(‘gpt2’, bos\_token=‘\<|startoftext|\>’, eos\_token=‘\<|endoftext|\>’, pad\_token=‘\<|paf|\>’)

x =

text1 = tokenizer(‘\<|startoftext|\>’ + ‘Question. A’ + ‘\<|endoftext|\>’, truncation=True, max\_length=20, padding=“max\_length”)  
x.append(text1[‘input\_ids’])

text2 = tokenizer(‘\<|startoftext|\>’ + ‘Question. B’ + ‘\<|endoftext|\>’, truncation=True, max\_length=20, padding=“max\_length”)  
x.append(text2[‘input\_ids’])

text3 = tokenizer(‘\<|startoftext|\>’ + ‘Question. C’ + ‘\<|endoftext|\>’, truncation=True, max\_length=20, padding=“max\_length”)  
x.append(text3[‘input\_ids’])

print(x)

and this is what I got:  
[[50257, 24361, 13, 317, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258], [50257, 24361, 13, 347, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258], [50257, 24361, 13, 327, 50256, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258, 50258]]

text1, text2, text3 always have the same length. Is there anyway to output the token of ‘A’, ‘B’, ‘C’? Is it right before 50256 which is eos\_token i guess?