# Generating NER data sample for RoBERTa model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-ner-data-sample-for-roberta-model%2F972%3E%5C
**Category:** 🤗Transformers\
**Created:** [September 3, 2020, 10:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-ner-data-sample-for-roberta-model%2F972 "2020-09-03T10:06:32Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![proxyht](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@proxyht](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fproxyht)%5C
**Post date:** [September 3, 2020, 10:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-ner-data-sample-for-roberta-model%2F972%2F1 "2020-09-03T10:06:32Z")

</div>

I have a trained RoBERTa model with a Byte Level BPE Encoding algorithm, which I want to benchmark on a custom NER dataset.  
Each sample is as followed:

Text: John is playing football  
Labels: B-PER O O O

The text could be run through the tokenizer to generate subword tokens. However, the total count of the subwords tokens may be different from the text tokens, and I don’t know how to align the labels accordingly

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 6, 2020, 8:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-ner-data-sample-for-roberta-model%2F972%2F2 "2020-09-06T08:16:06Z")

</div>

cc @stefan-it

---

<div class="post-metadata">

**Author:** ![proxyht](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@proxyht](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fproxyht)%5C
**Post date:** [September 14, 2020, 1:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-ner-data-sample-for-roberta-model%2F972%2F3 "2020-09-14T01:53:56Z")

</div>

Hello, it’s this topiv still relevant?

Vào 15:26, CN, 6 Th9, 2020 Suraj Patil via Hugging Face Forums \<[hellohellohello@discoursemail.com](mailto:hellohellohello@discoursemail.com)\> đã viết:

---

<div class="post-metadata">

**Author:** ![nbroad](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nbroad/32/26465_2.png) [@nbroad](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbroad)%5C
**Post date:** [September 14, 2020, 2:11am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fgenerating-ner-data-sample-for-roberta-model%2F972%2F4 "2020-09-14T02:11:16Z")

</div>

Maybe something in here can help you:

[https://github.com/huggingface/transformers/tree/master/examples/token-classification](https://github.com/huggingface/transformers/tree/master/examples/token-classification)