# Automatic sentence segmentation and encoding

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautomatic-sentence-segmentation-and-encoding%2F1479%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [October 12, 2020, 9:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautomatic-sentence-segmentation-and-encoding%2F1479 "2020-10-12T09:43:56Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vblagoje](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vblagoje/32/20818_2.png) [@vblagoje](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvblagoje)%5C
**Post date:** [October 12, 2020, 9:43am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fautomatic-sentence-segmentation-and-encoding%2F1479%2F1 "2020-10-12T09:43:56Z")

</div>

Hello everyone,

Is there a way to segment sentences and encode them using the tokenizers library automatically? I am currently using spacy and/or nltk for the sentence segmentation part and then tokenizers to encode the sentences.

Many thanks.