# Why is Trainer single-threaded during "Generating split..."?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-trainer-single-threaded-during-generating-split%2F82309%3E%5C
**Category:** 🤗Transformers\
**Created:** [April 17, 2024, 9:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-trainer-single-threaded-during-generating-split%2F82309 "2024-04-17T21:55:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jlmckins](https://avatars.discourse-cdn.com/v4/letter/j/f14d63/32.png) [@jlmckins](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fjlmckins)%5C
**Post date:** [April 17, 2024, 9:55pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fwhy-is-trainer-single-threaded-during-generating-split%2F82309%2F1 "2024-04-17T21:55:39Z")

</div>

I am training a 3B LLM from scratch with 1 million text samples, and it takes 1/2 hour to get through the “Generating split train…”. I get ~500 samples per second maximum in bursts. I’m wanting to scale to 200 Million samples, which says it will take 7 days. If I tokenize and pack the data myself using map(), with num\_proc of 64, and large batches, this can be done in less than 8 hours instead of days.

I see that you can pass in a sharded file if you have a streaming dataset, but using streaming gives other errors. How can I trick it to use multiple workers when Trainer is “Generating split” inside of Trainer or SFTTrainer?