# Run\_mlm.py: generate train split is very slow

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-generate-train-split-is-very-slow%2F23621%3E%5C
**Category:** Beginners\
**Created:** [September 26, 2022, 9:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-generate-train-split-is-very-slow%2F23621 "2022-09-26T09:54:27Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![timpal0l](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/timpal0l/32/16162_2.png) [@timpal0l](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ftimpal0l)%5C
**Post date:** [September 26, 2022, 9:54am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-generate-train-split-is-very-slow%2F23621%2F1 "2022-09-26T09:54:27Z")

</div>

I am running [run\_mlm.py](https://huggingface.co/datasets/openwebtext) on [openwebtext](https://huggingface.co/datasets/openwebtext), which is about 10gb. I am using 4 nodes on a Slurm cluster with 4 A100 gpus, so there is a lot of compute power, however to generate the train split it takes about 100 hours for only 10gb. How can this be speed up?

```auto
DISTRIBUTED_ARGS="--nproc_per_node $NPROC_PER_NODE \
                  --nnodes $SLURM_JOB_NUM_NODES \
                  --node_rank $SLURM_NODEID \
                  --master_addr $MASTER_ADDR \
                  --master_port $MASTER_PORT"

cmd1="torchrun $DISTRIBUTED_ARGS \
    transformers/examples/pytorch/language-modeling/run_mlm.py \
    --model_name_or_path "microsoft/deberta-v3-base" \
    --dataset_name "openwebtext" \
    --num_train_epochs 3.0 \
    --dataloader_num_workers 2 \
    --fp16 \
    --per_device_train_batch_size 10 \
    --per_device_eval_batch_size 10 \
    --do_train \
    --do_eval \
    --evaluation_strategy "steps" \
    --eval_steps 2000 \
    --report_to "wandb" \
    --overwrite_output_dir"

$cmd1

```

```auto
Generating train split: 1%|▏ | 101641/8013769 [1:08:17<99:51:59, 22.01 examples/s]
Generating train split: 1%|▏ | 101674/8013769 [1:08:19<101:24:22, 21.05 examples/s]
Generating train split: 1%|▏ | 101703/8013769 [1:08:20<104:04:45, 21.12 examples/s]

```

Latest pytorch and transformers with cuda 11.6. But I guess this is cpu / ram heavy. Is there any other way of generating the train splits? e.g before running the mlm script?

Thanks!

---

<div class="post-metadata">

**Author:** ![cog](https://avatars.discourse-cdn.com/v4/letter/c/df788c/32.png) [@cog](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fcog)%5C
**Post date:** [September 27, 2022, 1:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frun-mlm-py-generate-train-split-is-very-slow%2F23621%2F2 "2022-09-27T01:58:37Z")

</div>

hi @timpal0l ,

There dataloader worker only set ‘2’ is seem to be why pipeline is slow.

I think you can increase ‘dataloader worker number’ to train split speed up.

regards.