# Distribute training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistribute-training%2F26160%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 16, 2022, 9:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistribute-training%2F26160 "2022-11-16T09:40:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![liumaishen](https://avatars.discourse-cdn.com/v4/letter/l/6a8cbe/32.png) [@liumaishen](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fliumaishen)%5C
**Post date:** [November 16, 2022, 9:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdistribute-training%2F26160%2F1 "2022-11-16T09:40:19Z")

</div>

why the training args n\_gpu is set to 1 when use Trainer’s distributed training？  
which means only one device is allowed in a node？  
and the training parameters printed is calculated with the n\_gpus=1.  
I want to know what should i do when every node has multi gpus.

The script is [https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run\_mlm.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py)  
Here is the training parameters printed

 ![屏幕截图 2022-11-16 173247](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/1ea9a0cfad586ca3ed1e2262af656bba6fb7a9b6.png)  
Actually， I have two nodes both have 8 gpus，so the total batch size is 32 \* 8 \* 2 = 512 instead of 32 \* 2 = 64，which is calculated with the num\_gpus=1 in a node.