# Running a Trainer in DistributedDataParallel mode

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-a-trainer-in-distributeddataparallel-mode%2F1718%3E%5C
**Category:** 🤗Transformers\
**Created:** [October 23, 2020, 7:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-a-trainer-in-distributeddataparallel-mode%2F1718 "2020-10-23T19:16:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![deppen8](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/deppen8/32/516_2.png) [@deppen8](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdeppen8)%5C
**Post date:** [October 23, 2020, 7:16pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-a-trainer-in-distributeddataparallel-mode%2F1718%2F1 "2020-10-23T19:16:25Z")

</div>

I am trying to train a model on four GPUs (AWS ml.p3.8xlarge). As far as I can tell, to get my model to train in `DistributedDataParallel`, I only need to specify some integer value for `local_rank`. But my understanding is that this will only distribute the training across a single GPU (whichever I specify with `local_rank`).

What is the proper way to launch `DistributedDataParallel` training across all four GPUs using a `Trainer`? Do I have to launch something via the command line (as hinted at here [https://github.com/huggingface/transformers/issues/1651](https://github.com/huggingface/transformers/issues/1651))?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [October 24, 2020, 8:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Frunning-a-trainer-in-distributeddataparallel-mode%2F1718%2F2 "2020-10-24T08:24:39Z")

</div>

Hi @deppen8  
Yes, you’ll need to use `torch.distributed.launch` for distributed training.  
See [this command](https://github.com/huggingface/transformers/tree/master/examples/text-classification#distributed-training) for an example.