# Trainer is not using multiple GPUs in the DP setup **URL:** **Author:** ![vladyorsh](https://avatars.discourse-cdn.com/v4/letter/v/ee7513/32.png) [@vladyorsh](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvladyorsh)%5C **Post date:** [April 9, 2023, 3:51pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftrainer-is-not-using-multiple-gpus-in-the-dp-setup%2F36051%2F1 "2023-04-09T15:51:04Z") I’m trying to launch a custom model training through the Trainer API in the single-node-multi-GPU setup. I use the subclasssed Trainer, which modifies the `evaluation_loop()` function. My training script sees all the available GPUs through `torch.cuda` commands; however, I observe no speedup when launching the script as the ordinary python command. As I see from the code, the Trainer wraps the model into `nn.DataParallel`, so it should work, but it doesn’t. I do see the speedup in the DPP scenario, when launching through the `torchrun`; however, the RAM usage scales up as well, and I cannot use all the GPUs because I need to load the data directly into the memory – fetching from files is too slow. Switching to other libraries such as `accelerate` is also undesirable due to the effort of re-coding the needed Trainer API features. Is there anything that can help me to make the DP+Trainer setup into work? UPD: I suggest that with an increasing amount of GPUs the Trainer doesn’t split accumulation steps between individual GPUs, so the actual batch size is `acc_steps * n_gpus * per_device_batch_size`. However, I also suggest that the DPP setup must do this as well, which doesn’t explain the speedup.