# Speed expectations for production BERT models on CPU vs GPU?

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeed-expectations-for-production-bert-models-on-cpu-vs-gpu%2F452%3E%5C
**Category:** Beginners\
**Created:** [July 23, 2020, 7:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeed-expectations-for-production-bert-models-on-cpu-vs-gpu%2F452 "2020-07-23T19:10:15Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![wpietri](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wpietri/32/200_2.png) [@wpietri](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fwpietri)%5C
**Post date:** [July 23, 2020, 7:10pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeed-expectations-for-production-bert-models-on-cpu-vs-gpu%2F452%2F1 "2020-07-23T19:10:15Z")

</div>

Hi! I’m working with an ML model produced by a researcher. I’m trying to set it up to run economically in production on large volumes of text. I know a lot about production engineering, but next to nothing about ML. I’m getting some results that are surprising to me, and I’m hoping for pointers to explanations and advice.

Right now I have the key code broken out into 3 methods, so it’s easy to profile.

```
def _tokenize(self, text):
    return self.tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt').to(self.device)

def _run_model(self, model_input):
    return self.model(model_input['input_ids'], token_type_ids=model_input['token_type_ids'])[0]

def _extract_results(self, logits):
    return logits[0][0].item(), logits[0][1].item()

```

If I run this using my laptop CPU, I get numbers that make sense to me. For 1459 items, those three methods take 196.4 seconds, or about 135 ms per item. About 2.9 seconds is \_tokenize, and the rest is \_run\_model.

When I switch over to my laptop GPU, I get numbers that mystify me. The same data takes 131.8 seconds. 2.5 seconds to tokenize, and running the model takes 20.3 seconds. But extracting the result takes 108.8 seconds!

The \_extract\_results method costs the same whether I extract one logit or both. The first one that I extract is slow, whether that’s [0] or [1]. The second one is effectively free.

From nvidia-smi, I can see that the GPU is really being used, and my process is using ~850MB of the 2 GB of GPU RAM. So that seems fine. And if it matters, this is a GeForce 940MX on a Thinkpad t470.

Do these numbers make sense to more experienced hands? I was expecting the GPU runs to be much faster, but if I actually want to get the results, it’s only a little faster.

Thanks!

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Frgwatwormhill)%5C
**Post date:** [October 2, 2020, 11:53am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fspeed-expectations-for-production-bert-models-on-cpu-vs-gpu%2F452%2F2 "2020-10-02T11:53:59Z")

</div>

(probably too late to be useful…)

Is the GPU being used for all of the sections? (Maybe your researcher only put the middle section on GPU.)

Does the run-model step actually train and update the model, or does it just calculate answers for the data you entered?