# Finetuned model takes double inference time

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuned-model-takes-double-inference-time%2F32857%3E%5C
**Category:** 🤗Transformers\
**Created:** [March 2, 2023, 2:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuned-model-takes-double-inference-time%2F32857 "2023-03-02T02:41:38Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saltychtao](https://avatars.discourse-cdn.com/v4/letter/s/9de0a6/32.png) [@Saltychtao](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSaltychtao)%5C
**Post date:** [March 2, 2023, 2:41am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ffinetuned-model-takes-double-inference-time%2F32857%2F1 "2023-03-02T02:41:38Z")

</div>

I am finetuning a XGLM-7.5B as a translation model using transformers 4.26.0 + deepspeed 0.8.0. The generation time for 1000 sentences of the original XGLM model downloaded from huggingface is about 8 minutes. However, when the finetuned model takes about 15 minutes to finish the generation. Could anyone tell me how to fix this problem?

Code snippets that may be related:  
loading model(either pretrained or finetuned)

```auto
tokenizer.padding_side = 'left'
model = AutoModelForCausalLM.from_pretrained(args.model_name_or_path).cuda().half()

```

generation

```auto
with torch.no_grad():
            generated_ids = model.generate(
                **encoding,
                max_new_tokens=100,
                num_beams=4,
                early_stopping=True,
                )

```