**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [November 17, 2022, 8:01am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F2 "2022-11-17T08:01:47Z")
Hi,
The [pipeline() API](https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/pipelines) is created mostly for people who don’t care too much about the details of the underlying process, for people who just want to use a machine learning model without having to implement several details like pre- and postprocessing themselves. The pipeline API is created such that you get an easy-to-use abstraction over any ML model, which is great for inference. The [SummarizationPipeline](https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/pipelines#transformers.SummarizationPipeline) for instance uses generate() behind the scenes.
On the other hand, if you do care about the details, then it’s recommended to generate text yourself by calling [generate()](https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate) yourself and implement pre-and postprocessing yourself.
Also note that any text generation pipeline does provide a [generate\_kwargs](https://github.com/huggingface/transformers/blob/94b3f544a1f5e04b78d87a2ae32a7ac252e22e31/src/transformers/pipelines/text2text_generation.py#L138) argument, which means that technically you can forward any of the keyword arguments that generate() supports to the pipeline as well.
---
**Author:**  [@Z3K3](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FZ3K3)%5C
**Post date:** [November 17, 2022, 5:40pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F3 "2022-11-17T17:40:09Z")
Thank you for this response [nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr). This was what I wanted to know.
---
**Author:**  [@Saptarshi7](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FSaptarshi7)%5C
**Post date:** [August 16, 2023, 9:45pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F4 "2023-08-16T21:45:20Z")
Hello,
So I tested both recently and found a very peculiar behavior under similar parameter values. This was using Galactica’s 1.3B variant
```auto
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline, set_seed
import torch
checkpoint = "facebook/galactica-1.3b"
tokenizer = AutoTokenizer.from_pretrained(checkpoint, padding_side="left")
model = AutoModelForCausalLM.from_pretrained(checkpoint)
model.to('cuda')
generator = pipeline('text-generation', model=model, tokenizer=tokenizer, device=0)
#With pipeline
set_seed(42)
generator(['Is this', 'What is the matter'], renormalize_logits=True, do_sample=True, use_cache=True, max_new_tokens=10)
#With model.generate()
device=torch.device('cuda',0)
model.to(device)
tokenizer = AutoTokenizer.from_pretrained(checkpoint, padding_side="left")
tokenizer.pad_token = tokenizer.eos_token = ''
tokenized_prompts = tokenizer(['Is this', 'What is the matter'], padding=True, return_tensors='pt')
set_seed(42)
model_op = model.generate(input_ids=tokenized_prompts['input_ids'].to(device),
attention_mask=tokenized_prompts['attention_mask'].to(device),
renormalize_logits=False, do_sample=True,
use_cache=True, max_new_tokens=10)
tokenizer.batch_decode(model_op, skip_special_tokens=True)
```
Here is the result with each,
```auto
[{'generated_text': 'Is this method for dealing with multiple objects?\n\n\n'}],
[{'generated_text': 'What is the matter density of a star whose radius is equal to '}]
................
['Is this method for dealing with multiple objects?\n\n\n',
'What is the matter of this, I know that it isn’t']
```
As we can see, both methods are producing different outputs, even under the same settings. However, the first generation for each method seems to be the same & I tried it for a bunch of other prompts. That being said if we turn off do\_sample i.e.
> do\_sample = False (greedy decoding)
then, we get the same results. Thus, I believe this is related to the sampling method being employed which is producing different results. Does anyone have any thoughts on this?
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [December 25, 2023, 8:59pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F5 "2023-12-25T20:59:13Z")
Hi,
Well, sampling is exactly causing randomness 😃 you can set a seed to get reproducabile results even when using sampling:
```auto
from transformers import set_seed
set_seed(42)
```
Refer to the [generate blog post](https://huggingface.co/blog/how-to-generate) for more details.
---
**Author:**  [@brando](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fbrando)%5C
**Post date:** [December 5, 2024, 7:26pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F6 "2024-12-05T19:26:49Z")
> [@nielsr](#):
>
> The [pipeline() API](https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/pipelines) is created mostly for people who don’t care too much about the details of the underlying process, for people who just want to use a machine learning model without having to implement several details like pre- and postprocessing themselves.
Do you mind sharing a concrete example of what you mean by pre and postprocessing in this context? @nielsr
Thank you in advance.
---
**Author:**  [@nielsr](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnielsr)%5C
**Post date:** [December 29, 2024, 11:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F7 "2024-12-29T11:07:37Z")
By pre-processing, I mean turning a sentence into tokens, then turning those tokens into numbers (indices in the vocabulary of a Transformer model). The tokenizer can be used for this purpose, which automatically turns text into so-called `input_ids`. The pipeline uses a tokenizer behind the scenes.
As for post-processing, one needs to decode the generate id’s back into text. The tokenizer can also be used for this, using the `decode` or `batch_decode` methods. The pipeline also makes use of these methods to present the result as text.
---
**Author:**  [@hongyeliu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhongyeliu)%5C
**Post date:** [January 20, 2025, 2:24am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F8 "2025-01-20T02:24:33Z")
> [@nielsr](#):
>
> By pre-processing, I mean turning a sentence into tokens, then turning those tokens into numbers (indices in the vocabulary of a Transformer model). The tokenizer can be used for this purpose, which automatically turns text into so-called `input_ids`. The pipeline uses a tokenizer behind the scenes.
>
> As for post-processing, one needs to decode the generate id’s back into text. The tokenizer can also be used for this, using the `decode` or `batch_decode` methods. The pipeline also makes use of these methods to present the result as text
Thank you for your response earlier. I have a question regarding the [generate\_kwargs](https://github.com/huggingface/transformers/blob/94b3f544a1f5e04b78d87a2ae32a7ac252e22e31/src/transformers/pipelines/text2text_generation.py#L138) argument needed to make .generate perform equivalently to .pipeline.
Currently, I am using the model from [Meta-Llama-3.1-8B-Instruct-bnb-4bit](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit). When I use .generate, the output begins by repeating the input prompt before generating the desired output. Since my prompt is quite lengthy, I can only see a truncated version of it in the output.
However, when I use .pipeline, it outputs the desired response directly without repeating the prompt. I suspect the difference might be due to .generate using greedy search for decoding, while .pipeline applies additional configurations like penalty terms to avoid regenerating the prompt.
I understand from your response that this might be the case, but I am unsure how to inspect the configuration used by .pipeline and apply similar settings to the model.generation\_config. Could you provide an example code snippet illustrating how to achieve this?
Thank you for your help!
---
**Author:**  [@hongyeliu](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fhongyeliu)%5C
**Post date:** [February 17, 2025, 3:11pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F9 "2025-02-17T15:11:48Z")
@nielsr sry, forgot to @
---
**Author:**  [@Bendang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBendang)%5C
**Post date:** [July 5, 2025, 1:50pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F10 "2025-07-05T13:50:23Z")
> [@hongyeliu](#):
>
> suspect the difference might be due to .generat
I am having the same problem. Have you figured out how to do this?
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [July 6, 2025, 3:55am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F11 "2025-07-06T03:55:29Z")
For now, I think the default value in Pipeline is prioritized by [`generation_config.json`](https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit/blob/main/generation_config.json), followed by [the default value in `GenerationConfig`](https://huggingface.co/docs/transformers/en/main_classes/text_generation). If you reproduce this, you should get almost the same result. Probably like this:
```py
outputs = model.generate(input_ids, do_sample=True, top_k=50, top_p=0.9, temperature=0.6, repetition_penalty=1.0, max_length=131072, bos_token_id=128000, pad_token_id=128004, eos_token_id=[128001, 128008, 128009])
```
---
**Author:**  [@Bendang](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FBendang)%5C
**Post date:** [July 16, 2025, 4:28pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fpipeline-vs-model-generate%2F26203%2F12 "2025-07-16T16:28:57Z")
I found a workaround to make `model.generate` produce the same output as the `pipeline`. I ran the pipeline in debug mode and set a breakpoint [here](https://github.com/huggingface/transformers/blob/e68ebb695f9d1d990462397e284e79d8729aafea/src/transformers/pipelines/text2text_generation.py#L220C1-L221C1). At that point, I pickled the `generate_kwargs` used internally by the pipeline and reused them directly in my own call to `model.generate`. This way, I was able to replicate the exact same output as the pipeline.
Hope this helps anyone facing a similar issue.