# Seq2Seq Distillation: Methodology Questions
**URL:**
**Author:**  [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [September 27, 2020, 6:17pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F1 "2020-09-27T18:17:00Z")
This thread should be used to ask questions about how `examples/seq2seq/distillation.py` works, and to ask questions about the associated paper after it gets released.
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [September 28, 2020, 9:37am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F2 "2020-09-28T09:37:08Z")
What is the reasoning behind choosing alternating layers ?
no teacher distillation scores for XSUM ?
no teacher is working for non seq-2-seq task as well as we saw with MNLI, should we also see if it works other tasks as well ?
---
**Author:**  [@sshleifer](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsshleifer)%5C
**Post date:** [October 5, 2020, 1:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F3 "2020-10-05T13:24:05Z")
Alternating layers seems to perform the best by a moderate amount.
Definitely interested to see results for other tasks!
---
**Author:**  [@surajp](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsurajp)%5C
**Post date:** [December 17, 2020, 5:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F4 "2020-12-17T05:26:42Z")
relocated to `examples/research_projects/seq2seq-distillation/distillation.py` ?
---
**Author:**  [@valhalla](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fvalhalla)%5C
**Post date:** [December 17, 2020, 6:44am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F5 "2020-12-17T06:44:19Z")
Yes, that project is now moved to `research_projects` dir.
---
**Author:**  [@nbansal](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fnbansal)%5C
**Post date:** [June 25, 2021, 6:59am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F6 "2021-06-25T06:59:05Z")
Hey @sshleifer, I was trying to fine-tune the `distill-pegasus-cnn-16-4` [model](https://huggingface.co/sshleifer/distill-pegasus-cnn-16-4) provided by you but I am not sure of the hyper-parameters. Could you please share the hyper-parameters that you used to train this model (and achieve the results shown in Table 5 from your [paper](https://arxiv.org/pdf/2010.13002.pdf)?
Thanks a lot!
Naman
---
**Author:**  [@Aktsvigun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAktsvigun)%5C
**Post date:** [June 3, 2022, 4:24pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F7 "2022-06-03T16:24:59Z")
Hi! have a question regarding the article «Pre-trained Summarization Distillation» ([https://arxiv.org/pdf/2010.13002.pdf](https://arxiv.org/pdf/2010.13002.pdf)). In section 6.2, it is said «Table 10 shows results from fine-tuning **teacher** models…». However, throughout the paper it is stated that the experiments with pseudo-labeling only when fine-tuning the **student** model were performed. Is it a typo and the result from fine-tuning **student** models is indeed depicted?
Thanks in advance!
---
**Author:**  [@ppaudel](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fppaudel)%5C
**Post date:** [August 7, 2023, 4:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fseq2seq-distillation-methodology-questions%2F1270%2F8 "2023-08-07T16:12:56Z")
Hi @sshleifer . Any thoughts on if the T5 distillation would still be feasible with PEFT techniques such as LORA? I have a fine tuned T5-11B using LORA and want to distill this model to something feasible like T5-base or even T5-large. But I’m not sure if the teacher model , which essentially has a LoRA adapter work on a similar way ? Any thoughts / ideas regarding this would be great help. Thanks