# Encoder decoder model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoder-decoder-model%2F28328%3E%5C
**Category:** 🤗Transformers\
**Created:** [December 23, 2022, 10:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoder-decoder-model%2F28328 "2022-12-23T10:27:39Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JoAmps](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joamps/32/13066_2.png) [@JoAmps](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJoAmps)%5C
**Post date:** [December 23, 2022, 10:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoder-decoder-model%2F28328%2F1 "2022-12-23T10:27:39Z")

</div>

I want to use an encoder-decoder model, but I want to perform masked language modeling and casual language modeling (unsupervised learning) on the encoder-decoder so it understands the context I am using, before fine-tuning it. How best to go about this?