# Decoding latents to RGB without upscaling
**URL:**
**Author:**  [@keturn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fketurn)%5C
**Post date:** [September 16, 2022, 4:07am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F1 "2022-09-16T04:07:37Z")
For the purpose of visualizing work in progress on the image inference pipelines, I’d like to convert the latents to a display format at each step along the way. But I find that if I run them through the pipeline’s VAE decoder, that takes far too much time that could be better spent on the pipeline’s main task.
Because this is only for preview purposes, I don’t need to use an expensive method to scale them back up to 512×512. It would be sufficient to leave them at 64×64, letting the application do some naive upscaling if desired.
Is there a way to decode the 4-channel latent space to a 3-channel image format without upscaling? Will that be dramatically faster than the full decode+upscale method?
I tried fumbling around a bit creating an instance of AutoencoderKL configured similarly to that of the [pretrained model](https://huggingface.co/CompVis/stable-diffusion-v1-4/blob/main/vae/config.json), but without so many `UpDecoderBlock`s. Unfortunately that attempt didn’t lead to any useful results, and I’m not sure if that was even on the right track.
---
**Author:**  [@keturn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fketurn)%5C
**Post date:** [September 17, 2022, 1:33am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F2 "2022-09-17T01:33:03Z")
After some empirical tests, I have determined that I can get a useful approximation of the RGB output using a linear combination of the latent channels.
This approximation comes from multiplying the four latent channels by these factors:
```python
v1_4_rgb_latent_factors = [
# R G B
[0.298, 0.207, 0.208], # L1
[0.187, 0.286, 0.173], # L2
[-0.158, 0.189, 0.264], # L3
[-0.184, -0.271, -0.473], # L4
]
```
[This is for Stable Diffusion v1.4. I assume it’s not universally true.]
Here’s the output from the actual VAE decoder for comparison:
The approximation is a little undersaturated, maybe it could stand to have a bit of tuning, but it’s not bad considering it’s over two thousand times faster.
So that’s useful to know. But I certainly feel like I did this the hard way, by analyzing some outputs and developing a new approximation of them. Hopefully there’s an easier way to determine those values or some other similarly cheap approximation?
---
**Author:**  [@anton-l](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fanton-l)%5C
**Post date:** [September 21, 2022, 8:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F3 "2022-09-21T20:22:11Z")
Wow, that’s a very neat way to approximate the decoded image, never thought that the latents would end up so interpretable!
Will keep that in mind for future pipelines
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [September 22, 2022, 11:16am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F4 "2022-09-22T11:16:19Z")
That’s pretty amazing, do those numbers work for all types of images? I wonder why your initial experiment to use a smaller decoder wouldn’t work, it sounds like a reasonable idea to me!
---
**Author:**  [@keturn](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fketurn)%5C
**Post date:** [September 22, 2022, 4:46pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F5 "2022-09-22T16:46:20Z")
> [@pcuenq](#):
>
> do those numbers work for all types of images?
I fit them against some grayscale images, some bright saturated colors like the hot air balloon example above, and some mid-tones. The result seems _close enough_ for the images I’ve seen come out of it. Sometimes the color accuracy is worse than others, but it’s sufficient if you’re just trying to get a rough idea of the composition.
I’ve posted some demo code for both gradio and ipywidgets: [GitHub - keturn/sd-progress-demo: cheap views of intermediate Stable Diffusion results](https://github.com/keturn/sd-progress-demo/)
> [@pcuenq](#):
>
> I wonder why your initial experiment to use a smaller decoder wouldn’t work, it sounds like a reasonable idea to me!
I’m sure it would have gone better if I had _any_ idea what I was doing. 😆 But this is all my first project involving pytorch and neural networks. I have some more catching up to do on the fundamentals before I can even read how the decoder is put together, let alone understand how to modify it.
---
**Author:**  [@erucipe](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Ferucipe)%5C
**Post date:** [September 25, 2022, 2:58am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F7 "2022-09-25T02:58:57Z")
This works!
---
**Author:**  [@damian0815](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdamian0815)%5C
**Post date:** [October 4, 2022, 10:13am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F8 "2022-10-04T10:13:39Z")
Got this working on [my img2img doc branch of the InvokeUI fork](https://github.com/damian0815/InvokeAI/tree/document-img2img). you can pass `--write_intermediates` to the `dream>` prompt and it will write every latent step to a png file.
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [October 4, 2022, 11:45am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F9 "2022-10-04T11:45:03Z")
That’s very cool! Are you doing it without upscaling? Did you train a decoder, or did you do it differently? Can you please point me to the place in the code where that happens? 🙂 Thanks!
---
**Author:**  [@damian0815](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdamian0815)%5C
**Post date:** [October 4, 2022, 12:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F10 "2022-10-04T12:34:34Z")
No upscaling, it’s a tiny image (64x64 pixels at image size=512x512). [InvokeAI/dream.py at fe401e88a0a143f01d175b6e56e3a1bf5d60ee2d · damian0815/InvokeAI · GitHub](https://github.com/damian0815/InvokeAI/blob/fe401e88a0a143f01d175b6e56e3a1bf5d60ee2d/scripts/dream.py#L284). i’m just using keturn’s/erucipe’s method.
---
**Author:**  [@pcuenq](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fpcuenq)%5C
**Post date:** [October 4, 2022, 1:32pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F11 "2022-10-04T13:32:19Z")
Ah, I misunderstood, thought it was a different method 🙂
---
**Author:**  [@PaulTheHuman](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FPaulTheHuman)%5C
**Post date:** [January 14, 2023, 1:57pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F12 "2023-01-14T13:57:50Z")
Hi, this is very good! I solute you! o7
I notice you are using a linear approximation with no constant term x–\>Ax (where A is the matrix)
Other things people could try are:
(1) Linear approximation with constant term x-\>Ax+B
(2) A quadratic approximation x-\>Ax^2+Bx+C (this would need a total of 63 values to find!). A would be a 3x4x4 tensor.
(3) A cubic approximation x-\>Ax^3+Bx^2+Cx+D. (255 values)
The quadratic approximation should only be about 4 times slower but that should be fine since the process can be done in milliseconds.
The values should be able to be found using a simple neural network with random images as training data. If anyone wants to give it a go. Let us know the results 😃
---
**Author:**  [@Astropulse](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FAstropulse)%5C
**Post date:** [January 26, 2023, 6:34pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F13 "2023-01-26T18:34:53Z")
I would be very interested in a way to directly decode the latent with less noise and more accurate colors, if anyone has an idea on how to do that I would love to try it out.
---
**Author:**  [@madebyollin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmadebyollin)%5C
**Post date:** [April 23, 2023, 9:41pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fdecoding-latents-to-rgb-without-upscaling%2F23204%2F14 "2023-04-23T21:41:58Z")
Chiming in with some other options for latent → RGB previewing:
- BirchLabs posted a few more linear-layer-only decoders [here](https://birchlabs.co.uk/machine-learning#vae-distillation) (including a linear + bias version and a 3 layer MLP). These all produce RGB images that are the same size as the latents.
- I posted a tiny conv-only latent decoder [here](https://github.com/madebyollin/taesd), which produces full 512x512 preview images (it’s slower than the linear-layer-only options, but should still be way faster than the official decoder).