| --- |
| base_model: black-forest-labs/FLUX.1-dev |
| datasets: TIGER-Lab/OmniEdit-Filtered-1.2M |
| library_name: diffusers |
| license: other |
| inference: true |
| tags: |
| - flux |
| - flux-diffusers |
| - text-to-image |
| - diffusers |
| - control |
| - diffusers-training |
| widget: |
| - text: Give this the look of a traditional Japanese woodblock print. |
| output: |
| url: >- |
| https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_car.jpg |
| - text: transform the setting to a winter scene |
| output: |
| url: >- |
| https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_green_creature.jpg |
| - text: turn the color of mushroom to gray |
| output: |
| url: >- |
| https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_mushroom.jpg |
| - text: Change it to look like it's in the style of an impasto painting. |
| output: |
| url: >- |
| https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_norte_dam.jpg |
| --- |
| |
| # Flux Edit |
|
|
| <Gallery /> |
|
|
| These are the control weights trained on [black-forest-labs/FLUX.1-dev](htpss://hf.co/black-forest-labs/FLUX.1-dev) |
| and [TIGER-Lab/OmniEdit-Filtered-1.2M](https://huggingface.co/datasets/TIGER-Lab/OmniEdit-Filtered-1.2M) for image editing. We use the |
| [Flux Control framework](https://blackforestlabs.ai/flux-1-tools/) for fine-tuning. |
|
|
|
|
| ## License |
|
|
| Please adhere to the licensing terms as described [here](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) |
|
|
|
|
| ## Intended uses & limitations |
|
|
| ### Inference |
|
|
| ```py |
| from diffusers import FluxControlPipeline, FluxTransformer2DModel |
| from diffusers.utils import load_image |
| import torch |
| |
| path = "sayakpaul/FLUX.1-dev-edit-v0" |
| edit_transformer = FluxTransformer2DModel.from_pretrained(path, torch_dtype=torch.bfloat16) |
| pipeline = FluxControlPipeline.from_pretrained( |
| "black-forest-labs/FLUX.1-dev", transformer=edit_transformer, torch_dtype=torch.bfloat16 |
| ).to("cuda") |
| |
| url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/assets/mushroom.jpg" |
| image = load_image(url) # resize as needed. |
| print(image.size) |
| |
| prompt = "turn the color of mushroom to gray" |
| image = pipeline( |
| control_image=image, |
| prompt=prompt, |
| guidance_scale=30., # change this as needed. |
| num_inference_steps=50, # change this as needed. |
| max_sequence_length=512, |
| height=image.height, |
| width=image.width, |
| generator=torch.manual_seed(0) |
| ).images[0] |
| image.save("edited_image.png") |
| ``` |
|
|
| ### Speeding inference with a turbo LoRA |
|
|
| We can speed up the inference by reducing the `num_inference_steps` to produce a nice image by using turbo LoRA like [`ByteDance/Hyper-SD`](https://hf.co/ByteDance/Hyper-SD). |
|
|
| Make sure to install `peft` before running the code below: `pip install -U peft`. |
|
|
| <details> |
| <summary>Code</summary> |
|
|
| ```py |
| from diffusers import FluxControlPipeline, FluxTransformer2DModel |
| from diffusers.utils import load_image |
| from huggingface_hub import hf_hub_download |
| import torch |
| |
| path = "sayakpaul/FLUX.1-dev-edit-v0" |
| edit_transformer = FluxTransformer2DModel.from_pretrained(path, torch_dtype=torch.bfloat16) |
| pipeline = FluxControlPipeline.from_pretrained( |
| "black-forest-labs/FLUX.1-dev", transformer=edit_transformer, torch_dtype=torch.bfloat16 |
| ).to("cuda") |
| |
| # load the turbo LoRA |
| pipeline.load_lora_weights( |
| hf_hub_download("ByteDance/Hyper-SD", "Hyper-FLUX.1-dev-8steps-lora.safetensors"), adapter_name="hyper-sd" |
| ) |
| pipeline.set_adapters(["hyper-sd"], adapter_weights=[0.125]) |
| |
| |
| url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/assets/mushroom.jpg" |
| image = load_image(url) # resize as needed. |
| print(image.size) |
| |
| prompt = "turn the color of mushroom to gray" |
| image = pipeline( |
| control_image=image, |
| prompt=prompt, |
| guidance_scale=30., # change this as needed. |
| num_inference_steps=8, # change this as needed. |
| max_sequence_length=512, |
| height=image.height, |
| width=image.width, |
| generator=torch.manual_seed(0) |
| ).images[0] |
| image.save("edited_image.png") |
| ``` |
|
|
| </details> |
| <br> |
| <details> |
| <summary>Comparison</summary> |
|
|
| <table align="center"> |
| <tr> |
| <th>50 steps</th> |
| <th>8 steps</th> |
| </tr> |
| <tr> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_car.jpg" alt="50 steps 1" width="150"></td> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_8steps_car.jpg" alt="8 steps 1" width="150"></td> |
| </tr> |
| <tr> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_norte_dam.jpg" alt="50 steps 2" width="150"></td> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_8steps_norte_dam.jpg" alt="8 steps 2" width="150"></td> |
| </tr> |
| <tr> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_mushroom.jpg" alt="50 steps 3" width="150"></td> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_8steps_mushroom.jpg" alt="8 steps 3" width="150"></td> |
| </tr> |
| <tr> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_green_creature.jpg" alt="50 steps 4" width="150"></td> |
| <td align="center"><img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/edited_8steps_green_creature.jpg" alt="8 steps 4" width="150"></td> |
| </tr> |
| </table> |
| |
|
|
| </details> |
|
|
| You can also choose to perform quantization if the memory requirements cannot be satisfied further w.r.t your hardware. Refer to the [Diffusers documentation](https://huggingface.co/docs/diffusers/main/en/quantization/overview) to learn more. |
|
|
| `guidance_scale` also impacts the results: |
|
|
| <table align="center"> |
| <tr> |
| <th>Prompt</th> |
| <th>Collage (gs: 10)</th> |
| <th>Collage (gs: 20)</th> |
| <th>Collage (gs: 30)</th> |
| <th>Collage (gs: 40)</th> |
| </tr> |
| <tr> |
| <td align="center"> |
| <em>Give this the look of a traditional Japanese woodblock print.</em> |
| </td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_0.png" alt="Edited Image gs 10"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_1.png" alt="Edited Image gs 20"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_2.png" alt="Edited Image gs 30"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_3.png" alt="Edited Image gs 40"></td> |
| </tr> |
| <tr> |
| <td align="center"> |
| <em>transform the setting to a winter scene</em> |
| </td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_4.png" alt="Edited Image gs 10"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_5.png" alt="Edited Image gs 20"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_5.png" alt="Edited Image gs 30"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_6.png" alt="Edited Image gs 40"></td> |
| </tr> |
| <tr> |
| <td align="center"> |
| <em>turn the color of mushroom to gray</em> |
| </td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_12.png" alt="Edited Image gs 10"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_13.png" alt="Edited Image gs 20"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_14.png" alt="Edited Image gs 30"></td> |
| <td align="center"><img src="https://huggingface.co/sayakpaul/FLUX.1-dev-edit-v0/resolve/main/images_15.png" alt="Edited Image gs 40"></td> |
| </tr> |
| </table> |
| |
|
|
| ### Limitations and bias |
|
|
| Expect the model to perform underwhelmingly as we don't know the exact training details of Flux Control. |
|
|
| ## Training details |
|
|
| Fine-tuning codebase is [here](https://github.com/sayakpaul/flux-image-editing). Training hyperparameters: |
|
|
| * Per GPU batch size: 4 |
| * Gradient accumulation steps: 4 |
| * Guidance scale: 30 |
| * BF16 mixed-precision |
| * AdamW optimizer (8bit from `bitsandbytes`) |
| * Constant learning rate of 5e-5 |
| * Weight decay of 1e-6 |
| * 20000 training steps |
|
|
|
|
| Training was conducted using a node of 8xH100s. |
|
|
| We used a simplified flow mechanism to perform the linear interpolation. In pseudo-code, that looks like: |
|
|
| ```py |
| sigmas = torch.rand(batch_size) |
| timesteps = (sigmas * noise_scheduler.config.num_train_timesteps).long() |
| ... |
| |
| noisy_model_input = (1.0 - sigmas) * pixel_latents + sigmas * noise |
| ``` |
|
|
| where `pixel_latents` is computed from the source images and `noise` is drawn from a Gaussian distribution. For more details, [check out |
| the repository](https://github.com/sayakpaul/flux-image-editing/blob/b041f62df8f959dc3b2f324d2bfdcdf3a6388598/train.py#L403). |
