Generate-then-Instruct-Edit Chain

A text-to-image model generates a base image from a prompt, then a separate instruction-following inpaint model takes that image plus a mask and a text instruction and applies targeted changes to specific regions while leaving the rest intact.

Properties

family: generate-to-edit
modality: text -> image -> image
whyMultiModel: The generation and editing objectives require different conditioning: the base model maximizes prompt fidelity from noise; the edit model must preserve unchanged regions, understand natural-language deltas, and apply localized changes. Using the same model and prompt for both leads to either image drift (high denoise) or insufficient change (low denoise). FLUX.1-dev and FLUX.1-Fill are distinct model weights with different training objectives (velocity prediction on pure noise vs. masked inpainting), making this a genuine two-model chain. This is distinct from base-refiner which uses low-denoise image-to-image on the whole image for detail finishing, not semantic region editing.
steps: [object Object], [object Object]
controls: Mask precision (tight vs loose); inpaint denoise strength; prompt describing only the desired change; blending mode at mask boundary; VAE tiling for memory management.
exampleStack: FLUX.1-dev (generate) -> FLUX.1-Fill-dev (inpaint masked region); or SDXL base -> SDXL-Inpaint; or Stable Diffusion 1.5 -> SD-Inpaint.
useCases: [Generating a hero image then swapping background][Creating a base composition then iteratively adjusting individual objects][Product photography with post-generation surface or label edits]
pitfalls: Mask bleeding (inpainted region texture mismatches surroundings); inconsistent lighting between generated base and edited region; over-large masks cause the edit model to ignore the surrounding context.
source: ComfyUI Official Docs: Flux.1 Fill dev Example

Provenance

Source: https://docs.comfy.org/tutorials/flux/flux-1-fill-dev

Properties

family

generate-to-edit

modality

text -> image -> image

whyMultiModel

The generation and editing objectives require different conditioning: the base model maximizes prompt fidelity from noise; the edit model must preserve unchanged regions, understand natural-language deltas, and apply localized changes. Using the same model and prompt for both leads to either image drift (high denoise) or insufficient change (low denoise). FLUX.1-dev and FLUX.1-Fill are distinct model weights with different training objectives (velocity prediction on pure noise vs. masked inpainting), making this a genuine two-model chain. This is distinct from base-refiner which uses low-denoise image-to-image on the whole image for detail finishing, not semantic region editing.

steps

[object Object], [object Object]

controls

Mask precision (tight vs loose); inpaint denoise strength; prompt describing only the desired change; blending mode at mask boundary; VAE tiling for memory management.

exampleStack

FLUX.1-dev (generate) -> FLUX.1-Fill-dev (inpaint masked region); or SDXL base -> SDXL-Inpaint; or Stable Diffusion 1.5 -> SD-Inpaint.

useCases

[Generating a hero image then swapping background][Creating a base composition then iteratively adjusting individual objects][Product photography with post-generation surface or label edits]

pitfalls

Mask bleeding (inpainted region texture mismatches surroundings); inconsistent lighting between generated base and edited region; over-large masks cause the edit model to ignore the surrounding context.

source

ComfyUI Official Docs: Flux.1 Fill dev Example