Hermosa Research·Synthesis
Generative Media Pipeline Patterns
A catalogue of 76 generative media pipelines across 9 families and 223 sourced stages. It covers image, video, character consistency, audio-driven animation, and 3D, with the ordered models, controls, failure modes, and source for each chain.
- Published
- 3 July 2026
- Data as of
- 19 August 2026
- Research type
- [synthesis]Distils many external sources into a position. The rigor lives in source verification and per-claim attribution.[tentative]Reasonably supported, but parts are still unverified or liable to drift.
- Topics
- [generative-media][diffusion][image-generation][video-generation][3d-generation][ai-pipelines]
Method: We catalogued 67 chains across nine families and 194 stages. Each pattern names at least two distinct generative models, an ordered workflow, controls, failure modes, and a source covering the chain.
Background
Generative media workflows often split work among models. One pass establishes composition, another preserves identity, a third adds motion or detail, and a final pass restores or upscales the result. The order and handoff between those stages determine whether a result can be repeated.
This catalogue records 76 reusable chains across 9 families. Every entry names at least two generative models, orders the stages, identifies the control inputs and common failure, and links to a source that covers the chain.
Key findings
- 1Most chains separate exploration from finishing. A fast or flexible pass establishes the asset, then a slower or more specialised pass refines detail, motion, identity, or texture.
- 2Control inputs deserve their own stage. Identity, pose, structure, and style are more repeatable when supplied through references, adapters, masks, seeds, or control networks instead of prompt wording alone.
- 3Image-to-video starts with composition. Many motion workflows treat a strong still, first frame, or storyboard as the upstream asset, then pass it to a video model for movement and timing.
- 43D workflows often separate geometry from materials. Pipelines such as TRELLIS and Hunyuan3D generate or refine shape before producing texture, which makes each stage easier to inspect and replace.
- 5Audio-driven animation now extends beyond lip sync. Recent systems use speech or music to drive facial motion, gesture, or full-body movement. The dataset separates those workflows from face-only talking-head pipelines.
The pipeline catalogue
The 76 pipelines are grouped by workflow shape. Each record gives the ordered stages and models, controls, use cases, multi-model rationale, common failure, and source.
Draft to finish2
A rough pass, then a high-fidelity finishing pass.
Base + Refiner Draft-to-Finish
text -> imageA fast base model lays down composition and structure, then a high-fidelity refiner pass adds detail and fixes artifacts, optionally followed by an upscaler.
Stack, controls & moreHide
- Example stack
- SDXL base -> SDXL refiner -> 4x ESRGAN upscaler (or FLUX.1 schnell draft -> FLUX.1 dev refine).
- Controls
- Shared seed across passes; denoise strength on the refine step; base/refiner step-split ratio; optional control net on the draft.
- Why multi-model
- One model rarely optimizes both global composition and fine detail at once; splitting draft from refine lets a cheap fast pass explore many options and an expensive pass finish only the keepers.
- Use cases
- Hero art and key visuals, Batch concepting then finishing keepers, Print-resolution stills
- Pitfalls
- Over-denoising the refine step drifts away from the draft composition; a mismatched seed breaks continuity between passes.
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis (arXiv:2307.01952)
ComfyTextures Unreal create-refine-upscale-edit texture chain
render capture -> image -> image -> image -> imageUnreal Engine editor plugin ComfyTextures captures viewport render passes of selected actors and runs them through a ComfyUI-orchestrated chain: a fast SDXL+LCM-LoRA pass creates a low-res draft texture, a slower SDXL base+refiner pass refines it, an inpainting workflow allows targeted edits, and a 4x-UltraSharp upscale finishes the result, which is applied back as a new material/texture on the actor.
Stack, controls & moreHide
- Example stack
- UE viewport capture -> depth/canny ControlNet -> SDXL+LCM Create -> SDXL Base+Refiner Refine -> SDXL inpaint Edit -> 4x-UltraSharp Upscale -> applied as UE material
- Controls
- Actor/viewpoint selection in the UE editor, single or multi point-of-view projection, ControlNet weight (canny/depth), edit target (texture region vs whole object), workflow JSON is user-editable in ComfyUI
- Why multi-model
- Distinct from baseline base-refiner-draft-to-finish (generic T2I draft/refine/upscale) and from StableGen because it operates purely on existing Unreal actor geometry via viewport-camera depth/canny ControlNet projection (no mesh-generation stage), and chains four separately-named modes (Create/Refine/Edit/Upscale) each backed by a distinct model or LoRA variant, wired directly into the UE material graph.
- Use cases
- rapid environment art texturing directly inside the UE editor, iterative look-dev without leaving Unreal, targeted texture fixes on shipped game assets
- Pitfalls
- multi-viewpoint projection is explicitly marked work-in-progress in the plugin, so single-viewpoint consistency is the reliable path; needs 16GB+ VRAM recommended for SDXL workflows
Generate to edit9
Generate, then a targeted edit, inpaint, or variation.
Bernini MLLM-Planner + DiT-Renderer Video Edit
video + reference image + instruction -> edited videoByteDance's Bernini splits instruction-based video generation and editing into two models: an MLLM planner that predicts the target semantic representation in ViT embedding space, and a DiT diffusion renderer that synthesizes pixels conditioned on that plan plus source-video features, keeping unedited regions stable and preserving subject identity.
Stack, controls & moreHide
- Example stack
- Bernini planner (Qwen2.5-VL based MLLM) -> Bernini renderer (Wan2.2 DiT, HIGH + LOW backbones) -> Wan 2.1 VAE decode; ComfyUI graph adds a Wan-style T5 text encoder and the LightX2V LoRA pair.
- Controls
- Edit instruction wording; optional reference image for garment swap or subject insertion; source-feature strength (identity preservation); HIGH vs LOW phase split; LightX2V high_noise/low_noise LoRA motion adapters in the ComfyUI workflow.
- Why multi-model
- Semantic planning (what the edit should become) and pixel rendering (how it should look) are different learned problems: the MLLM reasons about the instruction in embedding space while the DiT renderer conditions on the plan and low-level source features. Neither model alone can both interpret the edit instruction and preserve fine visual detail.
- Use cases
- Instruction-based video editing with stable unedited regions, Reference-guided garment swap and object insertion in video, Subject-to-video: placing a person or character into a new scene, Identity-preserving scene and style changes
- Pitfalls
- The plan bounds the edit: if the MLLM misreads the instruction the renderer faithfully produces the wrong edit; the renderer weights are open (Apache 2.0), and the planner (Qwen2.5-VL-7B-Instruct) and renderer (Wan2.2-T2V-A14B) backbones are confirmed via the official bytedance/Bernini GitHub repo, but planner quality still gates the whole chain.
Bernini: Latent Semantic Planning for Video Diffusion (arXiv:2605.22344)
Bria native GenAI node chain in Nuke
text -> image -> image -> imageBria ships 11 GenAI nodes (Fibo Generate, Fibo Edit, RMBG, Erase, GenFill, Expand, Enhance, Upscale, Sequence Output) as native Nuke Group nodes that chain directly in the compositing node graph: text-to-image draft, then prompt-based inpaint/edit, then dedicated enhance/upscale, all at full resolution without leaving Nuke.
Stack, controls & moreHide
- Example stack
- Fibo Generate -> GenFill -> Upscale, all native Nuke Group nodes in one .nk script
- Controls
- Node-graph wiring only (standard Nuke Group nodes); prompt text, mask input on GenFill/Erase, aspect ratio and guidance on Fibo Generate; Sequence Output node batches across frame ranges
- Why multi-model
- Fibo Generate (text-to-image foundation model), GenFill (inpainting/ControlNet-based edit model), and Upscale/Enhance (dedicated super-resolution and Keymix quality models) are distinct Bria model families chained node-to-node; this is not one model with multiple presets.
- Use cases
- VFX plate generation without leaving the compositor, in-comp set extension/inpaint, batch frame-range AI touch-ups
- Pitfalls
- Multi-viewpoint/3D-aware consistency is not addressed (2D compositing only); each node runs full-resolution inference per frame which is costly for long sequences via Sequence Output
Generate-then-Instruct-Edit Chain
text -> image -> imageA text-to-image model generates a base image from a prompt, then a separate instruction-following inpaint model takes that image plus a mask and a text instruction and applies targeted changes to specific regions while leaving the rest intact.
Stack, controls & moreHide
- Example stack
- FLUX.1-dev (generate) -> FLUX.1-Fill-dev (inpaint masked region); or SDXL base -> SDXL-Inpaint; or Stable Diffusion 1.5 -> SD-Inpaint.
- Controls
- Mask precision (tight vs loose); inpaint denoise strength; prompt describing only the desired change; blending mode at mask boundary; VAE tiling for memory management.
- Why multi-model
- The generation and editing objectives require different conditioning: the base model maximizes prompt fidelity from noise; the edit model must preserve unchanged regions, understand natural-language deltas, and apply localized changes. Using the same model and prompt for both leads to either image drift (high denoise) or insufficient change (low denoise). FLUX.1-dev and FLUX.1-Fill are distinct model weights with different training objectives (velocity prediction on pure noise vs. masked inpainting), making this a genuine two-model chain. This is distinct from base-refiner which uses low-denoise image-to-image on the whole image for detail finishing, not semantic region editing.
- Use cases
- Generating a hero image then swapping background, Creating a base composition then iteratively adjusting individual objects, Product photography with post-generation surface or label edits
- Pitfalls
- Mask bleeding (inpainted region texture mismatches surroundings); inconsistent lighting between generated base and edited region; over-large masks cause the edit model to ignore the surrounding context.
IDM-VTON Virtual Try-On
person image + garment image -> imageRealistically transfer a garment from a product photo onto a person image, warping the cloth to body geometry and inpainting it in with a diffusion try-on model.
Stack, controls & moreHide
- Example stack
- IDM-VTON (or CatVTON/OOTDiffusion) -> 2x upscale.
- Controls
- Person and garment inputs; garment category; agnostic masks (pose, parse); try-on guidance scale.
- Why multi-model
- Try-on couples geometric garment warping with a diffusion inpaint conditioned on the warped garment; neither a pure generator nor a plain inpainter handles the two together.
- Use cases
- E-commerce product previews, Lookbook generation, Catalog at scale
- Pitfalls
- Extreme poses and occlusions warp the garment incorrectly; hair and hands crossing the garment region cause artifacts.
Improving Diffusion Models for Authentic Virtual Try-on in the Wild, IDM-VTON (arXiv:2403.05139)
Inpaint Anything: Segment-Remove-Replace
image + click -> mask -> removed or replaced region -> composited outputA user clicks an object, Segment Anything produces its mask, and depending on intent the hole is either filled by a fast non-diffusion inpainter (object removal) or filled by a text-guided diffusion model (object/background replacement).
Stack, controls & moreHide
- Example stack
- SAM -> LaMa (remove) or Stable Diffusion Inpainting (replace)
- Controls
- click/point prompt, mask dilation, text prompt (replace path only), inpainting strength, seed
- Why multi-model
- Segmentation and inpainting are different problem classes: SAM has no generative capability, LaMa cannot follow text prompts for novel content, and Stable Diffusion alone can't reliably target an arbitrary user-clicked region without an upstream mask.
- Use cases
- one-click object removal from photos, background replacement via text prompt, e-commerce photo cleanup
- Pitfalls
- LaMa struggles with large masks needing coherent new structure (shadows, reflections); Stable Diffusion replace path can hallucinate seams without careful mask feathering and colour matching
Inpaint Anything: Segment Anything Meets Image Inpainting, arXiv:2304.06790
InstructPix2Pix Instruction Edit
image + instruction -> imageEdit an existing image from a natural-language instruction ("make it sunset", "add sunglasses") without a mask, by conditioning the model jointly on the source image and the command.
Stack, controls & moreHide
- Example stack
- InstructPix2Pix (SD) or FLUX edit -> upscale.
- Controls
- Image-conditioning strength (how much of the original to keep); text-conditioning strength; instruction phrasing.
- Why multi-model
- Mask-free, instruction-driven editing needs a model trained to follow commands over an image condition, distinct from both pure generation and masked inpainting.
- Use cases
- Iterative art direction, No-mask quick edits, Batch style tweaks
- Pitfalls
- Over-strong image conditioning ignores the instruction; weak conditioning hallucinates changes the user did not ask for.
InstructPix2Pix: Learning to Follow Image Editing Instructions (arXiv:2211.09800)
Product Photo Scene Generation & Relight
image -> mask -> generated background -> relit compositeA product cutout is segmented from its original photo, dropped into a newly generated lifestyle background, then IC-Light relights the product to match the new scene's lighting so the composite reads as a single photograph.
Stack, controls & moreHide
- Example stack
- BiRefNet -> SDXL (background) -> IC-Light v2 (relight)
- Controls
- segmentation mask, background text prompt, IC-Light lighting-direction/text conditioning, denoise strength to preserve product detail, seed
- Why multi-model
- Segmentation, background generation, and physically-plausible relighting are three distinct capabilities: a single text-to-image model can generate a scene but cannot preserve the exact product geometry/texture, and it cannot relight an already-composited foreground to match a background it didn't originate.
- Use cases
- e-commerce product photography at scale, advertising creative variants, catalogue localisation with different scene moods
- Pitfalls
- IC-Light can drift product colour/texture at high denoise; frequency separation or detail-preserving masks are needed to keep label text and material sharpness intact
Product Photo Relight ComfyUI workflows (risunobushi, OpenArt)
Product-into-Scene Composite and Relight
product cutout image + background scene image -> lit, composited ad-ready imageA studio product photo and a separate background/scene image are composited by an image-editing model that also relights the product to match the new scene's light direction and color temperature, producing an advertising-ready packshot without a physical reshoot.
Stack, controls & moreHide
- Example stack
- RMBG-2.0 -> ByteDance Seedream 4.5 (composite + relight) -> Topaz Gigapixel upscale
- Controls
- background/scene reference image, lighting preset (golden hour, studio, dramatic directional), product placement/scale, upscale factor
- Why multi-model
- Compositing two independent images and physically-plausible relighting are distinct operations; the workflow depends on prior product segmentation/cutout plus a generative compositing-and-relighting model, and often a separate upscale pass for print/4K delivery, so no single call handles isolation, blend, and light-matching in one pass reliably.
- Use cases
- e-commerce catalog imagery at scale, seasonal ad creative reusing one product shoot across many scenes, swapping backgrounds for regional/localized campaigns
- Pitfalls
- Relighting can shift product color fidelity (brand color accuracy), and shadow direction mismatches between product and scene are a common tell if the compositing step skips explicit light-direction analysis.
Composite your Product + Scene and Relight (official ComfyUI workflow template)
SAM 2 to MatAnyone Rotoscoping Composite
video + point/text prompt -> propagated mask -> alpha matte -> compositeA subject is segmented in one frame via a promptable segmentation model, the mask is propagated across the clip, then a dedicated video matting model refines the rough mask into a per-pixel alpha matte for clean compositing onto a new background.
Stack, controls & moreHide
- Example stack
- SAM 2 (first-frame mask) -> MatAnyone 2 (alpha propagation) -> compositing tool (background swap)
- Controls
- click/box/text prompt for initial mask, mask correction clicks, matte quality evaluator threshold, background choice
- Why multi-model
- Promptable segmentation models produce binary, sometimes coarse masks that are not designed for fine alpha detail (hair, motion blur, semi-transparency); a separate matting model specialized in alpha propagation is needed to get broadcast-quality composites.
- Use cases
- greenscreen-free background replacement for UGC video, VFX rotoscoping for indie production, compositing AI-generated characters onto filmed plates
- Pitfalls
- SAM 2 wins on speed but loses edge fidelity on hair/motion blur versus a manual rotoscoper; the recommended fix is pairing with a dedicated matting model rather than shipping the raw SAM 2 mask
MatAnyone 2: Scaling Video Matting via a Learned Quality Evaluator (arXiv:2512.11782, CVPR 2026)
Style & control7
Condition structure or aesthetic with references and control nets.
ControlNet Structure-Lock Chain
image + text -> imageA preprocessor (canny, depth, or pose annotator) extracts a structural control map from a reference image; that map is then injected into a conditioned diffusion model to generate a new image that preserves the source structure.
Stack, controls & moreHide
- Example stack
- MiDaS depth -> ControlNet-v1.1-depth + SDXL; or OpenPose -> ControlNet-OpenPose + SD 1.5; or Canny -> FLUX.1 ControlNet (Jasper canny).
- Controls
- Annotator type (canny vs depth vs pose); ControlNet conditioning scale; text prompt; optional second ControlNet for combined conditions (e.g. depth + canny simultaneously).
- Why multi-model
- Two genuinely distinct model types run in sequence: an annotator/extractor model (e.g. HED edge detector, MiDaS depth, OpenPose keypoint) produces a dense control signal, and a separate ControlNet-augmented diffusion model consumes that signal during generation. Neither step can substitute for the other.
- Use cases
- Pose-controlled character generation, Preserving scene layout across style variations, Sketch / lineart to rendered image
- Pitfalls
- Over-conditioning (scale too high) makes the output look traced with no creative latitude; mismatched annotator and ControlNet model type produces incoherent outputs; depth annotator struggles with textureless or reflective surfaces.
Adding Conditional Control to Text-to-Image Diffusion Models (arXiv:2302.05543)
Depth/Pose-Guided Video-to-Video Restyle
video -> control signal (depth/canny/pose) -> videoA source video is passed through a preprocessor that extracts a structural control signal (depth map, Canny edges, or OpenPose skeleton). That signal is fed as conditioning to a video diffusion model which generates new video that preserves the original motion and geometry while applying a new appearance, style, or character. The result is a restyled video with motion inherited from the reference footage.
Stack, controls & moreHide
- Example stack
- Lotus Depth preprocessor -> LTX-2 ControlNet (guided video generation). OR OpenPose extractor -> Wan 2.1 Fun ControlNet.
- Controls
- Control signal type (depth vs. canny vs. pose); control strength; style and content prompts; sampler schedule (LTXVScheduler); frame dimensions.
- Why multi-model
- Video diffusion models cannot natively decompose appearance from structure; a separate ControlNet preprocessor model is required to extract the structural signal, which then constrains the generator. Neither the preprocessor nor the generator can do the other's job alone.
- Use cases
- Motion retargeting (human to character), Scene relighting and restyle, Product placement in existing footage, Previz from rough reference video
- Pitfalls
- Strong depth control preserves layout but can suppress texture creativity; OpenPose fails on non-standard human shapes; motion blur in source degrades extracted control quality.
RunComfy: LTX-2 ControlNet in ComfyUI - Depth-Controlled Video Workflow
InstantID Zero-Shot Identity-Style
face image + style text -> stylized portraitZero-shot identity-preserving generation: with only one reference face and a style prompt, generate that person in any style or scene, combining a face encoder, an IdentityNet, and a lightweight adapter.
Stack, controls & moreHide
- Example stack
- InstantID (SDXL) -> 2x upscale.
- Controls
- Reference face; style and prompt text; identity strength; ControlNet pose or edge.
- Why multi-model
- Locking a specific identity while changing style needs a dedicated identity encoder and IdentityNet, separate from the text-to-image backbone.
- Use cases
- Stylized profile portraits, Themed personal art, Brand persona variants
- Pitfalls
- Identity versus style is a trade-off set by the strength knob; non-face references give no signal.
InstantID: Zero-shot Identity-Preserving Generation in Seconds (arXiv:2401.07519)
Reference-Driven Line Art Video Colorization
line-art video + reference character image -> tracked correspondences -> colorized animationA raw line-art/sketch video sequence and a single colored reference image of the character are fed into a video diffusion model that uses point-tracking correspondences to transfer color and shading from the reference onto every sketch frame, producing a temporally consistent colored animation.
Stack, controls & moreHide
- Example stack
- CoTracker2 -> AniDoc sketch extractor -> Stable Video Diffusion (AniDoc checkpoint)
- Controls
- reference character image (color/design sheet), point-tracking correspondence maps, per-frame sketch/line-art control, frame count (14-frame native window)
- Why multi-model
- A single video diffusion model cannot both (a) infer temporally-stable pixel correspondences between a static reference pose and a moving sketch character and (b) generate coherent color; a dedicated point-tracker supplies explicit correspondence guidance the diffusion U-Net conditions on, and an SVD-based backbone handles temporal coherence and generation separately.
- Use cases
- 2D anime in-betweening automation, colorizing rough animatics from a character color sheet, reducing manual cel-coloring labor in production pipelines
- Pitfalls
- Large pose deviations between reference and sketch frames can cause color bleeding; native window is 14 frames so longer shots need chunking/stitching, which can introduce seams.
AniDoc: Animation Creation Made Easier (CVPR 2025, arXiv:2412.14173)
Restyled First-Frame Propagation
source video + style prompt -> restyled first frame -> structure-guided full-video restyleAn image-editing model restyles only the first frame of a source video into a target look; a video model then regenerates the entire clip by propagating that one already-styled frame forward through time, using lightweight structural maps only as an ancillary guardrail against geometry drift rather than as the source of style.
Stack, controls & moreHide
- Example stack
- Flux Kontext (first-frame restyle) -> Wan 2.2 Fun (video-to-video, first-frame + ControlNet conditioned)
- Controls
- style prompt / reference frame for the first-frame edit, choice of ancillary ControlNet preprocessor (depth/canny/pose), ControlNet conditioning strength, output resolution
- Why multi-model
- The style decision is made once, by an image-editing model, on a single frame where it can be art-directed cheaply; a text-to-video or per-frame ControlNet pass cannot reliably apply that same specific edited look to every frame without drifting, so a second model only propagates an already-finished still.
- Use cases
- Stylizing existing footage into a new visual genre while keeping the original motion, Converting live-action reference into anime or painterly renders, Brand-consistent restyle of stock/user video
- Pitfalls
- Style fidelity depends entirely on how well the first frame was restyled; a first frame that conflicts with the ancillary structure maps produces visible flicker or geometry fighting once propagated across the full clip.
Wan 2.2 Video Restyle: First Frame Style Transfer ComfyUI Workflow
StreamDiffusionTD live multi-ControlNet + StreamV2V temporal-consistency chain
live video/control signal -> image (real-time loop) -> temporally-consistent videoTouchDesigner tox StreamDiffusionTD runs SDXL-Turbo in real time driven by any TouchDesigner-generated control signal (camera feed, procedural TOP, depth/canny/normal maps built in TD) through multiple simultaneous TensorRT-accelerated ControlNets, then applies StreamV2V cached-attention video-to-video processing so consecutive live frames stay temporally coherent instead of flickering independently.
Stack, controls & moreHide
- Example stack
- TouchDesigner TOP control -> SDXL-Turbo + multi-ControlNet (StreamDiffusionTD) -> StreamV2V temporal pass
- Controls
- ControlNet on/off toggle mid-stream and per-unit weight, multiple simultaneous ControlNets, TensorRT engine build per model for real-time speed, any TD-authored conditioning image (audio-reactive, camera, procedural) wired into the second input
- Why multi-model
- Distinct from baseline depth-pose-guided-v2v-restyle (offline video-to-video restyle via LTX/Wan Fun) because this runs live/interactively at TouchDesigner frame rates with a different model pair, SDXL-Turbo for per-frame generation plus StreamV2V for cross-frame temporal consistency, driven by arbitrary TD-authored control signals rather than a pre-extracted depth/pose track on a fixed video file.
- Use cases
- live VJ/installation real-time AI restyle of camera feed, interactive projection-mapped generative visuals reacting to audio or sensors, real-time previz restyle driven by procedural TD signals
- Pitfalls
- requires per-model TensorRT engine builds (CUDA/TRT setup) before streaming; ControlNet must be enabled at stream start to be usable mid-session; real-time frame rate depends heavily on GPU (documented benchmarks are for high-end cards)
Style Reference Adapter Chain
style image + text -> imageA style reference image is encoded by an image adapter and its style features are injected selectively into style-specific attention blocks of a diffusion model, optionally combined with a tile ControlNet that preserves the spatial structure of a content image.
Stack, controls & moreHide
- Example stack
- IP-Adapter style + SDXL base (InstantStyle); or IP-Adapter + Tile ControlNet-SDXL (InstantStyle-Plus); or FLUX.1-dev + IP-Adapter FLUX style variant.
- Controls
- IP-Adapter style weight; decoupled cross-attention scale; tile ControlNet strength; content image inversion (DDIM/ReNoise) for structure fidelity; style guidance gradient weight at each denoising step.
- Why multi-model
- Three logically distinct model components run in sequence or parallel: a style-feature encoder (IP-Adapter or equivalent) that extracts appearance embeddings, a tile/spatial ControlNet that holds content structure, and the base diffusion model that synthesizes the output. No single model can simultaneously encode arbitrary style references AND enforce spatial structure AND generate high-fidelity output. Distinct from Reference-Locked Character Consistency (existing entry) which uses face-identity adapters (InstantID/FaceID) for person identity; this pattern targets visual style (brushwork, color palette, texture aesthetic) of non-identity references.
- Use cases
- Applying an artist's painting style to new photo subjects, Consistent visual look across a series of illustrations, Transferring color palette and texture without copying identity
- Pitfalls
- Style and content bleeding into each other when adapter weight is too high; tile ControlNet over-constrains composition when content and style differ in aspect; style from photorealistic references may transfer poorly to anime base models.
Restore & upscale3
Restoration, upscaling, and detail recovery passes.
Face-Restore + Background-Upscale Chain
image -> imageA face-restoration model (GFPGAN, CodeFormer) recovers degraded facial detail in the generated or photographed image, while a separate general-purpose upscaler (Real-ESRGAN) handles background and non-face regions.
Stack, controls & moreHide
- Example stack
- GFPGAN v1.4 (face regions) + Real-ESRGAN x4plus (background); or CodeFormer + Real-ESRGAN via A1111 Extra tab or ComfyUI FaceRestoreCF node.
- Controls
- CodeFormer fidelity weight (0=quality, 1=identity fidelity); Real-ESRGAN model variant (x4plus, x4plus-anime); face detection confidence threshold; blend mask feathering.
- Why multi-model
- Face restoration and general upscaling are fundamentally different tasks requiring specialized models: face models exploit facial priors (identity, geometry) that general upscalers lack, and general upscalers handle textures and backgrounds that face models cannot attend to. Base-refiner pipelines add a detail-finishing pass on the whole image; this pattern uses region-split specialist models where each model operates on a distinct pixel domain.
- Use cases
- Restoring small or degraded faces in generated images, Old photo revival (scan to high-res), Upscaling AI portraits without over-smoothing faces
- Pitfalls
- Face detection failures on unusual angles or heavy occlusion leave faces unrestored; high CodeFormer fidelity may preserve blur from the original; seam artifacts at the face/background boundary if blend mask is not feathered.
Flux Kontext Restore then Kling-Animate Vintage Photo
image -> image -> videoAn old/vintage photograph is first colorized and quality-enhanced with Flux Kontext, then the restored image is animated into a short video with Kling.
Stack, controls & moreHide
- Example stack
- n8n + imgbb hosting + FLUX Kontext (via API) + Kling Video AI
- Controls
- colorization strength, animation motion prompt, clip length (~5s)
- Why multi-model
- Two distinct generative-media models in sequence: Flux Kontext performs image-to-image restoration (colorization/enhancement) of a real historical photo, and Kling then performs image-to-video animation of the restored result. This differs from the baseline's fresh-generation keyframe-to-i2v pattern and from the baseline's ESRGAN+RIFE finishing chain because the source asset is a real degraded photograph being restored, not a freshly generated or already-clean keyframe, and no upscaler/interpolator is involved.
- Use cases
- bringing family archive photos to life for social media, memorial/nostalgia video content, heritage/history content creation
- Pitfalls
- colorization can introduce historically inaccurate colors; animation motion is generic and not guided by the actual scene depth/pose
Generated-Video Upscale + Frame-Interpolation Finishing
video -> upscaled video -> high-fps videoAI-generated video is typically low resolution (480p-720p) and low frame rate (12-24fps). A two-stage finishing chain first passes every frame through a spatial upscaler (Real-ESRGAN or similar GAN) to reach 2-4x resolution, then applies a temporal frame interpolation model (RIFE) to double or quadruple the frame rate, producing a smooth, high-resolution output suitable for delivery.
Stack, controls & moreHide
- Example stack
- Wan 2.1 (generated 480p 15fps video) -> Real-ESRGAN 4x upscale per frame -> RIFE VFI x2 interpolation -> 1080p 30fps output.
- Controls
- Upscale factor (2x, 4x); ESRGAN model variant (for film vs. animation); RIFE multiplier (x2, x4); FPS target; scale parameter in RIFE for VRAM-limited runs.
- Why multi-model
- Spatial upscaling and temporal interpolation are fundamentally different tasks: upscalers work frame-by-frame on spatial frequency, while interpolators predict optical flow between frames to synthesise new ones. No single model handles both, and applying interpolation before upscaling degrades flow estimation accuracy.
- Use cases
- Delivery-ready upres of AI short films, Slow-motion effects on generated clips, Social platform output requiring 1080p 30fps minimum
- Pitfalls
- RIFE struggles with large fast motions or scene cuts (ghosting artifacts); upscaling before interpolation is safer but requires more VRAM; temporal consistency breaks if source frames are already low-quality.
Real-ESRGAN: Practical Algorithms for General Image/Video Restoration (GitHub: xinntao/Real-ESRGAN)
Character consistency11
Identity locked across generations.
Diffusion-Transformer Garment-Preserving Video Try-On
person video + garment image -> garment-preserved try-on videoA video of a person and a flat garment reference image are fed into a diffusion-transformer model that disentangles garment texture/print from pose and injects it frame-by-frame, producing a video of the person wearing the new garment with consistent fabric detail and motion.
Stack, controls & moreHide
- Example stack
- MagicTryOn garment encoder -> MagicTryOn DiT (Wan 2.1 backbone) -> distilled 4-step sampler
- Controls
- garment reference image, person video, mask-aware loss region (garment area), sampling steps (distilled to 4)
- Why multi-model
- Garment appearance (fine texture/print) and body pose/motion are different signal types that must be extracted, disentangled and separately conditioned on; a single frame-level try-on model cannot maintain garment fidelity across motion without an explicit garment-preservation encoder plus a video-native temporal backbone with distillation for practical speed.
- Use cases
- e-commerce garment videos showing fabric drape/motion instead of static try-on stills, fashion social ads without reshoot per SKU, size/fit preview videos
- Pitfalls
- Fast motion or self-occlusion (arms crossing torso) causes garment texture drift across frames; front-back consistency during turns remains a known failure mode versus single-frame try-on.
HunyuanCustom Multi-Subject Video Customization
reference image(s) + text (+ optional driving video) -> subject-consistent customized videoTencent's HunyuanCustom fuses a LLaVA-based text-image understanding module with the HunyuanVideo DiT and an image-ID enhancement module (temporal concatenation) to keep one or more reference subjects visually consistent throughout a generated or subject-replaced video.
Stack, controls & moreHide
- Example stack
- LLaVA text-image fusion -> HunyuanVideo DiT with image-ID enhancement -> video-driven injection module (subject replacement)
- Controls
- one or more subject reference images, text prompt, optional driving/source video for replacement mode, dual-subject vs single-subject workflow selection
- Why multi-model
- The multimodal conditioning explicitly combines a separate vision-language model (LLaVA) for text-image fusion with the HunyuanVideo generation backbone, plus dedicated injection sub-networks (video-driven injection module for subject-replacement mode); no single model handles both the semantic fusion and the temporal identity reinforcement.
- Use cases
- virtual product/human advertisements with a consistent subject dropped into new video scenes, dual-subject consistent video generation, replacing a subject in existing footage with a reference identity while keeping motion
- Pitfalls
- dual/multi-subject workflows are more failure-prone than single-subject (identity bleed between subjects); the video-driven injection (replacement) path is sensitive to the compression/alignment of the source video and can introduce warping at subject boundaries
HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation (arXiv:2505.04512)
ID-V2V Identity-Preserving Video Restylization
source video + edited keyframe -> identity-preserved restyled videoA source video's first frame (and optionally further anchor frames) is restyled into a new look by an image-editing model, then a dedicated video-to-video model, ID-V2V, regenerates the full clip in that new look while explicitly holding facial likeness, expression, eye gaze and full-body performance constant, casting identity preservation as a video-relighting problem rather than leaving it to the same model that authored the style change.
Stack, controls & moreHide
- Example stack
- NanoBanana Pro (relit, restyled first frame) -> ID-V2V (video-to-video, identity- and performance-preserving propagation) -> restyled output video.
- Controls
- Number and placement of edited/anchor keyframes; whether NanoBanana or NanoBanana Pro (with relighting) is used for the keyframe edit; relit-facial-region and facial-normal-map conditioning strength; depth-sequence conditioning strength for the propagated edit.
- Why multi-model
- The keyframe restyle and the identity-preserving propagation are trained as two separate systems with different objectives: the image-editing model (NanoBanana / NanoBanana Pro) authors the new look at inference time but, per the project's own documentation, often cannot hold the exact pose or expression when restyling a frame, so the stylized first frame ends up misaligned with the source video's first frame; ID-V2V is a distinct video-to-video generative framework trained specifically to invert a relighting transform, using relit facial regions and facial normal maps to anchor identity while the edited keyframe and depth sequences drive the visual edit. Neither model can do the other's job: the image editor has no temporal or video-native identity-preservation training, and ID-V2V does not author style decisions itself, it only propagates an already-edited keyframe. This is distinct from the tracked 'Restyled First-Frame Propagation' pattern (Flux Kontext -> Wan 2.2 Fun), which uses generic depth/canny/pose maps purely as a geometry-drift guardrail and has no facial-relighting-based identity-preservation training objective.
- Use cases
- Shoot-first, restyle-later VFX workflows where the performance is locked but the look is decided later, Converting live-action footage into a new visual genre while keeping the actor's exact performance intact, Multi-person interaction scenes that need a new look without identity swap or drift between subjects
- Pitfalls
- Image-editing models like NanoBanana often can't hold the exact pose or expression when restyling a frame, so downstream quality depends on ID-V2V's relighting-based correction rather than trusting the keyframe outright; because training pairs are constructed via self-supervised relighting of a single video, restyles that go far outside that relighting distribution (extreme genre or geometry changes) can still degrade identity preservation.
ID-V2V: Identity-Preserving Video Restylization (SIGGRAPH Asia 2026, arXiv:2607.22830)
InfiniteYou Recraft to Animated Portrait
single face photo + text -> identity-recrafted photo -> animated clipByteDance InfiniteYou injects identity features through InfuseNet residual connections into a FLUX DiT to recraft a person's photo under a new prompt/pose/style while preserving likeness, then a downstream image-to-video model animates the recrafted still.
Stack, controls & moreHide
- Example stack
- InfiniteYou (InfuseNet) + FLUX.1 dev -> Wan 2.2 image-to-video
- Controls
- identity reference image, text prompt for pose/scene/style, base model swap (dev vs schnell) for quality/speed tradeoff, SFT stage controlling text-image alignment strength
- Why multi-model
- InfuseNet's identity injection is specific to still-image DiT generation and carries no temporal component; a distinct video diffusion backbone is required to add motion, and InfiniteYou's own docs describe it as plug-and-play specifically for composing with other methods/backbones.
- Use cases
- personalized avatar photo then short animated intro clip, recrafting a single reference photo into varied scenes before turning into a talking/moving asset, identity-locked marketing creative across image and short-video formats
- Pitfalls
- identity similarity vs. text-prompt alignment is a direct tradeoff; pushing pose/style far from the reference photo degrades likeness, and any facial artifact in the recrafted still propagates and amplifies once animated
InfiniteYou: Flexible Photo Recrafting While Preserving Your Identity (arXiv:2503.16418)
IP-Adapter Identity-Lock
image + text -> imageA lightweight image-prompt adapter injects a reference image as a decoupled visual condition, so a single subject stays recognizable across many re-prompts and styles without any fine-tuning.
Stack, controls & moreHide
- Example stack
- SDXL + IP-Adapter (or FLUX + IP-Adapter) -> 4x upscale.
- Controls
- Reference image and weight; prompt text and CFG; scale of the image-prompt contribution; compositional masks to keep layout text-driven.
- Why multi-model
- The base diffusion model has no notion of a specific identity; the adapter supplies a separate image-encoder pathway that the U-Net cannot reproduce from text alone.
- Use cases
- Brand mascots and recurring characters, Portrait stylization, Product-on-model with locked identity
- Pitfalls
- Too-high adapter weight overfits the reference and kills prompt adherence; low-fidelity reference images degrade identity rather than stabilize it.
PhotoMaker Custom-Identity Generation
images + text -> imageStack a few reference photos of a person into a unified identity embedding, then generate that exact person in any scene, style, or action from text alone, with no per-subject training.
Stack, controls & moreHide
- Example stack
- PhotoMaker (SDXL) -> upscaler.
- Controls
- Number and variety of reference photos; identity strength; text prompt and style tags.
- Why multi-model
- Identity fusion across multiple reference images needs a dedicated encoder and a merged embedding that a vanilla text-to-image model cannot represent.
- Use cases
- Personalized avatars, Same person across marketing scenes, Fast persona prototyping
- Pitfalls
- Too few or too similar references collapse identity diversity; strong style tags can overwhelm the identity signal.
PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding (arXiv:2312.04461)
Reference-Locked Character Consistency
reference image + text -> imageLock a character's identity across many generations by conditioning a base image model on a reference through an identity adapter, so new poses and scenes keep the same face and character.
Stack, controls & moreHide
- Example stack
- InstantID or IP-Adapter FaceID -> SDXL or FLUX -> GFPGAN/CodeFormer restore.
- Controls
- Identity-adapter weight and scale; reference image quality; pose or control-net guidance; LoRA strength.
- Why multi-model
- A base text-to-image model has no persistent identity; an identity adapter or trained LoRA injects the reference's features so the character stays consistent across prompts in a way a single model cannot.
- Use cases
- Consistent characters across a storyboard, Brand mascots, Comic and graphic-novel panels
- Pitfalls
- Too-high adapter weight copies the reference pose and lighting (overfit); too low loses identity; identity drifts under extreme pose changes.
InstantID: Zero-shot Identity-Preserving Generation in Seconds (arXiv:2401.07519)
Soul ID Locked Character to Veo 3 Talking Video
text/image -> image -> identity-lock -> video+voice -> polishGenerate or upload a character image (Soul or Nano Banana), train a portable digital-twin identity on it with Soul ID, then feed that locked identity into Veo 3 to generate a talking video with native synchronized dialogue and voice, finishing with VFX and camera-motion polish nodes, all inside Higgsfield's node-based Canvas without leaving the app or re-uploading references per shot.
Stack, controls & moreHide
- Example stack
- Soul or Nano Banana (still) -> Soul ID (identity lock) -> Veo 3 (talking video + voice) -> Higgsfield VFX/Camera Motion (polish)
- Controls
- Soul ID reference photo set for identity training; text prompt describing motion, setting and dialogue for Veo 3; VFX and camera-motion node parameters; export aspect ratio (9:16 or 16:9)
- Why multi-model
- Chains at least three distinct generative systems: an image generator (Soul or Nano Banana) for the base character still, Soul ID as a trained identity-lock layer that persists the face across generations without re-uploading a reference each time, and Veo 3 as a separate video-and-voice model that produces native lip-synced dialogue directly (no separate TTS or lip-sync model bolted on afterward). This differs from reference-locked-character-consistency (InstantID/IP-Adapter face lock on still images only, no video stage) and from voice-clone-to-talking-head / tts-video-diffusion-lipsync-chain (both of which pair a separate zero-shot TTS clone with a separate lip-sync/talking-head model rather than a video model with native audio generation).
- Use cases
- recurring branded spokesperson videos across campaign variations, UGC-style talking avatar ads without re-shooting per script, consistent narrator/host across a multi-episode video series
- Pitfalls
- Requires a Higgsfield subscription tier with Veo 3 access; identity lock quality depends on the base reference image; native Veo 3 audio/dialogue means separate TTS or lip-sync tooling is not used, so voice style is constrained to what Veo 3 can render from the prompt rather than a cloned voice.
Stand-In Lightweight Identity Video Control
single face photo + text (+ optional pose video / style LoRA) -> identity-locked videoStand-In adds a tiny conditional-image branch (~1% extra parameters) onto Wan2.1/2.2 that restricts self-attention with conditional position mapping, locking a face identity across an entire generated video clip while composing with pose-driven control or community style LoRAs.
Stack, controls & moreHide
- Example stack
- antelopev2 face encoder -> Stand-In adapter + Wan 2.1-14B text-to-video -> VACE pose control (optional)
- Controls
- single reference face image, text prompt, optional driving pose video (VACE), optional LoRA for stylization, denoising-strength adjustment for the experimental face-swap mode
- Why multi-model
- Identity extraction and video synthesis are handled by two separate networks: a dedicated face-recognition encoder produces the identity embedding, and a large pretrained video DiT (untouched in its core weights) consumes it; optional third-party pose control (VACE) or LoRA further compose in as independent modules.
- Use cases
- identity-locked talking/acting clips from one photo, face-swap-in-video variants, stylized (cartoon/anime) identity-consistent video via community LoRA composition
- Pitfalls
- trained only on real-person data, so cartoon/object generalization is a secondary, less-tested mode; combining pose control (VACE) with identity locking simultaneously can still trade off motion naturalness against strict facial consistency
Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generation (arXiv:2508.07901)
UNO Multi-Subject Compose to Wan Animate
reference images (subject + object) + text -> composed still -> animated videoByteDance UNO composes a character (and optional object/outfit reference) into a single consistent still via in-context multi-image conditioning on FLUX.1-dev, then a CLIP-Vision-encoded image-to-video model animates that still into a moving clip with the same identity and outfit.
Stack, controls & moreHide
- Example stack
- ByteDance UNO (FLUX.1 dev) -> CLIP Vision encode -> Wan 2.2 image-to-video
- Controls
- number/order of reference images, text prompt for scene, UnoPE position mapping (subject vs. object disambiguation), CLIP Vision reference averaging across multiple stills to stabilize identity, negative reference image to suppress unwanted traits
- Why multi-model
- UNO's DiT adapter only produces a single still frame from reference images; it has no temporal/motion prior. A separate video diffusion model with its own vision encoder is required to turn the consistent still into motion without re-drifting identity.
- Use cases
- product/character mockups that need to move, outfit-consistent social clips from a single compose, game/animation asset previsualization
- Pitfalls
- UNO can still show attribute confusion between subject and object on complex multi-reference prompts; if the composed still has inconsistent lighting/crop, the video model's identity embedding destabilizes and drifts within a few seconds of generated motion
Wan2.2 Animate Character Replacement & Motion Transfer
source video + character reference image -> pose/face control signals -> character-replaced videoExtracts whole-body pose and face keypoints from a source video, isolates the on-screen subject, encodes a separate reference character's identity, then fuses all three signals into a video diffusion model that replaces the on-screen performer with the reference character while preserving the original motion and expression timing. This is the current successor pattern to Animate Anyone / MimicMotion-style motion transfer.
Stack, controls & moreHide
- Example stack
- DWPose (pose/face extraction) -> Segment Anything 2 (subject mask) -> CLIP Vision clip_vision_h (identity encode) -> Wan2.2-Animate-14B (video-to-video character replacement) -> Wan 2.1 VAE decode.
- Controls
- Mix (character replacement) vs Move (animate-from-reference) mode selection; LightX2V LoRA for 4-step acceleration; UMT5-XXL text prompt; mask preserve-background toggle; pose/face-crop conditioning strength.
- Why multi-model
- A single video diffusion model has no mechanism to bind 'this exact pose/expression sequence' to 'this specific reference identity': a pose/face annotator (DWPose) produces the spatially-aligned skeleton and face-crop control signal the diffusion model cannot derive on its own, a segmentation pass isolates the subject mask, and a CLIP Vision encoder separately embeds the reference character's identity; only then does the Wan2.2-Animate-14B diffusion transformer fuse pose, mask, and identity embedding into the output. This is distinct from the tracked depth-pose-guided-v2v-restyle pattern (Lotus Depth/OpenPose -> LTX-2/Wan 2.1 Fun), which restyles the same on-screen person's appearance from a structure signal; here the CLIP-Vision-encoded reference image drives an actual identity swap of the on-screen subject for a different character, not a style change of the original person.
- Use cases
- Swapping an on-screen performer for a branded character/avatar in existing footage for ad or social content, Motion-transfer from a reference actor's performance onto a static character reference, Lip- and expression-accurate character replacement for dubbing-style localization
- Pitfalls
- Identity fidelity depends heavily on the quality and pose of the CLIP Vision reference image; fast motion or heavy occlusion in the source video degrades DWPose keypoint extraction, which propagates into warped limbs or misaligned expressions on the replaced character.
Image to video6
Stills driven into motion.
Depth-Guided Video Relighting Chain
video -> matte + depth -> per-frame relight -> temporally-stabilized videoA character is matted out of source footage, depth-mapped, then relit frame-by-frame with an IC-Light ControlNet driven by a hand-painted light map, with AnimateDiff-based motion modules used to stabilize the relit sequence.
Stack, controls & moreHide
- Example stack
- Robust Video Matting -> Depth Anything V2 -> IC-Light ControlNet (SD1.5) -> AnimateDiff
- Controls
- light map image, ControlNet strength/end-percent, ip-adapter reference for lighting style, AnimateDiff motion LoRA, seed
- Why multi-model
- IC-Light is an image relighting model with no native temporal awareness; matting isolates the subject from background contamination, depth conditioning keeps the relighting geometrically consistent, and a motion module is needed to prevent frame-to-frame lighting flicker.
- Use cases
- re-lighting talking-head footage for a new set, matching actor lighting to a virtual background, day-for-night conversions
- Pitfalls
- frame-independent relighting causes flicker without a temporal/motion module; matte edge errors bleed background light onto the subject
First-Last-Frame Interpolation (FLF2V)
image + image -> videoTwo keyframe images, a start frame and an end frame, are generated (or sourced) independently; a specialised first-last-frame-to-video model then fills in all intermediate frames, producing a coherent motion arc between the two endpoints. Chained passes can drive longer transformations by treating the last frame of one pass as the first frame of the next.
Stack, controls & moreHide
- Example stack
- FLUX.1 dev (start frame) + FLUX.1 dev (end frame) -> Wan 2.2 FLF2V interpolation -> optional RIFE 60fps pass. Alternative: FLUX.1 dev keyframes -> LTX-2.5 FLF2V (ComfyUI native workflow).
- Controls
- Positive/negative prompts to describe the in-between motion; video size and length; denoising schedule for early vs. late timesteps; chained FLF passes for longer sequences.
- Why multi-model
- Conventional i2v only anchors the start; FLF2V anchors both endpoints, so the transformation arc is determinate. The two keyframes are typically generated by a separate t2i model (FLUX, SDXL), making this a t2i -> FLF2V two-stage chain where each model handles only what it excels at.
- Use cases
- Transformation and morph effects, Matched cuts and transitions, Character pose changes
- Pitfalls
- Extreme pose or scene changes can cause incoherent mid-motion; chained passes must carefully manage extracted last-frame quality. FLF2V clips are short (up to ~5 s per pass).
Apatero: WAN 2.2 FLF First-Last Frame Video Guide - ComfyUI 2025
FLUX.1 Krea to Ovi Text-Image-to-Joint-Audio-Video (t2i2v)
text -> image -> synchronized audio + videoIn Ovi's t2i2v mode, FLUX.1 Krea first composes a starting keyframe from a text prompt; that image (plus the text prompt) then conditions Ovi, a twin-backbone diffusion model whose video branch is initialized from Wan2.2 and whose custom 5B audio branch was pretrained from scratch, which jointly denoises video and audio in a single pass to produce a synchronized clip with sound as a co-product, not a downstream add-on.
Stack, controls & moreHide
- Example stack
- FLUX.1 Krea [dev] (text -> starting image) -> Ovi t2i2v (image + text -> jointly denoised synchronized audio + video, Wan2.2-initialized video branch, custom 5B audio branch).
- Controls
- text prompt for the FLUX Krea starting image; Ovi generation mode (t2v, i2v, or t2i2v); resolution and clip duration; text prompt guiding the audio branch's soundscape/dialogue.
- Why multi-model
- FLUX.1 Krea and Ovi are separately trained, separately released model families solving different problems: FLUX.1 Krea composes a single still frame from text, while Ovi's twin-backbone architecture jointly diffuses video and audio conditioned on that image. Ovi's own repository documents the image-generation step as an optional call out to FLUX Krea rather than something Ovi's architecture performs itself. The closest tracked pattern is keyframe-drive-i2v (text-to-image keyframe -> image-to-video), which this chain matches stage-for-stage and modelType-for-modelType; the genuine novelty is that stage two here emits synchronized audio jointly with the video in one denoising pass rather than only motion, so no separate foley, TTS, or lip-sync model is needed afterward, unlike every image-to-video entry in the keyframe family. It is also distinct from soul-id-veo3-talking-character-pipeline, the tracked native-audio comparator (Ovi is self-described as "veo-3-like"): that chain is a hosted, identity-locked talking-head pipeline (Soul ID -> Veo 3), whereas this one is an open-weights joint audio-video denoiser seeded by a separate open text-to-image model, with no identity-training stage.
- Use cases
- Single-prompt synchronized dialogue or ambience clips without a separate TTS or foley pass, Open-weights, locally runnable alternative to closed joint audio-video models for quick previs, Turning a single still concept into a scene with matching native sound in one generation call
- Pitfalls
- Because video and audio are denoised jointly rather than muxed afterward, an image that conflicts with the intended audio scene (a quiet portrait paired with a loud-action prompt) can produce mismatched sound design that cannot be fixed by re-running just one branch; the audio branch is a from-scratch 5B model with less mature guidance controls than the more established Wan2.2-derived video branch.
Keyframe-Drive Image-to-Video
text -> image -> videoA text-to-image model generates one or more keyframe stills that lock composition, character, and lighting; those stills are then fed into an image-to-video model to produce motion. The i2v model inherits the keyframe's identity rather than hallucinating from text alone, giving tighter control over subject appearance and scene layout.
Stack, controls & moreHide
- Example stack
- ComfyUI KSampler + Character LoRA + IPAdapter (keyframe still) -> Kling API image-to-video (motion clip).
- Controls
- Seed consistency between t2i and i2v passes; motion prompt describing camera move or action; motion scale / CFG; clip duration.
- Why multi-model
- A single text-to-video model must simultaneously handle composition, identity, and motion; splitting the job lets a high-quality t2i model nail the still, then a separate i2v model concentrate on motion, preserving the still's identity in a way end-to-end t2v cannot.
- Use cases
- AI influencer and character video content, Product reveal shots, Character animation from concept art, Social media motion graphics
- Pitfalls
- i2v models can drift from the keyframe under strong motion prompts; over-long clips degrade identity. Short (2-6 s) clips per shot recommended. Source image quality matters more than generation settings.
Apatero: AI Influencer Image to Video - Kling AI ComfyUI Workflow 2025
Light-A-Video Progressive Light Fusion
video -> per-frame relit candidates -> temporally fused relit videoA training-free video relighting pipeline that repurposes an existing image relighting model with a Consistent Light Attention module across frames, then linearly blends source and relit appearance over time (Progressive Light Fusion) to remove flicker without any video-specific training.
Stack, controls & moreHide
- Example stack
- IC-Light (per-frame) -> Light-A-Video CLA (cross-frame attention) -> Light-A-Video PLF (temporal blend)
- Controls
- light source/text prompt for target illumination, blending ratio in PLF, base video diffusion backbone choice (e.g. VideoCrafter2, AnimateDiff)
- Why multi-model
- A still-image relighting model alone produces per-frame lighting-source drift when run independently on each frame; a second temporal-fusion mechanism operating over the video diffusion's attention and output space is required to enforce cross-frame consistency.
- Use cases
- relighting UGC or archival video without retraining, time-of-day changes for existing video clips, consistent studio-light simulation on handheld footage
- Pitfalls
- training-free temporal fusion can under-correct for large lighting-direction changes, causing residual low-frequency flicker on fast camera motion
Motion-Brush Trajectory-Controlled Image Animation
static image + hand-drawn motion paths -> per-element trajectory videoA still image is segmented into up to six user-selected regions, each assigned its own hand-painted motion trajectory (direction, curve, speed), and a commercial video diffusion model animates each region along its path while holding the rest of the frame static.
Stack, controls & moreHide
- Example stack
- Kling Motion Brush region selector -> Kling Motion Control path encoder -> Kling V2.6 Motion Control
- Controls
- brush-selected motion regions (up to 6), hand-drawn trajectory curves per element, static-lock brush for background, optional text prompt for action description
- Why multi-model
- Region selection/masking and trajectory-path encoding are handled by a separate interactive control layer distinct from the underlying video generator; the diffusion backbone alone has no mechanism to accept per-object drawn paths without an intermediate trajectory-to-motion-field encoder.
- Use cases
- product shots with a single element (e.g. bottle cap, fabric) animated in place, cinemagraph-style social ads, explainer graphics where only an icon/arrow should move
- Pitfalls
- Long or sharply curved trajectories can produce warping/limb-breaking artifacts; overlapping trajectories for adjacent regions often bleed into each other.
Multi-shot8
Storyboard to per-shot animation to a stitched sequence.
Character Sheet to Lettered Comic Panel Sequence
text prompt -> character reference sheet -> panel images -> paneled page + letteringA consistent character is first generated as a multi-view reference sheet (front/side/angled), then reused via identity+pose adapters to populate a sequence of panel images, which are finally arranged into a hierarchical comic-page layout with speech-bubble lettering added as a separate post-process.
Stack, controls & moreHide
- Example stack
- SDXL (character sheet) -> IPAdapter FaceID + ControlNet OpenPose (per-panel generation) -> ComfyUI Panels layout -> PanelForge lettering
- Controls
- character reference sheet (multi-view), IPAdapter identity weight, ControlNet pose skeleton per panel, panel-cut hierarchy (direction/count/angle), bubble placement/text
- Why multi-model
- Identity consistency across many panels, per-panel pose/composition control, and page-layout/lettering are three separate concerns: an identity adapter locks the face/design, a pose ControlNet governs per-panel staging, and a dedicated panel-layout/lettering engine (not a diffusion model) composites the final publishable page.
- Use cases
- webcomic/manga production pipelines needing consistent recurring characters, storyboard-to-comic adaptation, rapid pitch-deck comic mockups
- Pitfalls
- Identity drift creeps in over long panel sequences despite IPAdapter locking, especially with extreme pose or expression changes; automated lettering still typically needs human copyediting/placement pass before publishing.
DiffSensei MLLM Identity-Adapter Manga Generation
character reference image(s) + per-panel text + character/dialogue bounding boxes -> customized manga panelsA multimodal LLM (SEED-X), fine-tuned as a text-compatible identity adapter, reads one or more character reference images and adjusts their expression, pose, and action to match each panel's text cues; those adapted features feed an SDXL diffusion backbone through masked cross-attention alongside per-panel character and dialogue bounding boxes, producing layout-aware, identity-consistent manga pages.
Stack, controls & moreHide
- Example stack
- IP-Adapter-Plus-SDXL (character feature extractor) + SEED-X MLLM (LoRA + resampler, identity/pose adaptation) -> SDXL (masked cross-attention panel diffusion with layout + dialogue boxes).
- Controls
- per-panel text prompt; number and choice of character reference images; character bounding box per panel; dialogue bounding box per panel; LoRA/resampler fine-tuning stage.
- Why multi-model
- SDXL alone cannot hold multiple named characters' identities across panels while flexibly re-posing and re-expressing them per prompt; DiffSensei chains a separate SEED-X-initialized MLLM (updated via LoRA and a resampler) that performs the semantic identity-to-panel adaptation, an IP-Adapter-Plus-SDXL-initialized resampler that extracts the character features, and the SDXL diffusion generator that renders the panel. This differs from the tracked character-sheet-to-lettered-comic-panels pattern (IPAdapter FaceID + a static OpenPose ControlNet, with panel layout and lettering handled by separate non-generative ComfyUI tools): here the MLLM itself performs per-panel semantic pose/expression adaptation of the character before diffusion, rather than injecting a fixed identity embedding under a hand-picked pose skeleton.
- Use cases
- Multi-character manga/webtoon page generation with per-panel pose and expression control from reference art, Customized manga generation from a small set of character reference images, Layout-aware panel generation with dialogue-bubble placement built into the conditioning rather than added as a post-process
- Pitfalls
- Panel generation depends on correctly assigned per-panel character and dialogue bounding boxes; because the identity adapter's cross-attention is masked by these boxes, an incorrect box placement misaligns a character's features onto the wrong panel region.
LLM-Planned Multi-Scene POV Video Cascade (fal Live-in-Scene)
text->text(x6 prompts)->image(x6)->video(x6 w/audio)->merged videofal.ai's 'Live-in-Scene' workflow template takes a film name, has an LLM scene planner break it into 6 first-person POV scene prompts, generates a keyframe image for each with Nano Banana Pro, animates each into a 4-second clip with generated ambient audio via Seedance 1.5 Pro image-to-video, then merges all six clips into one ~24-second video.
Stack, controls & moreHide
- Example stack
- OpenRouter Gemini 2.5 Flash (scene planner LLM) -> Nano Banana Pro (per-scene text-to-image, x6) -> Seedance 1.5 Pro i2v (per-scene animate+audio, x6) -> fal ffmpeg-api merge-videos
- Controls
- film name input; LLM scene planner enforces one clear action per scene and ambient-only audio (no dialogue), sequencing scenes toward a narrative climax; ffmpeg merge stitches final order
- Why multi-model
- An LLM planning model drives per-shot prompt generation, a separate text-to-image model renders each keyframe, a separate image-to-video model (with native audio generation) animates each shot, and an ffmpeg merge step assembles the final cut, all as one chained endpoint.
- Use cases
- generating short immersive POV trailers/homages to a film's setting, rapid multi-scene sizzle reels from a single text seed, prototyping episodic short-form video from narrative prompts
- Pitfalls
- LLM-authored per-scene prompts can drift in visual continuity between the 6 independently generated keyframes since there is no shared identity/style-lock adapter across scenes; distinct from multi-shot-narrative-stitch (Qwen Image Edit + Wan, per-shot keyframe->i2v->concatenate) and storyboard-grid-to-multishot-video (single Nano Banana 2 storyboard grid then Kling 3.0 multi-shot animate) because here an LLM planner generates independent per-scene prompts and each scene is a separately generated keyframe+clip with native generated audio, not a single grid split into shots
fal-ai-community/skills WORKFLOWS.md reference (fal.ai Workflow Templates, Live-in-Scene)
Multi-Shot Narrative Stitch
text -> image (per shot) -> video (per shot) -> stitched videoA narrative or storyboard is broken into individual shots; a reference-conditioned image model generates a consistent keyframe per shot, an image-to-video model animates each keyframe into a short clip, and all clips are concatenated in order to form a full multi-shot sequence. Character and style consistency is maintained by conditioning each shot on a shared reference.
Stack, controls & moreHide
- Example stack
- Qwen Image Edit (scene stills, reference-conditioned) -> Wan 2.2 i2v (per shot) -> concatenate into final cut.
- Controls
- Per-shot motion and camera prompts; shared character reference weight; clip duration per shot; concatenation order and transition type.
- Why multi-model
- No single t2v model can reliably maintain character identity and narrative continuity across multiple shots. Decomposing into per-shot i2v passes with a shared reference adapter is the only practical way to hold consistency while exercising per-shot camera and motion control.
- Use cases
- Short-form narrative films, Brand story ads, Game cinematics, Automated social video
- Pitfalls
- Identity drift accumulates across shots without a strong reference signal; stitching transitions are abrupt if not bridged; inconsistent lighting between shots breaks cohesion.
RunComfy: Create Coherent Scenes - Qwen Image Edit & Wan 2.2 ComfyUI Workflow
Storyboard Grid to Multi-Shot Video
text concept -> storyboard image grid -> multi-shot videoA dedicated image model renders the full shot sequence as consistent staged panels (sometimes a single grid image) so composition and character silhouette are locked before any motion exists; a video model then animates the panels together, using them as reference images plus a timestamped multi-shot prompt to decide camera work and motion per shot.
Stack, controls & moreHide
- Example stack
- Nano Banana 2 (storyboard grid + detail fix) -> Kling 3.0 (multi-shot, omni reference)
- Controls
- storyboard panel order/count, per-shot text description, reference-image weighting (omni reference), timestamped multi-shot prompts, anchor-frame selection
- Why multi-model
- Image models hold composition, lighting, and character silhouette across many panels in one pass; video models are better at motion and camera dynamics but drift on composition if asked to invent both from text alone. Splitting what it looks like from how it moves across two model families is what stabilizes multi-shot output.
- Use cases
- Ad and trailer previz, Narrative shorts with consistent characters across scenes, Product commercials with multiple staged shots
- Pitfalls
- Consistency comes from a strong anchor frame, not the grid itself; a weak or ambiguous panel drifts its downstream shot, and product/face detail lost during grid rendering silently carries into the animated shot unless fixed first.
Nano Banana 2 and Kling 3.0: Cinematic AI Ad Workflow (2026)
StoryDiffusion Consistent Storyboard
text -> image sequence (+ video)Generate a multi-panel comic or storyboard where the same characters stay consistent across shots using consistent self-attention across a batch, then optionally animate the transitions between frames.
Stack, controls & moreHide
- Example stack
- StoryDiffusion (SDXL) -> Sora/Kling image-to-video for transitions.
- Controls
- Character reference prompts; number of panels; shared-attention on/off; motion strength for transitions.
- Why multi-model
- Cross-shot consistency needs attention shared across generations, plus a motion model to bridge frames; a single image pass cannot enforce identity across panels.
- Use cases
- Comic and graphic-novel pages, Ad storyboards, Children's book illustration
- Pitfalls
- Shared attention can homogenize distinct characters; long sequences drift as the consistency window is bounded.
Suno Song to Flux-Runway Generative Music Video
text -> audio -> image -> videoA text prompt drives Suno to write and generate a full song, then per-scene image stills are generated with Flux and animated into short clips with Runway, which are composited against the finished track into a music video.
Stack, controls & moreHide
- Example stack
- n8n + Suno API + Flux (BlackForest Labs/RapidAPI) + Runway ML + Creatomate
- Controls
- song/style prompt, per-scene image prompts, clip duration
- Why multi-model
- Chains three distinct generative-media models across two modalities: Suno generates the music track itself (not just narration), Flux generates the scene artwork, and Runway generates video motion from those stills; a rendering step composites them. This differs from the baseline's beat-synced music-video pattern (which pairs a generated song with a beat-synced *editing* tool over existing footage) because here every visual asset, not just the audio, is freshly generated per scene.
- Use cases
- AI music video generation, social lyric videos, artist promo clips from a single prompt
- Pitfalls
- per-scene image/video costs compound quickly across a full song length; visual continuity between scenes is not enforced by any identity/consistency model
n8n workflow template: Generate AI songs + music videos using Suno API, Flux, Runway and Creatomate
Text-to-Narrated Documentary Video (fal Documentary)
text->text(script/shots)->image(x6)->video(x12)->speech->merged narrated videofal.ai's 'Documentary' workflow template turns a text topic into a narrated short documentary: an LLM drafts the story/shot script, Dreamina v3.1 generates six scene images, Seedance 1/pro animates them into video clips, ElevenLabs eleven-v3 synthesizes voiceover narration, and ffmpeg merges audio with video and concatenates all shots into one final cut.
Stack, controls & moreHide
- Example stack
- LLM (fal-ai/any-llm, script + shot breakdown) -> Dreamina v3.1 text-to-image (x6 scenes) -> Seedance v1 pro image-to-video (per-scene clips) -> ElevenLabs eleven-v3 TTS narration -> fal ffmpeg-api merge-audio-video + merge-videos
- Controls
- text topic/prompt input drives the LLM script; narration voice via ElevenLabs eleven-v3; ffmpeg merge-audio-video then merge-videos assembles the final concatenated cut
- Why multi-model
- Combines an LLM scriptwriter, a text-to-image model, an image-to-video model, and a text-to-speech model, each a distinct hosted model, glued together by ffmpeg merge/concat utilities into a single narrated-video pipeline.
- Use cases
- turning a topic/prompt into a narrated explainer or mini-documentary video, auto-generating voiced short-form video essays, rapid narrated video drafts for social/education content
- Pitfalls
- narration timing must be aligned to clip lengths via the merge-audio-video step, mismatched pacing between the 6 generated shots and the continuous voiceover can produce dead air or overlap; distinct from text-to-song-beat-synced-music-video (Suno/Udio song-driven, beat-synced cuts, no spoken narration) and llm-orchestrated-talking-scene-assembly (Qwen-Image/Edit + Wan2.2 + InfiniteTalk + MiniMax, built around a talking on-camera subject) because this chain is narration-over-B-roll documentary style with ElevenLabs TTS and Dreamina/Seedance visuals, no talking-head lipsync and no music generation
Audio driven14
Audio or speech driving visuals (lip-sync, talking head).
Chain-of-Thought Reasoning Foley Editing
video -> MLLM reasoning trace -> object-targeted audio -> edited audioA multimodal LLM first reasons step-by-step about a video's objects, actions, and acoustic environment, producing a structured chain-of-thought plan that then steers a separate audio foundation model to generate foley, and the same MLLM can re-reason to guide targeted natural-language edits of specific sound elements.
Stack, controls & moreHide
- Example stack
- ThinkSound MLLM reasoning stage -> AudioCoT-guided audio foundation model -> instruction-guided re-edit pass
- Controls
- object-click targeting for interactive refinement, natural-language edit instructions, reasoning trace granularity
- Why multi-model
- The audio synthesis model has no native reasoning capacity to decide which objects should sound like what or when; a multimodal LLM is needed to decompose the scene into an interpretable plan (AudioCoT), and a distinct audio diffusion/foundation model executes that plan into actual waveforms, with the two iterating for object-level and instruction-level refinement.
- Use cases
- Iteratively directed foley for film post-production, Targeted SFX fixes without regenerating the whole soundtrack, Sound design review tools with explainable reasoning traces
- Pitfalls
- Reasoning quality bottlenecks final audio quality; a wrong CoT plan (misidentified object or action) propagates directly into an incorrect sound choice.
DALL-E Generated Character to ElevenLabs-Hedra Talking Podcast
text -> image -> audio -> videoA script is written by GPT-4, a wholly new photorealistic character portrait is generated from scratch with DALL-E, ElevenLabs synthesizes a voice reading the script, and Hedra animates the generated portrait's face to lip-sync and express in time with that audio.
Stack, controls & moreHide
- Example stack
- n8n + OpenAI GPT-4 + DALL-E + ElevenLabs + Hedra
- Controls
- character description prompt, script topic, voice selection, expression intensity
- Why multi-model
- Chains a script-writing LLM with three distinct generative-media models: DALL-E (text-to-image) creates the character portrait itself, ElevenLabs (text-to-audio) synthesizes the voice, and Hedra (audio-to-video) animates the face. This is structurally distinct from the baseline's voice-clone-to-talking-head pattern, which assumes an existing photo and a cloned voice from a reference sample; here both the character image and the voice are generated from nothing (no reference photo, no voice clone), making the character-creation step itself a generative-media stage rather than an input.
- Use cases
- novelty talking-character podcast clips, generated-mascot social content, no-reference-photo talking avatar content
- Pitfalls
- DALL-E-generated faces can drift between regenerations with no identity lock across episodes; Hedra's expression range is limited on non-frontal generated portraits
n8n workflow template: Create Animated Baby Podcast Videos with GPT, DALL-E, ElevenLabs and Hedra
Image to Scripted Voice-Clone UGC Video
image -> text (script) -> audio (cloned voice) -> videoA single product/character image is turned into a short lip-synced UGC-style video: an LLM writes a performance script and scene description from the image, ElevenLabs clones or selects a voice to synthesize the narration, then LTX-2.3 generates the video with native lip sync driven by that audio.
Stack, controls & moreHide
- Example stack
- Gemini (script) -> ElevenLabs Instant Voice Clone (narration) -> LTX-2.3 (video + native lipsync)
- Controls
- source image, script/expression tags from the LLM stage, voice selection or cloned voice sample, scene description prompt, LTX-2.3 duration and lipsync strength
- Why multi-model
- No single model both writes a grounded performance script from an image, clones/synthesizes a matching voice, and renders a lip-synced video; the chain needs an image-conditioned script generator, a separate voice-cloning TTS model, and a distinct audio-conditioned video generator.
- Use cases
- product explainer videos from a single photo, testimonial-style UGC ads, social media talking posts without filming
- Pitfalls
- LTX-2.3's native lipsync can drift on longer scripts or fast speech; cloned voice quality depends on the reference sample length and clarity, and mismatched scene description vs. script tone produces awkward framing
ComfyUI official template: Generate UGC Video With Voice Clone
LLM-Orchestrated Talking-Scene Assembly
character brief -> keyframes -> video + talking-head -> voiced final cutA general-purpose LLM first drafts the dialogue, per-shot prompts, and voice casting from a character brief; an image model then generates and edits successive scene keyframes, a video model animates each keyframe into a shot stitched via last-frame continuation, a talking-head model adds lip-synced performance and face consistency, and a voice model supplies the matching speech track.
Stack, controls & moreHide
- Example stack
- Qwen-Image -> Qwen-Image-Edit -> Wan 2.2 I2V -> InfiniteTalk -> MiniMax voice synthesis
- Controls
- character brief/persona, per-shot LLM-authored prompts, LoRA selection, last-frame continuation between segments, voice profile, audio-duration-driven video timing
- Why multi-model
- No single video model renders and edits a sequence of scene keyframes, animates those into shots, drives lip-synced facial performance, and synthesizes a timed voice track; each is a distinct trained model family chained by an LLM-authored shot plan that keeps them in sync via shared scene/character state.
- Use cases
- AI-hosted talking-scene shorts with dialogue, Synthetic interview/skit production, Multi-scene narrative shorts with lip-synced characters
- Pitfalls
- Because timing is driven by measured audio duration feeding back into video segment length, a mismatch anywhere in the LLM-authored shot plan propagates through every downstream stage and only surfaces as broken sync in the final assembly.
LTX-2 Audio-Conditioned Two-Stage Generation
audio + reference image + prompt -> base audio-video -> upscaled audio-videoLTX-2's dedicated audio-to-video pipeline generates a synchronized base clip directly conditioned on an input audio file (not text), jointly denoising audio and video in one diffusion pass, then a separate spatial-upscaler checkpoint sharpens the result in a second chained pass.
Stack, controls & moreHide
- Example stack
- ltx-2.3-22b-dev (A2VidPipelineTwoStage) -> ltx-2.3-spatial-upscaler-x2
- Controls
- input audio track (dialogue/music pacing drives motion energy), reference/face image, upscaler factor (x1.5 or x2), prompt for scene description
- Why multi-model
- The base diffusion transformer that jointly denoises audio+video latents is trained for temporal/audio coherence, not resolution; production output requires a distinct spatial-upscaler checkpoint chained afterward. These are two separately downloadable model weights, not one model run twice.
- Use cases
- Podcast/avatar audio-driven scenes, Voice-driven talking clips, Music-synced motion generation
- Pitfalls
- Because audio and video are denoised jointly rather than muxed afterward, mismatched or noisy input audio directly corrupts motion quality (not just lip sync), and re-running only the upscale stage cannot fix a bad base generation.
Lightricks/LTX-2 official repository (A2VidPipelineTwoStage, spatial upscaler checkpoints)
MiniMax H3 Context-IR to Base to Regenerate-2K
text/image/video/audio references -> refined context -> synchronized audio+video (768p) -> upscaled audio+video (2K)MiniMax H3 splits reference-to-video generation into three separately-hosted-or-downloadable stages: a hosted instruction-refinement model converts complex multimodal inputs into a Context Intermediate Representation, a locally-deployable Ref2VA (or FL2VA) diffusion checkpoint jointly generates synchronized audio and video at 768p from that context, and a hosted regeneration model feeds the 768p result plus the original context back through the system to produce a 2K output.
Stack, controls & moreHide
- Example stack
- H3-Context-IR (hosted instruction refinement) -> MiniMax-H3 Base Ref2VA (local, SGLang-deployed, 768p synchronized audio+video) -> H3-Regenerate-2K (hosted, context-conditioned 2K regeneration).
- Controls
- choice of FL2VA (text/first-last-frame) vs Ref2VA (multi-image/video/audio reference) checkpoint; number and type of reference assets fed to Context-IR; whether to run the hybrid local+hosted path for full 2K output or stop at the local 768p H3-Base result; native canvas capped at 768px short edge (max 768x1344) for the local stage.
- Why multi-model
- MiniMax's own H3 model card documents three named, separately versioned modules rather than one model run at different settings: H3-Context-IR (hosted-only instruction-refinement layer, not open-sourced), H3-Base (an open-weight Omni Transformer with distinct FL2VA and Ref2VA checkpoints that ships locally), and H3-Regenerate-2K (a hosted-only super-resolution regeneration model, also not open-sourced). The local H3-Base checkpoint cannot reach the documented 2K quality on its own; the card explicitly instructs users to 'combine a locally deployed SGLang service with the official H3-Context-IR and H3-Regenerate-2K APIs' to reproduce the full pipeline. This is distinct from the tracked ltx2-audio-conditioned-two-stage-upscale pattern (a single LTX-2.3 base checkpoint plus one spatial-upscaler checkpoint, no instruction-refinement stage) and from flux-krea-ovi-t2i2v-joint-audio-video (a separate open text-to-image model seeding an open joint audio-video diffuser, no context-refinement or regeneration stage at all).
- Use cases
- Reference-driven brand/product video where multiple images, a driving clip, and reference audio must be reconciled into one coherent shot, Localized or dubbed video that needs an audio reference plus a face/body reference reconciled before generation, High-resolution short-form video production where the free-tier 768p local model is generated first and only promising takes are sent to the paid 2K regeneration step
- Pitfalls
- The full 2K path depends on MiniMax's hosted Context-IR and Regenerate-2K APIs, which are 'not yet open-sourced,' so a fully local deployment is capped at 768p; the local H3-Base checkpoint 'uses full attention' rather than native sparse attention, making it computationally heavy at longer durations or higher resolutions even before the hosted regeneration step is invoked.
MultiTalk Multi-Person Dialogue Video
reference image + multi-stream audio + text prompt -> multi-person conversational videoGiven a reference image containing multiple people, separate audio streams per speaker, and a scene prompt, the pipeline generates a video where each character's lips and turn-taking match their own audio stream, driven by a large video diffusion backbone conditioned on per-person audio embeddings.
Stack, controls & moreHide
- Example stack
- Kokoro-82M (TTS for each character line) -> Chinese-Wav2Vec2 (audio embedding) -> MultiTalk / Wan 2.1-I2V-14B (multi-person audio-driven video)
- Controls
- per-speaker audio-person binding (L-RoPE), resolution (480p/720p), streaming vs clip mode, TeaCache acceleration, LoRA for style
- Why multi-model
- A single-speaker lip-sync model cannot bind multiple simultaneous audio streams to the correct face in a shared frame; MultiTalk needs a speech encoder to extract per-speaker embeddings, a Label Rotary Position Embedding scheme to bind each audio stream to its person, and a large video diffusion model to render coherent multi-person motion and interaction.
- Use cases
- Multi-character animated dialogue scenes, Podcast-to-video with two animated hosts, Cartoon character conversation generation
- Pitfalls
- Audio-person binding can swap identities when reference image faces are visually similar or poorly separated in frame; long streaming generations can drift in lip accuracy.
Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (arXiv:2505.22647)
OmniHuman Audio-to-Talking-Human
image/video + audio -> human-animation videoA one-stage model animates a full human (not just the face) from audio, scaled up by conditioning on portrait, pose, and body signals together so motion and lip-sync emerge from a single network.
Stack, controls & moreHide
- Example stack
- TTS voice clone -> OmniHuman -> 2x video upscale.
- Controls
- Source portrait or video; audio; body and pose conditioning; motion intensity.
- Why multi-model
- Full-body co-speech motion, gestures, and lip-sync run on different timescales; OmniHuman unifies them but still pairs with speech synthesis and upscaling in a real pipeline.
- Use cases
- Presenter and spokesperson videos, Multilingual dubbing with motion, Social-content avatars
- Pitfalls
- Hands and fine gestures can smear under fast motion; low-resolution inputs upscale with artifacts.
SadTalker Audio-to-Portrait
portrait + audio -> talking-head videoDrive a single portrait photo with speech by estimating realistic 3D motion coefficients (head pose and expression) from the audio, then rendering a lip-synced talking-head clip.
Stack, controls & moreHide
- Example stack
- SadTalker (or a commercial talking-head API) -> upscale.
- Controls
- Source portrait; audio clip; expression and pose amplification; stillness of background.
- Why multi-model
- Speech-to-motion mapping and photo-real rendering are separate problems; a dedicated motion-coefficient model feeds a renderer that a generator alone cannot.
- Use cases
- Narrator and explainer avatars, Localization and dubbing, Talking-photo gifts
- Pitfalls
- Extreme head turns break the single-image 3D assumption; fast speech outruns the expression model.
Text-to-Song Beat-Synced Music Video
lyrics + style prompt -> generated song -> beat/lyric analysis -> synced visual cutsLyrics and a style prompt are turned into a full song via a text-to-music model, then a separate analysis+generation stage detects tempo, beats, and lyric timing from the audio to drive cut points, transitions, and karaoke-style captions over AI-generated or stock visuals.
Stack, controls & moreHide
- Example stack
- Suno (lyrics+prompt -> song) -> Freebeat / One More Shot AI (beat detection + beat-synced scene generation)
- Controls
- beat-detection sensitivity, cut density per BPM, karaoke caption styling, aspect ratio export presets (16:9/9:16/1:1)
- Why multi-model
- The music generator produces audio only and has no visual or editing capability; a distinct beat/onset-detection and lyric-alignment stage must analyze the finished waveform, and a video assembly/generation stage must consume that timing data to place cuts and captions in sync.
- Use cases
- Independent artists generating music videos from a single song, Social-platform lyric videos with karaoke captions, Rapid pitch/demo music videos for unreleased tracks
- Pitfalls
- Beat detection can misfire on syncopated or tempo-shifting tracks, causing cuts that feel arbitrary; lyric alignment drifts on ad-libbed or heavily processed vocals.
One More Shot AI, beat-synced music-video generator (product docs)
Text/Pose-to-Video Avatar to Real-Time Lip-Sync
text/image/pose prompt -> generated avatar video -> real-time audio-driven lip-syncA text-to-video (or image-to-video / pose-to-video) model first generates the avatar's body motion and scene, and a separate real-time latent-space lip-sync model then re-renders just the mouth region at 30+ FPS to match streaming audio, decoupling body/scene generation from low-latency mouth synchronization.
Stack, controls & moreHide
- Example stack
- MuseV (text/image/pose -> base avatar video) -> MuseTalk (streaming audio -> real-time lip-synced output)
- Controls
- generation mode (text/image/pose-to-video) for the base clip, streaming audio chunk size, target FPS, latent inpainting region size
- Why multi-model
- The upstream video generator is optimized for motion and scene diversity, not phoneme-accurate low-latency lip sync, while the lip-sync model is optimized for real-time inpainting of the mouth region only and cannot generate body motion or backgrounds itself; combining them lets each specialize.
- Use cases
- Live streaming virtual presenters/VTubers, Real-time conversational avatar assistants, Low-latency dubbing of generated avatar video into a live audio feed
- Pitfalls
- Base video's head pose/angle range must stay within what the lip-sync inpainting model was trained on, or mouth-region inpainting artifacts appear during large head turns.
MuseTalk: Real-Time High Quality Lip Synchronization with Latent Space Inpainting
TTS-Driven Video-Diffusion Talking Character
text script -> synthesized speech -> generated base video -> lip-synced final videoA text script is converted to speech by a local TTS engine, an image-to-video model animates a character from a still reference into a base video clip, and a lip-sync model then re-syncs that generated video's mouth movements to the TTS audio, producing a fully text-driven talking character without any recorded voice or footage.
Stack, controls & moreHide
- Example stack
- Kokoro TTS (text -> speech) -> LTX-Video (image -> base video) -> Sonic or LatentSync (audio -> lip-synced final video)
- Controls
- voice selection/language in TTS, video model choice for quality vs speed tradeoff, lip-sync model choice (Sonic quality vs LatentSync speed), lips_expression intensity
- Why multi-model
- No single model both generates a moving video from a still image AND produces accurate phoneme-level lip motion; the pipeline needs a TTS model for speech, an image-to-video diffusion model to create believable body/head motion, and a dedicated audio-conditioned lip-sync model as a final pass because the video model's own lip movements are not driven by the actual phonemes.
- Use cases
- Fully synthetic talking-character shorts with no source footage or recorded voice, Localizing generated character videos into multiple languages/voices, Rapid iteration on scripted avatar content
- Pitfalls
- Compounding errors across three models: TTS prosody, base-video head motion, and lip-sync fidelity errors stack, and swapping the middle video-generation stage changes how well the final lip-sync pass locks on.
Video-to-Audio Foley Synthesis
video -> text prompt -> synced audioA silent generated (or real) video plus an optional text caption is fed into a joint video-audio-text diffusion model that synthesizes semantically matched, temporally synchronized sound effects and ambience directly from visual motion.
Stack, controls & moreHide
- Example stack
- Wan 2.1 (text-to-video) -> MMAudio (video+text -> audio) -> mux audio onto video
- Controls
- text caption strength, classifier-free guidance scale, audio duration alignment to video FPS, synchronization module weight
- Why multi-model
- The video generator has no concept of sound; a dedicated video-to-audio model trained on paired video-audio-text data is required to infer materials, impacts, and timing from pixels and produce synchronized waveforms, and it in turn depends on a video encoder distinct from the audio decoder.
- Use cases
- Auto-Foley for silent AI-generated b-roll, Sound design for product demo videos, Rapid SFX drafts for game trailers
- Pitfalls
- Struggles with style continuity across cuts (no reference-audio injection pathway) and can hallucinate plausible-but-wrong sound sources for ambiguous visual motion.
Voice-Clone TTS to Talking-Head Lip Sync
text + reference audio + reference face -> audio -> talking-head videoA zero-shot voice-cloning TTS model synthesises speech from a text script in the voice of a reference speaker; that generated audio is then fed into a separate lip-sync or talking-head model which drives realistic facial animation on a reference image or video, producing a complete avatar video without studio recording.
Stack, controls & moreHide
- Example stack
- Tortoise TTS (voice clone) -> Wav2Lip (lip sync) -> GFPGAN upscaler. Commercial equivalent: ElevenLabs voice clone -> HeyGen Avatar IV talking head.
- Controls
- TTS: speaker reference audio length and quality, speaking rate, emotional guidance. Lip-sync: face-crop padding, sync confidence threshold, video fps. Enhancement: denoise strength.
- Why multi-model
- Voice cloning (modelling prosody and speaker timbre from a reference clip) and talking-head animation (modelling facial dynamics from audio signals) are separate learned problems with different architectures and training data; chaining specialist models gives independent control over voice and face, and allows either to be swapped without retraining the other.
- Use cases
- Personalised AI avatar video from scripts, Video dubbing in a target speaker's voice, Automated marketing or training video production, Accessibility narration with a familiar voice
- Pitfalls
- Temporal jitter when generated audio phoneme boundaries mismatch the lip-sync model's training distribution; unnatural eye-blink and head-pose when the reference face is a still image; voice cloning requires clean reference audio (background noise degrades timbre matching).
3D16
Image or text to 3D mesh, texture, and multi-view chains.
Concept Image to Rigged, Motion-Ready Character (Scenario Uthana Chain)
text->image->3D mesh->rigged 3D->animated 3DScenario's platform workflow chain generates a character concept image with GPT Image 2, converts it to a 3D mesh with Hunyuan3D 3.1, runs the mesh through Uthana Character Rigging to auto-place a bipedal skeleton and skin weights in under 30 seconds, then animates the rigged character with Uthana Video-to-Motion (from a reference clip) or Uthana Text-to-Motion (from a text description).
Stack, controls & moreHide
- Example stack
- GPT Image 2 (text-to-image concept) -> Hunyuan3D 3.1 (image-to-3d mesh) -> Uthana Character Rigging (auto-rig: skeleton + skin weights, no valid modelType slot) -> Uthana Video-to-Motion or Text-to-Motion (animate, no valid modelType slot)
- Controls
- T-pose/A-pose mesh required (OBJ/GLB/FBX) for the auto-rigger to read shoulder line, hips, and spine cleanly; downstream motion source is either a reference video clip (Video-to-Motion) or a text description (Text-to-Motion)
- Why multi-model
- Chains a text-to-image model and a separate image-to-3D mesh model as the two genuinely distinct generative-media stages; the downstream Uthana auto-rigger and motion-application steps are utility/retrieval operations (skeleton fitting, library motion application), not additional generative models, but are included because they differentiate the full-chain vendor stack from the existing Tripo + Mixamo pattern.
- Use cases
- concept-to-animated game character without manual rigging, rapid prototyping of animation-ready NPCs from a single prompt, reusing one auto-generated rig across multiple motion passes
- Pitfalls
- auto-rigging requires a clean humanoid mesh in a T/A-pose with feet on the ground, non-bipedal or non-canonical-pose meshes from the upstream 3D step can fail to rig correctly; closest existing pattern is mesh-to-rig-to-animation-chain (Tripo Studio mesh + Mixamo auto-rig + animation library) but this chain uses an entirely different vendor stack throughout (GPT Image 2 concept generation, Hunyuan3D 3.1 reconstruction, and Uthana's auto-rigger plus Uthana's own video/text-to-motion generators instead of a Mixamo animation library), and adds an upstream text-to-image concept-generation stage that the Tripo+Mixamo chain does not include
Generate-then-Decompose PBR Material (CHORD)
text -> image (flat-lit texture) -> PBR map set (base color, normal, height, roughness, metalness)A text prompt is first turned into a seamless, tileable flat-lit texture by an image diffusion model, then Ubisoft La Forge's CHORD model decomposes that single texture into a full physically-based-rendering material map set via SVBRDF estimation.
Stack, controls & moreHide
- Example stack
- Z-Image-Turbo (text to tileable texture) -> Ubisoft CHORD (SVBRDF decomposition) -> ChordNormalToHeight (height map)
- Controls
- text prompt for the base texture, optional lineart/height conditioning input, group bypass toggles to run texture-gen and CHORD stages independently or feed in a user-provided texture directly to CHORD
- Why multi-model
- Generating a plausible texture and estimating physically accurate SVBRDF channels (base color, normal, height, roughness, metalness) from it are different learned tasks; CHORD is a dedicated decomposition/estimation model chained after a separate text-to-image generator rather than one model doing both.
- Use cases
- rapid PBR material authoring for game engines and DCC tools, converting AI-generated textures into production-ready material sets, iterating on material ideas without manual SVBRDF capture
- Pitfalls
- CHORD is released under a research-only license limiting production use; decomposition quality depends on the flatness/lighting of the generated source texture, poorly lit or shaded input textures produce inaccurate roughness/metalness estimates
Generated Mesh to Rigged, Animated Character
text/image -> 3D character mesh -> auto-rigged skeleton + skin weights -> retargeted animationTurns a prompt or reference image into a game-ready animated character by generating a T-posed base mesh with one model, automatically detecting a skeleton and computing skin weights with a dedicated auto-rigging service, then applying that service motion-capture library onto the resulting rig.
Stack, controls & moreHide
- Example stack
- Tripo Studio (T-pose mesh) -> Blender cleanup -> Mixamo (auto-rig + skin weights) -> Mixamo animation library -> Unity/Unreal
- Controls
- T-pose generation toggle, mesh cleanup (orientation, vertex merging), skin-weight smoothing, animation clip selection
- Why multi-model
- Image/text-to-3D generators produce static geometry only, with no joints or deformation; auto-rigging services are trained specifically to infer skeletal topology and per-vertex skin weights from arbitrary mesh geometry, and separately hold the animation clip library. Shape generation and rigging/animation are distinct systems chained through a mesh export/cleanup step.
- Use cases
- Indie game NPC pipelines from concept art to playable character, Rapid prototyping of animated avatars, Batch-rigging libraries of AI-generated characters
- Pitfalls
- Auto-rigging struggles with non-humanoid or heavily stylized meshes and can produce broken skin weights at joints; meshes that skip cleanup (non-manifold, wrong facing) fail upload; library clips often need per-character proportion adjustment.
How to Rig an AI-Generated Character for Mixamo: Auto-Rigging Guide (Tripo AI)
Geometry-Aware PBR Material Distillation
untextured 3D mesh + text -> lit multi-view renders -> decomposed PBR material mapsTakes an existing bare mesh and generates physically-based material maps (albedo, roughness, metallic) matched to its geometry, using a diffusion model conditioned on the mesh's own geometry and a chosen lighting environment, then distilling the shaded outputs into decomposed BRDF parameters via inverse rendering.
Stack, controls & moreHide
- Example stack
- Hunyuan3D 2.1 (bare mesh) -> DreamMat diffusion -> DreamMat PBR distillation -> Blender/Unreal import
- Controls
- environment light choice/rotation during conditioning, number of viewpoints rendered, per-material text prompt, distillation iteration count, mesh UV parameterization
- Why multi-model
- An image-to-3D model only produces baked-in RGB texture or bare geometry; it cannot separate real material properties from shading. DreamMat needs a geometry- and light-aware diffusion model to render plausible shaded views under controlled illumination, plus a separate inverse-rendering distillation to decompose those views into albedo/roughness/metallic maps free of baked-in lighting.
- Use cases
- Relightable game/film asset texturing, Converting scanned or generated bare meshes into engine-ready PBR assets, Batch material re-skinning of an existing mesh library
- Pitfalls
- Without light-aware conditioning, shading bakes into albedo and looks wrong under new lighting; multi-view inconsistency causes seams at UV boundaries; distillation is compute-heavy per asset (minutes, not seconds).
Geometry-Then-Texture 3D Asset Pipeline
image -> 3D mesh -> textured 3D assetA first model generates an untextured 3D geometry (mesh / TSDF) from a single image or text prompt; a second, separate diffusion model synthesises high-resolution texture maps conditioned on the produced geometry, yielding a fully textured, PBR-ready 3D asset.
Stack, controls & moreHide
- Example stack
- Step1X-3D geometry stage (VAE-DiT TSDF) -> Step1X-3D texture stage (SD-XL-fine-tuned multi-view generator) -> mesh export. Alternative: Hunyuan3D-DiT shape -> Hunyuan3D-Paint texture -> GLB.
- Controls
- Input image quality and field of view; geometry resolution cap; texture diffusion guidance scale; number of synthesis views; UV unwrap quality; PBR channel selection (albedo, metallic, roughness, normal).
- Why multi-model
- Geometry generation and texture synthesis have conflicting objectives: the shape model optimises for accurate 3D form while the texture model optimises for view-consistent, photorealistic appearance conditioned on that form. Splitting into two specialist models lets each solve its own problem; the texture model explicitly conditions on the finished geometry rather than trying to hallucinate both simultaneously.
- Use cases
- Game asset prototyping from concept art, Product visualisation from a single photo, 3D printing prep from reference images, Rapid 3D scene population for VFX
- Pitfalls
- Texture seams and occlusion holes where multi-view coverage is sparse; UV inpainting quality degrades for highly concave surfaces; geometry errors propagate into texture conditioning (garbage-in, garbage-out); PBR decomposition may bake lighting into albedo.
Hunyuan3D-2 Image-to-Mesh
image -> textured meshA two-stage shape-then-texture pipeline: a diffusion model produces a clean 3D shape from a single image, then a second model bakes PBR textures onto it for a production-ready mesh.
Stack, controls & moreHide
- Example stack
- Hunyuan3D-2 (shape + texture) -> remesh.
- Controls
- Input image; view conditioning; texture resolution; seed.
- Why multi-model
- Shape and texture have different distributions and resolutions; splitting them into a shape model and a texture model yields far cleaner meshes than a single joint generator.
- Use cases
- Printable miniatures, Game props, Product 3D shots
- Pitfalls
- Back and underside faces are guessed from a single view; complex topology retopologizes poorly.
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets (arXiv:2501.12202)
LocalMesh Single-Photo Splat-to-PBR-Mesh
single photo -> 3D Gaussian splat -> carved mesh -> PBR-textured meshA single photograph is lifted directly to a navigable 3D Gaussian splat by a feed-forward image-to-splat model, then a bundled local stack of a monocular geometry model, an inpainter, and material-decomposition models carves that splat into a textured, PBR-mapped quad mesh, all running on the user's own GPU with nothing uploaded to a server.
Stack, controls & moreHide
- Example stack
- TripoSplat (photo -> splat) -> MoGe-2 (geometry) -> mesh carve -> LaMa + Stable Diffusion 2.1 (texture completion) -> IDArb + SuperMat (PBR decomposition) -> OBJ/GLB export, entirely local on one GPU
- Controls
- capture angle and quality of the single input photo, splat point budget, mesh export format (OBJ/GLB) versus raw splat export (PLY), which PBR maps to export
- Why multi-model
- The vendor's own model-download list names six distinct component checkpoints (TripoSplat, MoGe-2, LaMa, IDArb, a Stable Diffusion 2.1 base, and SuperMat) that together turn one photo into a full PBR mesh, a task no single one of them performs alone: TripoSplat only produces an untextured, unfused Gaussian splat with no solid surface; MoGe-2 is a monocular geometry/depth estimator with no material or splat capability; LaMa and the SD 2.1-based pass are inpainting models with no 3D or material understanding; and IDArb/SuperMat are dedicated PBR material-decomposition models that require an already-complete texture to decompose. Note: LocalMesh's page lists these six checkpoints as the bundled model stack but does not spell out the exact stage-by-stage wiring; the specific role assigned to each model below (geometry vs. inpaint vs. material) is inferred from each model's independently documented purpose (TripoSplat = image-to-splat, MoGe-2 = monocular depth/normals, LaMa = inpainting, IDArb/SuperMat = PBR decomposition), not stated verbatim by the vendor.
- Use cases
- Offline, privacy-sensitive single-photo 3D asset capture for game or VFX props, Rapid PBR-ready mesh creation from a single product or object photo without a photogrammetry rig, Local 3D scanning where uploading source photos to a cloud service is undesirable
- Pitfalls
- Single-photo capture means unseen sides of the object are inpainted or hallucinated rather than truly reconstructed; PBR decomposition quality depends entirely on how well the inpainting stage completed occluded regions; requires a local NVIDIA GPU with at least 8GB VRAM; the vendor does not publish a technical writeup of the exact stage order, so the mechanism above is a reasoned inference from each bundled model's known purpose, not a confirmed architecture diagram.
Multi-Angle Turnaround to Rigged Game Asset (fal Game Assets)
text->image->image(xN)->3D meshfal.ai's 'Game Assets' workflow template generates an object image with Seedream, then uses Seedream Edit to produce left/right/back angle views of that same object, then feeds the multi-view set into Hyper3D Rodin v2 to reconstruct a textured 3D mesh in one API call.
Stack, controls & moreHide
- Example stack
- Seedream v4 (text-to-image) -> Seedream v4 Edit (multi-view turnaround, 3x) -> Hyper3D Rodin v2 (multi-view image-to-3d)
- Controls
- prompt for object concept; edit prompts specify angle (left side/right side/back view) and background/texture cleanup; Rodin multi-view concat mode input
- Why multi-model
- Chains a text-to-image generator, an instruction-based image editor (for consistent multi-view turnaround), and a dedicated multi-view-conditioned 3D reconstruction model, each a separate hosted model exposed as one workflow endpoint.
- Use cases
- game-ready prop/asset generation from a single concept prompt, turning a product concept into a 3D asset without a photo shoot, rapid prototyping of 3D collectibles/NPCs for prototyping
- Pitfalls
- edited turnaround views can drift in proportions/details between angles since Seedream Edit has no hard 3D consistency guarantee, which can degrade Rodin's reconstruction versus true multi-view-diffusion methods (e.g. Zero123/SyncDreamer) that are trained explicitly for view consistency; closest existing pattern is multiview-diffusion-to-3d-reconstruction (Zero123/SyncDreamer -> NeuS/InstantMesh) but this chain substitutes a general-purpose text-to-image + instruction-edit pair for the multi-view-diffusion step and Rodin v2 for the NeuS/InstantMesh reconstruction stage, a materially different model family and generation goal (single-prompt asset creation, not reconstruction from an existing photo)
Multi-View Diffusion to 3D Reconstruction
single image -> multi-view images -> 3D meshA multi-view diffusion model generates a set of geometrically consistent novel-view images of an object from a single reference image; those synthetic views are then passed to a neural reconstruction algorithm (NeuS, NeRF, or an instant-reconstruction model) to recover the full 3D mesh without needing real multi-view capture. Note: no 'multi-view' family enum value exists; draft-to-finish is the closest fit for this two-stage view-synthesis-then-reconstruction chain.
Stack, controls & moreHide
- Example stack
- SyncDreamer (16-view generation) -> NeuS mesh reconstruction -> marching cubes export. Alternative: Zero123++ views -> InstantMesh reconstruction.
- Controls
- Number of synthesised views; elevation and azimuth sampling angles; diffusion guidance scale; reconstruction iteration count; NeuS surface threshold; optional mask for background removal before reconstruction.
- Why multi-model
- A single-image-to-3D model must simultaneously understand 2D appearance and 3D structure, which is under-constrained. Splitting into a view-synthesis model (which inherits rich 2D priors from large diffusion training) and a reconstruction model (which specialises in 3D geometry from image sets) lets each focus on its strength, and the reconstruction model receives denser, consistent multi-view supervision.
- Use cases
- Single-photo 3D object digitisation for e-commerce, Game asset creation from concept art, 3D model generation for AR placement, Museum / heritage object 3D capture from photographs
- Pitfalls
- Multi-view consistency is imperfect for textureless or symmetric objects; NeuS reconstruction quality degrades when generated views have geometric drift; thin structures (wires, hair) are not recovered well by implicit surface methods.
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (project page)
Photogrammetry to Surface-Aligned Mesh
multi-view photos/video -> camera poses -> 3D Gaussian splats -> textured surface meshReconstructs a real-world object from ordinary photos or a video walkthrough: Structure-from-Motion recovers camera poses, Gaussian Splatting trains a photoreal radiance field on those poses, then a surface-extraction pass converts the volumetric splats into a clean, editable, UV-textured mesh.
Stack, controls & moreHide
- Example stack
- COLMAP -> 3D Gaussian Splatting -> SuGaR -> Blender/Unreal/Unity import
- Controls
- capture density/overlap, COLMAP matcher settings, 3DGS training iterations and densification thresholds, SuGaR regularization type, target mesh face count, Poisson depth
- Why multi-model
- COLMAP solves camera geometry but produces no renderable surface; 3D Gaussian Splatting renders photoreal views but is a volumetric point representation with no faces, UVs, or watertight geometry; SuGaR is a distinct surface-alignment and Poisson-extraction stage needed to pull an editable mesh out of the splats. Each stage solves a problem the others cannot.
- Use cases
- Photogrammetry asset capture from a phone video, Location scanning for virtual production, Cultural heritage digitization with editable, engine-ready output, AR/VR asset creation from real objects
- Pitfalls
- Poor image overlap or motion blur makes COLMAP fail pose estimation, propagating floaters and fractured geometry; reflective/transparent surfaces break both SfM matching and Gaussian convergence; mesh extraction can lose fine detail present in the raw splats.
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction (CVPR 2024)
Segment-Then-Reconstruct Scene Kitbashing
single cluttered photo -> per-object segmentation masks -> individually reconstructed posed 3D objectsExtracts every distinct object from one ordinary photo of a cluttered scene and reconstructs each as its own posed, textured 3D asset, by first running a general-purpose segmentation model to isolate object masks, then feeding each mask into a dedicated single-image-to-3D reconstruction model.
Stack, controls & moreHide
- Example stack
- SAM 3 (segmentation) -> SAM 3D Objects (per-object 3D reconstruction) -> scene assembly in Blender/Unity
- Controls
- mask selection/refinement (click or box prompts), per-object occlusion handling, output pose alignment to the original camera view, which detected objects to reconstruct
- Why multi-model
- A 2D segmentation model identifies what and where objects are but has no concept of 3D geometry; a single-image 3D reconstruction model lifts one already-isolated object into 3D but cannot first find and separate multiple objects in a cluttered scene. SAM 3D Objects reconstructs one selected object at a time and depends on an upstream segmentation step.
- Use cases
- Rapid kitbashing of game props from a single reference photo, AR product placement extracted from lifestyle photography, Building a 3D asset library from real-world photo references
- Pitfalls
- Reconstructs objects one at a time with no reasoning about physical interactions, so assembled scenes can show interpenetration or floating objects; heavy occlusion degrades reconstruction; segmentation errors propagate into missing or fused assets.
Introducing SAM 3D: 3D Reconstruction for Physical World Images (Meta AI)
StableGen TRELLIS mesh-gen to multi-ControlNet texture to PBR decomposition (Blender)
text/image -> 3d mesh -> image (x N views) -> PBR mapsBlender addon StableGen chains TRELLIS.2 (text/image-to-3D mesh generation) into a ComfyUI-driven multi-viewpoint diffusion texturing pass using SDXL/FLUX/Qwen-Image-Edit with simultaneous depth+canny+normal ControlNet and IPAdapter style guidance, then optionally decomposes the generated texture into full PBR material maps via Marigold/StableDelight, all inside one Blender workflow.
Stack, controls & moreHide
- Example stack
- TRELLIS.2 mesh -> multi-view depth+canny ControlNet + IPAdapter SDXL texturing -> Marigold/StableDelight PBR decomposition, driven from the StableGen Blender panel via a local ComfyUI server
- Controls
- ControlNet weight per unit (depth/canny/normal), IPAdapter reference image and weight, generation strategy (sequential/grid/separate per view), per-region local-edit masks, PBR map toggles
- Why multi-model
- Distinct from baseline's geometry-then-texture-3d and geometry-aware-pbr-material-distillation because it locks simultaneous multi-ControlNet + IPAdapter conditioning across many camera viewpoints in a single texturing pass (not a single depth pass), and unifies mesh generation, multi-view texturing, and in-addon PBR decomposition as one chain rather than separate tools; TRELLIS.2, SDXL/FLUX/Qwen diffusion, ControlNet, IPAdapter, and Marigold/StableDelight are five distinct model families invoked in sequence.
- Use cases
- game-ready PBR-textured props from a prompt, re-texturing existing game/VFX meshes with geometry-locked multi-view AI textures, local editing of specific texture regions without a full re-generation
- Pitfalls
- Multi-view seams/blending are projection-based and can show artifacts at UV boundaries; PBR decomposition is a heuristic estimate, not measured material data; requires a running local ComfyUI server plus VRAM for SDXL/FLUX
Text-to-Image-Depth Room Mesh Fusion
text -> per-view 2D renders -> per-view depth -> fused, inpainted textured room meshGenerates a room-scale 3D environment purely from a text prompt by iteratively rendering images at chosen camera poses, lifting each into 3D with monocular depth, and fusing/inpainting the growing mesh so each new view integrates seamlessly with previously generated geometry.
Stack, controls & moreHide
- Example stack
- Stable Diffusion -> monocular depth -> Stable Diffusion inpainting -> Text2Room fusion -> textured room mesh (OBJ)
- Controls
- text prompt per region/object, camera viewpoint trajectory/selection strategy, depth alignment tolerance, inpainting mask region, mesh simplification thresholds
- Why multi-model
- A text-to-image model produces flat 2D pixels with no camera-consistent geometry; a monocular depth model lifts a single image to a partial point cloud but cannot generate novel content or fill disocclusions; a text-conditioned inpainting model fills the gaps exposed as the camera moves, and a mesh-fusion algorithm stitches each frame into one consistent mesh.
- Use cases
- Rapid environment blockouts for game levels from text briefs, Virtual production background generation, Synthetic training environments for embodied AI, Architectural concept walkthroughs
- Pitfalls
- Depth misalignment across frames causes seams, ghosting, or duplicated geometry; viewpoint selection can miss occluded regions leaving holes; style can drift between frames; scales to room interiors but not large open scenes.
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models (ICCV 2023, arXiv:2303.11989)
TRELLIS Structured-3D-Latents Pipeline
text/image -> 3D (mesh, radiance field, 3DGS)Generate a 3D asset in a structured latent space, then decode it to whichever representation the pipeline needs (mesh, volume, or Gaussian splats) from a single forward pass.
Stack, controls & moreHide
- Example stack
- TRELLIS (or TRELLIS.2) -> texture refine -> retopo.
- Controls
- Input image or text; output format (mesh/3DGS/volume); guidance strength; seed.
- Why multi-model
- A single representation cannot serve every downstream use; the structured-latent decoder lets one model emit interchangeable mesh, voxel, and 3DGS outputs that each need different post-processing.
- Use cases
- Game and VFX assets, AR product previews, Rapid concept modeling
- Pitfalls
- Thin structures and interiors are under-represented; topology needs retopology before production use.
TRELLIS: Structured 3D Latents for Scalable and Versatile 3D Generation (arXiv:2412.01506)
WorldMesh Navigable Multi-Room Scene Generation
text -> mesh scaffold -> mesh-conditioned room renders -> segmented objects -> reconstructed 3D sceneA text description first drives construction of an explicit mesh scaffold (walls, floors) for an arbitrarily large, multi-room environment; that scaffold is rendered to condition a separate image-synthesis model per room, a promptable segmentation model masks individual objects room by room, a single-image-to-3D model reconstructs each masked object, and a depth model validates geometry before the scene is exported as a navigable, geometry-regularized 3D Gaussian Splat.
Stack, controls & moreHide
- Example stack
- Text prompt -> WorldMesh mesh scaffold (floor plan geometry) -> Nano Banana Pro or FLUX.2-klein-9b (mesh-conditioned room renders) -> SAM 3 (per-room object masks via Gradio UI) -> SAM 3D Objects (per-object 3D reconstruction) -> Depth Pro (depth validation) -> Nerfstudio Splatfacto/COLMAP (geometry-regularized 3DGS export).
- Controls
- Text prompt describing the floor plan and rooms; manual per-room object-mask clicks via the Gradio UI (Phase 1); choice of image model (cloud Nano Banana Pro vs. local FLUX.2-klein-9b distilled vs. undistilled base, the base being about 5x slower but higher quality); SAM 3 text/interactive segmentation prompts; depth-validation tolerance for geometry regularization.
- Why multi-model
- No single diffusion model can hold arbitrarily large, multi-room 3D consistency: WorldMesh decouples structural geometry (an explicit mesh scaffold) from appearance (a mesh-conditioned image model), then depends on a distinct promptable segmentation model to identify individual objects, a separate single-image 3D reconstruction model to lift each object into 3D, and a monocular metric-depth model to validate the resulting geometry, before a Gaussian-splat training stage produces the final navigable scene. Each stage solves a problem the others cannot: the scaffold has no appearance, the image model has no persistent 3D structure or object boundaries, the segmentation model has no 3D capability, and the reconstruction model needs an already-isolated object mask. This is distinct from the tracked text-to-image-depth-inpaint-room-mesh pattern (Text2Room: iterative per-view depth-lift + SD inpainting fused into one mesh, single room, no explicit floor-plan scaffold or per-object 3D reconstruction) and from segment-then-reconstruct-scene-kitbash (SAM 3 + SAM 3D Objects applied to a single existing cluttered photo, not to a text-generated, mesh-scaffolded, navigable multi-room environment).
- Use cases
- Navigable multi-room virtual-production or game-level generation from a text brief, Embodied-AI training environments needing consistent, arbitrarily-sized multi-room geometry, Architectural walkthrough prototyping generated purely from a text description, VR/AR scene prototyping with per-object 3D editability
- Pitfalls
- The published pipeline is two-phase and requires a manual masking step (a Gradio UI where the user clicks objects room by room) rather than being a single automated pass; the undistilled local FLUX.2-klein-9b-base variant is about 5x slower than the distilled variant for higher quality; reconstruction quality is bounded by SAM 3 segmentation accuracy in cluttered or occluded rooms, and errors in the mesh scaffold or depth validation propagate into the final Gaussian-splat geometry.
WorldMirror to SplatGuide Pose-Free Novel View Synthesis
unposed multi-view images -> feed-forward 3D Gaussian splat -> diffusion-refined novel viewsA feed-forward reconstruction model builds a 3D Gaussian Splatting scene directly from unposed images, no camera-pose recovery or COLMAP structure-from-motion step required; a separate multi-view diffusion model then reuses that one reconstruction pass in three ways (pixel-aligned renders, an occlusion-aware visibility voting map, and reconstruction feature tokens) to synthesize photorealistic novel views that fill in content the splat alone cannot render cleanly.
Stack, controls & moreHide
- Example stack
- WorldMirror (pose-free feed-forward 3DGS reconstruction) -> SplatGuide conditioning (renders + visibility voting map + reconstruction tokens) -> SEVA multi-view diffusion (novel-view refinement)
- Controls
- reconstruction backbone choice (WorldMirror or zero-shot AnySplat), number and arrangement of unposed source views, camera and register token conditioning, Plucker ray embeddings, occlusion voting-map threshold
- Why multi-model
- Feed-forward 3DGS reconstruction and multi-view diffusion solve different problems: the reconstruction backbone (WorldMirror) estimates geometry and camera poses in one forward pass but cannot hallucinate unseen or under-observed content, while the diffusion model (SEVA) can synthesize plausible novel content but has no native 3D geometric grounding on its own. The paper trains only the diffusion stage while keeping the reconstruction backbone frozen, and explicitly frames this as chaining feed-forward 3DGS with multi-view diffusion rather than a single joint model. This is distinct from the tracked photogrammetry-splat-to-surface-mesh pattern (COLMAP SfM -> 3DGS training -> SuGaR mesh extraction), which requires classical pose recovery on many photos and outputs a mesh; here pose and geometry come from one learned feed-forward pass on unposed images, and the second stage refines renderable views rather than extracting a surface mesh.
- Use cases
- Novel-view synthesis from casually captured, unposed photo or video sets, Real-estate and interior walkthroughs without a dedicated SfM capture pipeline, Sparse-view 3D content creation for previz where COLMAP pose recovery is impractical
- Pitfalls
- Only the diffusion stage is trained while the reconstruction backbone stays frozen, so novel-view quality is bounded by the reconstruction backbone's pose and geometry accuracy; heavy occlusion still challenges the visibility voting map; evaluated on RealEstate10K, DL3DV, Tanks-and-Temples and Mip-NeRF 360 as a research benchmark, not yet a shipped consumer tool.
SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis (arXiv:2608.16863)
Browse all pipelines, stages, sources, and CSV/JSON downloads →
What recurs across the catalogue
The models change quickly, but three implementation concerns recur: explicit controls, defined handoffs, and validation at stage boundaries.
Use explicit controls
References, control networks, masks, seeds, and identity embeddings make important constraints visible and reusable. Prompt text alone is a weak place to store them.
Model the handoff
Each stage has a defined input, output, and transformation. The dataset maps those stages to workflow-node categories so the chain can be inspected or implemented as a graph.
Failures cluster at handoffs
Seeds, masks, dimensions, colour, identity, and camera assumptions can drift between stages. A pipeline needs explicit contracts for the artifacts and controls it passes forward.
When a pipeline is warranted
Every additional stage adds cost, latency, and another failure point. Add one only when it addresses a named limitation: a finishing pass for detail, a reference stage for identity, a motion stage for video, or a texture stage for 3D. Define the input, output, controls, and acceptance check for that handoff before adding the model.