Hermosa AI
Loading...
Topics
Generative Media Pipelines
77 pipelines
[
fortnightly
]
2026-07-04
Base + Refiner Draft-to-Finish
A fast base model lays down composition and structure, then a high-fidelity refiner pass adds detail and fixes artifacts, optionally followed by an upscaler.
Reference-Locked Character Consistency
Lock a character's identity across many generations by conditioning a base image model on a reference through an identity adapter, so new poses and scenes keep the same face and character.
Geometry-Then-Texture 3D Asset Pipeline
A first model generates an untextured 3D geometry (mesh / TSDF) from a single image or text prompt; a second, separate diffusion model synthesises high-resolution texture maps conditioned on the produced geometry, yielding a fully textured, PBR-ready 3D asset.
Voice-Clone TTS to Talking-Head Lip Sync
A zero-shot voice-cloning TTS model synthesises speech from a text script in the voice of a reference speaker; that generated audio is then fed into a separate lip-sync or talking-head model which drives realistic facial animation on a reference image or video, producing a complete avatar video without studio recording.
Multi-View Diffusion to 3D Reconstruction
A multi-view diffusion model generates a set of geometrically consistent novel-view images of an object from a single reference image; those synthetic views are then passed to a neural reconstruction algorithm (NeuS, NeRF, or an instant-reconstruction model) to recover the full 3D mesh without needing real multi-view capture. Note: no 'multi-view' family enum value exists; draft-to-finish is the closest fit for this two-stage view-synthesis-then-reconstruction chain.
ControlNet Structure-Lock Chain
A preprocessor (canny, depth, or pose annotator) extracts a structural control map from a reference image; that map is then injected into a conditioned diffusion model to generate a new image that preserves the source structure.
Face-Restore + Background-Upscale Chain
A face-restoration model (GFPGAN, CodeFormer) recovers degraded facial detail in the generated or photographed image, while a separate general-purpose upscaler (Real-ESRGAN) handles background and non-face regions.
Generate-then-Instruct-Edit Chain
A text-to-image model generates a base image from a prompt, then a separate instruction-following inpaint model takes that image plus a mask and a text instruction and applies targeted changes to specific regions while leaving the rest intact.
Keyframe-Drive Image-to-Video
A text-to-image model generates one or more keyframe stills that lock composition, character, and lighting; those stills are then fed into an image-to-video model to produce motion. The i2v model inherits the keyframe's identity rather than hallucinating from text alone, giving tighter control over subject appearance and scene layout.
First-Last-Frame Interpolation (FLF2V)
Two keyframe images, a start frame and an end frame, are generated (or sourced) independently; a specialised first-last-frame-to-video model then fills in all intermediate frames, producing a coherent motion arc between the two endpoints. Chained passes can drive longer transformations by treating the last frame of one pass as the first frame of the next.
Multi-Shot Narrative Stitch
A narrative or storyboard is broken into individual shots; a reference-conditioned image model generates a consistent keyframe per shot, an image-to-video model animates each keyframe into a short clip, and all clips are concatenated in order to form a full multi-shot sequence. Character and style consistency is maintained by conditioning each shot on a shared reference.
Depth/Pose-Guided Video-to-Video Restyle
A source video is passed through a preprocessor that extracts a structural control signal (depth map, Canny edges, or OpenPose skeleton). That signal is fed as conditioning to a video diffusion model which generates new video that preserves the original motion and geometry while applying a new appearance, style, or character. The result is a restyled video with motion inherited from the reference footage.
Generated-Video Upscale + Frame-Interpolation Finishing
AI-generated video is typically low resolution (480p-720p) and low frame rate (12-24fps). A two-stage finishing chain first passes every frame through a spatial upscaler (Real-ESRGAN or similar GAN) to reach 2-4x resolution, then applies a temporal frame interpolation model (RIFE) to double or quadruple the frame rate, producing a smooth, high-resolution output suitable for delivery.
IP-Adapter Identity-Lock
A lightweight image-prompt adapter injects a reference image as a decoupled visual condition, so a single subject stays recognizable across many re-prompts and styles without any fine-tuning.
PhotoMaker Custom-Identity Generation
Stack a few reference photos of a person into a unified identity embedding, then generate that exact person in any scene, style, or action from text alone, with no per-subject training.
StoryDiffusion Consistent Storyboard
Generate a multi-panel comic or storyboard where the same characters stay consistent across shots using consistent self-attention across a batch, then optionally animate the transitions between frames.
InstructPix2Pix Instruction Edit
Edit an existing image from a natural-language instruction ("make it sunset", "add sunglasses") without a mask, by conditioning the model jointly on the source image and the command.
IDM-VTON Virtual Try-On
Realistically transfer a garment from a product photo onto a person image, warping the cloth to body geometry and inpainting it in with a diffusion try-on model.
TRELLIS Structured-3D-Latents Pipeline
Generate a 3D asset in a structured latent space, then decode it to whichever representation the pipeline needs (mesh, volume, or Gaussian splats) from a single forward pass.
Hunyuan3D-2 Image-to-Mesh
A two-stage shape-then-texture pipeline: a diffusion model produces a clean 3D shape from a single image, then a second model bakes PBR textures onto it for a production-ready mesh.
SadTalker Audio-to-Portrait
Drive a single portrait photo with speech by estimating realistic 3D motion coefficients (head pose and expression) from the audio, then rendering a lip-synced talking-head clip.
OmniHuman Audio-to-Talking-Human
A one-stage model animates a full human (not just the face) from audio, scaled up by conditioning on portrait, pose, and body signals together so motion and lip-sync emerge from a single network.
InstantID Zero-Shot Identity-Style
Zero-shot identity-preserving generation: with only one reference face and a style prompt, generate that person in any style or scene, combining a face encoder, an IdentityNet, and a lightweight adapter.
Product Photo Scene Generation & Relight
A product cutout is segmented from its original photo, dropped into a newly generated lifestyle background, then IC-Light relights the product to match the new scene's lighting so the composite reads as a single photograph.
Depth-Guided Video Relighting Chain
A character is matted out of source footage, depth-mapped, then relit frame-by-frame with an IC-Light ControlNet driven by a hand-painted light map, with AnimateDiff-based motion modules used to stabilize the relit sequence.
Light-A-Video Progressive Light Fusion
A training-free video relighting pipeline that repurposes an existing image relighting model with a Consistent Light Attention module across frames, then linearly blends source and relit appearance over time (Progressive Light Fusion) to remove flicker without any video-specific training.
SAM 2 to MatAnyone Rotoscoping Composite
A subject is segmented in one frame via a promptable segmentation model, the mask is propagated across the clip, then a dedicated video matting model refines the rough mask into a per-pixel alpha matte for clean compositing onto a new background.
Inpaint Anything: Segment-Remove-Replace
A user clicks an object, Segment Anything produces its mask, and depending on intent the hole is either filled by a fast non-diffusion inpainter (object removal) or filled by a text-guided diffusion model (object/background replacement).
Reference-Driven Line Art Video Colorization
A raw line-art/sketch video sequence and a single colored reference image of the character are fed into a video diffusion model that uses point-tracking correspondences to transfer color and shading from the reference onto every sketch frame, producing a temporally consistent colored animation.
Motion-Brush Trajectory-Controlled Image Animation
A still image is segmented into up to six user-selected regions, each assigned its own hand-painted motion trajectory (direction, curve, speed), and a commercial video diffusion model animates each region along its path while holding the rest of the frame static.
Product-into-Scene Composite and Relight
A studio product photo and a separate background/scene image are composited by an image-editing model that also relights the product to match the new scene's light direction and color temperature, producing an advertising-ready packshot without a physical reshoot.
Diffusion-Transformer Garment-Preserving Video Try-On
A video of a person and a flat garment reference image are fed into a diffusion-transformer model that disentangles garment texture/print from pose and injects it frame-by-frame, producing a video of the person wearing the new garment with consistent fabric detail and motion.
Character Sheet to Lettered Comic Panel Sequence
A consistent character is first generated as a multi-view reference sheet (front/side/angled), then reused via identity+pose adapters to populate a sequence of panel images, which are finally arranged into a hierarchical comic-page layout with speech-bubble lettering added as a separate post-process.
Video-to-Audio Foley Synthesis
A silent generated (or real) video plus an optional text caption is fed into a joint video-audio-text diffusion model that synthesizes semantically matched, temporally synchronized sound effects and ambience directly from visual motion.
MultiTalk Multi-Person Dialogue Video
Given a reference image containing multiple people, separate audio streams per speaker, and a scene prompt, the pipeline generates a video where each character's lips and turn-taking match their own audio stream, driven by a large video diffusion backbone conditioned on per-person audio embeddings.
Chain-of-Thought Reasoning Foley Editing
A multimodal LLM first reasons step-by-step about a video's objects, actions, and acoustic environment, producing a structured chain-of-thought plan that then steers a separate audio foundation model to generate foley, and the same MLLM can re-reason to guide targeted natural-language edits of specific sound elements.
Text-to-Song Beat-Synced Music Video
Lyrics and a style prompt are turned into a full song via a text-to-music model, then a separate analysis+generation stage detects tempo, beats, and lyric timing from the audio to drive cut points, transitions, and karaoke-style captions over AI-generated or stock visuals.
TTS-Driven Video-Diffusion Talking Character
A text script is converted to speech by a local TTS engine, an image-to-video model animates a character from a still reference into a base video clip, and a lip-sync model then re-syncs that generated video's mouth movements to the TTS audio, producing a fully text-driven talking character without any recorded voice or footage.
Text/Pose-to-Video Avatar to Real-Time Lip-Sync
A text-to-video (or image-to-video / pose-to-video) model first generates the avatar's body motion and scene, and a separate real-time latent-space lip-sync model then re-renders just the mouth region at 30+ FPS to match streaming audio, decoupling body/scene generation from low-latency mouth synchronization.
UNO Multi-Subject Compose to Wan Animate
ByteDance UNO composes a character (and optional object/outfit reference) into a single consistent still via in-context multi-image conditioning on FLUX.1-dev, then a CLIP-Vision-encoded image-to-video model animates that still into a moving clip with the same identity and outfit.
InfiniteYou Recraft to Animated Portrait
ByteDance InfiniteYou injects identity features through InfuseNet residual connections into a FLUX DiT to recraft a person's photo under a new prompt/pose/style while preserving likeness, then a downstream image-to-video model animates the recrafted still.
Stand-In Lightweight Identity Video Control
Stand-In adds a tiny conditional-image branch (~1% extra parameters) onto Wan2.1/2.2 that restricts self-attention with conditional position mapping, locking a face identity across an entire generated video clip while composing with pose-driven control or community style LoRAs.
HunyuanCustom Multi-Subject Video Customization
Tencent's HunyuanCustom fuses a LLaVA-based text-image understanding module with the HunyuanVideo DiT and an image-ID enhancement module (temporal concatenation) to keep one or more reference subjects visually consistent throughout a generated or subject-replaced video.
Photogrammetry to Surface-Aligned Mesh
Reconstructs a real-world object from ordinary photos or a video walkthrough: Structure-from-Motion recovers camera poses, Gaussian Splatting trains a photoreal radiance field on those poses, then a surface-extraction pass converts the volumetric splats into a clean, editable, UV-textured mesh.
Geometry-Aware PBR Material Distillation
Takes an existing bare mesh and generates physically-based material maps (albedo, roughness, metallic) matched to its geometry, using a diffusion model conditioned on the mesh's own geometry and a chosen lighting environment, then distilling the shaded outputs into decomposed BRDF parameters via inverse rendering.
Text-to-Image-Depth Room Mesh Fusion
Generates a room-scale 3D environment purely from a text prompt by iteratively rendering images at chosen camera poses, lifting each into 3D with monocular depth, and fusing/inpainting the growing mesh so each new view integrates seamlessly with previously generated geometry.
Generated Mesh to Rigged, Animated Character
Turns a prompt or reference image into a game-ready animated character by generating a T-posed base mesh with one model, automatically detecting a skeleton and computing skin weights with a dedicated auto-rigging service, then applying that service motion-capture library onto the resulting rig.
Segment-Then-Reconstruct Scene Kitbashing
Extracts every distinct object from one ordinary photo of a cluttered scene and reconstructs each as its own posed, textured 3D asset, by first running a general-purpose segmentation model to isolate object masks, then feeding each mask into a dedicated single-image-to-3D reconstruction model.
Storyboard Grid to Multi-Shot Video
A dedicated image model renders the full shot sequence as consistent staged panels (sometimes a single grid image) so composition and character silhouette are locked before any motion exists; a video model then animates the panels together, using them as reference images plus a timestamped multi-shot prompt to decide camera work and motion per shot.
LTX-2 Audio-Conditioned Two-Stage Generation
LTX-2's dedicated audio-to-video pipeline generates a synchronized base clip directly conditioned on an input audio file (not text), jointly denoising audio and video in one diffusion pass, then a separate spatial-upscaler checkpoint sharpens the result in a second chained pass.
LLM-Orchestrated Talking-Scene Assembly
A general-purpose LLM first drafts the dialogue, per-shot prompts, and voice casting from a character brief; an image model then generates and edits successive scene keyframes, a video model animates each keyframe into a shot stitched via last-frame continuation, a talking-head model adds lip-synced performance and face consistency, and a voice model supplies the matching speech track.
Restyled First-Frame Propagation
An image-editing model restyles only the first frame of a source video into a target look; a video model then regenerates the entire clip by propagating that one already-styled frame forward through time, using lightweight structural maps only as an ancillary guardrail against geometry drift rather than as the source of style.
Image to Scripted Voice-Clone UGC Video
A single product/character image is turned into a short lip-synced UGC-style video: an LLM writes a performance script and scene description from the image, ElevenLabs clones or selects a voice to synthesize the narration, then LTX-2.3 generates the video with native lip sync driven by that audio.
Generate-then-Decompose PBR Material (CHORD)
A text prompt is first turned into a seamless, tileable flat-lit texture by an image diffusion model, then Ubisoft La Forge's CHORD model decomposes that single texture into a full physically-based-rendering material map set via SVBRDF estimation.
Suno Song to Flux-Runway Generative Music Video
A text prompt drives Suno to write and generate a full song, then per-scene image stills are generated with Flux and animated into short clips with Runway, which are composited against the finished track into a music video.
Flux Kontext Restore then Kling-Animate Vintage Photo
An old/vintage photograph is first colorized and quality-enhanced with Flux Kontext, then the restored image is animated into a short video with Kling.
DALL-E Generated Character to ElevenLabs-Hedra Talking Podcast
A script is written by GPT-4, a wholly new photorealistic character portrait is generated from scratch with DALL-E, ElevenLabs synthesizes a voice reading the script, and Hedra animates the generated portrait's face to lip-sync and express in time with that audio.
Multi-Angle Turnaround to Rigged Game Asset (fal Game Assets)
fal.ai's 'Game Assets' workflow template generates an object image with Seedream, then uses Seedream Edit to produce left/right/back angle views of that same object, then feeds the multi-view set into Hyper3D Rodin v2 to reconstruct a textured 3D mesh in one API call.
LLM-Planned Multi-Scene POV Video Cascade (fal Live-in-Scene)
fal.ai's 'Live-in-Scene' workflow template takes a film name, has an LLM scene planner break it into 6 first-person POV scene prompts, generates a keyframe image for each with Nano Banana Pro, animates each into a 4-second clip with generated ambient audio via Seedance 1.5 Pro image-to-video, then merges all six clips into one ~24-second video.
Text-to-Narrated Documentary Video (fal Documentary)
fal.ai's 'Documentary' workflow template turns a text topic into a narrated short documentary: an LLM drafts the story/shot script, Dreamina v3.1 generates six scene images, Seedance 1/pro animates them into video clips, ElevenLabs eleven-v3 synthesizes voiceover narration, and ffmpeg merges audio with video and concatenates all shots into one final cut.
Concept Image to Rigged, Motion-Ready Character (Scenario Uthana Chain)
Scenario's platform workflow chain generates a character concept image with GPT Image 2, converts it to a 3D mesh with Hunyuan3D 3.1, runs the mesh through Uthana Character Rigging to auto-place a bipedal skeleton and skin weights in under 30 seconds, then animates the rigged character with Uthana Video-to-Motion (from a reference clip) or Uthana Text-to-Motion (from a text description).
Bria native GenAI node chain in Nuke
Bria ships 11 GenAI nodes (Fibo Generate, Fibo Edit, RMBG, Erase, GenFill, Expand, Enhance, Upscale, Sequence Output) as native Nuke Group nodes that chain directly in the compositing node graph: text-to-image draft, then prompt-based inpaint/edit, then dedicated enhance/upscale, all at full resolution without leaving Nuke.
StableGen TRELLIS mesh-gen to multi-ControlNet texture to PBR decomposition (Blender)
Blender addon StableGen chains TRELLIS.2 (text/image-to-3D mesh generation) into a ComfyUI-driven multi-viewpoint diffusion texturing pass using SDXL/FLUX/Qwen-Image-Edit with simultaneous depth+canny+normal ControlNet and IPAdapter style guidance, then optionally decomposes the generated texture into full PBR material maps via Marigold/StableDelight, all inside one Blender workflow.
ComfyTextures Unreal create-refine-upscale-edit texture chain
Unreal Engine editor plugin ComfyTextures captures viewport render passes of selected actors and runs them through a ComfyUI-orchestrated chain: a fast SDXL+LCM-LoRA pass creates a low-res draft texture, a slower SDXL base+refiner pass refines it, an inpainting workflow allows targeted edits, and a 4x-UltraSharp upscale finishes the result, which is applied back as a new material/texture on the actor.
StreamDiffusionTD live multi-ControlNet + StreamV2V temporal-consistency chain
TouchDesigner tox StreamDiffusionTD runs SDXL-Turbo in real time driven by any TouchDesigner-generated control signal (camera feed, procedural TOP, depth/canny/normal maps built in TD) through multiple simultaneous TensorRT-accelerated ControlNets, then applies StreamV2V cached-attention video-to-video processing so consecutive live frames stay temporally coherent instead of flickering independently.
Soul ID Locked Character to Veo 3 Talking Video
Generate or upload a character image (Soul or Nano Banana), train a portable digital-twin identity on it with Soul ID, then feed that locked identity into Veo 3 to generate a talking video with native synchronized dialogue and voice, finishing with VFX and camera-motion polish nodes, all inside Higgsfield's node-based Canvas without leaving the app or re-uploading references per shot.
Style Reference Adapter Chain
A style reference image is encoded by an image adapter and its style features are injected selectively into style-specific attention blocks of a diffusion model, optionally combined with a tile ControlNet that preserves the spatial structure of a content image.
Wan2.2 Animate Character Replacement & Motion Transfer
Extracts whole-body pose and face keypoints from a source video, isolates the on-screen subject, encodes a separate reference character's identity, then fuses all three signals into a video diffusion model that replaces the on-screen performer with the reference character while preserving the original motion and expression timing. This is the current successor pattern to Animate Anyone / MimicMotion-style motion transfer.
DiffSensei MLLM Identity-Adapter Manga Generation
A multimodal LLM (SEED-X), fine-tuned as a text-compatible identity adapter, reads one or more character reference images and adjusts their expression, pose, and action to match each panel's text cues; those adapted features feed an SDXL diffusion backbone through masked cross-attention alongside per-panel character and dialogue bounding boxes, producing layout-aware, identity-consistent manga pages.
ID-V2V Identity-Preserving Video Restylization
A source video's first frame (and optionally further anchor frames) is restyled into a new look by an image-editing model, then a dedicated video-to-video model, ID-V2V, regenerates the full clip in that new look while explicitly holding facial likeness, expression, eye gaze and full-body performance constant, casting identity preservation as a video-relighting problem rather than leaving it to the same model that authored the style change.
WorldMesh Navigable Multi-Room Scene Generation
A text description first drives construction of an explicit mesh scaffold (walls, floors) for an arbitrarily large, multi-room environment; that scaffold is rendered to condition a separate image-synthesis model per room, a promptable segmentation model masks individual objects room by room, a single-image-to-3D model reconstructs each masked object, and a depth model validates geometry before the scene is exported as a navigable, geometry-regularized 3D Gaussian Splat.
FLUX.1 Krea to Ovi Text-Image-to-Joint-Audio-Video (t2i2v)
In Ovi's t2i2v mode, FLUX.1 Krea first composes a starting keyframe from a text prompt; that image (plus the text prompt) then conditions Ovi, a twin-backbone diffusion model whose video branch is initialized from Wan2.2 and whose custom 5B audio branch was pretrained from scratch, which jointly denoises video and audio in a single pass to produce a synchronized clip with sound as a co-product, not a downstream add-on.
WorldMirror to SplatGuide Pose-Free Novel View Synthesis
A feed-forward reconstruction model builds a 3D Gaussian Splatting scene directly from unposed images, no camera-pose recovery or COLMAP structure-from-motion step required; a separate multi-view diffusion model then reuses that one reconstruction pass in three ways (pixel-aligned renders, an occlusion-aware visibility voting map, and reconstruction feature tokens) to synthesize photorealistic novel views that fill in content the splat alone cannot render cleanly.
LocalMesh Single-Photo Splat-to-PBR-Mesh
A single photograph is lifted directly to a navigable 3D Gaussian splat by a feed-forward image-to-splat model, then a bundled local stack of a monocular geometry model, an inpainter, and material-decomposition models carves that splat into a textured, PBR-mapped quad mesh, all running on the user's own GPU with nothing uploaded to a server.
MiniMax H3 Context-IR to Base to Regenerate-2K
MiniMax H3 splits reference-to-video generation into three separately-hosted-or-downloadable stages: a hosted instruction-refinement model converts complex multimodal inputs into a Context Intermediate Representation, a locally-deployable Ref2VA (or FL2VA) diffusion checkpoint jointly generates synchronized audio and video at 768p from that context, and a hosted regeneration model feeds the 768p result plus the original context back through the system to produce a 2K output.
Bernini MLLM-Planner + DiT-Renderer Video Edit
ByteDance's Bernini splits instruction-based video generation and editing into two models: an MLLM planner that predicts the target semantic representation in ViT embedding space, and a DiT diffusion renderer that synthesizes pixels conditioned on that plan plus source-video features, keeping unedited regions stable and preserving subject identity.
Depth-Video Character Reskin to Seedance 2.0
A source video's motion is flattened into a temporally-consistent depth video by a video-native depth model, and a single extracted frame is separately re-designed into a brand-new character and setting by an image editor. Seedance 2.0's reference-to-video mode then takes both the depth video and the new character image together and re-renders the original motion performed by the new character in the new world.
