LTX-2 Audio-Conditioned Two-Stage Generation

LTX-2's dedicated audio-to-video pipeline generates a synchronized base clip directly conditioned on an input audio file (not text), jointly denoising audio and video in one diffusion pass, then a separate spatial-upscaler checkpoint sharpens the result in a second chained pass.

Properties

family: audio-driven
modality: audio + reference image + prompt -> base audio-video -> upscaled audio-video
whyMultiModel: The base diffusion transformer that jointly denoises audio+video latents is trained for temporal/audio coherence, not resolution; production output requires a distinct spatial-upscaler checkpoint chained afterward. These are two separately downloadable model weights, not one model run twice.
steps: [object Object], [object Object]
controls: input audio track (dialogue/music pacing drives motion energy), reference/face image, upscaler factor (x1.5 or x2), prompt for scene description
exampleStack: ltx-2.3-22b-dev (A2VidPipelineTwoStage) -> ltx-2.3-spatial-upscaler-x2
useCases: [Podcast/avatar audio-driven scenes][Voice-driven talking clips][Music-synced motion generation]
pitfalls: Because audio and video are denoised jointly rather than muxed afterward, mismatched or noisy input audio directly corrupts motion quality (not just lip sync), and re-running only the upscale stage cannot fix a bad base generation.
source: Lightricks/LTX-2 official repository (A2VidPipelineTwoStage, spatial upscaler checkpoints)

Provenance

Source: https://github.com/Lightricks/LTX-2

LTX-2 Audio-Conditioned Two-Stage Generation

Properties

family: audio-driven
modality: audio + reference image + prompt -> base audio-video -> upscaled audio-video
whyMultiModel: The base diffusion transformer that jointly denoises audio+video latents is trained for temporal/audio coherence, not resolution; production output requires a distinct spatial-upscaler checkpoint chained afterward. These are two separately downloadable model weights, not one model run twice.
steps: [object Object], [object Object]
controls: input audio track (dialogue/music pacing drives motion energy), reference/face image, upscaler factor (x1.5 or x2), prompt for scene description
exampleStack: ltx-2.3-22b-dev (A2VidPipelineTwoStage) -> ltx-2.3-spatial-upscaler-x2
useCases: [Podcast/avatar audio-driven scenes][Voice-driven talking clips][Music-synced motion generation]
pitfalls: Because audio and video are denoised jointly rather than muxed afterward, mismatched or noisy input audio directly corrupts motion quality (not just lip sync), and re-running only the upscale stage cannot fix a bad base generation.
source: Lightricks/LTX-2 official repository (A2VidPipelineTwoStage, spatial upscaler checkpoints)

Provenance

Source: https://github.com/Lightricks/LTX-2