Video-to-Audio Foley Synthesis

A silent generated (or real) video plus an optional text caption is fed into a joint video-audio-text diffusion model that synthesizes semantically matched, temporally synchronized sound effects and ambience directly from visual motion.

Properties

family: audio-driven
modality: video -> text prompt -> synced audio
whyMultiModel: The video generator has no concept of sound; a dedicated video-to-audio model trained on paired video-audio-text data is required to infer materials, impacts, and timing from pixels and produce synchronized waveforms, and it in turn depends on a video encoder distinct from the audio decoder.
steps: [object Object], [object Object]
controls: text caption strength, classifier-free guidance scale, audio duration alignment to video FPS, synchronization module weight
exampleStack: Wan 2.1 (text-to-video) -> MMAudio (video+text -> audio) -> mux audio onto video
useCases: [Auto-Foley for silent AI-generated b-roll][Sound design for product demo videos][Rapid SFX drafts for game trailers]
pitfalls: Struggles with style continuity across cuts (no reference-audio injection pathway) and can hallucinate plausible-but-wrong sound sources for ambiguous visual motion.
source: MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (arXiv:2412.15322)

Provenance

Source: https://arxiv.org/abs/2412.15322

Video-to-Audio Foley Synthesis

Properties

family: audio-driven
modality: video -> text prompt -> synced audio
whyMultiModel: The video generator has no concept of sound; a dedicated video-to-audio model trained on paired video-audio-text data is required to infer materials, impacts, and timing from pixels and produce synchronized waveforms, and it in turn depends on a video encoder distinct from the audio decoder.
steps: [object Object], [object Object]
controls: text caption strength, classifier-free guidance scale, audio duration alignment to video FPS, synchronization module weight
exampleStack: Wan 2.1 (text-to-video) -> MMAudio (video+text -> audio) -> mux audio onto video
useCases: [Auto-Foley for silent AI-generated b-roll][Sound design for product demo videos][Rapid SFX drafts for game trailers]
pitfalls: Struggles with style continuity across cuts (no reference-audio injection pathway) and can hallucinate plausible-but-wrong sound sources for ambiguous visual motion.
source: MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis (arXiv:2412.15322)

Provenance

Source: https://arxiv.org/abs/2412.15322