Chain-of-Thought Reasoning Foley Editing

A multimodal LLM first reasons step-by-step about a video's objects, actions, and acoustic environment, producing a structured chain-of-thought plan that then steers a separate audio foundation model to generate foley, and the same MLLM can re-reason to guide targeted natural-language edits of specific sound elements.

Properties

family: audio-driven
modality: video -> MLLM reasoning trace -> object-targeted audio -> edited audio
whyMultiModel: The audio synthesis model has no native reasoning capacity to decide which objects should sound like what or when; a multimodal LLM is needed to decompose the scene into an interpretable plan (AudioCoT), and a distinct audio diffusion/foundation model executes that plan into actual waveforms, with the two iterating for object-level and instruction-level refinement.
steps: [object Object], [object Object], [object Object]
controls: object-click targeting for interactive refinement, natural-language edit instructions, reasoning trace granularity
exampleStack: ThinkSound MLLM reasoning stage -> AudioCoT-guided audio foundation model -> instruction-guided re-edit pass
useCases: [Iteratively directed foley for film post-production][Targeted SFX fixes without regenerating the whole soundtrack][Sound design review tools with explainable reasoning traces]
pitfalls: Reasoning quality bottlenecks final audio quality; a wrong CoT plan (misidentified object or action) propagates directly into an incorrect sound choice.
source: ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing (arXiv:2506.21448)

Provenance

Source: https://arxiv.org/abs/2506.21448

Chain-of-Thought Reasoning Foley Editing

Properties

family

audio-driven

modality

video -> MLLM reasoning trace -> object-targeted audio -> edited audio

whyMultiModel

The audio synthesis model has no native reasoning capacity to decide which objects should sound like what or when; a multimodal LLM is needed to decompose the scene into an interpretable plan (AudioCoT), and a distinct audio diffusion/foundation model executes that plan into actual waveforms, with the two iterating for object-level and instruction-level refinement.

steps

[object Object], [object Object], [object Object]

controls

object-click targeting for interactive refinement, natural-language edit instructions, reasoning trace granularity

exampleStack

ThinkSound MLLM reasoning stage -> AudioCoT-guided audio foundation model -> instruction-guided re-edit pass

useCases

[Iteratively directed foley for film post-production][Targeted SFX fixes without regenerating the whole soundtrack][Sound design review tools with explainable reasoning traces]

pitfalls

Reasoning quality bottlenecks final audio quality; a wrong CoT plan (misidentified object or action) propagates directly into an incorrect sound choice.

source

ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing (arXiv:2506.21448)