MultiTalk Multi-Person Dialogue Video

Given a reference image containing multiple people, separate audio streams per speaker, and a scene prompt, the pipeline generates a video where each character's lips and turn-taking match their own audio stream, driven by a large video diffusion backbone conditioned on per-person audio embeddings.

Properties

family: audio-driven
modality: reference image + multi-stream audio + text prompt -> multi-person conversational video
whyMultiModel: A single-speaker lip-sync model cannot bind multiple simultaneous audio streams to the correct face in a shared frame; MultiTalk needs a speech encoder to extract per-speaker embeddings, a Label Rotary Position Embedding scheme to bind each audio stream to its person, and a large video diffusion model to render coherent multi-person motion and interaction.
steps: [object Object], [object Object]
controls: per-speaker audio-person binding (L-RoPE), resolution (480p/720p), streaming vs clip mode, TeaCache acceleration, LoRA for style
exampleStack: Kokoro-82M (TTS for each character line) -> Chinese-Wav2Vec2 (audio embedding) -> MultiTalk / Wan 2.1-I2V-14B (multi-person audio-driven video)
useCases: [Multi-character animated dialogue scenes][Podcast-to-video with two animated hosts][Cartoon character conversation generation]
pitfalls: Audio-person binding can swap identities when reference image faces are visually similar or poorly separated in frame; long streaming generations can drift in lip accuracy.
source: Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (arXiv:2505.22647)

Provenance

Source: https://arxiv.org/abs/2505.22647

Properties

family

audio-driven

modality

reference image + multi-stream audio + text prompt -> multi-person conversational video

whyMultiModel

A single-speaker lip-sync model cannot bind multiple simultaneous audio streams to the correct face in a shared frame; MultiTalk needs a speech encoder to extract per-speaker embeddings, a Label Rotary Position Embedding scheme to bind each audio stream to its person, and a large video diffusion model to render coherent multi-person motion and interaction.

steps

[object Object], [object Object]

controls

per-speaker audio-person binding (L-RoPE), resolution (480p/720p), streaming vs clip mode, TeaCache acceleration, LoRA for style

exampleStack

Kokoro-82M (TTS for each character line) -> Chinese-Wav2Vec2 (audio embedding) -> MultiTalk / Wan 2.1-I2V-14B (multi-person audio-driven video)

useCases

[Multi-character animated dialogue scenes][Podcast-to-video with two animated hosts][Cartoon character conversation generation]

pitfalls

Audio-person binding can swap identities when reference image faces are visually similar or poorly separated in frame; long streaming generations can drift in lip accuracy.

source

Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation (arXiv:2505.22647)