Bernini MLLM-Planner + DiT-Renderer Video Edit

ByteDance's Bernini splits instruction-based video generation and editing into two models: an MLLM planner that predicts the target semantic representation in ViT embedding space, and a DiT diffusion renderer that synthesizes pixels conditioned on that plan plus source-video features, keeping unedited regions stable and preserving subject identity.

Properties

family: generate-to-edit
modality: video + reference image + instruction -> edited video
whyMultiModel: Semantic planning (what the edit should become) and pixel rendering (how it should look) are different learned problems: the MLLM reasons about the instruction in embedding space while the DiT renderer conditions on the plan and low-level source features. Neither model alone can both interpret the edit instruction and preserve fine visual detail.
steps: [object Object], [object Object]
controls: Edit instruction wording; optional reference image for garment swap or subject insertion; source-feature strength (identity preservation); HIGH vs LOW phase split; LightX2V high_noise/low_noise LoRA motion adapters in the ComfyUI workflow.
exampleStack: Bernini planner (Qwen2.5-VL based MLLM) -> Bernini renderer (Wan2.2 DiT, HIGH + LOW backbones) -> Wan 2.1 VAE decode; ComfyUI graph adds a Wan-style T5 text encoder and the LightX2V LoRA pair.
useCases: [Instruction-based video editing with stable unedited regions][Reference-guided garment swap and object insertion in video][Subject-to-video: placing a person or character into a new scene][Identity-preserving scene and style changes]
pitfalls: The plan bounds the edit: if the MLLM misreads the instruction the renderer faithfully produces the wrong edit; the renderer weights are open (Apache 2.0), and the planner (Qwen2.5-VL-7B-Instruct) and renderer (Wan2.2-T2V-A14B) backbones are confirmed via the official bytedance/Bernini GitHub repo, but planner quality still gates the whole chain.
source: Bernini: Latent Semantic Planning for Video Diffusion (arXiv:2605.22344)

Provenance

Source: https://arxiv.org/abs/2605.22344

Bernini MLLM-Planner + DiT-Renderer Video Edit

Properties

family

generate-to-edit

modality

video + reference image + instruction -> edited video

whyMultiModel

Semantic planning (what the edit should become) and pixel rendering (how it should look) are different learned problems: the MLLM reasons about the instruction in embedding space while the DiT renderer conditions on the plan and low-level source features. Neither model alone can both interpret the edit instruction and preserve fine visual detail.

steps

[object Object], [object Object]

controls

Edit instruction wording; optional reference image for garment swap or subject insertion; source-feature strength (identity preservation); HIGH vs LOW phase split; LightX2V high_noise/low_noise LoRA motion adapters in the ComfyUI workflow.

exampleStack

Bernini planner (Qwen2.5-VL based MLLM) -> Bernini renderer (Wan2.2 DiT, HIGH + LOW backbones) -> Wan 2.1 VAE decode; ComfyUI graph adds a Wan-style T5 text encoder and the LightX2V LoRA pair.

useCases

[Instruction-based video editing with stable unedited regions][Reference-guided garment swap and object insertion in video][Subject-to-video: placing a person or character into a new scene][Identity-preserving scene and style changes]

pitfalls

The plan bounds the edit: if the MLLM misreads the instruction the renderer faithfully produces the wrong edit; the renderer weights are open (Apache 2.0), and the planner (Qwen2.5-VL-7B-Instruct) and renderer (Wan2.2-T2V-A14B) backbones are confirmed via the official bytedance/Bernini GitHub repo, but planner quality still gates the whole chain.

source

Bernini: Latent Semantic Planning for Video Diffusion (arXiv:2605.22344)