StoryDiffusion Consistent Storyboard

Generate a multi-panel comic or storyboard where the same characters stay consistent across shots using consistent self-attention across a batch, then optionally animate the transitions between frames.

Properties

family: multi-shot
modality: text -> image sequence (+ video)
whyMultiModel: Cross-shot consistency needs attention shared across generations, plus a motion model to bridge frames; a single image pass cannot enforce identity across panels.
steps: [object Object], [object Object]
controls: Character reference prompts; number of panels; shared-attention on/off; motion strength for transitions.
exampleStack: StoryDiffusion (SDXL) -> Sora/Kling image-to-video for transitions.
useCases: [Comic and graphic-novel pages][Ad storyboards][Children's book illustration]
pitfalls: Shared attention can homogenize distinct characters; long sequences drift as the consistency window is bounded.
source: StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (arXiv:2405.01434)

Provenance

Source: https://arxiv.org/abs/2405.01434

StoryDiffusion Consistent Storyboard

Properties

family: multi-shot
modality: text -> image sequence (+ video)
whyMultiModel: Cross-shot consistency needs attention shared across generations, plus a motion model to bridge frames; a single image pass cannot enforce identity across panels.
steps: [object Object], [object Object]
controls: Character reference prompts; number of panels; shared-attention on/off; motion strength for transitions.
exampleStack: StoryDiffusion (SDXL) -> Sora/Kling image-to-video for transitions.
useCases: [Comic and graphic-novel pages][Ad storyboards][Children's book illustration]
pitfalls: Shared attention can homogenize distinct characters; long sequences drift as the consistency window is bounded.
source: StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (arXiv:2405.01434)

Provenance

Source: https://arxiv.org/abs/2405.01434