WorldMirror to SplatGuide Pose-Free Novel View Synthesis

A feed-forward reconstruction model builds a 3D Gaussian Splatting scene directly from unposed images, no camera-pose recovery or COLMAP structure-from-motion step required; a separate multi-view diffusion model then reuses that one reconstruction pass in three ways (pixel-aligned renders, an occlusion-aware visibility voting map, and reconstruction feature tokens) to synthesize photorealistic novel views that fill in content the splat alone cannot render cleanly.

Properties

family: 3d
modality: unposed multi-view images -> feed-forward 3D Gaussian splat -> diffusion-refined novel views
whyMultiModel: Feed-forward 3DGS reconstruction and multi-view diffusion solve different problems: the reconstruction backbone (WorldMirror) estimates geometry and camera poses in one forward pass but cannot hallucinate unseen or under-observed content, while the diffusion model (SEVA) can synthesize plausible novel content but has no native 3D geometric grounding on its own. The paper trains only the diffusion stage while keeping the reconstruction backbone frozen, and explicitly frames this as chaining feed-forward 3DGS with multi-view diffusion rather than a single joint model. This is distinct from the tracked photogrammetry-splat-to-surface-mesh pattern (COLMAP SfM -> 3DGS training -> SuGaR mesh extraction), which requires classical pose recovery on many photos and outputs a mesh; here pose and geometry come from one learned feed-forward pass on unposed images, and the second stage refines renderable views rather than extracting a surface mesh.
steps: [object Object], [object Object], [object Object]
controls: reconstruction backbone choice (WorldMirror or zero-shot AnySplat), number and arrangement of unposed source views, camera and register token conditioning, Plucker ray embeddings, occlusion voting-map threshold
exampleStack: WorldMirror (pose-free feed-forward 3DGS reconstruction) -> SplatGuide conditioning (renders + visibility voting map + reconstruction tokens) -> SEVA multi-view diffusion (novel-view refinement)
useCases: [Novel-view synthesis from casually captured, unposed photo or video sets][Real-estate and interior walkthroughs without a dedicated SfM capture pipeline][Sparse-view 3D content creation for previz where COLMAP pose recovery is impractical]
pitfalls: Only the diffusion stage is trained while the reconstruction backbone stays frozen, so novel-view quality is bounded by the reconstruction backbone's pose and geometry accuracy; heavy occlusion still challenges the visibility voting map; evaluated on RealEstate10K, DL3DV, Tanks-and-Temples and Mip-NeRF 360 as a research benchmark, not yet a shipped consumer tool.
source: SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis (arXiv:2608.16863)

Provenance

Source: https://arxiv.org/abs/2608.16863

WorldMirror to SplatGuide Pose-Free Novel View Synthesis

Properties

family

modality

unposed multi-view images -> feed-forward 3D Gaussian splat -> diffusion-refined novel views

whyMultiModel

Feed-forward 3DGS reconstruction and multi-view diffusion solve different problems: the reconstruction backbone (WorldMirror) estimates geometry and camera poses in one forward pass but cannot hallucinate unseen or under-observed content, while the diffusion model (SEVA) can synthesize plausible novel content but has no native 3D geometric grounding on its own. The paper trains only the diffusion stage while keeping the reconstruction backbone frozen, and explicitly frames this as chaining feed-forward 3DGS with multi-view diffusion rather than a single joint model. This is distinct from the tracked photogrammetry-splat-to-surface-mesh pattern (COLMAP SfM -> 3DGS training -> SuGaR mesh extraction), which requires classical pose recovery on many photos and outputs a mesh; here pose and geometry come from one learned feed-forward pass on unposed images, and the second stage refines renderable views rather than extracting a surface mesh.

steps

[object Object], [object Object], [object Object]

controls

reconstruction backbone choice (WorldMirror or zero-shot AnySplat), number and arrangement of unposed source views, camera and register token conditioning, Plucker ray embeddings, occlusion voting-map threshold

exampleStack

WorldMirror (pose-free feed-forward 3DGS reconstruction) -> SplatGuide conditioning (renders + visibility voting map + reconstruction tokens) -> SEVA multi-view diffusion (novel-view refinement)

useCases

[Novel-view synthesis from casually captured, unposed photo or video sets][Real-estate and interior walkthroughs without a dedicated SfM capture pipeline][Sparse-view 3D content creation for previz where COLMAP pose recovery is impractical]

pitfalls

Only the diffusion stage is trained while the reconstruction backbone stays frozen, so novel-view quality is bounded by the reconstruction backbone's pose and geometry accuracy; heavy occlusion still challenges the visibility voting map; evaluated on RealEstate10K, DL3DV, Tanks-and-Temples and Mip-NeRF 360 as a research benchmark, not yet a shipped consumer tool.

source

SplatGuide: Geometric Priors from 3D Gaussians for Pose-Free Novel View Synthesis (arXiv:2608.16863)