Segment-Then-Reconstruct Scene Kitbashing

Extracts every distinct object from one ordinary photo of a cluttered scene and reconstructs each as its own posed, textured 3D asset, by first running a general-purpose segmentation model to isolate object masks, then feeding each mask into a dedicated single-image-to-3D reconstruction model.

Properties

family: 3d
modality: single cluttered photo -> per-object segmentation masks -> individually reconstructed posed 3D objects
whyMultiModel: A 2D segmentation model identifies what and where objects are but has no concept of 3D geometry; a single-image 3D reconstruction model lifts one already-isolated object into 3D but cannot first find and separate multiple objects in a cluttered scene. SAM 3D Objects reconstructs one selected object at a time and depends on an upstream segmentation step.
steps: [object Object], [object Object]
controls: mask selection/refinement (click or box prompts), per-object occlusion handling, output pose alignment to the original camera view, which detected objects to reconstruct
exampleStack: SAM 3 (segmentation) -> SAM 3D Objects (per-object 3D reconstruction) -> scene assembly in Blender/Unity
useCases: [Rapid kitbashing of game props from a single reference photo][AR product placement extracted from lifestyle photography][Building a 3D asset library from real-world photo references]
pitfalls: Reconstructs objects one at a time with no reasoning about physical interactions, so assembled scenes can show interpenetration or floating objects; heavy occlusion degrades reconstruction; segmentation errors propagate into missing or fused assets.
source: Introducing SAM 3D: 3D Reconstruction for Physical World Images (Meta AI)

Provenance

Source: https://ai.meta.com/blog/sam-3d/

Properties

family

modality

single cluttered photo -> per-object segmentation masks -> individually reconstructed posed 3D objects

whyMultiModel

A 2D segmentation model identifies what and where objects are but has no concept of 3D geometry; a single-image 3D reconstruction model lifts one already-isolated object into 3D but cannot first find and separate multiple objects in a cluttered scene. SAM 3D Objects reconstructs one selected object at a time and depends on an upstream segmentation step.

steps

[object Object], [object Object]

controls

mask selection/refinement (click or box prompts), per-object occlusion handling, output pose alignment to the original camera view, which detected objects to reconstruct

exampleStack

SAM 3 (segmentation) -> SAM 3D Objects (per-object 3D reconstruction) -> scene assembly in Blender/Unity

useCases

[Rapid kitbashing of game props from a single reference photo][AR product placement extracted from lifestyle photography][Building a 3D asset library from real-world photo references]

pitfalls

Reconstructs objects one at a time with no reasoning about physical interactions, so assembled scenes can show interpenetration or floating objects; heavy occlusion degrades reconstruction; segmentation errors propagate into missing or fused assets.

source

Introducing SAM 3D: 3D Reconstruction for Physical World Images (Meta AI)