Gaussian splatting / Embodied agents / Simulation
Captured appearance.
Editable worlds.
A Gaussian splat captures how a workspace looks. Splat2Sim explores how AI agents can turn it into separate objects that can be moved, inspected, and connected through joints.
One finished splat · Agent-directed scene construction
1 Kaiju Solutions Inc. · 2 School of Computer Science, University of Windsor · 3 WASP Labs
4 Production and Operations Analytics Research Lab, Department of Mechanical, Automotive & Materials Engineering, University of Windsor
The gap between seeing and simulating
Visual detail is only
part of a digital twin.
Three-dimensional Gaussian splatting represents a scene with many small, semi-transparent elements. These reproduce appearance, but do not identify physical objects, collision shapes, mass, friction, or joints.
Splat2Sim gives multimodal agents tools to explore the reconstruction, select its Gaussians, build object groups, and test a proposed simulation model. Gaussian identities stay fixed: editing changes ownership, not the original capture.
The goal is the spatial foundation for a manufacturing digital twin—not a claim that a visual reconstruction alone establishes real-world physics.
Construct / Inspect / Reconsider
One scene. Three perspectives.
Shared tools connect image-based exploration to object editing and explicit physics tests.
- 01 / Construct
Builder
Explores the splat, separates objects and moving parts, creates collision shapes and joints, and reports attempted repairs and remaining limitations.
- 02 / Inspect
Supervisor
Reviews the scene in a separate conversation. It inspects objects in place, alone, and removed, then approves the result or requests concrete changes.
- 03 / Reconsider
Private judge
When changes are requested, checks the evidence and previous attempts. It can retain worthwhile requests, dismiss resolved or exhausted issues, or stop a whole-scene review.
Review inspections restore the builder’s state. The judge stays private from the other agents and receives no turn limits or resource counters. An external limit can interrupt the workflow without approval.
Inside the case study
Take the scene apart.
Matched renders reveal which objects were extracted—and what remains when they are hidden.
These controls select recorded photographs, not a live simulation. Background geometry remains visible in every view. Images are unchanged; object-only views use saved poses before gravity.
- 150,000
- Gaussians, each assigned once
- 10
- Foreground object / part groups
- 4
- Proposed robot hinges
- 500
- Model turns at the external limit
Progress, with visible limits.
The single Genesis-authored scene produced editable objects and connected robot motion. The run reached its 500-turn ceiling without supervisor approval or a judge stop verdict. Contact support failed, collision clearance remained unverified, and the proposed joints did not recover the reference Panda mechanism. These are construction and consistency results, not measured physical or segmentation accuracy.
Movement as an inspection tool
Change the pose.
Expose what was missed.
Joint motion helps reveal attached regions and material left behind. It also makes the difference between a visual model and validated mechanics clear.
These saved configurations set joint positions directly within the authored limits. They do not step physics or certify collision-free motion. A small stationary remnant near the original end position remains visible.
Toward simulation-to-real
Keep the captured context.
Improve the parts that matter.
Blurred boundaries and unseen surfaces limit what reassignment can recover. Future work could combine Splat2Sim’s exploration and inspection tools with asset generation, including NVIDIA’s GATOR, to fit cleaner models within the captured workspace.
Refining Gaussians or completing hidden backgrounds could address gaps exposed by object removal, with inferred content kept separate from captured evidence. Physical validation and robot-task evaluation remain necessary before using these environments for simulation-to-real training.
Explore further