(arXiv Preprint) SM4RT: Learning Structured Motion Geometry for 4D Reconstruction

Geometry Foundation Models (GFMs) have substantially advanced monocular 3D reconstruction, yet extending this capability to 4D dynamic understanding remains a fundamental challenge. Most existing motion perception methods (e.g., sparse tracking, dense point-wise flow) treat motion as independent point-wise displacements, ignoring the structured nature of physical motion. However, real-world objects usually obey rigid-body kinematics, and points thus usually move collectively, not in isolation. Motion itself possesses geometric structure: physical objects undergo a set of rigid-body transformations governed by $\mathrm{SE}(3)$, rather than unstructured point-wise displacements. Building on this insight, we propose \textbf{SM4RT}, a \textbf{S}tructured \textbf{M}otion \textbf{4}D \textbf{R}econstruction \textbf{T}ransformer for end-to-end 3D reconstruction and structured motion perception. SM4RT introduces \textit{Structure-of-Motion} (SoM) to represent scene dynamics, where scene motion is decomposed into a compact set of motion bases, each represented as a temporal sequence of 6D twists in $\mathfrak{se}(3)$. Dense scene motion is then recovered by sparse, time-shared per-pixel assignment weights over these bases, ensuring points on the same object share a common rigid-body motion trajectory. SM4RT introduces a parallel motion geometry encoder and decoder that jointly infer 3D geometry, world-coordinate motion, and scene kinematic structure in a single forward pass from monocular RGB video. SM4RT achieves strong motion reconstruction performance while preserving the geometric structure of scene motion.

Recommended citation: S.-H. J. Lin, W. Zheng, D. Zhuo, Y. Wu, J. Zhou, J. Lu, "SM4RT: Learning Structured Motion Geometry for 4D Reconstruction," in arXiv preprint 2607.22534, 2026.
Download Paper